Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: say when a screenshot goes out unredacted

Emre Sokullu4a5f0cce930e
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index b0ee08f96..054df9d60 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -138,7 +138,7 @@ Right before input goes out, both builds read what is under the pointer with `vi
- `drag_at` is refused when it is shorter than 6 CSS px, when the press and release would click a control (their nearest common ancestor is or is inside a button, link, label, similar control or pointer-cursor element, unless the drag starts on a slider, scrollbar, resizer, draggable item or grab handle), when either end is inside a frame it cannot inspect (another site's frame, or one drawn rotated, skewed, flipped or in 3D), when the target cannot be read, or when the click submit probe reads its start point as a submit, send, delete or payment control (a drag never gets the confirmation a click gets). On a protected messaging page `drag_at` is blocked outright, like `iframe_click`.
- `scroll_at` is refused over a focused number field, which takes the wheel as +/-.
-Every CDP input passes `cdpClient`'s page-agent fence and every content action passes `beforePageAgentDispatch`, so the page monitor attributes the events to the agent. `drag_at` is gated as a click (`TOOL_CAPABILITY`); captures and pointer results are untrusted page content (`UNTRUSTED_CONTENT_TOOLS`), and every image label says that text in it is data, never instructions. Screenshot redaction (Settings) applies to frames and zooms.
+Every CDP input passes `cdpClient`'s page-agent fence and every content action passes `beforePageAgentDispatch`, so the page monitor attributes the events to the agent. `drag_at` is gated as a click (`TOOL_CAPABILITY`); captures and pointer results are untrusted page content (`UNTRUSTED_CONTENT_TOOLS`), and every image label says that text in it is data, never instructions. Screenshot redaction (Settings) applies to frames, patches (cut from the redacted frame) and zooms, and is best-effort there as everywhere: an image it could not run on is still sent. Right after a navigation the page's scan is often not ready (its content script starts when the document is idle), so while the tab is loading WebBrain waits up to 1.5 s for the scan and captures again. When an image does go out unredacted (no scan, an undecodable capture, or a region that could not be pixelated, which fails the whole redaction), the trace records `vision_redaction_skipped` with the reason (`code`) and the image kind (`context`), and the side panel warns once per page and run (`_visionReportUnredacted`).
## Guards in a vision run
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 4dccbc13f..f4a161a97 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -13493,6 +13493,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
return promptTier !== 'compact' || (this._userAttachmentHandles.get(tabId)?.size || 0) > 0;
}
+ // Whether the tab is still loading: the redaction scan starts when the
+ // document is idle (vision-runtime.js _visionCaptureFrame waits for it).
+ async _visionTabLoading(tabId) {
+ try { return (await chrome.tabs.get(tabId))?.status === 'loading'; }
+ catch { return false; }
+ }
+
// WhatsApp Web and Telegram keep the tree path (vision-mode.js).
_visionNativeChatPage(tabId) {
const url = this._lastTabUrls?.get(tabId) || '';
@@ -19633,6 +19640,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
* @returns {Promise<string>}
*/
async _redactScreenshotDataUrl(tabId, dataUrl, opts = {}) {
+ // `opts.outcome`, when given, receives why the image came back as it did:
+ // 'redacted', 'no_regions' or 'no_regions_in_view' (nothing to hide), or
+ // 'decode_failed', 'scan_unavailable' or 'pixelate_failed' (the image is
+ // returned unredacted; vision mode reports those).
+ const report = status => { if (opts.outcome && typeof opts.outcome === 'object') opts.outcome.status = status; };
if (!this.screenshotRedaction || !dataUrl) return dataUrl;
// Resolve the captured image's pixel dimensions. Callers that already
@@ -19654,6 +19666,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
try { bmp.close?.(); } catch {}
}
} catch {
+ report('decode_failed');
return dataUrl;
}
}
@@ -19662,7 +19675,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
const snapshot = opts.redactionSnapshot
? this._normalizeScreenshotRedactionSnapshot(opts.redactionSnapshot, coordinateSpace)
: await this._captureScreenshotRedactionSnapshot(tabId, { coordinateSpace });
- if (!snapshot) return dataUrl;
+ if (!snapshot) {
+ report('scan_unavailable');
+ return dataUrl;
+ }
// The captured CSS box (CSS px) in the SAME space as the element rects.
// A bounded full-page capture can be shorter than the live document, so
@@ -19688,7 +19704,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
: (Number.isFinite(opts.offsetY) ? opts.offsetY : 0);
const regions = snapshot.regions;
- if (!regions.length) return dataUrl;
+ if (!regions.length) {
+ report('no_regions');
+ return dataUrl;
+ }
const imageRegions = mapRegionsToImage(regions, {
scaleX,
@@ -19698,10 +19717,18 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
imageWidth,
imageHeight,
});
- if (!imageRegions.length) return dataUrl;
+ if (!imageRegions.length) {
+ report('no_regions_in_view');
+ return dataUrl;
+ }
const redacted = await pixelateDataUrl(dataUrl, imageRegions);
- if (redacted === dataUrl) return dataUrl; // pixelation was a no-op/failed
+ if (redacted === dataUrl) {
+ // Pixelation was a no-op or failed.
+ report('pixelate_failed');
+ return dataUrl;
+ }
+ report('redacted');
// `pixelateDataUrl` re-encodes at a fixed quality, which can push the
// result back over the provider's base64 ceiling (callers have already
diff --git a/src/chrome/src/agent/screenshot-redaction.js b/src/chrome/src/agent/screenshot-redaction.js
index b521a2346..468211620 100644
--- a/src/chrome/src/agent/screenshot-redaction.js
+++ b/src/chrome/src/agent/screenshot-redaction.js
@@ -427,7 +427,9 @@ export async function pixelateDataUrl(dataUrl, regions, opts = {}) {
const sh = Math.max(1, Math.floor(h / block));
const tmp = new OffscreenCanvas(sw, sh);
const tctx = tmp.getContext('2d');
- if (!tctx) continue;
+ // One region left sharp would pass for a redacted image: fail whole,
+ // so the caller sees an unredacted result.
+ if (!tctx) return dataUrl;
tctx.imageSmoothingEnabled = false;
tctx.drawImage(canvas, x, y, w, h, 0, 0, sw, sh);
// Step 2: upscale back with smoothing off → blocky, unreadable pixels.
diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.js
index f51394376..ea49c9e52 100644
--- a/src/chrome/src/agent/vision-runtime.js
+++ b/src/chrome/src/agent/vision-runtime.js
@@ -57,6 +57,16 @@ const SETTLE_MAX_EXTRA_CAPTURES = 3;
// this many times: a page whose upload never succeeds goes back to vision
// and then reports the chooser instead of handing off again.
const UPLOAD_HANDOFF_DECISIONS = 5;
+// Screenshot redaction scans the page from a content script that starts when
+// the document is idle: a capture right after a navigation often finds no
+// scan yet. While the tab loads, WebBrain waits this long for one, then
+// captures again.
+const REDACTION_SCAN_WAIT_MS = 1500;
+const REDACTION_SCAN_POLL_MS = 250;
+// Redaction outcomes (agent _redactScreenshotDataUrl) that leave the image as
+// captured although redaction is on. Redaction is best-effort: the image is
+// still sent, and the user and the trace are told (_visionReportUnredacted).
+const UNREDACTED_OUTCOMES = new Set(['decode_failed', 'scan_unavailable', 'pixelate_failed', 'error']);
const UPLOAD_HANDOFFS_PER_RUN = 2;
// A change this small between two captures (a blinking caret, a ticking
// clock, a spinner: a few 16 px cells) is not the page still reacting to an
@@ -498,14 +508,16 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
* return any size (device pixels, emulated DPR, zoom); rasterizeFrame
* resamples to the plan so the coordinate map is exact.
*/
- proto._visionCaptureFrame = async function _visionCaptureFrame(tabId, state) {
+ proto._visionCaptureFrame = async function _visionCaptureFrame(tabId, state, { redactionRetry = true } = {}) {
const probe = await this._visionPlatformProbe(tabId);
if (!probe || !(probe.cssWidth > 0) || !(probe.cssHeight > 0)) throw new Error('The page viewport could not be measured.');
const profile = VISION_PROFILES[state.profileId] || VISION_PROFILES.default;
const size = planVisionImageSize({ cssWidth: probe.cssWidth, cssHeight: probe.cssHeight, profile, quality: state.quality });
let dataUrl = await this._visionPlatformCapture(tabId, { probe, outWidth: size.width, outHeight: size.height });
if (!dataUrl) throw new Error('The browser returned no screenshot.');
+ let redaction = null;
if (this.screenshotRedaction && typeof this._redactScreenshotDataUrl === 'function') {
+ const outcome = {};
try {
// The capture spans the visual viewport (no scrollbars), so the
// redactor gets that box in page space rather than its own
@@ -513,12 +525,23 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
dataUrl = await this._redactScreenshotDataUrl(tabId, dataUrl, {
coordinateSpace: 'page',
capturedCssBounds: { x: probe.scrollX || 0, y: probe.scrollY || 0, width: probe.cssWidth, height: probe.cssHeight },
+ outcome,
});
- } catch { /* keep the capture */ }
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ outcome.status = 'error';
+ }
+ // No scan yet on a loading page: wait for it, then capture again so the
+ // scan and the image show the same page.
+ if (outcome.status === 'scan_unavailable' && redactionRetry && await this._visionTabLoading?.(tabId) === true) {
+ await this._visionAwaitRedactionScan(tabId);
+ return this._visionCaptureFrame(tabId, state, { redactionRetry: false });
+ }
+ redaction = outcome.status || 'error';
}
const raster = await this._visionRasterize(dataUrl, size.width, size.height);
if (probe.url) (this._lastTabUrls ??= new Map()).set(tabId, String(probe.url));
- return { raster, probe, size, profile };
+ return { raster, probe, size, profile, redaction };
};
// Indirections over vision-encoder.js so the Node suite can substitute
@@ -531,6 +554,42 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
return encodeCanvasRegion(canvas, region, options);
};
+ /** Waits up to REDACTION_SCAN_WAIT_MS for the page's redaction scan to answer. */
+ proto._visionAwaitRedactionScan = async function _visionAwaitRedactionScan(tabId) {
+ const deadline = Date.now() + REDACTION_SCAN_WAIT_MS;
+ while (Date.now() < deadline) {
+ await sleep(REDACTION_SCAN_POLL_MS);
+ this._throwIfAborted?.(this._runAbortSignal?.(tabId));
+ let snapshot = null;
+ try { snapshot = await this._captureScreenshotRedactionSnapshot?.(tabId, { coordinateSpace: 'page' }); }
+ catch { snapshot = null; }
+ if (snapshot) return true;
+ }
+ return false;
+ };
+
+ /**
+ * Images sent to the model although screenshot redaction is on and could
+ * not run on them (UNREDACTED_OUTCOMES): each gets a coded trace note, and
+ * the side panel warns once per page and run. `entries` are
+ * { kind, redaction } for the images of one update.
+ */
+ proto._visionReportUnredacted = function _visionReportUnredacted(tabId, onUpdate, entries, url = '') {
+ const unredacted = entries.filter(entry => UNREDACTED_OUTCOMES.has(entry?.redaction));
+ if (!unredacted.length) return;
+ for (const entry of unredacted) this._visionTraceNote(tabId, 'vision_redaction_skipped', { code: entry.redaction, context: entry.kind });
+ const snapshot = this._visionRunSnapshots?.get(tabId);
+ const page = pageKey(url);
+ if (snapshot) {
+ snapshot.redactionWarned ??= new Set();
+ if (snapshot.redactionWarned.has(page)) return;
+ snapshot.redactionWarned.add(page);
+ }
+ try {
+ onUpdate?.('warning', { message: 'Screenshot redaction could not run on this page, so a screenshot was sent to the model unredacted. Redaction is best-effort; see Settings.' });
+ } catch { /* UI updates are best effort */ }
+ };
+
/**
* Capture until two consecutive frames match (bounded). Micro changes (a
* caret, a clock, a spinner) count as a match: they never stop, and waiting
@@ -653,6 +712,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
captureWidth: captured.raster.sourceWidth, captureHeight: captured.raster.sourceHeight,
bytes: encoded.bytes, mimeType: encoded.mimeType, estimatedTokens: tokens, profile: captured.profile.id,
});
+ this._visionReportUnredacted(tabId, onUpdate, [{ kind: 'keyframe', redaction: captured.redaction }], keyframe.url);
await this._visionRecordObservation(tabId, keyframe.url);
return keyframe;
};
@@ -728,6 +788,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
{ epoch: session.epoch, kind: 'zoom', ids: zooms.map(zoom => zoom.id) }));
this._visionEmitUpdate(tabId, onUpdate, { success: true, method: 'image_attach', vision: { kind: 'zoom', imageIds: zooms.map(zoom => zoom.id) } },
zooms.map(zoom => ({ id: zoom.id, kind: 'zoom', dataUrl: zoom.dataUrl })));
+ this._visionReportUnredacted(tabId, onUpdate, zooms.map(zoom => ({ kind: 'zoom', redaction: zoom.redaction })), session.keyframe?.url || '');
return { kind: 'zoom' };
}
if (!observe) return null;
@@ -805,6 +866,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
if (zooms.length) {
messages.push(this._visionImageMessage([note, ...zooms.map(zoom => zoom.note)].join('\n'), zooms,
{ epoch: session.epoch, kind: 'zoom', ids: zooms.map(zoom => zoom.id) }));
+ this._visionReportUnredacted(tabId, onUpdate, zooms.map(zoom => ({ kind: 'zoom', redaction: zoom.redaction })), frame.probe.url);
} else {
messages.push(visionUserMessage(note, { epoch: session.epoch, kind: 'none' }));
}
@@ -825,6 +887,8 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
const images = patches.map((patch, index) => ({ id: patch.id, kind: 'patch', dataUrl: encodedPatches[index].encoded.dataUrl }));
messages.push(this._visionImageMessage([note, ...zooms.map(zoom => zoom.note)].join('\n'), [...images, ...zooms],
{ epoch: session.epoch, kind: 'patch', ids: [...patches.map(patch => patch.id), ...zooms.map(zoom => zoom.id)] }));
+ this._visionReportUnredacted(tabId, onUpdate, [{ kind: 'patch', redaction: frame.redaction },
+ ...zooms.map(zoom => ({ kind: 'zoom', redaction: zoom.redaction }))], frame.probe.url);
const bytes = encodedPatches.reduce((sum, item) => sum + item.encoded.bytes, 0);
this._visionEmitUpdate(tabId, onUpdate, {
success: true,
@@ -1107,7 +1171,9 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
const plan = planZoomImageSize({ regionWidth: region.cssRegion.width, regionHeight: region.cssRegion.height, devicePixelRatio: probe?.devicePixelRatio || 1, profile });
let dataUrl = await this._visionPlatformCapture(tabId, { probe, cssRegion: region.cssRegion, outWidth: plan.width, outHeight: plan.height });
if (!dataUrl) throw new Error('The browser returned no screenshot.');
+ let redaction = null;
if (this.screenshotRedaction && typeof this._redactScreenshotDataUrl === 'function') {
+ const outcome = {};
try {
// A crop is a page-space box; the redactor maps element rects into it.
dataUrl = await this._redactScreenshotDataUrl(tabId, dataUrl, {
@@ -1118,8 +1184,14 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
width: region.cssRegion.width,
height: region.cssRegion.height,
},
+ outcome,
});
- } catch { /* keep the capture */ }
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ outcome.status = 'error';
+ }
+ // Reported when the zoom is sent (_visionReportUnredacted).
+ redaction = outcome.status || 'error';
}
const raster = await this._visionRasterize(dataUrl, plan.width, plan.height);
const encoding = await this._visionEncoding(profile);
@@ -1129,6 +1201,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
id: zoom.id, kind: 'zoom', dataUrl: encoded.dataUrl,
note: visionZoomNote(zoom, session.keyframe, region.keyframeRegion, { coordinates: session.coordinates }),
...(base ? { base } : {}),
+ ...(redaction ? { redaction } : {}),
});
this._visionTraceNote(tabId, 'vision_frame', { kind: 'zoom', id: zoom.id, region: region.keyframeRegion, width: plan.width, height: plan.height, bytes: encoded.bytes });
return { success: true, image: zoom.id, width: plan.width, height: plan.height, note: `Zoom ${zoom.id} follows this result.` };
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 8ba266c1e..df197b11a 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -11848,6 +11848,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
} catch (e) { return ''; }
}
+ // Whether the tab is still loading: the redaction scan starts when the
+ // document is idle (vision-runtime.js _visionCaptureFrame waits for it).
+ async _visionTabLoading(tabId) {
+ try { return (await browser.tabs.get(tabId))?.status === 'loading'; }
+ catch { return false; }
+ }
+
// WhatsApp Web and Telegram keep the tree path (vision-mode.js).
_visionNativeChatPage(tabId) {
const url = this._lastTabUrls?.get(tabId) || '';
@@ -17886,6 +17893,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
* @returns {Promise<string>}
*/
async _redactScreenshotDataUrl(tabId, dataUrl, opts = {}) {
+ // `opts.outcome`, when given, receives why the image came back as it did:
+ // 'redacted', 'no_regions' or 'no_regions_in_view' (nothing to hide), or
+ // 'decode_failed', 'scan_unavailable' or 'pixelate_failed' (the image is
+ // returned unredacted; vision mode reports those).
+ const report = status => { if (opts.outcome && typeof opts.outcome === 'object') opts.outcome.status = status; };
if (!this.screenshotRedaction || !dataUrl) return dataUrl;
let imageWidth = opts.imageWidth;
@@ -17902,6 +17914,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
try { bmp.close?.(); } catch {}
}
} catch {
+ report('decode_failed');
return dataUrl;
}
}
@@ -17910,7 +17923,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
const snapshot = opts.redactionSnapshot
? this._normalizeScreenshotRedactionSnapshot(opts.redactionSnapshot, coordinateSpace)
: await this._captureScreenshotRedactionSnapshot(tabId, { coordinateSpace });
- if (!snapshot) return dataUrl;
+ if (!snapshot) {
+ report('scan_unavailable');
+ return dataUrl;
+ }
const suppliedBounds = opts.capturedCssBounds;
const hasCapturedBounds = coordinateSpace === 'page' &&
@@ -17931,7 +17947,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
: (Number.isFinite(opts.offsetY) ? opts.offsetY : 0);
const regions = snapshot.regions;
- if (!regions.length) return dataUrl;
+ if (!regions.length) {
+ report('no_regions');
+ return dataUrl;
+ }
const imageRegions = mapRegionsToImage(regions, {
scaleX,
@@ -17941,10 +17960,18 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
imageWidth,
imageHeight,
});
- if (!imageRegions.length) return dataUrl;
+ if (!imageRegions.length) {
+ report('no_regions_in_view');
+ return dataUrl;
+ }
const redacted = await pixelateDataUrl(dataUrl, imageRegions);
- if (redacted === dataUrl) return dataUrl;
+ if (redacted === dataUrl) {
+ // Pixelation was a no-op or failed.
+ report('pixelate_failed');
+ return dataUrl;
+ }
+ report('redacted');
try {
return await this._compressJpegToByteCeiling(redacted);
} catch {
diff --git a/src/firefox/src/agent/screenshot-redaction.js b/src/firefox/src/agent/screenshot-redaction.js
index b521a2346..468211620 100644
--- a/src/firefox/src/agent/screenshot-redaction.js
+++ b/src/firefox/src/agent/screenshot-redaction.js
@@ -427,7 +427,9 @@ export async function pixelateDataUrl(dataUrl, regions, opts = {}) {
const sh = Math.max(1, Math.floor(h / block));
const tmp = new OffscreenCanvas(sw, sh);
const tctx = tmp.getContext('2d');
- if (!tctx) continue;
+ // One region left sharp would pass for a redacted image: fail whole,
+ // so the caller sees an unredacted result.
+ if (!tctx) return dataUrl;
tctx.imageSmoothingEnabled = false;
tctx.drawImage(canvas, x, y, w, h, 0, 0, sw, sh);
// Step 2: upscale back with smoothing off → blocky, unreadable pixels.
diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.js
index f51394376..ea49c9e52 100644
--- a/src/firefox/src/agent/vision-runtime.js
+++ b/src/firefox/src/agent/vision-runtime.js
@@ -57,6 +57,16 @@ const SETTLE_MAX_EXTRA_CAPTURES = 3;
// this many times: a page whose upload never succeeds goes back to vision
// and then reports the chooser instead of handing off again.
const UPLOAD_HANDOFF_DECISIONS = 5;
+// Screenshot redaction scans the page from a content script that starts when
+// the document is idle: a capture right after a navigation often finds no
+// scan yet. While the tab loads, WebBrain waits this long for one, then
+// captures again.
+const REDACTION_SCAN_WAIT_MS = 1500;
+const REDACTION_SCAN_POLL_MS = 250;
+// Redaction outcomes (agent _redactScreenshotDataUrl) that leave the image as
+// captured although redaction is on. Redaction is best-effort: the image is
+// still sent, and the user and the trace are told (_visionReportUnredacted).
+const UNREDACTED_OUTCOMES = new Set(['decode_failed', 'scan_unavailable', 'pixelate_failed', 'error']);
const UPLOAD_HANDOFFS_PER_RUN = 2;
// A change this small between two captures (a blinking caret, a ticking
// clock, a spinner: a few 16 px cells) is not the page still reacting to an
@@ -498,14 +508,16 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
* return any size (device pixels, emulated DPR, zoom); rasterizeFrame
* resamples to the plan so the coordinate map is exact.
*/
- proto._visionCaptureFrame = async function _visionCaptureFrame(tabId, state) {
+ proto._visionCaptureFrame = async function _visionCaptureFrame(tabId, state, { redactionRetry = true } = {}) {
const probe = await this._visionPlatformProbe(tabId);
if (!probe || !(probe.cssWidth > 0) || !(probe.cssHeight > 0)) throw new Error('The page viewport could not be measured.');
const profile = VISION_PROFILES[state.profileId] || VISION_PROFILES.default;
const size = planVisionImageSize({ cssWidth: probe.cssWidth, cssHeight: probe.cssHeight, profile, quality: state.quality });
let dataUrl = await this._visionPlatformCapture(tabId, { probe, outWidth: size.width, outHeight: size.height });
if (!dataUrl) throw new Error('The browser returned no screenshot.');
+ let redaction = null;
if (this.screenshotRedaction && typeof this._redactScreenshotDataUrl === 'function') {
+ const outcome = {};
try {
// The capture spans the visual viewport (no scrollbars), so the
// redactor gets that box in page space rather than its own
@@ -513,12 +525,23 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
dataUrl = await this._redactScreenshotDataUrl(tabId, dataUrl, {
coordinateSpace: 'page',
capturedCssBounds: { x: probe.scrollX || 0, y: probe.scrollY || 0, width: probe.cssWidth, height: probe.cssHeight },
+ outcome,
});
- } catch { /* keep the capture */ }
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ outcome.status = 'error';
+ }
+ // No scan yet on a loading page: wait for it, then capture again so the
+ // scan and the image show the same page.
+ if (outcome.status === 'scan_unavailable' && redactionRetry && await this._visionTabLoading?.(tabId) === true) {
+ await this._visionAwaitRedactionScan(tabId);
+ return this._visionCaptureFrame(tabId, state, { redactionRetry: false });
+ }
+ redaction = outcome.status || 'error';
}
const raster = await this._visionRasterize(dataUrl, size.width, size.height);
if (probe.url) (this._lastTabUrls ??= new Map()).set(tabId, String(probe.url));
- return { raster, probe, size, profile };
+ return { raster, probe, size, profile, redaction };
};
// Indirections over vision-encoder.js so the Node suite can substitute
@@ -531,6 +554,42 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
return encodeCanvasRegion(canvas, region, options);
};
+ /** Waits up to REDACTION_SCAN_WAIT_MS for the page's redaction scan to answer. */
+ proto._visionAwaitRedactionScan = async function _visionAwaitRedactionScan(tabId) {
+ const deadline = Date.now() + REDACTION_SCAN_WAIT_MS;
+ while (Date.now() < deadline) {
+ await sleep(REDACTION_SCAN_POLL_MS);
+ this._throwIfAborted?.(this._runAbortSignal?.(tabId));
+ let snapshot = null;
+ try { snapshot = await this._captureScreenshotRedactionSnapshot?.(tabId, { coordinateSpace: 'page' }); }
+ catch { snapshot = null; }
+ if (snapshot) return true;
+ }
+ return false;
+ };
+
+ /**
+ * Images sent to the model although screenshot redaction is on and could
+ * not run on them (UNREDACTED_OUTCOMES): each gets a coded trace note, and
+ * the side panel warns once per page and run. `entries` are
+ * { kind, redaction } for the images of one update.
+ */
+ proto._visionReportUnredacted = function _visionReportUnredacted(tabId, onUpdate, entries, url = '') {
+ const unredacted = entries.filter(entry => UNREDACTED_OUTCOMES.has(entry?.redaction));
+ if (!unredacted.length) return;
+ for (const entry of unredacted) this._visionTraceNote(tabId, 'vision_redaction_skipped', { code: entry.redaction, context: entry.kind });
+ const snapshot = this._visionRunSnapshots?.get(tabId);
+ const page = pageKey(url);
+ if (snapshot) {
+ snapshot.redactionWarned ??= new Set();
+ if (snapshot.redactionWarned.has(page)) return;
+ snapshot.redactionWarned.add(page);
+ }
+ try {
+ onUpdate?.('warning', { message: 'Screenshot redaction could not run on this page, so a screenshot was sent to the model unredacted. Redaction is best-effort; see Settings.' });
+ } catch { /* UI updates are best effort */ }
+ };
+
/**
* Capture until two consecutive frames match (bounded). Micro changes (a
* caret, a clock, a spinner) count as a match: they never stop, and waiting
@@ -653,6 +712,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
captureWidth: captured.raster.sourceWidth, captureHeight: captured.raster.sourceHeight,
bytes: encoded.bytes, mimeType: encoded.mimeType, estimatedTokens: tokens, profile: captured.profile.id,
});
+ this._visionReportUnredacted(tabId, onUpdate, [{ kind: 'keyframe', redaction: captured.redaction }], keyframe.url);
await this._visionRecordObservation(tabId, keyframe.url);
return keyframe;
};
@@ -728,6 +788,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
{ epoch: session.epoch, kind: 'zoom', ids: zooms.map(zoom => zoom.id) }));
this._visionEmitUpdate(tabId, onUpdate, { success: true, method: 'image_attach', vision: { kind: 'zoom', imageIds: zooms.map(zoom => zoom.id) } },
zooms.map(zoom => ({ id: zoom.id, kind: 'zoom', dataUrl: zoom.dataUrl })));
+ this._visionReportUnredacted(tabId, onUpdate, zooms.map(zoom => ({ kind: 'zoom', redaction: zoom.redaction })), session.keyframe?.url || '');
return { kind: 'zoom' };
}
if (!observe) return null;
@@ -805,6 +866,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
if (zooms.length) {
messages.push(this._visionImageMessage([note, ...zooms.map(zoom => zoom.note)].join('\n'), zooms,
{ epoch: session.epoch, kind: 'zoom', ids: zooms.map(zoom => zoom.id) }));
+ this._visionReportUnredacted(tabId, onUpdate, zooms.map(zoom => ({ kind: 'zoom', redaction: zoom.redaction })), frame.probe.url);
} else {
messages.push(visionUserMessage(note, { epoch: session.epoch, kind: 'none' }));
}
@@ -825,6 +887,8 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
const images = patches.map((patch, index) => ({ id: patch.id, kind: 'patch', dataUrl: encodedPatches[index].encoded.dataUrl }));
messages.push(this._visionImageMessage([note, ...zooms.map(zoom => zoom.note)].join('\n'), [...images, ...zooms],
{ epoch: session.epoch, kind: 'patch', ids: [...patches.map(patch => patch.id), ...zooms.map(zoom => zoom.id)] }));
+ this._visionReportUnredacted(tabId, onUpdate, [{ kind: 'patch', redaction: frame.redaction },
+ ...zooms.map(zoom => ({ kind: 'zoom', redaction: zoom.redaction }))], frame.probe.url);
const bytes = encodedPatches.reduce((sum, item) => sum + item.encoded.bytes, 0);
this._visionEmitUpdate(tabId, onUpdate, {
success: true,
@@ -1107,7 +1171,9 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
const plan = planZoomImageSize({ regionWidth: region.cssRegion.width, regionHeight: region.cssRegion.height, devicePixelRatio: probe?.devicePixelRatio || 1, profile });
let dataUrl = await this._visionPlatformCapture(tabId, { probe, cssRegion: region.cssRegion, outWidth: plan.width, outHeight: plan.height });
if (!dataUrl) throw new Error('The browser returned no screenshot.');
+ let redaction = null;
if (this.screenshotRedaction && typeof this._redactScreenshotDataUrl === 'function') {
+ const outcome = {};
try {
// A crop is a page-space box; the redactor maps element rects into it.
dataUrl = await this._redactScreenshotDataUrl(tabId, dataUrl, {
@@ -1118,8 +1184,14 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
width: region.cssRegion.width,
height: region.cssRegion.height,
},
+ outcome,
});
- } catch { /* keep the capture */ }
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ outcome.status = 'error';
+ }
+ // Reported when the zoom is sent (_visionReportUnredacted).
+ redaction = outcome.status || 'error';
}
const raster = await this._visionRasterize(dataUrl, plan.width, plan.height);
const encoding = await this._visionEncoding(profile);
@@ -1129,6 +1201,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
id: zoom.id, kind: 'zoom', dataUrl: encoded.dataUrl,
note: visionZoomNote(zoom, session.keyframe, region.keyframeRegion, { coordinates: session.coordinates }),
...(base ? { base } : {}),
+ ...(redaction ? { redaction } : {}),
});
this._visionTraceNote(tabId, 'vision_frame', { kind: 'zoom', id: zoom.id, region: region.keyframeRegion, width: plan.width, height: plan.height, bytes: encoded.bytes });
return { success: true, image: zoom.id, width: plan.width, height: plan.height, note: `Zoom ${zoom.id} follows this result.` };
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index da27faf04..e1088cf76 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -527,6 +527,83 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
assert.deepEqual(Object.keys(checked.vision.focusFollowUp).sort(), ['x', 'y']);
});
+ // Screenshot redaction stubbed per test: the helper reports its outcome
+ // (agent _redactScreenshotDataUrl `opts.outcome`).
+ function redactWith(ctx, outcomes) {
+ const updates = [];
+ const notes = [];
+ let index = 0;
+ ctx.agent.screenshotRedaction = true;
+ ctx.agent._redactScreenshotDataUrl = async (_tab, dataUrl, opts) => {
+ opts.outcome.status = typeof outcomes === 'function' ? outcomes(index++) : outcomes;
+ return dataUrl;
+ };
+ const traceNote = ctx.agent._visionTraceNote;
+ ctx.agent._visionTraceNote = function (tab, note, data) { notes.push({ note, ...data }); return traceNote.call(this, tab, note, data); };
+ return { updates, notes, onUpdate: (type, data) => updates.push({ type, data }) };
+ }
+ const redactionWarnings = updates => updates.filter(update => update.type === 'warning' && /Screenshot redaction could not run on this page/.test(update.data?.message));
+
+ test(`${build}: a screenshot redaction could not run on is still sent, with one warning per page and a coded trace note per image`, async () => {
+ let clicks = 0;
+ const ctx = runAgent(step => {
+ if (step === 1) return { toolCalls: [call('click_at', { x: 300, y: 200 })] };
+ if (step === 2) return { toolCalls: [call('zoom', { x: 0, y: 0, width: 400, height: 240 })] };
+ if (step === 3) return { toolCalls: [call('click_at', { x: 320, y: 220 })] };
+ if (step === 4) return { toolCalls: [call('done', { summary: 'Done.', outcome: 'success' })] };
+ return { content: 'done' };
+ }, { onDispatch: (name, _args, scene) => {
+ if (name !== 'click') return;
+ scene.rects.push({ x: 900, y: 600 - clicks * 40, w: 80, h: 30, color: [0, 120, 0] });
+ if (clicks++ === 1) scene.url = `${PAGE}?next`;
+ } });
+ const { updates, notes, onUpdate } = redactWith(ctx, 'scan_unavailable');
+ ctx.agent._visionTabLoading = async () => false;
+ await ctx.agent.processMessage(nextTab++, 'Fill the form.', onUpdate, 'act', [], { detachedRequestId: `vision-redaction-${build}` });
+ ctx.scene.url = PAGE;
+ assert.equal(redactionWarnings(updates).length, 2, 'once for each page');
+ const skipped = notes.filter(note => note.note === 'vision_redaction_skipped');
+ assert.deepEqual(skipped.map(note => note.context), ['keyframe', 'patch', 'zoom', 'keyframe']);
+ assert.ok(skipped.every(note => note.code === 'scan_unavailable'));
+ // Best-effort: the screenshots still reached the model.
+ assert.ok(totalImages(ctx.requests[0]) === 1 && totalImages(ctx.requests[1]) >= 2);
+ });
+
+ test(`${build}: a redacted screenshot or one with nothing to hide gives no warning`, async () => {
+ const ctx = runAgent(step => {
+ if (step === 1) return { toolCalls: [call('click_at', { x: 300, y: 200 })] };
+ if (step === 2) return { toolCalls: [call('done', { summary: 'Done.', outcome: 'success' })] };
+ return { content: 'done' };
+ }, { onDispatch: (name, _args, scene) => { if (name === 'click') scene.rects.push({ x: 900, y: 600, w: 80, h: 30, color: [0, 120, 0] }); } });
+ const { updates, notes, onUpdate } = redactWith(ctx, index => (index === 0 ? 'redacted' : 'no_regions_in_view'));
+ await ctx.agent.processMessage(nextTab++, 'Fill the form.', onUpdate, 'act', [], { detachedRequestId: `vision-redacted-${build}` });
+ assert.equal(redactionWarnings(updates).length, 0);
+ assert.equal(notes.filter(note => note.note === 'vision_redaction_skipped').length, 0);
+ });
+
+ test(`${build}: on a loading page the frame waits for the redaction scan and is captured again`, async () => {
+ const ctx = runAgent(step => (step === 1 ? { toolCalls: [call('done', { summary: 'Looked.', outcome: 'success' })] } : { content: 'done' }));
+ const { updates, notes, onUpdate } = redactWith(ctx, index => (index === 0 ? 'scan_unavailable' : 'redacted'));
+ ctx.agent._visionTabLoading = async () => true;
+ let scans = 0;
+ ctx.agent._captureScreenshotRedactionSnapshot = async () => (++scans < 2 ? null : { coordinateSpace: 'page', viewport: { width: 1440, height: 900 }, regions: [] });
+ const started = Date.now();
+ await ctx.agent.processMessage(nextTab++, 'Look.', onUpdate, 'act', [], { detachedRequestId: `vision-scan-wait-${build}` });
+ assert.equal(scans, 2, 'polled until the scan answered');
+ assert.equal(ctx.scene.captures, 2, 'captured again once the scan was ready');
+ assert.ok(Date.now() - started >= 2 * 250);
+ assert.equal(redactionWarnings(updates).length, 0);
+ assert.equal(notes.filter(note => note.note === 'vision_redaction_skipped').length, 0);
+ // A page that has finished loading gets no wait: the frame is sent and reported.
+ const loaded = runAgent(step => (step === 1 ? { toolCalls: [call('done', { summary: 'Looked.', outcome: 'success' })] } : { content: 'done' }));
+ const report = redactWith(loaded, 'scan_unavailable');
+ loaded.agent._visionTabLoading = async () => false;
+ loaded.agent._captureScreenshotRedactionSnapshot = async () => { throw new Error('no wait on a loaded page'); };
+ await loaded.agent.processMessage(nextTab++, 'Look.', report.onUpdate, 'act', [], { detachedRequestId: `vision-scan-loaded-${build}` });
+ assert.equal(loaded.scene.captures, 1);
+ assert.equal(redactionWarnings(report.updates).length, 1);
+ });
+
test(`${build}: a new run never shows or uses the previous run's frame when its first capture fails`, async () => {
const ctx = runAgent(step => {
if (step === 1) return { toolCalls: [call('done', { summary: 'Looked.', outcome: 'success' })] };
diff --git a/test/vision-mode-guards.mjs b/test/vision-mode-guards.mjs
index 17575403e..95dcf6e64 100644
--- a/test/vision-mode-guards.mjs
+++ b/test/vision-mode-guards.mjs
@@ -487,6 +487,26 @@ for (const build of ['chrome', 'firefox']) {
agent._endVisionRun(tab, null);
});
+ test(`${build}: the redaction helper says why an image came back unredacted`, async () => {
+ const agent = agentFor();
+ agent.screenshotRedaction = true;
+ const image = 'data:image/png;base64,AAAA';
+ const snapshot = regions => ({ coordinateSpace: 'page', viewport: { width: 100, height: 50 }, regions });
+ const redact = async (scan, opts = { imageWidth: 100, imageHeight: 50 }) => {
+ agent._captureScreenshotRedactionSnapshot = async () => scan;
+ const outcome = {};
+ const out = await agent._redactScreenshotDataUrl(1, image, { coordinateSpace: 'page', capturedCssBounds: { x: 0, y: 0, width: 100, height: 50 }, outcome, ...opts });
+ assert.equal(out, image);
+ return outcome.status;
+ };
+ assert.equal(await redact(null), 'scan_unavailable');
+ assert.equal(await redact(snapshot([])), 'no_regions');
+ assert.equal(await redact(snapshot([{ kind: 'input', rect: { x: 400, y: 400, w: 20, h: 10 } }])), 'no_regions_in_view');
+ // Node has no image decoding or canvas: the decode or the pixelation fails.
+ assert.equal(await redact(snapshot([{ kind: 'input', rect: { x: 10, y: 10, w: 20, h: 10 } }])), 'pixelate_failed');
+ assert.equal(await redact(snapshot([{ kind: 'input', rect: { x: 10, y: 10, w: 20, h: 10 } }]), {}), 'decode_failed');
+ });
+
test(`${build}: Gmail's result counter, which takes no target, stays in the vision tool list`, () => {
// Offered on a Gmail result list (getGmailResultCountPolicy).
const names = getToolsForMode('act', { tier: 'full', gmailResultCounting: true, vision: { coordinates: 'pixel' } }).map(tool => tool.function.name);
diff --git a/test/vision-mode.mjs b/test/vision-mode.mjs
index fc7bf8cf5..39a5d9874 100644
--- a/test/vision-mode.mjs
+++ b/test/vision-mode.mjs
@@ -683,6 +683,8 @@ test('both loops of both builds start or end an upload handoff before anything i
assert.ok(tick < resumes[index], `${build}: tick before the tools`);
}
}
+ // Screenshot redaction's pixelation is the same module in both builds.
+ assert.equal(read('src/chrome/src/agent/screenshot-redaction.js'), read('src/firefox/src/agent/screenshot-redaction.js'));
// Chrome's protocol guard marks a chooser only it saw.
assert.match(read('src/chrome/src/cdp/cdp-client.js'), /selector: null,\s*protocolOnly: true,/);
});
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index ddf44ddc9..e439aaeb3 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -165,6 +165,40 @@ for (const dpr of [1, 2, 3]) {
});
}
+test('screenshot redaction pixelates every region, or returns the image untouched when one region cannot be', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await (await browser.newContext()).newPage();
+ await page.goto(`${origin}/fixture`);
+ const result = await page.evaluate(async () => {
+ const { pixelateDataUrl } = await import('/agent/screenshot-redaction.js');
+ const canvas = new OffscreenCanvas(200, 100);
+ const context = canvas.getContext('2d');
+ // Fine stripes: pixelation averages them away.
+ for (let x = 0; x < 200; x += 2) { context.fillStyle = x % 4 ? '#000' : '#fff'; context.fillRect(x, 0, 2, 100); }
+ const blob = await canvas.convertToBlob({ type: 'image/png' });
+ const dataUrl = await new Promise(resolve => { const reader = new FileReader(); reader.onload = () => resolve(reader.result); reader.readAsDataURL(blob); });
+ const regions = [{ rect: { x: 0, y: 0, w: 60, h: 40 } }, { rect: { x: 100, y: 40, w: 60, h: 40 } }];
+ const redacted = await pixelateDataUrl(dataUrl, regions);
+ // The second region's scratch canvas cannot draw.
+ const getContext = OffscreenCanvas.prototype.getContext;
+ let scratch = 0;
+ OffscreenCanvas.prototype.getContext = function (...args) {
+ if (this.width < 50 && ++scratch === 2) return null;
+ return getContext.apply(this, args);
+ };
+ const partial = await pixelateDataUrl(dataUrl, regions);
+ OffscreenCanvas.prototype.getContext = getContext;
+ return { changed: redacted !== dataUrl, untouched: partial === dataUrl, scratch };
+ });
+ assert.equal(result.changed, true, 'both regions pixelated');
+ assert.equal(result.scratch, 2, 'the failure hit the second region');
+ assert.equal(result.untouched, true, 'a region left sharp fails the whole redaction');
+ } finally {
+ await browser.close();
+ }
+});
+
test('content-script vision input (Firefox path) clicks, double-clicks, focuses, scrolls and drags at the exact point', async () => {
const browser = await chromium.launch({ channel: 'chromium' });
try {