Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode: zoom-correct captures, dispatch checks, run and error safety
Emre Sokullu40b0344876ec
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex 9ddd4eaad..bc73d049d 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -39,13 +39,18 @@ The vision prompt (`buildVisionActPrompt`, used through `getVisionActPrompt` inEvery image is labeled with an id: `f<n>` for a full frame, `p<n>` for a patch, `z<n>` for a zoom. The model gives a point in the pixels of the image it read it from and names that image with `image` when it is a patch or zoom; with no `image` the point is read from the current screen (latest frame plus its patches). `VisionSession` (`vision-session.js`) keeps each image's map to the frame and the frame's map to CSS pixels, so the model never rescales anything.-Gemini and Qwen3-VL point in relative 0..1000 coordinates; for them the tools and prompt say so and the session converts (`coordinates: 'normalized'`). Every other model uses image pixels.+Gemini, Qwen3-VL, GLM-4.xV and InternVL point in relative 0..1000 coordinates; for them the tools and prompt say so and the session converts (`coordinates: 'normalized'`). Every other model uses image pixels. An `image` value that is not an exact id (`"z2.png"`, `"zoom z2"`) is refused (`unknown_image`) instead of being read as the current screen.A point is refused before any dispatch when:- its image belongs to an earlier frame (`stale_image`), or the area around it changed in a later patch (`stale_area`);- it lies outside the image;-- it is the second or later coordinate action of one model turn and the live screen changed near it after the earlier action (`_visionStabilityBlock`, checked right before dispatch, after every gate).+- it is the second or later coordinate action of one model turn and the live screen changed near it after the earlier action, or the screen could not be re-captured to check (`_visionStabilityBlock`, checked right before dispatch, after every gate);+- the run has no frame of its own yet (`no_frame`).++A refused point also stops the rest of the model's batch: the calls after it (a `type_text`, say) were planned on that point.++Every run starts on a frame it captured itself: `_beginVisionRun` retires the previous run's screen, so if the first capture fails the model is told so, earlier screenshots leave the request, and coordinates are refused until `take_screenshot` succeeds. Image ids and epochs continue across sessions of one conversation (another model, a restarted worker), and the current screen is the last full frame in the history by position.### Capture, retina and zoom@@ -58,6 +63,7 @@ A point is refused before any dispatch when:| Gemini | ≤ 1536, ≤ 1.2 MP; 0..1000 points | WebP || Qwen2/2.5-VL, UI-TARS | ≤ 1.0 MP, multiples of 28 (so `smart_resize` is the identity) | JPEG || Qwen3-VL / Qwen3.5+ | ≤ 1.05 MP, multiples of 32; 0..1000 points | JPEG |+| GLM-4.xV, InternVL | ≤ 1280, ≤ 1.0 MP, multiples of 28; 0..1000 points | JPEG || Local (llama.cpp, Ollama, LM Studio, WebGPU) | ≤ 1280, ≤ 1.0 MP, multiples of 28 | JPEG (llama.cpp cannot decode WebP) |An image never has more pixels than the CSS viewport: a retina capture is downscaled locally, and `zoom` is the path to fine detail. WebP is ~30% smaller than JPEG for UI screenshots, so uploads are faster; providers bill images by pixels, so the token cost is the same.@@ -65,9 +71,11 @@ An image never has more pixels than the CSS viewport: a retina capture is downscChrome captures with CDP `Page.captureScreenshot`. Two properties of that call drive the implementation (verified in `test/vision-pipeline-browser.mjs`):- **Clip coordinates are document coordinates.** A clip at `0,0` on a scrolled page captures the unrendered top of the document, a blank frame. The clip starts at the visual viewport's page offset (`Page.getLayoutMetrics`).-- **The output is clip × scale × device pixel ratio.** The requested scale divides the DPR out (browser zoom is part of the DPR), and whatever size actually comes back is resampled to the plan (`rasterizeFrame`), so emulated DPRs and compositor quirks cannot change the coordinate map.+- **The clip is in device-independent pixels**, CSS pixels times the browser zoom (`Page.getLayoutMetrics().cssVisualViewport.zoom`), and **the output is clip × scale × device scale factor.** With the DPR (device scale factor × zoom) divided out of the scale, the image comes back at the planned size; whatever size actually comes back is resampled to the plan (`rasterizeFrame`), so emulated DPRs and compositor quirks cannot change the coordinate map. The legacy auto-screenshot clip (`_viewportCaptureClip`) applies the same zoom factor.++Firefox captures with `tabs.captureTab`, whose `scale` is image pixels per CSS pixel (no DPR factor) and whose `rect` is page-relative. The rect is always the visual viewport: without one Firefox captures the whole content area, classic scrollbars included.-Firefox captures with `tabs.captureTab`, whose `scale` is image pixels per CSS pixel (no DPR factor) and whose `rect` is page-relative.+Screenshot redaction maps element boxes into the captured visual-viewport box (page space), so a field next to a scrollbar is still covered.### Diffs@@ -81,6 +89,8 @@ The "seen" buffer copies only patched regions, so changes below the threshold acOnly the current frame, its patches and zooms stay in the model request (`pruneVisionImages`); older images become a short placeholder. Screen updates are app-owned user turns (`webbrainAppOwned`), never mistaken for a new request.+A capture, encoding or input failure never ends the run: a screen update that cannot be prepared becomes a one-line note and the next update is a full frame; a vision tool that fails returns a tool error (an unknown outcome once input may have gone out).+## Input dispatch and safety`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. Only the final dispatch differs:@@ -88,11 +98,17 @@ Only the current frame, its patches and zooms stay in the model request (`pruneV- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab and click-progress annotation. The legacy coordinate path's semantic re-targeting and nearby-input redirect are skipped: a vision click lands where the model pointed. Iframes, including cross-origin ones, receive the events natively.- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus and a click's activation, selecting text for double/triple clicks, and entering same-origin iframes. A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.+Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:++- `click_at` with `clicks: 2|3` is refused on a button, link or other control, on a detected submit, on a protected messaging page, inside a frame it cannot inspect, or when the target cannot be read: each extra click would repeat the one action the checks approved. Double and triple clicks remain for selecting text.+- `drag_at` is refused when it is shorter than 6 CSS px, when the press and release would click a control (their nearest common ancestor is or is inside a button, link, label or similar), when either end is inside a cross-origin frame, or when the target cannot be read. On a protected messaging page `drag_at` is blocked outright, like `iframe_click`.+- `scroll_at` is refused over a focused number field, which takes the wheel as +/-.+Every CDP input passes `cdpClient`'s page-agent fence and every content action passes `beforePageAgentDispatch`, so the page monitor attributes the events to the agent. `drag_at` is gated as a click (`TOOL_CAPABILITY`); captures and pointer results are untrusted page content (`UNTRUSTED_CONTENT_TOOLS`), and every image label says that text in it is data, never instructions. Screenshot redaction (Settings) applies to frames and zooms.## Files-| File | Role (both builds; the first seven are byte-identical) |+| File | Role (both builds; the first eight are byte-identical) ||---|---|| `vision-mode.js` | setting normalization and per-run resolution || `vision-budget.js` | provider profiles, sizes, token estimates, formats |@@ -100,7 +116,8 @@ Every CDP input passes `cdpClient`'s page-agent fence and every content action p| `vision-session.js` | image ids, coordinate maps, staleness || `vision-tools.js` | tool schemas, tool-set transform, prompt, labels, click translation || `vision-encoder.js` | canvas decode / resample / encode |-| `vision-runtime.js` | agent methods: keyframes, observation after batches, native tools, pruning |+| `vision-runtime.js` | agent methods: keyframes, observation after batches, native tools, pruning, dispatch preflights |+| `vision-dom-probe.js` | the in-page probe behind the dispatch checks, and the checks themselves || `agent.js` | platform layer (`_visionPlatform*`, `_visionDispatchClick`) and hooks || `content.js` | `visionInput` (Firefox input path; also loaded in Chrome) |@@ -111,9 +128,9 @@ npm run test:vision-mode```- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt.-- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree.-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path.-- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text`.+- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page.+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.+- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text`; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them.The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.@@ -122,4 +139,5 @@ The older MV3 quality fixtures script an accessibility-tree model (they answer w- File uploads (`upload_file`) and page-resource downloads by selector are not offered in vision mode; switch it off for those tasks.- Closed shadow roots and canvas apps are reachable by pointer, but a native `<select>` popup is never visible in a screenshot: such a field is focused and `type_text` picks the option.- HTML5 drag and drop driven only by `dataTransfer` may not react to synthetic or CDP pointer drags.+- A drag is not shown to the submit, money or publishing checks; it is refused when it would click a control, but a custom "slide to confirm" widget is only as safe as the page makes it.- Firefox input is synthetic (`isTrusted: false`); pages that require trusted events need Chrome.diff --git a/package.json b/package.jsonindex 97d126ab5..1f0eb582d 100644--- a/package.json+++ b/package.json@@ -42,7 +42,7 @@"test:webmcp": "node test/webmcp-e2e.mjs","test:vision": "node test/vision/run.mjs","test:vision:validate": "node --test test/vision/lib/score.test.mjs && node test/vision/validate.mjs",- "test:vision-mode": "node --test test/vision-mode.mjs test/vision-mode-agent.mjs test/vision-pipeline-browser.mjs && node test/quality/vision-mode-mv3.mjs && node test/quality/vision-mode-mv3.mjs --dpr=2 && node test/quality/vision-mode-mv3.mjs --dpr=2 --zoom=1.5",+ "test:vision-mode": "node --test test/vision-mode.mjs test/vision-mode-agent.mjs test/vision-pipeline-browser.mjs && node test/quality/vision-mode-mv3.mjs && node test/quality/vision-mode-mv3.mjs --dpr=2 && node test/quality/vision-mode-mv3.mjs --dpr=2 --zoom=1.5 && node test/quality/vision-mode-mv3.mjs --zoom=1.25 --scroll=100","test:llm-tiny:validate": "node --test test/llm-tiny/cases.test.mjs test/llm-tiny/lib/adapters.test.mjs && node --check test/llm-tiny/run.mjs && node test/llm-tiny/run.mjs --help","test:anonymous": "node test/anonymous/run.mjs","test:ci": "node ci/test.mjs && node ci/cloud-capture.test.mjs",diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 6d2765009..4800e0ad1 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -22,6 +22,7 @@ import { mastodonPublicationIntentMessages, normalizeMastodonPublicationIntent,import { AGENT_TOOLS, AGENT_TOOL_NAMES, RESERVED_AGENT_TOOL_NAMES, RETIRED_AGENT_TOOL_NAMES, getToolsForMode, SYSTEM_PROMPT_BATCHING, SYSTEM_PROMPT_ASK, SYSTEM_PROMPT_ACT, SYSTEM_PROMPT_ACT_COMPACT, SYSTEM_PROMPT_ACT_MID, SYSTEM_PROMPT_LEAN_NOTE, SYSTEM_PROMPT_GENERATIVE_MEDIA, SYSTEM_PROMPT_DEV_APPENDIX, SYSTEM_PROMPT_WEBMCP_ASK, SYSTEM_PROMPT_WEBMCP_ACT, getVisionActPrompt } from './tools.js';import { installVisionRuntime } from './vision-runtime.js';import { VISION_NATIVE_TOOLS } from './vision-tools.js';+import { visionDomProbeExpression } from './vision-dom-probe.js';import { normalizeVisionModeSetting } from './vision-mode.js';import { rejectedCompletionRecovery } from './completion-recovery.js';import { FailureRecoveryRegistry, modelFailureRecovery } from './failure-recovery.js';@@ -16103,6 +16104,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (tc._vision?.error) {const recovery = await recordPreparationFailure(toolIndex, tc._vision.tool || fnName, this._parseToolCallArgs(tc).args || {}, tc._vision.error);if (recovery) return recovery;+ // The rest of the batch was planned on that point: type_text would+ // type into whatever has focus. The model decides again.+ if (interruptFailedBrowserAction(toolIndex, tc._vision.tool || fnName)) { navNotices.length = 0; break; }continue;}// Where this call's time goes before dispatch (_predispatchTimer), from@@ -19322,11 +19326,14 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d*/_viewportCaptureClip(probe, cssW, cssH, scale = 1) {const dpr = Math.max(0.25, Number(probe?.dpr) || 1);+ // The clip is in device-independent pixels: CSS px times browser zoom+ // (the DPR already contains the zoom, so the scale stays scale / DPR).+ const zoom = Math.max(0.25, Number(probe?.zoom) || 1);return {- x: Math.max(0, Number(probe?.scrollX) || 0),- y: Math.max(0, Number(probe?.scrollY) || 0),- width: cssW,- height: cssH,+ x: Math.max(0, Number(probe?.scrollX) || 0) * zoom,+ y: Math.max(0, Number(probe?.scrollY) || 0) * zoom,+ width: cssW * zoom,+ height: cssH * zoom,scale: scale / dpr,};}@@ -20123,7 +20130,15 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d};})()`);- return probe?.result?.value || null;+ const value = probe?.result?.value || null;+ if (value) {+ // Browser zoom: capture clips are in CSS px x zoom (_viewportCaptureClip).+ try {+ const zoom = Number((await cdpClient.sendCommand(tabId, 'Page.getLayoutMetrics'))?.cssVisualViewport?.zoom);+ if (zoom > 0) value.zoom = zoom;+ } catch {}+ }+ return value;} catch {return null;}@@ -20609,15 +20624,19 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dscrollX: Number.isFinite(Number(visual?.pageX)) ? Number(visual.pageX) : (Number(value.sx) || 0),scrollY: Number.isFinite(Number(visual?.pageY)) ? Number(visual.pageY) : (Number(value.sy) || 0),devicePixelRatio: Number(value.dpr) || 1,+ // Browser zoom (CSS px -> device-independent px); part of the DPR.+ zoom: Number(visual?.zoom) > 0 ? Number(visual.zoom) : 1,url,};}/*** Capture the viewport (or `cssRegion` of it) close to outWidth x- * outHeight. The compositor renders clip x scale x device pixel ratio, so- * the requested scale divides the DPR out; whatever size actually comes- * back is resampled to the plan by the runtime.+ * outHeight. The clip is in device-independent pixels (CSS px x browser+ * zoom) and the compositor renders clip x scale x device scale factor, so+ * with the DPR (= device scale factor x zoom) divided out of the scale the+ * output is outWidth wide. Whatever size actually comes back is resampled+ * to the plan by the runtime.*/async _visionPlatformCapture(tabId, { probe, cssRegion = null, outWidth, outHeight }) {await cdpClient.attach(tabId);@@ -20625,16 +20644,17 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dawait this._preparePageForCapture(tabId);const region = cssRegion || { x: 0, y: 0, width: probe.cssWidth, height: probe.cssHeight };const dpr = Math.max(0.25, Number(probe?.devicePixelRatio) || 1);+ const zoom = Math.max(0.25, Number(probe?.zoom) || 1);const scale = Math.max(0.05, Math.min(4, Number(outWidth) / (Math.max(1, region.width) * dpr)));const shot = await this._withIndicatorsHidden(tabId, () => cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {format: 'jpeg',quality: 90,fromSurface: true,clip: {- x: (Number(probe?.scrollX) || 0) + region.x,- y: (Number(probe?.scrollY) || 0) + region.y,- width: Math.max(1, region.width),- height: Math.max(1, region.height),+ x: ((Number(probe?.scrollX) || 0) + region.x) * zoom,+ y: ((Number(probe?.scrollY) || 0) + region.y) * zoom,+ width: Math.max(1, region.width) * zoom,+ height: Math.max(1, region.height) * zoom,scale,},}));@@ -20662,8 +20682,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// pointer already rests at the point.await cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', { type: 'mouseMoved', x: Math.max(0, x - 1), y: Math.max(0, y - 1), button: 'none', buttons: 0 });await cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', { type: 'mouseMoved', x, y, button: 'none', buttons: 0 });- const hit = await this._visionHitTest(tabId, x, y);- return { success: true, dispatched: true, method: 'cdp-hover', ...(hit?.target ? { target: hit.target } : {}) };+ const point = (await this._visionDomProbe(tabId, { points: [{ x, y }] }))?.points?.[0];+ const target = point?.found ? { tag: point.tag, ...(point.role ? { role: point.role } : {}) } : null;+ return { success: true, dispatched: true, method: 'cdp-hover', ...(target ? { target } : {}) };}async _visionPlatformDrag(tabId, { from, to, steps = 12 }) {@@ -20691,30 +20712,46 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dreturn { success: true, dispatched: pressed, method: 'cdp-drag' };}- /** Tag and role under a CSS point (no accessible name: it can identify the signed-in user). */- async _visionHitTest(tabId, x, y) {+ /**+ * What is under CSS points (vision-dom-probe.js), evaluated in an isolated+ * world so page scripts cannot redefine the DOM methods the checks use. The+ * world is created once per document (frame loader) and reused.+ */+ async _visionPlatformDomProbe(tabId, request) {+ await cdpClient.attach(tabId);+ await cdpClient.sendCommand(tabId, 'Runtime.enable');+ if (!this._visionProbeContexts) this._visionProbeContexts = new Map();+ let contextId = null;try {- const response = await cdpClient.evaluate(tabId, `(() => {- let el = document.elementFromPoint(${Number(x)}, ${Number(y)});- for (let depth = 0; el?.shadowRoot && depth < 8; depth++) {- const inner = el.shadowRoot.elementFromPoint(${Number(x)}, ${Number(y)});- if (!inner || inner === el) break;- el = inner;- }- if (!el) return null;- const select = el.closest ? el.closest('select') : null;- if (select) {- select.focus();- return { isSelect: true, current: select.options[select.selectedIndex]?.text?.trim() || '',- options: Array.from(select.options).slice(0, 60).map(option => option.text.trim()), target: { tag: 'select' } };- }- const role = el.getAttribute && el.getAttribute('role');- return { isSelect: false, target: { tag: String(el.tagName || '').toLowerCase(), ...(role ? { role: role.slice(0, 32) } : {}) } };- })()`);- return response?.result?.value || null;- } catch {- return null;+ await cdpClient.sendCommand(tabId, 'Page.enable');+ const frame = (await cdpClient.sendCommand(tabId, 'Page.getFrameTree'))?.frameTree?.frame;+ const cached = this._visionProbeContexts.get(tabId);+ if (frame?.id && cached?.frameId === frame.id && cached.loaderId === frame.loaderId) {+ contextId = cached.contextId;+ } else if (frame?.id) {+ contextId = (await cdpClient.sendCommand(tabId, 'Page.createIsolatedWorld', {+ frameId: frame.id, worldName: 'webbrain-vision-probe', grantUniveralAccess: false,+ }))?.executionContextId || null;+ if (contextId) this._visionProbeContexts.set(tabId, { frameId: frame.id, loaderId: frame.loaderId, contextId });+ }+ } catch (error) {+ if (error?.code === 'page_feedback_pending') throw error;+ contextId = null;+ }+ let response;+ try {+ response = await cdpClient.sendCommand(tabId, 'Runtime.evaluate', {+ expression: visionDomProbeExpression(request),+ returnByValue: true,+ ...(contextId ? { contextId } : {}),+ });+ } catch (error) {+ // A context that went away with its document is made again next time.+ this._visionProbeContexts.delete(tabId);+ throw error;}+ if (response?.exceptionDetails) return null;+ return response?.result?.value || null;}/**@@ -20728,6 +20765,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dvision = null,messageRecipientGuardRequired = false,messageRecipientDispatchBinding = null,+ detectedSubmitAction = null,throwIfAborted = () => {},markDispatched = () => {},cancelPressedPointer = async () => {},@@ -20739,17 +20777,17 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dthrowIfAborted();await cdpClient.attach(tabId);throwIfAborted();- // A native <select> opens an OS popup no screenshot can show; it is- // focused instead so type_text picks the option.- const hit = await this._visionHitTest(tabId, x, y);+ // A native <select> opens an OS popup no screenshot can show (it is+ // focused for type_text instead), and a double/triple click on a control+ // would repeat the one action the checks approved.+ const preflight = await this._visionClickPreflight(tabId, {+ x, y, button, clickCount,+ recipientGuarded: messageRecipientGuardRequired === true,+ submitDetected: detectedSubmitAction?.isSubmit === true,+ });throwIfAborted();- if (hit?.isSelect && button === 'left') {- return {- success: false, dispatched: false, noDispatch: true, nativeSelect: true, tag: 'SELECT', text: hit.current,- options: hit.options,- error: `That point is a native dropdown (current: "${hit.current}"). It is focused now: choose an option with type_text({text: "<option label>"}). Options: ${hit.options.join(', ')}`,- };- }+ if (preflight.refusal) return preflight.refusal;+ const hit = { target: preflight.target };const beforeTabIds = new Set((await chrome.tabs.query({})).map(tab => tab.id));const progressBefore = await this._clickProgressSnapshot(tabId);throwIfAborted();@@ -32254,7 +32292,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dasync _messageRecipientGuardBlock(tabId, toolName, args = {}, pageUrl = '', executionContext = null) {const name = String(toolName || '');const ordinaryGuardedTools = new Set(['click', 'click_ax', 'set_field', 'press_keys', 'upload_file']);- const unbindableDispatchTools = new Set(['iframe_click', 'execute_js', 'execute_webmcp_tool']);+ // drag_at (vision mode) presses and releases where the model points; no+ // recipient binding covers that, and a short drag is a click.+ const unbindableDispatchTools = new Set(['iframe_click', 'execute_js', 'execute_webmcp_tool', 'drag_at']);if (!ordinaryGuardedTools.has(name) && !unbindableDispatchTools.has(name)) return null;if (name === 'press_keys' && String(args?.key || '') !== 'Enter') return null;if (name === 'set_field' && args?.submit !== true) return null;@@ -32304,7 +32344,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d? (!uploadSourceKey? 'Attachment blocked: use exactly one attachmentId, downloadId, or absolute filePath. Remove fallback sources and retry with the same authorized recipient.': 'Attachment blocked: choose an exact, unique file-input selector in the active authorized conversation. Re-read the file-input inventory before retrying.')- : `Message action blocked: ${name} cannot bind its effects to the verified active recipient. Use click, click_ax, set_field({submit:true}), or a single Enter press on the visible composer instead.`,+ : name === 'drag_at'+ ? 'Message action blocked: drag_at cannot bind its effects to the verified active recipient. Use click_at on the control, or a single Enter press in the visible composer, instead.'+ : `Message action blocked: ${name} cannot bind its effects to the verified active recipient. Use click, click_ax, set_field({submit:true}), or a single Enter press on the visible composer instead.`,};}@@ -37469,6 +37511,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dvoid cdpClient.cleanupTab(tabId).catch(() => {});this._cancelPendingPlans(tabId, 'tab closed');this._isPdfTabCache.delete(tabId);+ this._visionForgetTab?.(tabId);this._lastTypeFieldIdent?.delete(tabId);this._lastTypeFieldEpoch?.delete(tabId);// A cleared conversation or closed tab takes its filled credentials along.@@ -51006,6 +51049,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dvision: dispatchContext._vision,messageRecipientGuardRequired,messageRecipientDispatchBinding,+ detectedSubmitAction: dispatchContext.detectedSubmitAction || null,throwIfAborted: throwIfEarlyCdpAborted,markDispatched: markEarlyCdpDispatched,cancelPressedPointer: cancelEarlyCdpPressedPointer,@@ -55730,6 +55774,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// Lean mode is resolved once, now that the run's mode and provider are// known, and holds for every tool list, prompt and guard of this run.const previousLeanRun = this._leanRuns?.get(tabId);+ const previousVisionRun = this._visionRunSnapshots?.get(tabId);this._beginLeanRun(tabId, mode);const previousBotMemory = this._botMemorySessions.get(tabId);let botMemorySession = null;@@ -55773,7 +55818,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}this._planExecutionGuards.delete(tabId);this._runModeOverrides.delete(tabId);- this._endVisionRun(tabId);+ this._endVisionRun(tabId, previousVisionRun);if (previousLeanRun) this._leanRuns.set(tabId, previousLeanRun);else this._leanRuns?.delete(tabId);if (previousRunProviderOverride) this._runProviderOverrides.set(tabId, previousRunProviderOverride);@@ -57779,6 +57824,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// Lean mode is resolved once, now that the run's mode and provider are// known, and holds for every tool list, prompt and guard of this run.const previousLeanRun = this._leanRuns?.get(tabId);+ const previousVisionRun = this._visionRunSnapshots?.get(tabId);this._beginLeanRun(tabId, mode);const previousBotMemory = this._botMemorySessions.get(tabId);let botMemorySession = null;@@ -57821,7 +57867,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}this._planExecutionGuards.delete(tabId);this._runModeOverrides.delete(tabId);- this._endVisionRun(tabId);+ this._endVisionRun(tabId, previousVisionRun);if (previousLeanRun) this._leanRuns.set(tabId, previousLeanRun);else this._leanRuns?.delete(tabId);if (previousRunProviderOverride) this._runProviderOverrides.set(tabId, previousRunProviderOverride);diff --git a/src/chrome/src/agent/vision-budget.js b/src/chrome/src/agent/vision-budget.jsindex 3f8c8807d..277a9ac29 100644--- a/src/chrome/src/agent/vision-budget.js+++ b/src/chrome/src/agent/vision-budget.js@@ -52,6 +52,13 @@ export const VISION_PROFILES = Object.freeze({maxLongEdge: Infinity, maxPixels: 1_048_576, maxShortEdge: Infinity, align: 32,format: 'jpeg', quality: 0.8,}),+ relative1000: Object.freeze({+ id: 'relative1000', tokenModel: 'patch28', coordinates: 'normalized',+ // GLM-4.xV and InternVL are trained on boxes normalized to 0..1000.+ // JPEG because they are often served by local runtimes too.+ maxLongEdge: 1280, maxPixels: 1_003_520, maxShortEdge: Infinity, align: 28,+ format: 'jpeg', quality: 0.8,+ }),local: Object.freeze({id: 'local', tokenModel: 'patch28', coordinates: 'pixel',// llama.cpp's image loader (stb_image) cannot decode WebP; JPEG is the@@ -85,6 +92,7 @@ export function detectVisionProfile(provider = {}) {const category = lower(config.category || provider?.category);// Qwen3-VL and the natively multimodal Qwen3.5+ point in relative 0..1000.if (/qwen3(\.\d+)?-(vl|omni)|qwen3\.[5-9]/.test(model)) return VISION_PROFILES.qwen3;+ if (/glm-4(\.\d+)?v|internvl/.test(model)) return VISION_PROFILES.relative1000;if (/gemini/.test(model)) return VISION_PROFILES.gemini;if (/claude|anthropic/.test(model) || ['anthropic', 'vertex-anthropic', 'oauth-claude', 'claude'].includes(name)) {return VISION_PROFILES.anthropic;diff --git a/src/chrome/src/agent/vision-dom-probe.js b/src/chrome/src/agent/vision-dom-probe.jsnew file mode 100644index 000000000..7e46c14c6--- /dev/null+++ b/src/chrome/src/agent/vision-dom-probe.js@@ -0,0 +1,200 @@+// What is under a viewport point, for vision-mode dispatch checks: a double+// click that would repeat a button's action, a drag that is really a click,+// a wheel over a focused number field, a native <select>, an embedded frame+// whose content cannot be inspected. Shared verbatim by both builds.+//+// visionDomProbe runs inside the page (Chrome: an isolated world through CDP;+// Firefox: tabs.executeScript in the content-script world, so page scripts+// cannot redefine the DOM methods it calls). It is serialized with+// toString(), so it must stay self-contained: no imports, no outer names.++/**+ * @param {{points: {x:number,y:number}[], pair?: boolean, focusSelect?: boolean}} request+ * points are CSS pixels of the top-level viewport. `pair` asks what a+ * press at points[0] and a release at points[1] would click. `focusSelect`+ * focuses a native <select> under points[0] (a click on it is answered+ * with type_text instead of an OS popup no screenshot can show).+ */+export function visionDomProbe(request) {+ const CLICKABLE = [+ 'a[href]', 'area[href]', 'button', 'summary', 'label', 'select', 'option', '[onclick]',+ '[role="button"]', '[role="link"]', '[role="menuitem"]', '[role="menuitemcheckbox"]',+ '[role="menuitemradio"]', '[role="tab"]', '[role="checkbox"]', '[role="radio"]',+ '[role="switch"]', '[role="option"]',+ 'input[type="submit"]', 'input[type="button"]', 'input[type="reset"]', 'input[type="image"]',+ 'input[type="checkbox"]', 'input[type="radio"]', 'input[type="file"]', 'input[type="color"]',+ ].join(',');+ // A click inside a text field only places the caret, even when a <label>+ // or link wraps the field, so the walk up stops there.+ const TEXT_ENTRY = [+ 'textarea', '[contenteditable=""]', '[contenteditable="true"]', '[contenteditable="plaintext-only"]',+ 'input:not([type])', 'input[type="text"]', 'input[type="search"]', 'input[type="email"]',+ 'input[type="url"]', 'input[type="tel"]', 'input[type="password"]', 'input[type="number"]',+ 'input[type="date"]', 'input[type="datetime-local"]', 'input[type="time"]', 'input[type="month"]',+ 'input[type="week"]', 'input[type="range"]',+ ].join(',');+ const req = request && typeof request === 'object' ? request : {};+ // Composed parent: through shadow roots to their host and out of+ // same-origin frames to the frame element.+ const up = node => {+ if (!node) return null;+ if (node.nodeType === 9) {+ try { return (node.defaultView && node.defaultView.frameElement) || null; } catch (e) { return null; }+ }+ const parent = node.parentNode;+ if (parent && parent.nodeType === 11) return parent.host || null;+ return parent || null;+ };+ const hit = (x, y) => {+ let doc = document;+ let cx = x;+ let cy = y;+ for (let depth = 0; depth < 8; depth++) {+ let el = doc.elementFromPoint(cx, cy);+ for (let i = 0; el && el.shadowRoot && i < 8; i++) {+ const inner = el.shadowRoot.elementFromPoint(cx, cy);+ if (!inner || inner === el) break;+ el = inner;+ }+ if (!el) return { el: null, opaque: false };+ const tag = String(el.tagName || '').toUpperCase();+ if (tag === 'IFRAME' || tag === 'FRAME' || tag === 'OBJECT' || tag === 'EMBED') {+ let inner = null;+ try { inner = el.contentDocument; } catch (e) { inner = null; }+ if (inner && inner.documentElement && (tag === 'IFRAME' || tag === 'FRAME')) {+ const rect = el.getBoundingClientRect();+ cx -= rect.left + (el.clientLeft || 0);+ cy -= rect.top + (el.clientTop || 0);+ doc = inner;+ continue;+ }+ // A cross-origin frame (or plugin): its content is not inspectable.+ return { el, opaque: true };+ }+ return { el, opaque: false };+ }+ return { el: null, opaque: false };+ };+ const clickableOf = node => {+ for (let n = node, i = 0; n && i < 256; n = up(n), i++) {+ if (n.nodeType === 1 && typeof n.matches === 'function') {+ try {+ if (n.matches(CLICKABLE)) return n;+ if (n.matches(TEXT_ENTRY)) return null;+ } catch (e) { /* invalid in this document */ }+ }+ }+ return null;+ };+ const points = Array.isArray(req.points) ? req.points.slice(0, 2) : [];+ const found = points.map(point => hit(Number(point && point.x), Number(point && point.y)));+ const infos = found.map((item, index) => {+ const el = item.el;+ if (!el) return { found: false };+ const control = clickableOf(el);+ const info = {+ found: true,+ tag: String(el.tagName || '').toLowerCase(),+ clickable: !!control,+ opaqueFrame: item.opaque === true,+ };+ const role = el.getAttribute ? el.getAttribute('role') : '';+ if (role) info.role = String(role).slice(0, 32);+ if (control) info.control = String(control.tagName || '').toLowerCase();+ if (typeof el.matches === 'function' && el.matches('input[type="number"]')) {+ const root = el.getRootNode ? el.getRootNode() : el.ownerDocument;+ info.focusedNumberInput = !!root && root.activeElement === el;+ }+ const select = el.closest ? el.closest('select') : null;+ if (select && !select.multiple && !(select.size > 1)) {+ const options = Array.from(select.options || []).slice(0, 60).map(option => String(option.text || '').trim());+ const current = select.options && select.options[select.selectedIndex]+ ? String(select.options[select.selectedIndex].text || '').trim() : '';+ info.select = { current, options };+ if (index === 0 && req.focusSelect === true) {+ try { select.focus(); } catch (e) { /* focus is best effort */ }+ }+ }+ return info;+ });+ const result = { points: infos };+ if (req.pair === true && found.length === 2) {+ if (!found[0].el || !found[1].el) {+ result.pair = { known: false };+ } else {+ // The click a press and release produce goes to their nearest common+ // ancestor; it activates a control when that ancestor is (inside) one.+ const ancestors = new Set();+ for (let n = found[1].el, i = 0; n && i < 256; n = up(n), i++) ancestors.add(n);+ let common = null;+ for (let n = found[0].el, i = 0; n && i < 256; n = up(n), i++) {+ if (ancestors.has(n)) { common = n; break; }+ }+ const control = common ? clickableOf(common) : null;+ result.pair = { known: true, clickEquivalent: !!control };+ if (control) result.pair.control = String(control.tagName || '').toLowerCase();+ }+ }+ return result;+}++/** An expression that evaluates visionDomProbe(request) in a page. */+export function visionDomProbeExpression(request) {+ return `(${visionDomProbe.toString()})(${JSON.stringify(request || {})})`;+}++/** Minimum press-to-release distance (CSS px) for drag_at. */+export const VISION_MIN_DRAG_CSS_PX = 6;++/**+ * Dispatch checks shared by both builds. Each returns a no-dispatch result+ * to send back to the model, or null when the action may go ahead.+ */+export function visionMultiClickRefusal(probe, { clickCount = 1, recipientGuarded = false, submitDetected = false } = {}) {+ if (!(clickCount > 1)) return null;+ const point = probe?.points?.[0];+ const reason = recipientGuarded ? 'this is a protected messaging page'+ : submitDetected ? 'the point is a submit control'+ : !point || !point.found ? 'WebBrain could not check what is under the pointer'+ : point.opaqueFrame ? 'the point is inside an embedded frame WebBrain cannot inspect'+ : point.clickable ? `the point is on a ${point.control || 'control'}`+ : '';+ if (!reason) return null;+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'repeated_click',+ error: `Not dispatched: a ${clickCount === 2 ? 'double' : 'triple'} click was refused because ${reason}, where every extra click would repeat the control's action (for example submit twice). Use click_at with clicks: 1. Double and triple clicks are for selecting text.`,+ };+}++export function visionDragRefusal(probe, from, to) {+ const distance = Math.hypot((Number(to?.x) || 0) - (Number(from?.x) || 0), (Number(to?.y) || 0) - (Number(from?.y) || 0));+ const refuse = (code, reason) => ({+ success: false, dispatched: false, noDispatch: true, visionInputRefused: code,+ error: `Not dispatched: ${reason}`,+ });+ if (distance < VISION_MIN_DRAG_CSS_PX) {+ return refuse('drag_too_short', 'a drag this short is a click. Use click_at to click.');+ }+ if (!probe?.points || probe.points.length < 2 || !probe.pair) {+ return refuse('drag_unchecked', 'WebBrain could not check what is under the pointer. Take a screenshot and try again, or use click_at / scroll_at.');+ }+ if (probe.points.some(point => point?.opaqueFrame)) {+ return refuse('drag_opaque_frame', 'the drag starts or ends inside an embedded frame whose content WebBrain cannot inspect, so it could act as a click there. Use click_at or scroll_at instead.');+ }+ if (!probe.pair.known) {+ return refuse('drag_unchecked', 'there is nothing under one end of the drag. Choose points on the page.');+ }+ if (probe.pair.clickEquivalent) {+ return refuse('drag_is_click', `pressing and releasing there would click the ${probe.pair.control || 'control'} under both ends. Use click_at to click it; drag_at is for sliders, reordering and selecting.`);+ }+ return null;+}++export function visionWheelRefusal(probe) {+ const point = probe?.points?.[0];+ if (!point?.focusedNumberInput) return null;+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'wheel_number_input',+ error: 'Not dispatched: the point is over a focused number field, and the mouse wheel would change its value instead of scrolling. Scroll at another point, or press Escape / click elsewhere first.',+ };+}diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.jsindex 962668c24..b6ecc119a 100644--- a/src/chrome/src/agent/vision-runtime.js+++ b/src/chrome/src/agent/vision-runtime.js@@ -27,8 +27,24 @@ import {VISION_SCROLL_STEP_CSS_PX, VISION_DEFAULT_SCROLL_STEPS, VISION_MAX_WAIT_SECONDS,translateVisionClick, visionKeyframeNote, visionPatchNote, visionNoChangeNote, visionZoomNote,} from './vision-tools.js';+import {+ VISION_MIN_DRAG_CSS_PX, visionMultiClickRefusal, visionDragRefusal, visionWheelRefusal,+} from './vision-dom-probe.js';const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));++// Errors that steer the run (stop, deadlines, page feedback) pass through;+// anything else in a vision tool or screen update becomes a result or a note+// instead of ending the run.+function isControlFlowError(error) {+ return error?.name === 'AbortError'+ || error?.code === 'content_action_timeout'+ || error?.code === 'page_feedback_pending';+}++function errorText(error) {+ return String(error?.message || error || 'unknown error').slice(0, 200);+}// A point within this many keyframe px of a change made earlier in the same// batch is refused: the model chose it on the screen before that change.const STABILITY_MARGIN_PX = 12;@@ -55,32 +71,53 @@ function regionLabel(region) {return { x: Math.round(region.x), y: Math.round(region.y), width: Math.round(region.width), height: Math.round(region.height) };}+/** Highest epoch and image number a history already used (session seeds). */+export function visionHistoryCounters(messages) {+ let epoch = 0;+ let counter = 0;+ if (Array.isArray(messages)) {+ for (const message of messages) {+ const tag = visionFrameTag(message);+ if (!tag) continue;+ epoch = Math.max(epoch, Number(tag.epoch) || 0);+ for (const id of Array.isArray(tag.ids) ? tag.ids : []) {+ const number = Number(String(id).slice(1));+ if (Number.isFinite(number)) counter = Math.max(counter, number);+ }+ }+ }+ return { epoch, counter };+}+/**- * Keep only the current vision epoch's images (latest frame, its patches and- * zooms). Exported for the unit suite; agents call it via _pruneOldImages.+ * Keep only the current screen's images: the LAST full frame in the history+ * and the patches and zooms after it. Position decides, not the epoch+ * number, which starts over in a new session (another model, a restarted+ * worker). Exported for the unit suite; agents call it via _pruneOldImages.* Returns { messages, latestEpoch } where `messages` is a shallow copy.*/export function pruneVisionImages(messages) {if (!Array.isArray(messages)) return { messages, latestEpoch: 0 };- let latestEpoch = 0;let latestKeyIndex = -1;- messages.forEach((message, index) => {- const tag = visionFrameTag(message);- if (!tag) return;- const epoch = Number(tag.epoch) || 0;- if (epoch >= latestEpoch) latestEpoch = epoch;- if (tag.kind === 'key' && epoch >= latestEpoch) latestKeyIndex = index;- });- if (!latestEpoch) return { messages, latestEpoch: 0 };+ let retireIndex = -1;+ for (let index = messages.length - 1; index >= 0 && latestKeyIndex < 0; index--) {+ const tag = visionFrameTag(messages[index]);+ if (tag?.kind === 'key') latestKeyIndex = index;+ else if (tag?.kind === 'retire' && retireIndex < 0) retireIndex = index;+ }+ if (latestKeyIndex < 0) return { messages, latestEpoch: 0 };+ const latestEpoch = Math.max(1, Number(visionFrameTag(messages[latestKeyIndex]).epoch) || 1);+ // A run whose first capture failed retires the last frame: it showed an+ // earlier run's screen, and no point may be read from it any more.+ const boundary = retireIndex > latestKeyIndex ? retireIndex : latestKeyIndex;const out = messages.map((message, index) => {const tag = visionFrameTag(message);if (!Array.isArray(message?.content)) return message;- const keep = tag ? (Number(tag.epoch) || 0) === latestEpoch : false;- if (tag && keep) return message;+ if (tag && index >= boundary) return message;// Untagged screenshots older than the current frame (an Ask-mode capture// before an automatic switch to Act) are superseded by it too. User// attachments are never touched.- if (!tag && (index > latestKeyIndex || message.role !== 'user')) return message;+ if (!tag && (index > boundary || message.role !== 'user')) return message;let inAttachments = false;let changed = false;const content = message.content.map(block => {@@ -121,31 +158,90 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {return state.active ? { coordinates: state.coordinates } : null;};- /** Freeze the setting for the length of one run (see vision-mode.js). */+ /**+ * Freeze the setting for the length of one run (see vision-mode.js). The+ * previous run's screen is retired: a run acts only on frames it captured+ * itself, so a failed first capture can never leave the model aiming at+ * the last run's page.+ */proto._beginVisionRun = function _beginVisionRun(tabId) {if (!this._visionRunSnapshots) this._visionRunSnapshots = new Map();this._visionRunSnapshots.set(tabId, {setting: normalizeVisionModeSetting(this.visionMode, 'off'),quality: normalizeVisionQuality(this.visionQuality),});+ this._visionSessions?.get(tabId)?.retireKeyframe?.();};- proto._endVisionRun = function _endVisionRun(tabId) {- this._visionRunSnapshots?.delete(tabId);- this._visionBatches?.delete(tabId);+ /** `previous` is the snapshot of an enclosing run on the same tab, if any. */+ proto._endVisionRun = function _endVisionRun(tabId, previous = null) {+ if (previous) this._visionRunSnapshots?.set(tabId, previous);+ else this._visionRunSnapshots?.delete(tabId);+ };++ /** A closed tab or cleared conversation drops its screen state. */+ proto._visionForgetTab = function _visionForgetTab(tabId) {+ this._visionSessions?.delete(tabId);+ this._visionProbeContexts?.delete(tabId);};proto._visionSessionFor = function _visionSessionFor(tabId, state) {if (!this._visionSessions) this._visionSessions = new Map();- let session = this._visionSessions.get(tabId);- if (!session || session.coordinates !== state.coordinates || session.profileId !== state.profileId) {- session = new VisionSession({ coordinates: state.coordinates });- session.profileId = state.profileId;- this._visionSessions.set(tabId, session);+ const previous = this._visionSessions.get(tabId);+ if (previous && previous.coordinates === state.coordinates && previous.profileId === state.profileId) return previous;+ // A new session (first use, another model, a restarted worker) continues+ // the history's numbering so its ids and epochs never repeat.+ const history = visionHistoryCounters(this.conversations?.get?.(tabId));+ const session = new VisionSession({+ coordinates: state.coordinates,+ startEpoch: Math.max(history.epoch, previous?.epoch || 0),+ startCounter: Math.max(history.counter, previous?.counter || 0),+ });+ session.profileId = state.profileId;+ if (previous) {+ for (const [id, retired] of previous.retired) session.retired.set(id, retired);+ for (const [id, image] of previous.images) session.retired.set(id, { epoch: image.epoch, replacedBy: null });}+ this._visionSessions.set(tabId, session);return session;};+ /** What is under CSS points (vision-dom-probe.js); null when it cannot be read. */+ proto._visionDomProbe = async function _visionDomProbe(tabId, request) {+ if (typeof this._visionPlatformDomProbe !== 'function') return null;+ try {+ return await this._visionPlatformDomProbe(tabId, request);+ } catch (error) {+ if (isControlFlowError(error)) throw error;+ return null;+ }+ };++ /**+ * Checks right before a vision click is dispatched, in both builds: a+ * native <select> is focused for type_text (its popup never shows in a+ * screenshot), and a double/triple click that would repeat a control's+ * action is refused. Returns { refusal } or { target }.+ */+ proto._visionClickPreflight = async function _visionClickPreflight(tabId, {+ x, y, button = 'left', clickCount = 1, recipientGuarded = false, submitDetected = false,+ } = {}) {+ const probe = await this._visionDomProbe(tabId, { points: [{ x, y }], focusSelect: button === 'left' });+ const point = probe?.points?.[0] || null;+ if (point?.select && button === 'left') {+ return {+ refusal: {+ success: false, dispatched: false, noDispatch: true, nativeSelect: true, tag: 'SELECT',+ text: point.select.current, options: point.select.options,+ error: `That point is a native dropdown (current: "${point.select.current}"). It is focused now: choose an option with type_text({text: "<option label>"}). Options: ${point.select.options.join(', ')}`,+ },+ };+ }+ const refusal = visionMultiClickRefusal(probe, { clickCount, recipientGuarded, submitDetected });+ if (refusal) return { refusal };+ return { target: point?.found ? { tag: point.tag, ...(point.role ? { role: point.role } : {}) } : null };+ };+proto._visionTraceNote = function _visionTraceNote(tabId, note, data) {const runId = this.currentRunId?.get?.(tabId);if (!runId || !trace?.recordNote) return;@@ -171,7 +267,15 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {let dataUrl = await this._visionPlatformCapture(tabId, { probe, outWidth: size.width, outHeight: size.height });if (!dataUrl) throw new Error('The browser returned no screenshot.');if (this.screenshotRedaction && typeof this._redactScreenshotDataUrl === 'function') {- try { dataUrl = await this._redactScreenshotDataUrl(tabId, dataUrl, { coordinateSpace: 'viewport' }); } catch { /* keep the capture */ }+ try {+ // The capture spans the visual viewport (no scrollbars), so the+ // redactor gets that box in page space rather than its own+ // innerWidth-based viewport.+ dataUrl = await this._redactScreenshotDataUrl(tabId, dataUrl, {+ coordinateSpace: 'page',+ capturedCssBounds: { x: probe.scrollX || 0, y: probe.scrollY || 0, width: probe.cssWidth, height: probe.cssHeight },+ });+ } catch { /* keep the capture */ }}const raster = await this._visionRasterize(dataUrl, size.width, size.height);return { raster, probe, size, profile };@@ -288,8 +392,11 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {try {return await this._visionPushKeyframe(tabId, messages, onUpdate, { state, reason });} catch (error) {- const message = `[SCREEN: the first screenshot could not be captured (${String(error?.message || error).slice(0, 160)}). Call take_screenshot to try again.]`;- messages.push(visionUserMessage(message, { epoch: 0, kind: 'error' }));+ if (isControlFlowError(error)) throw error;+ // The run starts with no screen (_beginVisionRun retired the last one);+ // the tag retires the last frame in the history for the model too.+ const message = `[SCREEN: the first screenshot could not be captured (${errorText(error).slice(0, 160)}). Earlier screenshots show an older screen and cannot be used for coordinates. Call take_screenshot to try again.]`;+ messages.push(visionUserMessage(message, { epoch: 0, kind: 'retire' }));this._visionTraceNote(tabId, 'vision_frame_error', { stage: 'start', error: String(error?.message || error).slice(0, 200) });return null;}@@ -323,16 +430,32 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {return { kind: 'zoom' };}if (!observe) return null;+ const failed = (stage, error) => {+ // The next update is a full frame; until then the model knows the+ // screen it last saw may be out of date.+ session.forceKeyframe = true;+ messages.push(visionUserMessage(`[SCREEN: the screen could not be ${stage === 'observe' ? 'captured' : 'prepared'} after the action (${errorText(error).slice(0, 160)}). Call take_screenshot to look again.]`,+ { epoch: session.epoch, kind: 'error' }));+ this._visionTraceNote(tabId, 'vision_frame_error', { stage, error: errorText(error) });+ return { kind: 'error' };+ };let frame;try {frame = await this._visionCaptureSettled(tabId, state);} catch (error) {- session.forceKeyframe = true;- messages.push(visionUserMessage(`[SCREEN: the screen could not be captured after the action (${String(error?.message || error).slice(0, 160)}). Call take_screenshot to look again.]`,- { epoch: session.epoch, kind: 'error' }));- this._visionTraceNote(tabId, 'vision_frame_error', { stage: 'observe', error: String(error?.message || error).slice(0, 200) });- return { kind: 'error' };+ if (isControlFlowError(error)) throw error;+ return failed('observe', error);}+ try {+ return await this._visionObserveFrame(tabId, messages, onUpdate, { state, session, frame });+ } catch (error) {+ if (isControlFlowError(error)) throw error;+ return failed('encode', error);+ }+ };++ /** The part of _visionObserveAfterBatch after a settled capture. */+ proto._visionObserveFrame = async function _visionObserveFrame(tabId, messages, onUpdate, { state, session, frame }) {const meta = {cssWidth: frame.probe.cssWidth, cssHeight: frame.probe.cssHeight,devicePixelRatio: frame.probe.devicePixelRatio, url: frame.probe.url,@@ -437,13 +560,19 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {const state = this._visionRunState(tabId);const session = this._visionSessions?.get(tabId);if (!state.active || !session?.keyframe || !session.seenPixels) return null;- let frame;- try { frame = await this._visionCaptureFrame(tabId, state); }- catch { return null; }const refuse = reason => ({success: false, dispatched: false, noDispatch: true, visionScreenChanged: true,error: `Not dispatched: ${reason} after your previous action in this turn. Look at the updated screen WebBrain shows next and choose the point again.`,});+ let frame;+ try { frame = await this._visionCaptureFrame(tabId, state); }+ catch (error) {+ if (isControlFlowError(error)) throw error;+ // The point was chosen before the earlier action; without a fresh+ // capture nothing shows it is still right.+ session.forceKeyframe = true;+ return refuse('WebBrain could not re-check the screen');+ }if (frame.probe.url !== session.keyframe.url) return refuse('the page navigated');if (frame.size.width !== session.keyframe.width || frame.size.height !== session.keyframe.height) return refuse('the viewport changed');const d = diffFrames(session.seenPixels, frame.raster.pixels, frame.size.width, frame.size.height);@@ -468,8 +597,28 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {return { resolved, session, state };};- /** Execute take_screenshot, zoom, scroll_at, hover_at, drag_at and wait. */+ /**+ * Execute take_screenshot, zoom, scroll_at, hover_at, drag_at and wait. A+ * failure becomes a tool result (no dispatch, or an unknown outcome once+ * input may have been sent) instead of ending the run.+ */proto._executeVisionTool = async function _executeVisionTool(tabId, name, args = {}, context = {}) {+ const dispatch = { started: false };+ try {+ return await this._executeVisionToolInner(tabId, name, args, context, dispatch);+ } catch (error) {+ if (isControlFlowError(error)) throw error;+ this._visionTraceNote(tabId, 'vision_tool_error', { tool: name, error: errorText(error), dispatched: dispatch.started });+ if (dispatch.started) {+ const session = this._visionSessions?.get(tabId);+ if (session) session.forceKeyframe = true;+ return { success: false, outcomeUnknown: true, error: `${name} failed while sending input (${errorText(error)}). Look at the next screen before repeating it.` };+ }+ return { success: false, dispatched: false, noDispatch: true, error: `${name} failed before any input was sent (${errorText(error)}).` };+ }+ };++ proto._executeVisionToolInner = async function _executeVisionToolInner(tabId, name, args, context, dispatch) {const vision = context?._vision || null;const state = this._visionRunState(tabId);if (!state.active) {@@ -493,6 +642,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {const probe = await this._visionPlatformProbe(tabId);const plan = planZoomImageSize({ regionWidth: region.cssRegion.width, regionHeight: region.cssRegion.height, devicePixelRatio: probe?.devicePixelRatio || 1, profile });let dataUrl = await this._visionPlatformCapture(tabId, { probe, cssRegion: region.cssRegion, outWidth: plan.width, outHeight: plan.height });+ if (!dataUrl) throw new Error('The browser returned no screenshot.');if (this.screenshotRedaction && typeof this._redactScreenshotDataUrl === 'function') {try {// A crop is a page-space box; the redactor maps element rects into it.@@ -532,9 +682,13 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {const probe = await this._visionPlatformProbe(tabId);point = { x: (probe?.cssWidth || 800) / 2, y: (probe?.cssHeight || 600) / 2 };}+ // A focused number field under the pointer takes the wheel as +/-.+ const refused = visionWheelRefusal(await this._visionDomProbe(tabId, { points: [point] }));+ if (refused) return { ...refused, at: point };const distance = steps * VISION_SCROLL_STEP_CSS_PX;const deltaX = direction === 'left' ? -distance : direction === 'right' ? distance : 0;const deltaY = direction === 'up' ? -distance : direction === 'down' ? distance : 0;+ dispatch.started = true;const result = await this._visionPlatformWheel(tabId, { x: point.x, y: point.y, deltaX, deltaY });return { success: result?.success !== false, dispatched: result?.dispatched !== false, direction, amount: steps, at: point, ...(result || {}) };}@@ -542,6 +696,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {const resolved = await this._visionResolveForTool(tabId, { x: args.x, y: args.y, image: args.image }, vision);if (resolved.error) return resolved.error;const point = { x: resolved.resolved.cssX, y: resolved.resolved.cssY };+ dispatch.started = true;const result = await this._visionPlatformHover(tabId, point);return { success: result?.success !== false, at: point, ...(result || {}) };}@@ -552,6 +707,13 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {if (!end.ok) return { success: false, dispatched: false, noDispatch: true, visionCoordinateError: end.code, error: end.error };const from = { x: start.resolved.cssX, y: start.resolved.cssY };const to = { x: end.cssX, y: end.cssY };+ // A press and release on the same control is a click that no click+ // guard has seen; a frame WebBrain cannot inspect could hide one.+ const longEnough = Math.hypot(to.x - from.x, to.y - from.y) >= VISION_MIN_DRAG_CSS_PX;+ const probe = longEnough ? await this._visionDomProbe(tabId, { points: [from, to], pair: true }) : null;+ const refused = visionDragRefusal(probe, from, to);+ if (refused) return { ...refused, from, to };+ dispatch.started = true;const result = await this._visionPlatformDrag(tabId, { from, to, steps: 12 });return { success: result?.success !== false, from, to, ...(result || {}) };}diff --git a/src/chrome/src/agent/vision-session.js b/src/chrome/src/agent/vision-session.jsindex f89164d89..e9c5c4c9b 100644--- a/src/chrome/src/agent/vision-session.js+++ b/src/chrome/src/agent/vision-session.js@@ -36,10 +36,15 @@ export function normalizeImageId(value) {}export class VisionSession {- constructor({ coordinates = 'pixel' } = {}) {+ /**+ * `startEpoch` / `startCounter` continue the numbering of an earlier+ * session of the same conversation (another model, a restarted worker), so+ * ids and epochs never repeat within one history.+ */+ constructor({ coordinates = 'pixel', startEpoch = 0, startCounter = 0 } = {}) {this.coordinates = VISION_COORDINATE_FORMATS.includes(coordinates) ? coordinates : 'pixel';- this.counter = 0;- this.epoch = 0;+ this.counter = Math.max(0, Math.floor(finite(startCounter) || 0));+ this.epoch = Math.max(0, Math.floor(finite(startEpoch) || 0));this.updateSeq = 0;this.keyframe = null;this.images = new Map();@@ -108,6 +113,22 @@ export class VisionSession {return this.keyframe;}+ /**+ * Forget the current screen (a new run starts, or the screen can no longer+ * be trusted): its images become "from an earlier screen" and coordinate+ * tools are refused until the next keyframe.+ */+ retireKeyframe() {+ for (const [id, image] of this.images) this.retired.set(id, { epoch: image.epoch, replacedBy: null });+ this.images = new Map();+ this.keyframe = null;+ this.seenPixels = null;+ this.changes = [];+ this.patchChainLength = 0;+ this.pendingZooms = [];+ this.forceKeyframe = true;+ }+/** Record patches (keyframe px regions) the model is about to receive. */registerPatches(regions = [], pixels = null) {if (!this.keyframe) throw new Error('A keyframe is required before patches.');@@ -178,8 +199,13 @@ export class VisionSession {}_imageFor(imageId) {- const id = normalizeImageId(imageId);- if (!id) return { image: this.keyframe, id: this.keyframe?.id || '', composite: true };+ const raw = imageId == null ? '' : String(imageId).trim();+ // No image named: the current screen. A value that is not an image id+ // ("zoom z2", "z2.png") is refused rather than read as the current screen,+ // which would map a zoom's point through the keyframe.+ if (!raw) return { image: this.keyframe, id: this.keyframe?.id || '', composite: true };+ const id = normalizeImageId(raw);+ if (!id) return { image: null, id: raw.slice(0, 40), retired: null };const image = this.images.get(id);if (image) return { image, id, composite: false };return { image: null, id, retired: this.retired.get(id) || null };@@ -199,7 +225,7 @@ export class VisionSession {if (lookup.retired) {return { ok: false, code: 'stale_image', error: `Image ${lookup.id} is from an earlier screen. The current full frame is ${this.keyframe.id}; read the point from it or from a later patch.` };}- return { ok: false, code: 'unknown_image', error: `Unknown image "${image}". Valid images now: ${[...this.images.keys()].join(', ')}.` };+ return { ok: false, code: 'unknown_image', error: `Unknown image "${lookup.id}". Name an image exactly as labeled (valid now: ${[...this.images.keys()].join(', ')}), or omit image for the current screen.` };}const target = lookup.image;const px = finite(x);@@ -261,9 +287,13 @@ export class VisionSession {const bottomRight = this.resolvePoint({ x: bx, y: by, image });if (!bottomRight.ok && bottomRight.code !== 'stale_area') return bottomRight;const k0 = topLeft.keyframePoint;+ // The rectangle's size in the target image's own pixels (normalized+ // coordinates are 0..1000 of that image).+ const wImage = this.coordinates === 'normalized' ? (w / NORMALIZED_COORDINATE_SCALE) * target.width : w;+ const hImage = this.coordinates === 'normalized' ? (h / NORMALIZED_COORDINATE_SCALE) * target.height : h;const k1 = bottomRight.ok ? bottomRight.keyframePoint : {- x: Math.min(this.keyframe.width, k0.x + w * target.scaleX),- y: Math.min(this.keyframe.height, k0.y + h * target.scaleY),+ x: Math.min(this.keyframe.width, k0.x + wImage * target.scaleX),+ y: Math.min(this.keyframe.height, k0.y + hImage * target.scaleY),};const kfRegion = {x: Math.max(0, Math.floor(k0.x)),diff --git a/src/chrome/src/agent/vision-tools.js b/src/chrome/src/agent/vision-tools.jsindex e09d02cb4..4fcf3f31f 100644--- a/src/chrome/src/agent/vision-tools.js+++ b/src/chrome/src/agent/vision-tools.js@@ -81,7 +81,7 @@ export function visionToolDefinitions({ coordinates = 'pixel' } = {}) {type: 'function',function: {name: 'click_at',- description: `Click at a point with a real mouse event. x/y are ${words.unit}. WebBrain converts the point to the page (retina displays and browser zoom included); never rescale it yourself. Aim at the visual centre of the target. clicks:2 double-clicks, clicks:3 selects a line or paragraph, button:"right" opens a context menu.`,+ description: `Click at a point with a real mouse event. x/y are ${words.unit}. WebBrain converts the point to the page (retina displays and browser zoom included); never rescale it yourself. Aim at the visual centre of the target. clicks:2 double-clicks and clicks:3 selects a line or paragraph, for selecting text only: repeated clicks on a button, link or other control are refused. button:"right" opens a context menu.`,parameters: {type: 'object',properties: {@@ -127,7 +127,7 @@ export function visionToolDefinitions({ coordinates = 'pixel' } = {}) {type: 'function',function: {name: 'drag_at',- description: `Press the left mouse button at the start point, move to the end point and release (sliders, reordering, selecting text, drawing). Coordinates are ${words.unit}; both points come from the same image.`,+ description: `Press the left mouse button at the start point, move to the end point and release (sliders, reordering, selecting text, drawing). Coordinates are ${words.unit}; both points come from the same image. Not a way to click: a drag of a few pixels, or one that presses and releases on the same button or link, is refused; use click_at.`,parameters: {type: 'object',properties: {@@ -238,15 +238,20 @@ export function translateVisionClick(args = {}, session) {};}const { button, clicks } = normalizeClickArgs(args);+ // The coordinate click path reads x,y both within 0..1 as a normalized+ // fraction; the top-left CSS pixel is nudged off that range.+ const corner = point.cssX <= 1 && point.cssY <= 1;+ const cssX = corner ? 1.5 : point.cssX;+ const cssY = corner ? 1.5 : point.cssY;return {ok: true,name: 'click',- args: { x: point.cssX, y: point.cssY, coordinate_space: 'css' },+ args: { x: cssX, y: cssY, coordinate_space: 'css' },vision: {tool: 'click_at',button,clickCount: clicks,- cssPoint: { x: point.cssX, y: point.cssY },+ cssPoint: { x: cssX, y: cssY },keyframePoint: point.keyframePoint,imageId: point.imageId,keyframeId: point.keyframeId,diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 4a10341a6..8dfad3c5c 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -21,6 +21,7 @@ import { mastodonPublicationIntentMessages, normalizeMastodonPublicationIntent,import { AGENT_TOOLS, AGENT_TOOL_NAMES, RESERVED_AGENT_TOOL_NAMES, RETIRED_AGENT_TOOL_NAMES, getToolsForMode, SYSTEM_PROMPT_BATCHING, SYSTEM_PROMPT_ASK, SYSTEM_PROMPT_ACT, SYSTEM_PROMPT_ACT_COMPACT, SYSTEM_PROMPT_ACT_MID, SYSTEM_PROMPT_LEAN_NOTE, SYSTEM_PROMPT_GENERATIVE_MEDIA, SYSTEM_PROMPT_DEV_APPENDIX, getVisionActPrompt } from './tools.js';import { installVisionRuntime } from './vision-runtime.js';import { VISION_NATIVE_TOOLS } from './vision-tools.js';+import { visionDomProbeExpression } from './vision-dom-probe.js';import { normalizeVisionModeSetting } from './vision-mode.js';import { rejectedCompletionRecovery } from './completion-recovery.js';import { FailureRecoveryRegistry, modelFailureRecovery } from './failure-recovery.js';@@ -14331,6 +14332,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (tc._vision?.error) {const recovery = await recordPreparationFailure(toolIndex, tc._vision.tool || fnName, this._parseToolCallArgs(tc).args || {}, tc._vision.error);if (recovery) return recovery;+ // The rest of the batch was planned on that point: type_text would+ // type into whatever has focus. The model decides again.+ if (interruptFailedBrowserAction(toolIndex, tc._vision.tool || fnName)) { navNotices.length = 0; break; }continue;}// Where this call's time goes before dispatch (_predispatchTimer), from@@ -17025,29 +17029,54 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d/*** captureTab renders at `scale` image pixels per CSS pixel (not times the- * DPR), so the plan's scale is passed directly. A region is a page-space- * rect (captureTab's rect is relative to the document).+ * DPR), so the plan's scale is passed directly. The rect is always given+ * (page-relative, like captureTab's rect): without one Firefox captures+ * the whole content area, classic scrollbars included, while the probe's+ * visual viewport excludes them, and points would drift by the scrollbar.*/async _visionPlatformCapture(tabId, { probe, cssRegion = null, outWidth, outHeight }) {const region = cssRegion || { x: 0, y: 0, width: probe.cssWidth, height: probe.cssHeight };const scale = Math.max(0.05, Math.min(4, Number(outWidth) / Math.max(1, region.width)));- const options = { format: 'jpeg', quality: 90, scale };- if (cssRegion) {- options.rect = {+ const options = {+ format: 'jpeg',+ quality: 90,+ scale,+ rect: {x: (Number(probe?.scrollX) || 0) + region.x,y: (Number(probe?.scrollY) || 0) + region.y,width: Math.max(1, region.width),height: Math.max(1, region.height),- };- }+ },+ };return this._withIndicatorsHidden(tabId, () => browser.tabs.captureTab(tabId, options));}+ /**+ * What is under CSS points (vision-dom-probe.js), evaluated in the+ * content-script world, whose DOM methods page scripts cannot redefine.+ */+ async _visionPlatformDomProbe(tabId, request) {+ const results = await browser.tabs.executeScript(tabId, { code: visionDomProbeExpression(request), frameId: 0 });+ return Array.isArray(results) ? results[0] || null : null;+ }+async _visionContentInput(tabId, params, kind = 'click') {await beforePageAgentDispatch(browser, tabId, {kind, frameId: 0, navigationCandidate: kind === 'click' && params.type === 'click', fenceOnly: true,});- const response = await browser.tabs.sendMessage(tabId, { target: 'content', action: 'vision_input', params }, { frameId: 0 });+ let response;+ try {+ response = await browser.tabs.sendMessage(tabId, { target: 'content', action: 'vision_input', params }, { frameId: 0 });+ } catch (error) {+ const message = String(error?.message || error).slice(0, 160);+ // No content script answered (still loading, restricted page): nothing+ // was sent. Any other failure may come after the input went out (the+ // click navigated away), so its outcome is unknown.+ if (/receiving end does not exist|could not establish connection/i.test(message)) {+ return { success: false, dispatched: false, noDispatch: true, error: `The page could not receive the input yet (${message}). Take a screenshot and try again.` };+ }+ return { success: false, outcomeUnknown: true, error: `The page stopped answering during the input (${message}). Look at the next screen before repeating it.` };+ }return response || { success: false, dispatched: false, noDispatch: true, error: 'The page did not answer the vision input.' };}@@ -17068,28 +17097,17 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d* point. Batch-level gates and guards already ran on this `click`; a* native <select> is focused for type_text instead of opening its popup.*/- async _visionDispatchClick(tabId, args, { vision = null, abortSignal = null, dispatchState = { started: false } } = {}) {+ async _visionDispatchClick(tabId, args, { vision = null, abortSignal = null, dispatchState = { started: false }, detectedSubmitAction = null } = {}) {const x = Number(args.x);const y = Number(args.y);this._throwIfAborted(abortSignal);- let select = null;- try {- const results = await browser.tabs.executeScript(tabId, { code: `(() => {- const el = document.elementFromPoint(${x}, ${y});- const select = el && el.closest ? el.closest('select') : null;- if (!select) return null;- select.focus();- return { current: select.options[select.selectedIndex] ? select.options[select.selectedIndex].text.trim() : '',- options: Array.from(select.options).slice(0, 60).map(option => option.text.trim()) };- })()` });- select = Array.isArray(results) ? results[0] : null;- } catch { select = null; }- if (select && (vision?.button || 'left') === 'left') {- return {- success: false, dispatched: false, noDispatch: true, nativeSelect: true, tag: 'SELECT', text: select.current, options: select.options,- error: `That point is a native dropdown (current: "${select.current}"). It is focused now: choose an option with type_text({text: "<option label>"}). Options: ${select.options.join(', ')}`,- };- }+ // A native <select> is focused for type_text instead of opening a popup+ // no screenshot shows; a repeated click on a control is refused.+ const preflight = await this._visionClickPreflight(tabId, {+ x, y, button: vision?.button || 'left', clickCount: vision?.clickCount || 1,+ submitDetected: detectedSubmitAction?.isSubmit === true,+ });+ if (preflight.refusal) return preflight.refusal;this._throwIfAborted(abortSignal);dispatchState.started = true;const response = await this._visionContentInput(tabId, {@@ -29102,7 +29120,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dasync _messageRecipientGuardBlock(tabId, toolName, args = {}, pageUrl = '', executionContext = null) {const name = String(toolName || '');const ordinaryGuardedTools = new Set(['click', 'click_ax', 'set_field', 'press_keys', 'upload_file']);- const unbindableDispatchTools = new Set(['iframe_click', 'execute_js', 'execute_webmcp_tool']);+ // drag_at (vision mode) presses and releases where the model points; no+ // recipient binding covers that, and a short drag is a click.+ const unbindableDispatchTools = new Set(['iframe_click', 'execute_js', 'execute_webmcp_tool', 'drag_at']);if (!ordinaryGuardedTools.has(name) && !unbindableDispatchTools.has(name)) return null;if (name === 'press_keys' && String(args?.key || '') !== 'Enter') return null;if (name === 'set_field' && args?.submit !== true) return null;@@ -29152,7 +29172,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d? (!uploadSourceKey? 'Attachment blocked: use exactly one attachmentId, downloadId, or absolute filePath. Remove fallback sources and retry with the same authorized recipient.': 'Attachment blocked: choose an exact, unique file-input selector in the active authorized conversation. Re-read the file-input inventory before retrying.')- : `Message action blocked: ${name} cannot bind its effects to the verified active recipient. Use click, click_ax, set_field({submit:true}), or a single Enter press on the visible composer instead.`,+ : name === 'drag_at'+ ? 'Message action blocked: drag_at cannot bind its effects to the verified active recipient. Use click_at on the control, or a single Enter press in the visible composer, instead.'+ : `Message action blocked: ${name} cannot bind its effects to the verified active recipient. Use click, click_ax, set_field({submit:true}), or a single Enter press on the visible composer instead.`,};}@@ -35281,6 +35303,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dthis._cancelPendingPlans(tabId, 'tab closed');this._clarificationAuthorizationGuards.delete(tabId);this._isPdfTabCache.delete(tabId);+ this._visionForgetTab?.(tabId);this._lastTypeFieldIdent?.delete(tabId);this._lastTypeFieldEpoch?.delete(tabId);// A cleared conversation or closed tab takes its filled credentials along.@@ -46281,6 +46304,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dvision: dispatchContext._vision,abortSignal: contentPipelineAbortSignal,dispatchState: contentPipelineDispatchState,+ detectedSubmitAction: dispatchContext.detectedSubmitAction || null,});}if (coordinatePoint && !dispatchBinding?.token) {@@ -47088,6 +47112,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// Lean mode is resolved once, now that the run's mode and provider are// known, and holds for every tool list, prompt and guard of this run.const previousLeanRun = this._leanRuns?.get(tabId);+ const previousVisionRun = this._visionRunSnapshots?.get(tabId);this._beginLeanRun(tabId, mode);const previousBotMemory = this._botMemorySessions.get(tabId);let botMemorySession = null;@@ -47123,7 +47148,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}this._planExecutionGuards.delete(tabId);this._runModeOverrides.delete(tabId);- this._endVisionRun(tabId);+ this._endVisionRun(tabId, previousVisionRun);if (previousLeanRun) this._leanRuns.set(tabId, previousLeanRun);else this._leanRuns?.delete(tabId);if (previousStandaloneChatRun) this._standaloneChatRunTabs.add(tabId);@@ -48930,6 +48955,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// Lean mode is resolved once, now that the run's mode and provider are// known, and holds for every tool list, prompt and guard of this run.const previousLeanRun = this._leanRuns?.get(tabId);+ const previousVisionRun = this._visionRunSnapshots?.get(tabId);this._beginLeanRun(tabId, mode);const previousBotMemory = this._botMemorySessions.get(tabId);let botMemorySession = null;@@ -48964,7 +48990,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}this._planExecutionGuards.delete(tabId);this._runModeOverrides.delete(tabId);- this._endVisionRun(tabId);+ this._endVisionRun(tabId, previousVisionRun);if (previousLeanRun) this._leanRuns.set(tabId, previousLeanRun);else this._leanRuns?.delete(tabId);if (previousStandaloneChatRun) this._standaloneChatRunTabs.add(tabId);diff --git a/src/firefox/src/agent/vision-budget.js b/src/firefox/src/agent/vision-budget.jsindex 3f8c8807d..277a9ac29 100644--- a/src/firefox/src/agent/vision-budget.js+++ b/src/firefox/src/agent/vision-budget.js@@ -52,6 +52,13 @@ export const VISION_PROFILES = Object.freeze({maxLongEdge: Infinity, maxPixels: 1_048_576, maxShortEdge: Infinity, align: 32,format: 'jpeg', quality: 0.8,}),+ relative1000: Object.freeze({+ id: 'relative1000', tokenModel: 'patch28', coordinates: 'normalized',+ // GLM-4.xV and InternVL are trained on boxes normalized to 0..1000.+ // JPEG because they are often served by local runtimes too.+ maxLongEdge: 1280, maxPixels: 1_003_520, maxShortEdge: Infinity, align: 28,+ format: 'jpeg', quality: 0.8,+ }),local: Object.freeze({id: 'local', tokenModel: 'patch28', coordinates: 'pixel',// llama.cpp's image loader (stb_image) cannot decode WebP; JPEG is the@@ -85,6 +92,7 @@ export function detectVisionProfile(provider = {}) {const category = lower(config.category || provider?.category);// Qwen3-VL and the natively multimodal Qwen3.5+ point in relative 0..1000.if (/qwen3(\.\d+)?-(vl|omni)|qwen3\.[5-9]/.test(model)) return VISION_PROFILES.qwen3;+ if (/glm-4(\.\d+)?v|internvl/.test(model)) return VISION_PROFILES.relative1000;if (/gemini/.test(model)) return VISION_PROFILES.gemini;if (/claude|anthropic/.test(model) || ['anthropic', 'vertex-anthropic', 'oauth-claude', 'claude'].includes(name)) {return VISION_PROFILES.anthropic;diff --git a/src/firefox/src/agent/vision-dom-probe.js b/src/firefox/src/agent/vision-dom-probe.jsnew file mode 100644index 000000000..7e46c14c6--- /dev/null+++ b/src/firefox/src/agent/vision-dom-probe.js@@ -0,0 +1,200 @@+// What is under a viewport point, for vision-mode dispatch checks: a double+// click that would repeat a button's action, a drag that is really a click,+// a wheel over a focused number field, a native <select>, an embedded frame+// whose content cannot be inspected. Shared verbatim by both builds.+//+// visionDomProbe runs inside the page (Chrome: an isolated world through CDP;+// Firefox: tabs.executeScript in the content-script world, so page scripts+// cannot redefine the DOM methods it calls). It is serialized with+// toString(), so it must stay self-contained: no imports, no outer names.++/**+ * @param {{points: {x:number,y:number}[], pair?: boolean, focusSelect?: boolean}} request+ * points are CSS pixels of the top-level viewport. `pair` asks what a+ * press at points[0] and a release at points[1] would click. `focusSelect`+ * focuses a native <select> under points[0] (a click on it is answered+ * with type_text instead of an OS popup no screenshot can show).+ */+export function visionDomProbe(request) {+ const CLICKABLE = [+ 'a[href]', 'area[href]', 'button', 'summary', 'label', 'select', 'option', '[onclick]',+ '[role="button"]', '[role="link"]', '[role="menuitem"]', '[role="menuitemcheckbox"]',+ '[role="menuitemradio"]', '[role="tab"]', '[role="checkbox"]', '[role="radio"]',+ '[role="switch"]', '[role="option"]',+ 'input[type="submit"]', 'input[type="button"]', 'input[type="reset"]', 'input[type="image"]',+ 'input[type="checkbox"]', 'input[type="radio"]', 'input[type="file"]', 'input[type="color"]',+ ].join(',');+ // A click inside a text field only places the caret, even when a <label>+ // or link wraps the field, so the walk up stops there.+ const TEXT_ENTRY = [+ 'textarea', '[contenteditable=""]', '[contenteditable="true"]', '[contenteditable="plaintext-only"]',+ 'input:not([type])', 'input[type="text"]', 'input[type="search"]', 'input[type="email"]',+ 'input[type="url"]', 'input[type="tel"]', 'input[type="password"]', 'input[type="number"]',+ 'input[type="date"]', 'input[type="datetime-local"]', 'input[type="time"]', 'input[type="month"]',+ 'input[type="week"]', 'input[type="range"]',+ ].join(',');+ const req = request && typeof request === 'object' ? request : {};+ // Composed parent: through shadow roots to their host and out of+ // same-origin frames to the frame element.+ const up = node => {+ if (!node) return null;+ if (node.nodeType === 9) {+ try { return (node.defaultView && node.defaultView.frameElement) || null; } catch (e) { return null; }+ }+ const parent = node.parentNode;+ if (parent && parent.nodeType === 11) return parent.host || null;+ return parent || null;+ };+ const hit = (x, y) => {+ let doc = document;+ let cx = x;+ let cy = y;+ for (let depth = 0; depth < 8; depth++) {+ let el = doc.elementFromPoint(cx, cy);+ for (let i = 0; el && el.shadowRoot && i < 8; i++) {+ const inner = el.shadowRoot.elementFromPoint(cx, cy);+ if (!inner || inner === el) break;+ el = inner;+ }+ if (!el) return { el: null, opaque: false };+ const tag = String(el.tagName || '').toUpperCase();+ if (tag === 'IFRAME' || tag === 'FRAME' || tag === 'OBJECT' || tag === 'EMBED') {+ let inner = null;+ try { inner = el.contentDocument; } catch (e) { inner = null; }+ if (inner && inner.documentElement && (tag === 'IFRAME' || tag === 'FRAME')) {+ const rect = el.getBoundingClientRect();+ cx -= rect.left + (el.clientLeft || 0);+ cy -= rect.top + (el.clientTop || 0);+ doc = inner;+ continue;+ }+ // A cross-origin frame (or plugin): its content is not inspectable.+ return { el, opaque: true };+ }+ return { el, opaque: false };+ }+ return { el: null, opaque: false };+ };+ const clickableOf = node => {+ for (let n = node, i = 0; n && i < 256; n = up(n), i++) {+ if (n.nodeType === 1 && typeof n.matches === 'function') {+ try {+ if (n.matches(CLICKABLE)) return n;+ if (n.matches(TEXT_ENTRY)) return null;+ } catch (e) { /* invalid in this document */ }+ }+ }+ return null;+ };+ const points = Array.isArray(req.points) ? req.points.slice(0, 2) : [];+ const found = points.map(point => hit(Number(point && point.x), Number(point && point.y)));+ const infos = found.map((item, index) => {+ const el = item.el;+ if (!el) return { found: false };+ const control = clickableOf(el);+ const info = {+ found: true,+ tag: String(el.tagName || '').toLowerCase(),+ clickable: !!control,+ opaqueFrame: item.opaque === true,+ };+ const role = el.getAttribute ? el.getAttribute('role') : '';+ if (role) info.role = String(role).slice(0, 32);+ if (control) info.control = String(control.tagName || '').toLowerCase();+ if (typeof el.matches === 'function' && el.matches('input[type="number"]')) {+ const root = el.getRootNode ? el.getRootNode() : el.ownerDocument;+ info.focusedNumberInput = !!root && root.activeElement === el;+ }+ const select = el.closest ? el.closest('select') : null;+ if (select && !select.multiple && !(select.size > 1)) {+ const options = Array.from(select.options || []).slice(0, 60).map(option => String(option.text || '').trim());+ const current = select.options && select.options[select.selectedIndex]+ ? String(select.options[select.selectedIndex].text || '').trim() : '';+ info.select = { current, options };+ if (index === 0 && req.focusSelect === true) {+ try { select.focus(); } catch (e) { /* focus is best effort */ }+ }+ }+ return info;+ });+ const result = { points: infos };+ if (req.pair === true && found.length === 2) {+ if (!found[0].el || !found[1].el) {+ result.pair = { known: false };+ } else {+ // The click a press and release produce goes to their nearest common+ // ancestor; it activates a control when that ancestor is (inside) one.+ const ancestors = new Set();+ for (let n = found[1].el, i = 0; n && i < 256; n = up(n), i++) ancestors.add(n);+ let common = null;+ for (let n = found[0].el, i = 0; n && i < 256; n = up(n), i++) {+ if (ancestors.has(n)) { common = n; break; }+ }+ const control = common ? clickableOf(common) : null;+ result.pair = { known: true, clickEquivalent: !!control };+ if (control) result.pair.control = String(control.tagName || '').toLowerCase();+ }+ }+ return result;+}++/** An expression that evaluates visionDomProbe(request) in a page. */+export function visionDomProbeExpression(request) {+ return `(${visionDomProbe.toString()})(${JSON.stringify(request || {})})`;+}++/** Minimum press-to-release distance (CSS px) for drag_at. */+export const VISION_MIN_DRAG_CSS_PX = 6;++/**+ * Dispatch checks shared by both builds. Each returns a no-dispatch result+ * to send back to the model, or null when the action may go ahead.+ */+export function visionMultiClickRefusal(probe, { clickCount = 1, recipientGuarded = false, submitDetected = false } = {}) {+ if (!(clickCount > 1)) return null;+ const point = probe?.points?.[0];+ const reason = recipientGuarded ? 'this is a protected messaging page'+ : submitDetected ? 'the point is a submit control'+ : !point || !point.found ? 'WebBrain could not check what is under the pointer'+ : point.opaqueFrame ? 'the point is inside an embedded frame WebBrain cannot inspect'+ : point.clickable ? `the point is on a ${point.control || 'control'}`+ : '';+ if (!reason) return null;+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'repeated_click',+ error: `Not dispatched: a ${clickCount === 2 ? 'double' : 'triple'} click was refused because ${reason}, where every extra click would repeat the control's action (for example submit twice). Use click_at with clicks: 1. Double and triple clicks are for selecting text.`,+ };+}++export function visionDragRefusal(probe, from, to) {+ const distance = Math.hypot((Number(to?.x) || 0) - (Number(from?.x) || 0), (Number(to?.y) || 0) - (Number(from?.y) || 0));+ const refuse = (code, reason) => ({+ success: false, dispatched: false, noDispatch: true, visionInputRefused: code,+ error: `Not dispatched: ${reason}`,+ });+ if (distance < VISION_MIN_DRAG_CSS_PX) {+ return refuse('drag_too_short', 'a drag this short is a click. Use click_at to click.');+ }+ if (!probe?.points || probe.points.length < 2 || !probe.pair) {+ return refuse('drag_unchecked', 'WebBrain could not check what is under the pointer. Take a screenshot and try again, or use click_at / scroll_at.');+ }+ if (probe.points.some(point => point?.opaqueFrame)) {+ return refuse('drag_opaque_frame', 'the drag starts or ends inside an embedded frame whose content WebBrain cannot inspect, so it could act as a click there. Use click_at or scroll_at instead.');+ }+ if (!probe.pair.known) {+ return refuse('drag_unchecked', 'there is nothing under one end of the drag. Choose points on the page.');+ }+ if (probe.pair.clickEquivalent) {+ return refuse('drag_is_click', `pressing and releasing there would click the ${probe.pair.control || 'control'} under both ends. Use click_at to click it; drag_at is for sliders, reordering and selecting.`);+ }+ return null;+}++export function visionWheelRefusal(probe) {+ const point = probe?.points?.[0];+ if (!point?.focusedNumberInput) return null;+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'wheel_number_input',+ error: 'Not dispatched: the point is over a focused number field, and the mouse wheel would change its value instead of scrolling. Scroll at another point, or press Escape / click elsewhere first.',+ };+}diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.jsindex 962668c24..b6ecc119a 100644--- a/src/firefox/src/agent/vision-runtime.js+++ b/src/firefox/src/agent/vision-runtime.js@@ -27,8 +27,24 @@ import {VISION_SCROLL_STEP_CSS_PX, VISION_DEFAULT_SCROLL_STEPS, VISION_MAX_WAIT_SECONDS,translateVisionClick, visionKeyframeNote, visionPatchNote, visionNoChangeNote, visionZoomNote,} from './vision-tools.js';+import {+ VISION_MIN_DRAG_CSS_PX, visionMultiClickRefusal, visionDragRefusal, visionWheelRefusal,+} from './vision-dom-probe.js';const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));++// Errors that steer the run (stop, deadlines, page feedback) pass through;+// anything else in a vision tool or screen update becomes a result or a note+// instead of ending the run.+function isControlFlowError(error) {+ return error?.name === 'AbortError'+ || error?.code === 'content_action_timeout'+ || error?.code === 'page_feedback_pending';+}++function errorText(error) {+ return String(error?.message || error || 'unknown error').slice(0, 200);+}// A point within this many keyframe px of a change made earlier in the same// batch is refused: the model chose it on the screen before that change.const STABILITY_MARGIN_PX = 12;@@ -55,32 +71,53 @@ function regionLabel(region) {return { x: Math.round(region.x), y: Math.round(region.y), width: Math.round(region.width), height: Math.round(region.height) };}+/** Highest epoch and image number a history already used (session seeds). */+export function visionHistoryCounters(messages) {+ let epoch = 0;+ let counter = 0;+ if (Array.isArray(messages)) {+ for (const message of messages) {+ const tag = visionFrameTag(message);+ if (!tag) continue;+ epoch = Math.max(epoch, Number(tag.epoch) || 0);+ for (const id of Array.isArray(tag.ids) ? tag.ids : []) {+ const number = Number(String(id).slice(1));+ if (Number.isFinite(number)) counter = Math.max(counter, number);+ }+ }+ }+ return { epoch, counter };+}+/**- * Keep only the current vision epoch's images (latest frame, its patches and- * zooms). Exported for the unit suite; agents call it via _pruneOldImages.+ * Keep only the current screen's images: the LAST full frame in the history+ * and the patches and zooms after it. Position decides, not the epoch+ * number, which starts over in a new session (another model, a restarted+ * worker). Exported for the unit suite; agents call it via _pruneOldImages.* Returns { messages, latestEpoch } where `messages` is a shallow copy.*/export function pruneVisionImages(messages) {if (!Array.isArray(messages)) return { messages, latestEpoch: 0 };- let latestEpoch = 0;let latestKeyIndex = -1;- messages.forEach((message, index) => {- const tag = visionFrameTag(message);- if (!tag) return;- const epoch = Number(tag.epoch) || 0;- if (epoch >= latestEpoch) latestEpoch = epoch;- if (tag.kind === 'key' && epoch >= latestEpoch) latestKeyIndex = index;- });- if (!latestEpoch) return { messages, latestEpoch: 0 };+ let retireIndex = -1;+ for (let index = messages.length - 1; index >= 0 && latestKeyIndex < 0; index--) {+ const tag = visionFrameTag(messages[index]);+ if (tag?.kind === 'key') latestKeyIndex = index;+ else if (tag?.kind === 'retire' && retireIndex < 0) retireIndex = index;+ }+ if (latestKeyIndex < 0) return { messages, latestEpoch: 0 };+ const latestEpoch = Math.max(1, Number(visionFrameTag(messages[latestKeyIndex]).epoch) || 1);+ // A run whose first capture failed retires the last frame: it showed an+ // earlier run's screen, and no point may be read from it any more.+ const boundary = retireIndex > latestKeyIndex ? retireIndex : latestKeyIndex;const out = messages.map((message, index) => {const tag = visionFrameTag(message);if (!Array.isArray(message?.content)) return message;- const keep = tag ? (Number(tag.epoch) || 0) === latestEpoch : false;- if (tag && keep) return message;+ if (tag && index >= boundary) return message;// Untagged screenshots older than the current frame (an Ask-mode capture// before an automatic switch to Act) are superseded by it too. User// attachments are never touched.- if (!tag && (index > latestKeyIndex || message.role !== 'user')) return message;+ if (!tag && (index > boundary || message.role !== 'user')) return message;let inAttachments = false;let changed = false;const content = message.content.map(block => {@@ -121,31 +158,90 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {return state.active ? { coordinates: state.coordinates } : null;};- /** Freeze the setting for the length of one run (see vision-mode.js). */+ /**+ * Freeze the setting for the length of one run (see vision-mode.js). The+ * previous run's screen is retired: a run acts only on frames it captured+ * itself, so a failed first capture can never leave the model aiming at+ * the last run's page.+ */proto._beginVisionRun = function _beginVisionRun(tabId) {if (!this._visionRunSnapshots) this._visionRunSnapshots = new Map();this._visionRunSnapshots.set(tabId, {setting: normalizeVisionModeSetting(this.visionMode, 'off'),quality: normalizeVisionQuality(this.visionQuality),});+ this._visionSessions?.get(tabId)?.retireKeyframe?.();};- proto._endVisionRun = function _endVisionRun(tabId) {- this._visionRunSnapshots?.delete(tabId);- this._visionBatches?.delete(tabId);+ /** `previous` is the snapshot of an enclosing run on the same tab, if any. */+ proto._endVisionRun = function _endVisionRun(tabId, previous = null) {+ if (previous) this._visionRunSnapshots?.set(tabId, previous);+ else this._visionRunSnapshots?.delete(tabId);+ };++ /** A closed tab or cleared conversation drops its screen state. */+ proto._visionForgetTab = function _visionForgetTab(tabId) {+ this._visionSessions?.delete(tabId);+ this._visionProbeContexts?.delete(tabId);};proto._visionSessionFor = function _visionSessionFor(tabId, state) {if (!this._visionSessions) this._visionSessions = new Map();- let session = this._visionSessions.get(tabId);- if (!session || session.coordinates !== state.coordinates || session.profileId !== state.profileId) {- session = new VisionSession({ coordinates: state.coordinates });- session.profileId = state.profileId;- this._visionSessions.set(tabId, session);+ const previous = this._visionSessions.get(tabId);+ if (previous && previous.coordinates === state.coordinates && previous.profileId === state.profileId) return previous;+ // A new session (first use, another model, a restarted worker) continues+ // the history's numbering so its ids and epochs never repeat.+ const history = visionHistoryCounters(this.conversations?.get?.(tabId));+ const session = new VisionSession({+ coordinates: state.coordinates,+ startEpoch: Math.max(history.epoch, previous?.epoch || 0),+ startCounter: Math.max(history.counter, previous?.counter || 0),+ });+ session.profileId = state.profileId;+ if (previous) {+ for (const [id, retired] of previous.retired) session.retired.set(id, retired);+ for (const [id, image] of previous.images) session.retired.set(id, { epoch: image.epoch, replacedBy: null });}+ this._visionSessions.set(tabId, session);return session;};+ /** What is under CSS points (vision-dom-probe.js); null when it cannot be read. */+ proto._visionDomProbe = async function _visionDomProbe(tabId, request) {+ if (typeof this._visionPlatformDomProbe !== 'function') return null;+ try {+ return await this._visionPlatformDomProbe(tabId, request);+ } catch (error) {+ if (isControlFlowError(error)) throw error;+ return null;+ }+ };++ /**+ * Checks right before a vision click is dispatched, in both builds: a+ * native <select> is focused for type_text (its popup never shows in a+ * screenshot), and a double/triple click that would repeat a control's+ * action is refused. Returns { refusal } or { target }.+ */+ proto._visionClickPreflight = async function _visionClickPreflight(tabId, {+ x, y, button = 'left', clickCount = 1, recipientGuarded = false, submitDetected = false,+ } = {}) {+ const probe = await this._visionDomProbe(tabId, { points: [{ x, y }], focusSelect: button === 'left' });+ const point = probe?.points?.[0] || null;+ if (point?.select && button === 'left') {+ return {+ refusal: {+ success: false, dispatched: false, noDispatch: true, nativeSelect: true, tag: 'SELECT',+ text: point.select.current, options: point.select.options,+ error: `That point is a native dropdown (current: "${point.select.current}"). It is focused now: choose an option with type_text({text: "<option label>"}). Options: ${point.select.options.join(', ')}`,+ },+ };+ }+ const refusal = visionMultiClickRefusal(probe, { clickCount, recipientGuarded, submitDetected });+ if (refusal) return { refusal };+ return { target: point?.found ? { tag: point.tag, ...(point.role ? { role: point.role } : {}) } : null };+ };+proto._visionTraceNote = function _visionTraceNote(tabId, note, data) {const runId = this.currentRunId?.get?.(tabId);if (!runId || !trace?.recordNote) return;@@ -171,7 +267,15 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {let dataUrl = await this._visionPlatformCapture(tabId, { probe, outWidth: size.width, outHeight: size.height });if (!dataUrl) throw new Error('The browser returned no screenshot.');if (this.screenshotRedaction && typeof this._redactScreenshotDataUrl === 'function') {- try { dataUrl = await this._redactScreenshotDataUrl(tabId, dataUrl, { coordinateSpace: 'viewport' }); } catch { /* keep the capture */ }+ try {+ // The capture spans the visual viewport (no scrollbars), so the+ // redactor gets that box in page space rather than its own+ // innerWidth-based viewport.+ dataUrl = await this._redactScreenshotDataUrl(tabId, dataUrl, {+ coordinateSpace: 'page',+ capturedCssBounds: { x: probe.scrollX || 0, y: probe.scrollY || 0, width: probe.cssWidth, height: probe.cssHeight },+ });+ } catch { /* keep the capture */ }}const raster = await this._visionRasterize(dataUrl, size.width, size.height);return { raster, probe, size, profile };@@ -288,8 +392,11 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {try {return await this._visionPushKeyframe(tabId, messages, onUpdate, { state, reason });} catch (error) {- const message = `[SCREEN: the first screenshot could not be captured (${String(error?.message || error).slice(0, 160)}). Call take_screenshot to try again.]`;- messages.push(visionUserMessage(message, { epoch: 0, kind: 'error' }));+ if (isControlFlowError(error)) throw error;+ // The run starts with no screen (_beginVisionRun retired the last one);+ // the tag retires the last frame in the history for the model too.+ const message = `[SCREEN: the first screenshot could not be captured (${errorText(error).slice(0, 160)}). Earlier screenshots show an older screen and cannot be used for coordinates. Call take_screenshot to try again.]`;+ messages.push(visionUserMessage(message, { epoch: 0, kind: 'retire' }));this._visionTraceNote(tabId, 'vision_frame_error', { stage: 'start', error: String(error?.message || error).slice(0, 200) });return null;}@@ -323,16 +430,32 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {return { kind: 'zoom' };}if (!observe) return null;+ const failed = (stage, error) => {+ // The next update is a full frame; until then the model knows the+ // screen it last saw may be out of date.+ session.forceKeyframe = true;+ messages.push(visionUserMessage(`[SCREEN: the screen could not be ${stage === 'observe' ? 'captured' : 'prepared'} after the action (${errorText(error).slice(0, 160)}). Call take_screenshot to look again.]`,+ { epoch: session.epoch, kind: 'error' }));+ this._visionTraceNote(tabId, 'vision_frame_error', { stage, error: errorText(error) });+ return { kind: 'error' };+ };let frame;try {frame = await this._visionCaptureSettled(tabId, state);} catch (error) {- session.forceKeyframe = true;- messages.push(visionUserMessage(`[SCREEN: the screen could not be captured after the action (${String(error?.message || error).slice(0, 160)}). Call take_screenshot to look again.]`,- { epoch: session.epoch, kind: 'error' }));- this._visionTraceNote(tabId, 'vision_frame_error', { stage: 'observe', error: String(error?.message || error).slice(0, 200) });- return { kind: 'error' };+ if (isControlFlowError(error)) throw error;+ return failed('observe', error);}+ try {+ return await this._visionObserveFrame(tabId, messages, onUpdate, { state, session, frame });+ } catch (error) {+ if (isControlFlowError(error)) throw error;+ return failed('encode', error);+ }+ };++ /** The part of _visionObserveAfterBatch after a settled capture. */+ proto._visionObserveFrame = async function _visionObserveFrame(tabId, messages, onUpdate, { state, session, frame }) {const meta = {cssWidth: frame.probe.cssWidth, cssHeight: frame.probe.cssHeight,devicePixelRatio: frame.probe.devicePixelRatio, url: frame.probe.url,@@ -437,13 +560,19 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {const state = this._visionRunState(tabId);const session = this._visionSessions?.get(tabId);if (!state.active || !session?.keyframe || !session.seenPixels) return null;- let frame;- try { frame = await this._visionCaptureFrame(tabId, state); }- catch { return null; }const refuse = reason => ({success: false, dispatched: false, noDispatch: true, visionScreenChanged: true,error: `Not dispatched: ${reason} after your previous action in this turn. Look at the updated screen WebBrain shows next and choose the point again.`,});+ let frame;+ try { frame = await this._visionCaptureFrame(tabId, state); }+ catch (error) {+ if (isControlFlowError(error)) throw error;+ // The point was chosen before the earlier action; without a fresh+ // capture nothing shows it is still right.+ session.forceKeyframe = true;+ return refuse('WebBrain could not re-check the screen');+ }if (frame.probe.url !== session.keyframe.url) return refuse('the page navigated');if (frame.size.width !== session.keyframe.width || frame.size.height !== session.keyframe.height) return refuse('the viewport changed');const d = diffFrames(session.seenPixels, frame.raster.pixels, frame.size.width, frame.size.height);@@ -468,8 +597,28 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {return { resolved, session, state };};- /** Execute take_screenshot, zoom, scroll_at, hover_at, drag_at and wait. */+ /**+ * Execute take_screenshot, zoom, scroll_at, hover_at, drag_at and wait. A+ * failure becomes a tool result (no dispatch, or an unknown outcome once+ * input may have been sent) instead of ending the run.+ */proto._executeVisionTool = async function _executeVisionTool(tabId, name, args = {}, context = {}) {+ const dispatch = { started: false };+ try {+ return await this._executeVisionToolInner(tabId, name, args, context, dispatch);+ } catch (error) {+ if (isControlFlowError(error)) throw error;+ this._visionTraceNote(tabId, 'vision_tool_error', { tool: name, error: errorText(error), dispatched: dispatch.started });+ if (dispatch.started) {+ const session = this._visionSessions?.get(tabId);+ if (session) session.forceKeyframe = true;+ return { success: false, outcomeUnknown: true, error: `${name} failed while sending input (${errorText(error)}). Look at the next screen before repeating it.` };+ }+ return { success: false, dispatched: false, noDispatch: true, error: `${name} failed before any input was sent (${errorText(error)}).` };+ }+ };++ proto._executeVisionToolInner = async function _executeVisionToolInner(tabId, name, args, context, dispatch) {const vision = context?._vision || null;const state = this._visionRunState(tabId);if (!state.active) {@@ -493,6 +642,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {const probe = await this._visionPlatformProbe(tabId);const plan = planZoomImageSize({ regionWidth: region.cssRegion.width, regionHeight: region.cssRegion.height, devicePixelRatio: probe?.devicePixelRatio || 1, profile });let dataUrl = await this._visionPlatformCapture(tabId, { probe, cssRegion: region.cssRegion, outWidth: plan.width, outHeight: plan.height });+ if (!dataUrl) throw new Error('The browser returned no screenshot.');if (this.screenshotRedaction && typeof this._redactScreenshotDataUrl === 'function') {try {// A crop is a page-space box; the redactor maps element rects into it.@@ -532,9 +682,13 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {const probe = await this._visionPlatformProbe(tabId);point = { x: (probe?.cssWidth || 800) / 2, y: (probe?.cssHeight || 600) / 2 };}+ // A focused number field under the pointer takes the wheel as +/-.+ const refused = visionWheelRefusal(await this._visionDomProbe(tabId, { points: [point] }));+ if (refused) return { ...refused, at: point };const distance = steps * VISION_SCROLL_STEP_CSS_PX;const deltaX = direction === 'left' ? -distance : direction === 'right' ? distance : 0;const deltaY = direction === 'up' ? -distance : direction === 'down' ? distance : 0;+ dispatch.started = true;const result = await this._visionPlatformWheel(tabId, { x: point.x, y: point.y, deltaX, deltaY });return { success: result?.success !== false, dispatched: result?.dispatched !== false, direction, amount: steps, at: point, ...(result || {}) };}@@ -542,6 +696,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {const resolved = await this._visionResolveForTool(tabId, { x: args.x, y: args.y, image: args.image }, vision);if (resolved.error) return resolved.error;const point = { x: resolved.resolved.cssX, y: resolved.resolved.cssY };+ dispatch.started = true;const result = await this._visionPlatformHover(tabId, point);return { success: result?.success !== false, at: point, ...(result || {}) };}@@ -552,6 +707,13 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {if (!end.ok) return { success: false, dispatched: false, noDispatch: true, visionCoordinateError: end.code, error: end.error };const from = { x: start.resolved.cssX, y: start.resolved.cssY };const to = { x: end.cssX, y: end.cssY };+ // A press and release on the same control is a click that no click+ // guard has seen; a frame WebBrain cannot inspect could hide one.+ const longEnough = Math.hypot(to.x - from.x, to.y - from.y) >= VISION_MIN_DRAG_CSS_PX;+ const probe = longEnough ? await this._visionDomProbe(tabId, { points: [from, to], pair: true }) : null;+ const refused = visionDragRefusal(probe, from, to);+ if (refused) return { ...refused, from, to };+ dispatch.started = true;const result = await this._visionPlatformDrag(tabId, { from, to, steps: 12 });return { success: result?.success !== false, from, to, ...(result || {}) };}diff --git a/src/firefox/src/agent/vision-session.js b/src/firefox/src/agent/vision-session.jsindex f89164d89..e9c5c4c9b 100644--- a/src/firefox/src/agent/vision-session.js+++ b/src/firefox/src/agent/vision-session.js@@ -36,10 +36,15 @@ export function normalizeImageId(value) {}export class VisionSession {- constructor({ coordinates = 'pixel' } = {}) {+ /**+ * `startEpoch` / `startCounter` continue the numbering of an earlier+ * session of the same conversation (another model, a restarted worker), so+ * ids and epochs never repeat within one history.+ */+ constructor({ coordinates = 'pixel', startEpoch = 0, startCounter = 0 } = {}) {this.coordinates = VISION_COORDINATE_FORMATS.includes(coordinates) ? coordinates : 'pixel';- this.counter = 0;- this.epoch = 0;+ this.counter = Math.max(0, Math.floor(finite(startCounter) || 0));+ this.epoch = Math.max(0, Math.floor(finite(startEpoch) || 0));this.updateSeq = 0;this.keyframe = null;this.images = new Map();@@ -108,6 +113,22 @@ export class VisionSession {return this.keyframe;}+ /**+ * Forget the current screen (a new run starts, or the screen can no longer+ * be trusted): its images become "from an earlier screen" and coordinate+ * tools are refused until the next keyframe.+ */+ retireKeyframe() {+ for (const [id, image] of this.images) this.retired.set(id, { epoch: image.epoch, replacedBy: null });+ this.images = new Map();+ this.keyframe = null;+ this.seenPixels = null;+ this.changes = [];+ this.patchChainLength = 0;+ this.pendingZooms = [];+ this.forceKeyframe = true;+ }+/** Record patches (keyframe px regions) the model is about to receive. */registerPatches(regions = [], pixels = null) {if (!this.keyframe) throw new Error('A keyframe is required before patches.');@@ -178,8 +199,13 @@ export class VisionSession {}_imageFor(imageId) {- const id = normalizeImageId(imageId);- if (!id) return { image: this.keyframe, id: this.keyframe?.id || '', composite: true };+ const raw = imageId == null ? '' : String(imageId).trim();+ // No image named: the current screen. A value that is not an image id+ // ("zoom z2", "z2.png") is refused rather than read as the current screen,+ // which would map a zoom's point through the keyframe.+ if (!raw) return { image: this.keyframe, id: this.keyframe?.id || '', composite: true };+ const id = normalizeImageId(raw);+ if (!id) return { image: null, id: raw.slice(0, 40), retired: null };const image = this.images.get(id);if (image) return { image, id, composite: false };return { image: null, id, retired: this.retired.get(id) || null };@@ -199,7 +225,7 @@ export class VisionSession {if (lookup.retired) {return { ok: false, code: 'stale_image', error: `Image ${lookup.id} is from an earlier screen. The current full frame is ${this.keyframe.id}; read the point from it or from a later patch.` };}- return { ok: false, code: 'unknown_image', error: `Unknown image "${image}". Valid images now: ${[...this.images.keys()].join(', ')}.` };+ return { ok: false, code: 'unknown_image', error: `Unknown image "${lookup.id}". Name an image exactly as labeled (valid now: ${[...this.images.keys()].join(', ')}), or omit image for the current screen.` };}const target = lookup.image;const px = finite(x);@@ -261,9 +287,13 @@ export class VisionSession {const bottomRight = this.resolvePoint({ x: bx, y: by, image });if (!bottomRight.ok && bottomRight.code !== 'stale_area') return bottomRight;const k0 = topLeft.keyframePoint;+ // The rectangle's size in the target image's own pixels (normalized+ // coordinates are 0..1000 of that image).+ const wImage = this.coordinates === 'normalized' ? (w / NORMALIZED_COORDINATE_SCALE) * target.width : w;+ const hImage = this.coordinates === 'normalized' ? (h / NORMALIZED_COORDINATE_SCALE) * target.height : h;const k1 = bottomRight.ok ? bottomRight.keyframePoint : {- x: Math.min(this.keyframe.width, k0.x + w * target.scaleX),- y: Math.min(this.keyframe.height, k0.y + h * target.scaleY),+ x: Math.min(this.keyframe.width, k0.x + wImage * target.scaleX),+ y: Math.min(this.keyframe.height, k0.y + hImage * target.scaleY),};const kfRegion = {x: Math.max(0, Math.floor(k0.x)),diff --git a/src/firefox/src/agent/vision-tools.js b/src/firefox/src/agent/vision-tools.jsindex e09d02cb4..4fcf3f31f 100644--- a/src/firefox/src/agent/vision-tools.js+++ b/src/firefox/src/agent/vision-tools.js@@ -81,7 +81,7 @@ export function visionToolDefinitions({ coordinates = 'pixel' } = {}) {type: 'function',function: {name: 'click_at',- description: `Click at a point with a real mouse event. x/y are ${words.unit}. WebBrain converts the point to the page (retina displays and browser zoom included); never rescale it yourself. Aim at the visual centre of the target. clicks:2 double-clicks, clicks:3 selects a line or paragraph, button:"right" opens a context menu.`,+ description: `Click at a point with a real mouse event. x/y are ${words.unit}. WebBrain converts the point to the page (retina displays and browser zoom included); never rescale it yourself. Aim at the visual centre of the target. clicks:2 double-clicks and clicks:3 selects a line or paragraph, for selecting text only: repeated clicks on a button, link or other control are refused. button:"right" opens a context menu.`,parameters: {type: 'object',properties: {@@ -127,7 +127,7 @@ export function visionToolDefinitions({ coordinates = 'pixel' } = {}) {type: 'function',function: {name: 'drag_at',- description: `Press the left mouse button at the start point, move to the end point and release (sliders, reordering, selecting text, drawing). Coordinates are ${words.unit}; both points come from the same image.`,+ description: `Press the left mouse button at the start point, move to the end point and release (sliders, reordering, selecting text, drawing). Coordinates are ${words.unit}; both points come from the same image. Not a way to click: a drag of a few pixels, or one that presses and releases on the same button or link, is refused; use click_at.`,parameters: {type: 'object',properties: {@@ -238,15 +238,20 @@ export function translateVisionClick(args = {}, session) {};}const { button, clicks } = normalizeClickArgs(args);+ // The coordinate click path reads x,y both within 0..1 as a normalized+ // fraction; the top-left CSS pixel is nudged off that range.+ const corner = point.cssX <= 1 && point.cssY <= 1;+ const cssX = corner ? 1.5 : point.cssX;+ const cssY = corner ? 1.5 : point.cssY;return {ok: true,name: 'click',- args: { x: point.cssX, y: point.cssY, coordinate_space: 'css' },+ args: { x: cssX, y: cssY, coordinate_space: 'css' },vision: {tool: 'click_at',button,clickCount: clicks,- cssPoint: { x: point.cssX, y: point.cssY },+ cssPoint: { x: cssX, y: cssY },keyframePoint: point.keyframePoint,imageId: point.imageId,keyframeId: point.keyframeId,diff --git a/test/quality/vision-mode-mv3.mjs b/test/quality/vision-mode-mv3.mjsindex 429499b98..61e50c253 100644--- a/test/quality/vision-mode-mv3.mjs+++ b/test/quality/vision-mode-mv3.mjs@@ -15,8 +15,12 @@ import { fixtureFixedChoiceReply } from '../semantic-decision-stub.mjs';// node test/quality/vision-mode-mv3.mjs DPR 1// node test/quality/vision-mode-mv3.mjs --dpr=2 retina// node test/quality/vision-mode-mv3.mjs --zoom=1.25 browser zoom 125%+// node test/quality/vision-mode-mv3.mjs --scroll=100 page scrolled 100 CSS px+// Two colored markers (one fixed, one in the document) must appear in the+// first frame exactly where the frame label's scale puts their CSS boxes.const dpr = Number((process.argv.find(arg => arg.startsWith('--dpr=')) || '--dpr=1').slice(6)) || 1;const zoom = Number((process.argv.find(arg => arg.startsWith('--zoom=')) || '--zoom=1').slice(7)) || 1;+const scroll = Number((process.argv.find(arg => arg.startsWith('--scroll=')) || '--scroll=0').slice(9)) || 0;const out = path.resolve(process.env.WEBBRAIN_QUALITY_ARTIFACTS || 'ci/artifacts/vision-mode-mv3');await fs.mkdir(out, { recursive: true });@@ -24,15 +28,24 @@ await fs.mkdir(out, { recursive: true });const LAYOUT = {name: { x: 60, y: 140, w: 320, h: 36 },submit: { x: 60, y: 220, w: 180, h: 44 },+ // Viewport-fixed and document-positioned markers for the pixel check.+ fixedMarker: { x: 700, y: 60, w: 80, h: 40 },+ pageMarker: { x: 600, y: 230, w: 60, h: 50 },};const body = `<!doctype html><meta charset="utf-8"><title>Vision MV3 fixture</title><style>body{margin:0;font:16px sans-serif}h1{position:absolute;left:60px;top:40px;margin:0}#name{position:absolute;left:${LAYOUT.name.x}px;top:${LAYOUT.name.y}px;width:${LAYOUT.name.w}px;height:${LAYOUT.name.h}px;box-sizing:border-box;font:16px sans-serif}#submit{position:absolute;left:${LAYOUT.submit.x}px;top:${LAYOUT.submit.y}px;width:${LAYOUT.submit.w}px;height:${LAYOUT.submit.h}px;background:#1a73e8;color:#fff;border:0;font:16px sans-serif}#result{position:absolute;left:60px;top:300px;width:400px;height:30px}-label{position:absolute;left:60px;top:112px}</style>+label{position:absolute;left:60px;top:112px}+#fixed-marker{position:fixed;left:${LAYOUT.fixedMarker.x}px;top:${LAYOUT.fixedMarker.y}px;width:${LAYOUT.fixedMarker.w}px;height:${LAYOUT.fixedMarker.h}px;background:rgb(255,0,255)}+#page-marker{position:absolute;left:${LAYOUT.pageMarker.x}px;top:${LAYOUT.pageMarker.y}px;width:${LAYOUT.pageMarker.w}px;height:${LAYOUT.pageMarker.h}px;background:rgb(0,255,255)}+#tall{position:absolute;left:0;top:0;width:1px;height:2400px}</style>+<div id="tall"></div><div id="fixed-marker"></div><div id="page-marker"></div><h1>Vision form</h1><label for="name">Name</label><input id="name" name="name" autocomplete="off"><button id="submit" type="button">Send name</button><div id="result" role="status"></div>-<script>document.getElementById('submit').addEventListener('click', async () => {+<script>// A hostile page cannot blind the dispatch checks: they run in an isolated world.+Document.prototype.elementFromPoint = function () { return null; };+document.getElementById('submit').addEventListener('click', async () => {const value = document.getElementById('name').value;await fetch('/submit', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ name: value }) });document.getElementById('result').textContent = 'Saved: ' + value;@@ -73,7 +86,7 @@ let keyframe = null;// label declares, as a model reading the label would.const centre = (rect, image) => ({x: Math.round((rect.x + rect.w / 2) / image.ratio),- y: Math.round((rect.y + rect.h / 2) / image.ratio),+ y: Math.round((rect.y - scroll + rect.h / 2) / image.ratio),});const server = http.createServer(async (request, response) => {@@ -101,17 +114,27 @@ const server = http.createServer(async (request, response) => {const key = /SCREENSHOT (f\d+)/.exec(label)?.[1];if (key && !frames.some(frame => frame.id === key)) {const declared = /(\d+)x(\d+) image; 1 image px = ([\d.]+) CSS px/.exec(label);- frames.push({ id: key, size, declared: declared ? { width: Number(declared[1]), height: Number(declared[2]), ratio: Number(declared[3]) } : null, call: providerCalls.length });+ frames.push({ id: key, size, declared: declared ? { width: Number(declared[1]), height: Number(declared[2]), ratio: Number(declared[3]) } : null, call: providerCalls.length, ...(frames.length ? {} : { dataUrl: url }) });}}}keyframe = frames.at(-1)?.declared || null;const step = actionCalls++;+ // Decide from the history, not a step count: WebBrain may set a+ // response aside when the page changed while the model was thinking.+ const results = new Map(input.messages.filter(message => message.role === 'tool').map(message => [message.tool_call_id, String(message.content)]));+ const pairs = input.messages.filter(message => message.role === 'assistant').flatMap(message => message.tool_calls || [])+ .map(item => ({ name: item.function.name, ok: /"success":true/.test(results.get(item.id) || '') }));+ const typedAt = pairs.findLastIndex(pair => pair.name === 'type_text' && pair.ok);+ const submitted = typedAt >= 0 && pairs.slice(typedAt + 1).some(pair => pair.name === 'click_at' && pair.ok);let call;if (!keyframe) call = { name: 'take_screenshot', args: {} };- else if (step === 0) call = { name: 'click_at', args: centre(LAYOUT.name, keyframe) };- else if (step === 1) call = { name: 'type_text', args: { text: 'Vision QA' } };- else if (step === 2) call = { name: 'click_at', args: centre(LAYOUT.submit, keyframe) };+ else if (typedAt < 0) {+ const last = pairs.at(-1);+ call = last?.name === 'click_at' && last.ok+ ? { name: 'type_text', args: { text: 'Vision QA' } }+ : { name: 'click_at', args: centre(LAYOUT.name, keyframe) };+ } else if (!submitted) call = { name: 'click_at', args: centre(LAYOUT.submit, keyframe) };else call = { name: 'done', args: { summary: 'The name was sent.', outcome: 'success' } };completion = { role: 'assistant', content: '', tool_calls: [{ id: `vision-${step}`, type: 'function', function: { name: call.name, arguments: JSON.stringify(call.args) } }] };}@@ -151,14 +174,18 @@ try {await harness.evaluate(async ({ tabId, zoom }) => chrome.tabs.setZoom(tabId, zoom), { tabId, zoom });await page.waitForFunction(expected => Math.abs(window.devicePixelRatio - expected) < 0.05, dpr * zoom);}+ if (scroll) {+ await page.evaluate(y => window.scrollTo(0, y), scroll);+ await page.waitForFunction(y => Math.abs(window.scrollY - y) < 1, scroll);+ }// What the scripted model knows about the screen: its CSS size.- cssViewport = await page.evaluate(() => ({ width: document.documentElement.clientWidth, height: document.documentElement.clientHeight, dpr: devicePixelRatio }));+ cssViewport = await page.evaluate(() => ({ width: document.documentElement.clientWidth, height: document.documentElement.clientHeight, dpr: devicePixelRatio, scrollY: window.scrollY }));const started = performance.now();const result = await message('chat', { tabId, text: 'Type Vision QA into the Name field and press Send name.', mode: 'act', requestId: 'vision-mv3', locale: 'en' });const elapsedMs = Math.round(performance.now() - started);const events = await harness.evaluate(() => events);const value = await page.locator('#name').inputValue();- await fs.writeFile(path.join(out, `evidence-dpr${dpr}-zoom${zoom}.json`), JSON.stringify({ result, value, submissions, frames, cssViewport, elapsedMs, events, providerCalls },+ await fs.writeFile(path.join(out, `evidence-dpr${dpr}-zoom${zoom}-scroll${scroll}.json`), JSON.stringify({ result, value, submissions, frames, cssViewport, elapsedMs, events, providerCalls },(_key, item) => typeof item === 'string' && item.startsWith('data:image/') ? `[image ${JSON.stringify(imageSize(item))}]` : item, 2));const actionRequests = providerCalls.filter(call => Array.isArray(call.tools) && call.tools.length);@@ -174,13 +201,52 @@ try {assert.equal(first.size.type, 'webp', 'Anthropic-family models get WebP');// Never device pixels: at most one image px per CSS px.assert.ok(first.size.width <= cssViewport.width, `image ${first.size.width}px for a ${cssViewport.width}px CSS viewport at DPR ${cssViewport.dpr}`);+ // The pixels show the page where the label's scale says: each marker's box+ // in the image, times the declared CSS px per image px, is its CSS box in+ // the viewport (browser zoom and scrolling included).+ const boxes = await page.evaluate(async dataUrl => {+ const bitmap = await createImageBitmap(await (await fetch(dataUrl)).blob());+ const canvas = new OffscreenCanvas(bitmap.width, bitmap.height);+ const context = canvas.getContext('2d');+ context.drawImage(bitmap, 0, 0);+ const { data, width, height } = context.getImageData(0, 0, bitmap.width, bitmap.height);+ const find = test => {+ let x0 = Infinity, y0 = Infinity, x1 = -1, y1 = -1;+ for (let y = 0; y < height; y++) for (let x = 0; x < width; x++) {+ const i = (y * width + x) * 4;+ if (test(data[i], data[i + 1], data[i + 2])) { x0 = Math.min(x0, x); y0 = Math.min(y0, y); x1 = Math.max(x1, x); y1 = Math.max(y1, y); }+ }+ return x1 < 0 ? null : { x0, y0, x1: x1 + 1, y1: y1 + 1 };+ };+ return {+ fixed: find((r, g, b) => r > 200 && g < 90 && b > 200),+ page: find((r, g, b) => r < 90 && g > 200 && b > 200),+ };+ }, first.dataUrl);+ const ratio = first.declared.ratio;+ for (const [key, rect, offsetY] of [['fixed', LAYOUT.fixedMarker, 0], ['page', LAYOUT.pageMarker, cssViewport.scrollY]]) {+ const box = boxes[key];+ assert.ok(box, `${key} marker visible in f1`);+ const css = { x0: box.x0 * ratio, y0: box.y0 * ratio, x1: box.x1 * ratio, y1: box.y1 * ratio };+ // Clipped to the viewport (at 150% zoom it is only 853 CSS px wide).+ const expected = {+ x0: Math.max(0, rect.x), y0: Math.max(0, rect.y - offsetY),+ x1: Math.min(cssViewport.width, rect.x + rect.w), y1: Math.min(cssViewport.height, rect.y - offsetY + rect.h),+ };+ for (const side of ['x0', 'y0', 'x1', 'y1']) {+ assert.ok(Math.abs(css[side] - expected[side]) <= 3, `${key} marker ${side}: image says CSS ${css[side].toFixed(1)}, page has ${expected[side]} (${JSON.stringify({ box, ratio, cssViewport })})`);+ }+ }assert.equal(value, 'Vision QA', `the click focused the field and typing reached it: ${JSON.stringify(result).slice(0, 600)}`);assert.equal(submissions.length, 1, JSON.stringify(events.filter(event => event.type === 'tool_result').map(event => event.data), null, 1).slice(0, 3000));assert.deepEqual(submissions[0], { name: 'Vision QA' });// The model-facing click results (tool messages) name the exact-point path.const toolResults = actionRequests.at(-1).messages.filter(message => message.role === 'tool').map(message => String(message.content));const clickResults = toolResults.filter(text => /"method":"cdp-vision-pointer"/.test(text));- assert.equal(clickResults.length, 2, toolResults.join('\n').slice(0, 2000));+ assert.ok(clickResults.length >= 2, toolResults.join('\n').slice(0, 2000));+ // The pre-dispatch probe saw the real button although the page replaced+ // elementFromPoint in its own world.+ assert.ok(clickResults.some(text => /"target":\{"tag":"button"\}/.test(text)), clickResults.join('\n'));// After the first frame the model got patches or "no visible change", not// a new full frame per action.const laterImages = actionRequests.slice(1).flatMap(call => call.messages.filter(message => Array.isArray(message.content))@@ -189,7 +255,7 @@ try {const lastRequest = actionRequests.at(-1);const imagesInLast = lastRequest.messages.flatMap(message => (Array.isArray(message.content) ? message.content : [])).filter(block => block.type === 'image_url').length;assert.ok(imagesInLast <= 6, `old images were pruned (${imagesInLast} in the last request)`);- console.log(JSON.stringify({ passed: true, dpr, zoom, cssViewport, firstFrame: first, frames: frames.length, actionRequests: actionRequests.length, submitted: submissions.length, elapsedMs }));+ console.log(JSON.stringify({ passed: true, dpr, zoom, scroll, cssViewport, firstFrame: { ...first, dataUrl: undefined }, markers: boxes, frames: frames.length, actionRequests: actionRequests.length, submitted: submissions.length, elapsedMs }));} finally {await context?.close(); await new Promise(resolve => server.close(resolve));const resolved = path.resolve(profile);diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjsindex d66fe2869..adc3d83f6 100644--- a/test/vision-mode-agent.mjs+++ b/test/vision-mode-agent.mjs@@ -37,6 +37,14 @@ function createScene({ cssWidth = 1440, cssHeight = 900, devicePixelRatio = 2 }cssWidth, cssHeight, devicePixelRatio, url: PAGE, scrollX: 0, scrollY: 0,rects: [{ x: 100, y: 100, w: 300, h: 60, color: [30, 90, 200] }],captures: 0,+ probes: [],+ // What the DOM probe reports under points (vision-dom-probe.js shape).+ probe(request) {+ return {+ points: (request.points || []).map(() => ({ found: true, tag: 'div', clickable: false, opaqueFrame: false })),+ ...(request.pair ? { pair: { known: true, clickEquivalent: false } } : {}),+ };+ },render(width, height) {const data = new Uint8ClampedArray(width * height * 4).fill(255);const sx = width / this.cssWidth;@@ -98,6 +106,7 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))_visionPlatformWheel: async (_tab, input) => { platform.push({ kind: 'wheel', ...input }); return { success: true, dispatched: true }; },_visionPlatformHover: async (_tab, input) => { platform.push({ kind: 'hover', ...input }); return { success: true, dispatched: true }; },_visionPlatformDrag: async (_tab, input) => { platform.push({ kind: 'drag', ...input }); return { success: true, dispatched: true }; },+ _visionPlatformDomProbe: async (_tab, request) => { scene.probes.push(request); return scene.probe(request); },});for (const name of ['_chromeProtectedPageFailure', '_captchaMutationPreflight', '_workflowPreSubmitDispatchBlock','_messageRecipientGuardBlock', '_detectLikelySubmitAction', '_socialPublicationPreSubmitBlock']) agent[name] = async () => null;@@ -113,7 +122,7 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))onDispatch?.(name, args, scene);return { success: true, dispatched: true };};- return { agent, requests, dispatched, platform, scene };+ return { agent, requests, dispatched, platform, scene, provider: scripted };}const runLoop = (agent, tab, streaming, text = 'Fill the form.') => streaming? agent.processMessageStream(tab, text, () => {}, 'act', { detachedRequestId: `vision-${tab}` })@@ -276,6 +285,160 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))});}+ const toolTexts = request => request.messages.filter(message => message.role === 'tool').map(message => String(message.content));+ const totalImages = request => request.messages.reduce((sum, message) => sum + imageCount(message), 0);++ test(`${build}: a refused click_at stops the rest of its batch`, async () => {+ const { agent, requests, dispatched } = runAgent(step => {+ if (step === 1) return { toolCalls: [call('click_at', { x: 5000, y: 5000 }), call('type_text', { text: 'hunter2' })] };+ if (step === 2) return { toolCalls: [call('done', { summary: 'Stopped.', outcome: 'partial' })] };+ return { content: 'done' };+ });+ await runLoop(agent, nextTab++, false);+ // type_text types into whatever has focus; it was planned on the refused point.+ assert.ok(!dispatched.some(entry => entry.name === 'type_text'), JSON.stringify(dispatched.map(entry => entry.name)));+ assert.ok(toolTexts(requests[1]).some(text => /outside f1/.test(text)), toolTexts(requests[1]).join('\n'));+ });++ test(`${build}: a vision tool that throws returns a tool error and the run goes on`, async () => {+ const ctx = runAgent(step => {+ if (step === 1) return { toolCalls: [call('scroll_at', { direction: 'down' })] };+ if (step === 2) return { toolCalls: [call('done', { summary: 'Could not scroll.', outcome: 'partial' })] };+ return { content: 'done' };+ });+ ctx.agent._visionPlatformWheel = async () => { throw new Error('Could not establish connection. Receiving end does not exist.'); };+ const result = await runLoop(ctx.agent, nextTab++, false);+ assert.equal(ctx.requests.length, 2, `the model decided again: ${JSON.stringify(result).slice(0, 300)}`);+ assert.ok(toolTexts(ctx.requests[1]).some(text => /scroll_at failed while sending input/.test(text)), toolTexts(ctx.requests[1]).join('\n'));+ });++ test(`${build}: a screen update that cannot be encoded leaves a note, then a full frame follows`, async () => {+ let failPatch = true;+ const ctx = runAgent(step => {+ if (step === 1) return { toolCalls: [call('click_at', { x: 648, y: 405 })] };+ if (step === 2) return { toolCalls: [call('click_at', { x: 200, y: 150 })] };+ if (step === 3) return { toolCalls: [call('done', { summary: 'Clicked twice.', outcome: 'success' })] };+ return { content: 'done' };+ }, { onDispatch: (name, _args, scene) => { if (name === 'click') scene.rects.push({ x: 600 + scene.rects.length * 20, y: 420, w: 60, h: 40, color: [200, 40, 40] }); } });+ const encode = ctx.agent._visionEncodeRegion;+ ctx.agent._visionEncodeRegion = async (canvas, region, options) => {+ if (region && failPatch) { failPatch = false; throw new Error('convertToBlob failed'); }+ return encode(canvas, region, options);+ };+ await runLoop(ctx.agent, nextTab++, false);+ assert.equal(ctx.requests.length, 3);+ assert.match(textOf(visionMessages(ctx.requests[1].messages).at(-1)), /could not be prepared after the action/);+ const latest = visionMessages(ctx.requests[2].messages).at(-1);+ assert.equal(latest.webbrainVisionFrame.kind, 'key', textOf(latest));+ assert.match(textOf(latest), /full viewport \(requested\)/);+ });++ test(`${build}: a new run never shows or uses the previous run's frame when its first capture fails`, async () => {+ const ctx = runAgent(step => {+ if (step === 1) return { toolCalls: [call('done', { summary: 'Looked.', outcome: 'success' })] };+ if (step === 2) return { toolCalls: [call('click_at', { x: 648, y: 405 })] };+ if (step === 3) return { toolCalls: [call('done', { summary: 'Could not see.', outcome: 'partial' })] };+ return { content: 'done' };+ });+ const tab = nextTab++;+ await runLoop(ctx.agent, tab, false, 'Look at the page.');+ ctx.agent._visionPlatformCapture = async () => { throw new Error('capture failed'); };+ await runLoop(ctx.agent, tab, false, 'Click the button.');+ assert.ok(!ctx.dispatched.some(entry => entry.name === 'click'), 'nothing was clicked');+ const firstOfRun2 = ctx.requests[1];+ assert.match(textOf(visionMessages(firstOfRun2.messages).at(-1)), /first screenshot could not be captured/);+ assert.equal(totalImages(firstOfRun2), 0, 'the previous run\'s frame is not shown');+ assert.ok(toolTexts(ctx.requests[2]).some(text => /No screenshot of this page has been taken yet/.test(text)), toolTexts(ctx.requests[2]).join('\n'));+ });++ test(`${build}: after a model switch between runs the new frame is kept and ids never repeat`, async () => {+ const ctx = runAgent(step => {+ if (step === 1) return { toolCalls: [call('click_at', { x: 648, y: 405 })] };+ if (step === 2) return { toolCalls: [call('done', { summary: 'Run one.', outcome: 'success' })] };+ if (step === 3) return { toolCalls: [call('done', { summary: 'Run two.', outcome: 'success' })] };+ return { content: 'done' };+ }, { onDispatch: (_name, _args, scene) => { scene.url = `${PAGE}?n=${scene.captures}`; scene.rects = [{ x: 0, y: 0, w: 1440, h: 900, color: [20, 20, 20] }]; } });+ const tab = nextTab++;+ await runLoop(ctx.agent, tab, false, 'Run one.');+ // Another family: a new vision session (other image sizes) for run two.+ Object.assign(ctx.provider, { name: 'openai', model: 'gpt-4.1', config: { category: 'cloud', type: 'openai', model: 'gpt-4.1' } });+ ctx.scene.rects = [{ x: 50, y: 50, w: 100, h: 100, color: [0, 120, 0] }];+ await runLoop(ctx.agent, tab, false, 'Run two.');+ const run2 = ctx.requests[2];+ const keys = visionMessages(run2.messages).filter(message => message.webbrainVisionFrame.kind === 'key');+ assert.ok(keys.length >= 3, `frames of both runs are in the history (${keys.length})`);+ assert.equal(imageCount(keys.at(-1)), 1, 'the new run\'s frame is sent');+ assert.equal(totalImages(run2), 1, 'and nothing older');+ const ids = keys.map(message => message.webbrainVisionFrame.ids[0]);+ assert.equal(new Set(ids).size, ids.length, `ids never repeat: ${ids}`);+ });++ test(`${build}: drag_at refuses click-length drags and drags that would click a control`, async () => {+ const ctx = runAgent(step => {+ if (step === 1) return { toolCalls: [call('drag_at', { start_x: 300, start_y: 300, end_x: 302, end_y: 301 })] };+ if (step === 2) return { toolCalls: [call('drag_at', { start_x: 300, start_y: 300, end_x: 360, end_y: 300 })] };+ if (step === 3) return { toolCalls: [call('done', { summary: 'No drag.', outcome: 'partial' })] };+ return { content: 'done' };+ });+ ctx.scene.probe = request => ({+ points: request.points.map(() => ({ found: true, tag: 'button', clickable: true, control: 'button', opaqueFrame: false })),+ ...(request.pair ? { pair: { known: true, clickEquivalent: true, control: 'button' } } : {}),+ });+ await runLoop(ctx.agent, nextTab++, false);+ assert.equal(ctx.platform.filter(entry => entry.kind === 'drag').length, 0, 'no drag was dispatched');+ assert.ok(toolTexts(ctx.requests[1]).some(text => /a drag this short is a click/.test(text)), toolTexts(ctx.requests[1]).join('\n'));+ assert.ok(toolTexts(ctx.requests[2]).some(text => /would click the button under both ends/.test(text)), toolTexts(ctx.requests[2]).join('\n'));+ // The probe was asked about the press and release points together.+ assert.ok(ctx.scene.probes.some(request => request.pair === true && request.points.length === 2));+ });++ test(`${build}: scroll_at over a focused number field is refused`, async () => {+ const ctx = runAgent(step => {+ if (step === 1) return { toolCalls: [call('scroll_at', { direction: 'down', x: 648, y: 405 })] };+ if (step === 2) return { toolCalls: [call('done', { summary: 'No scroll.', outcome: 'partial' })] };+ return { content: 'done' };+ });+ ctx.scene.probe = request => ({ points: request.points.map(() => ({ found: true, tag: 'input', clickable: false, opaqueFrame: false, focusedNumberInput: true })) });+ await runLoop(ctx.agent, nextTab++, false);+ assert.equal(ctx.platform.filter(entry => entry.kind === 'wheel').length, 0);+ assert.ok(toolTexts(ctx.requests[1]).some(text => /focused number field/.test(text)), toolTexts(ctx.requests[1]).join('\n'));+ });++ test(`${build}: drag_at is blocked on a protected messaging page`, async () => {+ const agent = new Agent({});+ const block = await agent._messageRecipientGuardBlock(1, 'drag_at', { start_x: 1, start_y: 1, end_x: 50, end_y: 1 }, 'https://web.whatsapp.com/');+ assert.equal(block?.reasonCode, 'recipient_unverifiable_dispatch_path', JSON.stringify(block));+ assert.match(block.error, /drag_at cannot bind its effects/);+ assert.equal(await agent._messageRecipientGuardBlock(1, 'drag_at', {}, 'https://owned.example.test/form'), null);+ });++ test(`${build}: the click preflight refuses repeated clicks on controls and focuses native selects`, async () => {+ const agent = new Agent({});+ let answer = null;+ const requests = [];+ agent._visionPlatformDomProbe = async (_tab, request) => { requests.push(request); return answer; };+ const point = extra => ({ points: [{ found: true, tag: 'span', clickable: false, opaqueFrame: false, ...extra }] });+ answer = point({ clickable: true, control: 'button' });+ assert.equal((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 2 })).refusal?.visionInputRefused, 'repeated_click');+ assert.equal((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 1 })).refusal, undefined);+ answer = point({});+ assert.deepEqual((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 2 })).target, { tag: 'span' }, 'text may be double-clicked');+ assert.ok((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 3, recipientGuarded: true })).refusal);+ assert.ok((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 2, submitDetected: true })).refusal);+ answer = point({ opaqueFrame: true });+ assert.ok((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 2 })).refusal);+ answer = null;+ assert.ok((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 2 })).refusal, 'unknown target: no repeated click');+ assert.equal((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 1 })).refusal, undefined, 'a single click still goes');+ answer = point({ tag: 'select', select: { current: 'Red', options: ['Red', 'Blue'] } });+ const select = await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 1, button: 'left' });+ assert.equal(select.refusal?.nativeSelect, true);+ assert.match(select.refusal.error, /native dropdown \(current: "Red"\).*Options: Red, Blue/);+ assert.equal(requests.at(-1).focusSelect, true);+ assert.equal((await agent._visionClickPreflight(1, { x: 5, y: 5, clickCount: 1, button: 'right' })).refusal, undefined);+ assert.equal(requests.at(-1).focusSelect, false, 'a right click does not focus the select');+ });+test(`${build}: _pruneOldImages keeps the current frame with its patches and zooms only`, () => {const agent = new Agent({});const img = id => ({ type: 'image_url', image_url: { url: `data:image/webp;base64,${id}` } });diff --git a/test/vision-mode.mjs b/test/vision-mode.mjsindex 4c1f6809b..39bc14556 100644--- a/test/vision-mode.mjs+++ b/test/vision-mode.mjs@@ -10,7 +10,7 @@ import { test } from 'node:test';const ROOT = fileURLToPath(new URL('../', import.meta.url));const read = file => fs.readFileSync(path.join(ROOT, file), 'utf8');-const SHARED = ['vision-budget.js', 'vision-diff.js', 'vision-session.js', 'vision-tools.js', 'vision-mode.js', 'vision-encoder.js', 'vision-runtime.js'];+const SHARED = ['vision-budget.js', 'vision-diff.js', 'vision-session.js', 'vision-tools.js', 'vision-mode.js', 'vision-encoder.js', 'vision-runtime.js', 'vision-dom-probe.js'];const area = { get: async () => ({}), set: async () => {}, remove: async () => {} };const api = {@@ -423,3 +423,106 @@ for (const build of ['chrome', 'firefox']) {{ schema_version: 1, vision_mode: true, vision_mode_setting: 'on' });});}++const probeModule = await import('../src/chrome/src/agent/vision-dom-probe.js');+const runtimeModule = await import('../src/chrome/src/agent/vision-runtime.js');++test('session: numbering continues across sessions and a retired screen refuses points', () => {+ const session = new VisionSession({ startEpoch: 3, startCounter: 7 });+ const frame = session.registerKeyframe({ imageWidth: 1000, imageHeight: 500, cssWidth: 1000, cssHeight: 500 });+ assert.equal(frame.id, 'f8');+ assert.equal(frame.epoch, 4);+ assert.ok(session.resolvePoint({ x: 10, y: 10 }).ok);+ session.retireKeyframe();+ assert.equal(session.resolvePoint({ x: 10, y: 10 }).code, 'no_frame');+ assert.equal(session.forceKeyframe, true);+ const next = session.registerKeyframe({ imageWidth: 1000, imageHeight: 500, cssWidth: 1000, cssHeight: 500 });+ assert.equal(next.id, 'f9');+ assert.equal(session.resolvePoint({ x: 10, y: 10, image: 'f8' }).code, 'stale_image');+});++test('session: an image value that is not an id is refused, not read as the current screen', () => {+ const session = new VisionSession();+ session.registerKeyframe({ imageWidth: 1000, imageHeight: 500, cssWidth: 2000, cssHeight: 1000 });+ const zoom = session.registerZoom({ x: 500, y: 250, width: 250, height: 125 }, 500, 250);+ assert.equal(zoom.id, 'z2');+ for (const value of ['zoom z2', 'z2.png', 'Z-2', 'image z2']) {+ const point = session.resolvePoint({ x: 200, y: 100, image: value });+ assert.equal(point.code, 'unknown_image', value);+ assert.match(point.error, /Name an image exactly as labeled/);+ }+ // Case and whitespace are forgiven; the zoom maps through its own scale.+ const exact = session.resolvePoint({ x: 200, y: 100, image: ' Z2 ' });+ assert.ok(exact.ok);+ assert.deepEqual({ x: exact.cssX, y: exact.cssY }, { x: 1200, y: 600 });+ for (const value of [undefined, null, '']) assert.equal(session.resolvePoint({ x: 10, y: 10, image: value }).imageId, 'f1');+});++test('session: a normalized zoom whose far corner changed keeps its requested size', () => {+ const session = new VisionSession({ coordinates: 'normalized' });+ session.registerKeyframe({ imageWidth: 1280, imageHeight: 720, cssWidth: 1280, cssHeight: 720 });+ // The bottom-right area changed after f1: the far corner is stale.+ session.registerPatches([{ x: 600, y: 300, width: 400, height: 300 }]);+ const region = session.resolveRegion({ x: 250, y: 250, width: 250, height: 250, image: 'f1' });+ assert.ok(region.ok, JSON.stringify(region));+ // 250/1000 of 1280x720, not 250 keyframe px.+ assert.deepEqual({ width: region.keyframeRegion.width, height: region.keyframeRegion.height }, { width: 320, height: 180 });+});++test('pruning: the last frame in the history is current, whatever its epoch number', () => {+ const img = id => ({ type: 'image_url', image_url: { url: `data:image/webp;base64,${id}` } });+ const frame = (epoch, kind, ids) => ({ role: 'user', content: [{ type: 'text', text: ids.join(',') }, ...ids.map(img)], webbrainVisionFrame: { epoch, kind, ids } });+ // Session A (epochs 1..3), then session B restarted at epoch 1.+ const messages = [frame(1, 'key', ['f1']), frame(1, 'patch', ['p2']), frame(3, 'key', ['f3']), frame(3, 'zoom', ['z4']), frame(1, 'key', ['f1']), frame(1, 'patch', ['p2'])];+ const kept = list => list.flatMap(message => message.content).filter(block => block.type === 'image_url').length;+ const { messages: pruned, latestEpoch } = runtimeModule.pruneVisionImages(messages);+ assert.equal(latestEpoch, 1);+ assert.equal(kept(pruned.slice(0, 4)), 0, 'session A images are omitted');+ assert.equal(kept(pruned.slice(4)), 2, 'session B frame and patch are kept');+ // A failed first capture retires the last frame for the model too.+ const retired = runtimeModule.pruneVisionImages([...messages, { role: 'user', content: 'no frame', webbrainVisionFrame: { epoch: 0, kind: 'retire' } }]).messages;+ assert.equal(kept(retired), 0);+ assert.deepEqual(runtimeModule.visionHistoryCounters(messages), { epoch: 3, counter: 4 });+});++test('dispatch checks: repeated clicks, drags that click, wheels over number fields', () => {+ const at = extra => ({ points: [{ found: true, tag: 'span', clickable: false, opaqueFrame: false, ...extra }] });+ const { visionMultiClickRefusal, visionDragRefusal, visionWheelRefusal } = probeModule;+ assert.equal(visionMultiClickRefusal(at({ clickable: true, control: 'button' }), { clickCount: 1 }), null);+ assert.match(visionMultiClickRefusal(at({ clickable: true, control: 'button' }), { clickCount: 2 }).error, /double click was refused because the point is on a button/);+ assert.equal(visionMultiClickRefusal(at({}), { clickCount: 3 }), null);+ assert.ok(visionMultiClickRefusal(at({}), { clickCount: 2, recipientGuarded: true }));+ assert.ok(visionMultiClickRefusal(null, { clickCount: 2 }));+ const pair = (pairInfo, extra = {}) => ({ points: [at(extra).points[0], at(extra).points[0]], pair: pairInfo });+ const from = { x: 10, y: 10 };+ assert.equal(visionDragRefusal(pair({ known: true, clickEquivalent: false }), from, { x: 12, y: 13 }).visionInputRefused, 'drag_too_short');+ assert.equal(visionDragRefusal(pair({ known: true, clickEquivalent: false }), from, { x: 200, y: 10 }), null);+ assert.equal(visionDragRefusal(pair({ known: true, clickEquivalent: true, control: 'a' }), from, { x: 200, y: 10 }).visionInputRefused, 'drag_is_click');+ assert.equal(visionDragRefusal(pair({ known: true, clickEquivalent: false }, { opaqueFrame: true }), from, { x: 200, y: 10 }).visionInputRefused, 'drag_opaque_frame');+ assert.equal(visionDragRefusal(null, from, { x: 200, y: 10 }).visionInputRefused, 'drag_unchecked');+ assert.equal(visionWheelRefusal(at({ focusedNumberInput: true })).visionInputRefused, 'wheel_number_input');+ assert.equal(visionWheelRefusal(at({ focusedNumberInput: false })), null);+ assert.equal(visionWheelRefusal(null), null, 'an unreadable target does not block scrolling');+ // The probe travels as text: it must stay self-contained.+ const expression = probeModule.visionDomProbeExpression({ points: [{ x: 1, y: 2 }], pair: false });+ assert.match(expression, /^\(function visionDomProbe\(request\)/);+ assert.match(expression, /\(\{"points":\[\{"x":1,"y":2\}\],"pair":false\}\)$/);+});++test('profiles: GLM-4.xV and InternVL point in relative 0..1000', () => {+ for (const model of ['z-ai/glm-4.5v', 'zai-org/GLM-4.6V', 'THUDM/GLM-4.1V-9B-Thinking', 'OpenGVLab/InternVL3-78B']) {+ const profile = budget.detectVisionProfile({ name: 'openrouter', config: { model } });+ assert.equal(profile.id, 'relative1000', model);+ assert.equal(profile.coordinates, 'normalized');+ assert.equal(profile.format, 'jpeg');+ }+ assert.equal(budget.detectVisionProfile({ name: 'openrouter', config: { model: 'z-ai/glm-4.6' } }).id, 'default', 'text GLM is unaffected');+});++test('click_at at the top-left CSS pixel is nudged off the normalized-fraction range', () => {+ const session = new VisionSession();+ session.registerKeyframe({ imageWidth: 1000, imageHeight: 500, cssWidth: 1000, cssHeight: 500 });+ const result = visionTools.translateVisionClick({ x: 0, y: 1 }, session);+ assert.deepEqual(result.args, { x: 1.5, y: 1.5, coordinate_space: 'css' });+ assert.deepEqual(visionTools.translateVisionClick({ x: 0, y: 40 }, session).args, { x: 0, y: 40, coordinate_space: 'css' });+});diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex f06a49f95..762d5519a 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -215,3 +215,60 @@ test('content-script vision input (Firefox path) clicks, double-clicks, focuses,await browser.close();}});++test('the DOM probe behind the dispatch checks reads controls, text fields, focus, selects and frames', async () => {+ const { visionDomProbeExpression } = await import('../src/chrome/src/agent/vision-dom-probe.js');+ const browser = await chromium.launch({ channel: 'chromium' });+ try {+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });+ await page.setContent(`<!doctype html><style>body{margin:0;font:14px sans-serif}*{box-sizing:border-box}+ #btn{position:absolute;left:100px;top:100px;width:200px;height:50px}+ #lab{position:absolute;left:100px;top:180px}#txt{width:200px}+ #num{position:absolute;left:100px;top:240px;width:120px}+ #sel{position:absolute;left:100px;top:300px;width:120px}+ #card{position:absolute;left:100px;top:380px;width:250px;height:100px;display:block;background:#eee}+ #slider{position:absolute;left:400px;top:100px;width:300px;height:30px;background:#ddd}+ #same{position:absolute;left:400px;top:200px;width:300px;height:150px;border:0}+ #cross{position:absolute;left:400px;top:400px;width:200px;height:100px;border:0}</style>+ <button id="btn"><span>Send</span></button>+ <label id="lab">Name <input id="txt" type="text" value="hello world"></label>+ <input id="num" type="number" value="10">+ <select id="sel"><option>Red</option><option>Blue</option></select>+ <a id="card" href="#card">Card <span>inner</span></a>+ <div id="slider" role="slider" aria-valuenow="1"><div style="width:20px;height:30px;background:#333"></div></div>+ <iframe id="same" srcdoc="<style>body{margin:0}</style><button style='position:absolute;left:20px;top:20px;width:100px;height:40px'>In frame</button>"></iframe>+ <iframe id="cross" src="data:text/html,<button>x</button>"></iframe>`);+ await page.waitForFunction(() => document.getElementById('same').contentDocument?.querySelector('button'));+ const probe = request => page.evaluate(visionDomProbeExpression(request));+ const one = async (x, y, extra = {}) => (await probe({ points: [{ x, y }], ...extra })).points[0];++ const onButton = await one(150, 125);+ assert.ok(['button', 'span'].includes(onButton.tag), JSON.stringify(onButton));+ assert.deepEqual({ ...onButton, tag: 'x' }, { found: true, tag: 'x', clickable: true, opaqueFrame: false, control: 'button' });+ const text = await one(200, 190);+ assert.equal(text.tag, 'input');+ assert.equal(text.clickable, false, 'a click in a text field inside a label only places the caret');+ await page.focus('#num');+ assert.equal((await one(130, 250)).focusedNumberInput, true);+ await page.evaluate(() => document.activeElement.blur());+ assert.equal((await one(130, 250)).focusedNumberInput, false);+ const select = await one(130, 310, { focusSelect: true });+ assert.deepEqual(select.select, { current: 'Red', options: ['Red', 'Blue'] });+ assert.equal(await page.evaluate(() => document.activeElement.id), 'sel', 'the select is focused for type_text');+ const inFrame = await one(400 + 50, 200 + 30);+ assert.equal(inFrame.tag, 'button');+ assert.equal(inFrame.clickable, true);+ assert.equal(inFrame.opaqueFrame, false, 'same-origin frames are entered');+ assert.equal((await one(450, 430)).opaqueFrame, true, 'a cross-origin frame is opaque');++ const pair = async (a, b) => (await probe({ points: [a, b], pair: true })).pair;+ assert.deepEqual(await pair({ x: 110, y: 110 }, { x: 290, y: 140 }), { known: true, clickEquivalent: true, control: 'button' });+ assert.deepEqual(await pair({ x: 150, y: 125 }, { x: 600, y: 115 }), { known: true, clickEquivalent: false }, 'press on a button, release elsewhere');+ assert.equal((await pair({ x: 110, y: 400 }, { x: 330, y: 470 })).control, 'a', 'inside one link card');+ assert.deepEqual(await pair({ x: 405, y: 115 }, { x: 650, y: 115 }), { known: true, clickEquivalent: false }, 'a slider drag');+ assert.equal((await pair({ x: 425, y: 225 }, { x: 515, y: 255 })).clickEquivalent, true, 'inside a button in a same-origin frame');+ assert.deepEqual(await pair({ x: 150, y: 125 }, { x: 5000, y: 10 }), { known: false });+ } finally {+ await browser.close();+ }+});