Open-source AI browser agent for Chrome and Firefox (monorepo) π§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode: hover, monitor binding, frames and resizes
Emre Sokullu18eeb9cbbafb
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex de7521b2d..cdb7d0bb0 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -48,7 +48,8 @@ A point is refused before any dispatch when:- its image belongs to an earlier frame (`stale_image`), or the area around it changed in a later patch (`stale_area`); a zoom taken after an earlier action of its batch already shows what that action changed, so it counts from the update the batch produced and only a change after the zoom's capture makes its points stale (a frame-scale capture beside the zoom tells, `_visionRefreshZooms`);- it lies outside the image;- it is the second or later coordinate action of one model turn and the live screen changed near it after the earlier action, or the screen could not be re-captured to check (`_visionStabilityBlock`, checked right before dispatch, after every gate; a `drag_at` is checked at both ends against one capture);-- the run has no frame of its own yet (`no_frame`).+- the run has no frame of its own yet (`no_frame`);+- the browser window was resized or zoomed (CSS viewport size or device pixel ratio) after its frame was taken (`viewport_changed`, `_visionStaleLayoutRefusal`): checked once before each batch that uses points, so a resize page feedback has not reported yet is caught too; a full frame follows. A resize page feedback did report while the model decided sets the decision aside for a fresh full frame before that (`visionScreenFeedback`).A refused point also stops the rest of the model's batch: the calls after it (a `type_text`, say) were planned on that point.@@ -98,7 +99,7 @@ A capture, encoding or input failure never ends the run: a screen update that caThe page monitor (`content/page-monitor.js`) and page feedback (`agent/page-feedback.js`) watch the page between decisions as in a tree run. A vision run changes what they show the model and when they set a decision aside:- **No tree text reaches the model.** The pre-decision capture still binds keyboard input to the focused field, but its tree text is not added to the conversation; feedback messages carry the events and the URL without a page dump, and applying feedback skips the accessibility-tree read altogether.-- **Page churn does not supersede screenshot points.** `click_at`, `hover_at` and `drag_at` are `vision_point` calls and `scroll_at` a `scroll` call (`page-feedback-policy.js`), so automatic page activity while the model was thinking (a ticker, a live timeline, a spinner) keeps the decision. The pixels around the point are compared with the model's frame right before dispatch (`_visionStabilityBlock`, mandatory once anything changed), and the monitor rebases the prepared operation over passive or unattributed churn only while the element under the point is the prepared one and has not moved (`rebaseVisionPreparation`; a wheel binds no element). A person's input, a navigation or another document still voids it.+- **Page churn does not supersede screenshot points.** `click_at`, `hover_at` and `drag_at` are `vision_point` calls and `scroll_at` a `scroll` call (`page-feedback-policy.js`), so automatic page activity while the model was thinking (a ticker, a live timeline, a spinner) keeps the decision. The pixels around the point are compared with the model's frame right before dispatch (`_visionStabilityBlock`, mandatory once anything changed), and the monitor rebases the prepared operation over passive or unattributed churn only while the element under the point is the prepared one and has not moved (`rebaseVisionPreparation`: a click's point, a `hover_at`'s, a `drag_at`'s start, whose end the pixel check holds; a wheel binds no element). A person's input, a navigation or another document still voids it.- **Typing rides on its click.** `type_text` and `press_keys` right after a `click_at` of the same response are checked by focus: the element focused at the clicked point must be a text field (`_visionFocusFollowsClick`, the probe's `focus` reading). The page reacting to that focus (a hint, a composer expanding) no longer blocks the typing.- **The wheel stays the agent's.** Every CDP wheel notch passes the fence, and the monitor counts the wheel events those commands own, so a notch the browser delivers after the next one was fenced is not taken for a person's scroll.- **A set-aside decision is remade on the current screen** (`_visionObserveNow`: a patch, a new frame or "no visible change"), never on tree text.@@ -118,14 +119,14 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. A guard that binds a consequential click to the control and payload it checked (a deletion, a commit after a correction) refuses coordinate clicks on the tree path, where a ref or selector click exists; in a vision run, where the point is all there is, it binds the click to the control the submit probe read at that point (`_resourceMutationActionKey`), with the same record, stage, sibling and confirmation checks and the same card, and `_visionDispatchClick` reads the point again right before input goes out (`_visionMutationBindingBlock`): when the authorized control is no longer there, nothing is sent (`resource_changed_before_dispatch`) and the model chooses again on the updated screen. A guard's advice names what the run offers (`click_at`), never a ref or selector. Only the final dispatch differs:- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively.-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute and each `:hover` rule the content script can read is copied, keyed to that attribute, into a style sheet the document or shadow root adopts, which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (`recipient_bound_click`) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls. It maps a point into a same-origin frame as the Firefox input does; a frame no scale maps counts as one it cannot inspect. (The click submit probe's Chrome frame mapping likewise takes the frame's border, padding and drawn scale into account.)- `click_at` with `clicks: 2|3` is refused on a button, link or other control (including an element drawn with a pointer cursor, which is how a framework button made of a bare `<div>` shows itself), on a detected submit, on a protected messaging page, inside a frame it cannot inspect, or when the target cannot be read: each extra click would repeat the one action the checks approved. Double and triple clicks remain for selecting text.-- `drag_at` is refused when it is shorter than 6 CSS px, when the press and release would click a control (their nearest common ancestor is or is inside a button, link, label, similar control or pointer-cursor element, unless the drag starts on a slider, scrollbar, resizer, draggable item or grab handle), when either end is inside a cross-origin frame, when the target cannot be read, or when the click submit probe reads its start point as a submit, send, delete or payment control (a drag never gets the confirmation a click gets). On a protected messaging page `drag_at` is blocked outright, like `iframe_click`.+- `drag_at` is refused when it is shorter than 6 CSS px, when the press and release would click a control (their nearest common ancestor is or is inside a button, link, label, similar control or pointer-cursor element, unless the drag starts on a slider, scrollbar, resizer, draggable item or grab handle), when either end is inside a frame it cannot inspect (another site's frame, or one drawn rotated, skewed, flipped or in 3D), when the target cannot be read, or when the click submit probe reads its start point as a submit, send, delete or payment control (a drag never gets the confirmation a click gets). On a protected messaging page `drag_at` is blocked outright, like `iframe_click`.- `scroll_at` is refused over a focused number field, which takes the wheel as +/-.Every CDP input passes `cdpClient`'s page-agent fence and every content action passes `beforePageAgentDispatch`, so the page monitor attributes the events to the agent. `drag_at` is gated as a click (`TOOL_CAPABILITY`); captures and pointer results are untrusted page content (`UNTRUSTED_CONTENT_TOOLS`), and every image label says that text in it is data, never instructions. Screenshot redaction (Settings) applies to frames and zooms.@@ -164,12 +165,12 @@ npm run test:vision-mode```- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.-- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.+- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action, on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch.- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard, CSS `:hover` revealed under the pointer (a CSS-only menu, a rule inside `@media`, a shadow root's `:host(:hover)`) and ended by a person's own pointer, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame and another origin's frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.@@ -181,4 +182,4 @@ The older MV3 quality fixtures script an accessibility-tree model (they answer w- HTML5 drag and drop: in Firefox a drag that starts on a `draggable="true"` item runs the drag-and-drop event sequence itself with a constructed `DataTransfer`; in Chrome the CDP pointer drag is left to the browser, as for `drag_drop`, so a page that relies only on `dataTransfer` may not react there.- A drag is not shown to the submit, money or publishing checks; it is refused when it would click a control, but a custom "slide to confirm" widget is only as safe as the page makes it.- The 45-second window that holds back a second click on the same committing caption covers text clicks (`click({text})`) only. Like a ref click on the tree path, a repeated vision click relies on the submit confirmation and the deletion and publication guards instead.-- Firefox input is synthetic (`isTrusted: false`); pages that require trusted events need Chrome.+- Firefox input is synthetic (`isTrusted: false`); pages that require trusted events need Chrome. Its emulated `:hover` covers the style sheets the content script can read: a `:hover` rule in another origin's style sheet does not apply, and page scripts that test `:hover` themselves see no hover.diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex e8d0213e8..24789b31b 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -15730,8 +15730,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout denabled: this.browserBatchEnabled !== false, tier: this._resolvePromptTier(provider), allowedToolNames, toolSchemas: batchSchemas,});// Vision mode: click_at becomes an ordinary coordinate click here, before- // any gate or guard runs (vision-runtime.js).- toolCalls = this._visionTranslateToolCalls(tabId, toolCalls);+ // any gate or guard runs (vision-runtime.js). Points read from a frame the+ // window was resized or zoomed since are refused first.+ const staleLayout = await this._visionStaleLayoutRefusal(tabId, toolCalls);+ toolCalls = this._visionTranslateToolCalls(tabId, toolCalls, { staleLayout });const originalUpdate = onUpdate;const cancellationState = { active: null };// Begin wrapper rows at their execution position, including rejected batches.diff --git a/src/chrome/src/agent/page-feedback.js b/src/chrome/src/agent/page-feedback.jsindex 4f2370660..629f6d895 100644--- a/src/chrome/src/agent/page-feedback.js+++ b/src/chrome/src/agent/page-feedback.js@@ -1989,8 +1989,10 @@ export function installPageFeedback(Agent) {params: { ...safeAction(name, args), runToken: run.token, operationId, allowPassiveRebase: allowPassiveRebase === true,// The monitor rebases these over passive churn while the element// at the CSS point (or the focused field) stays the same; a wheel- // binds no element. A person's input still voids all three.- ...(policy.kind === 'vision_point' && vision?.cssPoint ? { visionPoint: true, x: vision.cssPoint.x, y: vision.cssPoint.y } : {}),+ // binds no element. A person's input still voids all three. A+ // native hover_at / drag_at gives its point as monitorPoint.+ ...(policy.kind === 'vision_point' && (vision?.cssPoint || vision?.monitorPoint)+ ? { visionPoint: true, x: (vision.cssPoint || vision.monitorPoint).x, y: (vision.cssPoint || vision.monitorPoint).y } : {}),...(focusFollowUp ? { visionFocusFollowUp: true } : {}),...(name === 'scroll_at' ? { visionWheel: true } : {}),...(batchId ? { batchId } : {}),diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.jsindex 7ffb988de..e8414e1d3 100644--- a/src/chrome/src/agent/vision-runtime.js+++ b/src/chrome/src/agent/vision-runtime.js@@ -61,6 +61,13 @@ const MICRO_CHANGE_MIN_CELLS = 6;// as does the coordinate click click_at becomes.const VISION_LOOP_SCREEN_TOOLS = new Set([...VISION_NATIVE_TOOLS, 'type_text', 'press_keys']);+// Does this vision call place something at points of a screenshot (a click,+// hover, drag, zoom region, or a wheel at a point)?+function visionCallReadsPoints(name, args = {}) {+ if (['click_at', 'hover_at', 'drag_at', 'zoom'].includes(name)) return true;+ return name === 'scroll_at' && args?.x != null && args?.y != null;+}+export function isMicroScreenChange(diff) {return diff?.comparable === true && Number.isFinite(diff.changedCells)&& diff.changedCells <= Math.max(MICRO_CHANGE_MIN_CELLS, Math.round((Number(diff.totalCells) || 0) * MICRO_CHANGE_CELL_RATIO));@@ -715,7 +722,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {* coordinate calls follow an earlier action in the same batch (those are* re-checked against the live screen right before dispatch).*/- proto._visionTranslateToolCalls = function _visionTranslateToolCalls(tabId, toolCalls) {+ proto._visionTranslateToolCalls = function _visionTranslateToolCalls(tabId, toolCalls, { staleLayout = null } = {}) {const state = this._visionRunState(tabId);if (!state.active || !Array.isArray(toolCalls) || !toolCalls.some(call => VISION_TOOL_NAMES.has(call?.function?.name))) return toolCalls;const session = this._visionSessionFor(tabId, state);@@ -734,7 +741,12 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {let args = {};try { args = typeof call.function.arguments === 'string' ? JSON.parse(call.function.arguments || '{}') : (call.function.arguments || {}); }catch { args = null; }- if (args && VISION_TRANSLATED_TOOLS.has(name)) {+ if (args && staleLayout && visionCallReadsPoints(name, args)) {+ // Read from a frame the window was resized or zoomed since+ // (_visionStaleLayoutRefusal): refused, and the rest of the batch+ // with it, before any gate runs.+ translated = { ...call, _vision: { tool: name, error: staleLayout } };+ } else if (args && VISION_TRANSLATED_TOOLS.has(name)) {const result = translateVisionClick(args, session);translated = result.ok? { ...call, function: { ...call.function, name: result.name, arguments: JSON.stringify(result.args) },@@ -742,7 +754,15 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {: { ...call, _vision: { tool: name, error: result.result } };clickedAt = result.ok && result.vision.button === 'left' ? { ...result.vision.cssPoint } : null;} else if (args) {- translated = { ...call, _vision: { tool: name, needsStabilityCheck: recheck } };+ // hover_at and drag_at bind the page monitor to the element at+ // their point (a drag's start; its end is held by the pixel check),+ // as a translated click does, so unrelated churn before dispatch+ // does not void them (page-feedback.js, rebaseVisionPreparation).+ const at = name === 'hover_at' ? { x: args.x, y: args.y, image: args.image }+ : name === 'drag_at' ? { x: args.start_x, y: args.start_y, image: args.image } : null;+ const point = at ? session.resolvePoint(at) : null;+ translated = { ...call, _vision: { tool: name, needsStabilityCheck: recheck,+ ...(point?.ok ? { monitorPoint: { x: point.cssX, y: point.cssY } } : {}) } };}} else if ((name === 'type_text' || name === 'press_keys') && clickedAt) {translated = { ...call, _vision: { tool: name, focusFollowUp: { ...clickedAt } } };@@ -755,6 +775,45 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {});};+ /**+ * Before a batch that uses screenshot points: has the window been resized+ * or zoomed (CSS viewport size or device pixel ratio) since the frame they+ * were read from? Page feedback sets a decision aside for a resize it+ * reported while the model decided; this catches one it had not reported+ * yet, which would otherwise send the points through the old geometry.+ * Returns the refusal for those points (a full frame follows), or null.+ */+ proto._visionStaleLayoutRefusal = async function _visionStaleLayoutRefusal(tabId, toolCalls) {+ const state = this._visionRunState(tabId);+ const session = state.active ? this._visionSessions?.get(tabId) : null;+ const keyframe = session?.keyframe;+ if (!keyframe || !Array.isArray(toolCalls)) return null;+ const reads = toolCalls.some(call => {+ const name = call?.function?.name || '';+ if (!VISION_TOOL_NAMES.has(name)) return false;+ try { return visionCallReadsPoints(name, typeof call.function.arguments === 'string' ? JSON.parse(call.function.arguments || '{}') : (call.function.arguments || {})); }+ catch { return false; }+ });+ if (!reads) return null;+ let probe = null;+ try { probe = await this._visionPlatformProbe(tabId); }+ catch (error) {+ if (isControlFlowError(error)) throw error;+ return null;+ }+ if (!probe || !session.layoutChanged({ cssWidth: probe.cssWidth, cssHeight: probe.cssHeight, devicePixelRatio: probe.devicePixelRatio })) return null;+ session.forceKeyframe = true;+ const size = (width, height) => `${Math.round(Number(width) || 0)}x${Math.round(Number(height) || 0)}`;+ this._visionTraceNote(tabId, 'vision_layout_changed', {+ frame: keyframe.id, was: size(keyframe.cssWidth, keyframe.cssHeight), now: size(probe.cssWidth, probe.cssHeight),+ dpr: [keyframe.devicePixelRatio, probe.devicePixelRatio],+ });+ return {+ success: false, dispatched: false, noDispatch: true, visionCoordinateError: 'viewport_changed',+ error: `Not dispatched: the browser window was resized or zoomed after ${keyframe.id} was taken (viewport ${size(keyframe.cssWidth, keyframe.cssHeight)} CSS px then, ${size(probe.cssWidth, probe.cssHeight)} now), so points read from it no longer land where you saw them. A fresh full frame follows; read the point again there.`,+ };+ };+/*** Keyboard input right after a vision click: does the focused element sit* at the clicked point and take text? Then the click's own page effectsdiff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex 8aa563781..987e3c35f 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -2068,6 +2068,127 @@return chain;}+ // ββ :hover for the synthetic pointer ββ+ // Synthetic events run a page's hover listeners but give nothing :hover,+ // so a menu or tooltip only CSS reveals (`.menu:hover .submenu`) would stay+ // hidden. While the vision pointer rests on an element, that element and+ // its ancestors (through shadow roots and out of same-origin frames) carry+ // an attribute, and a copy of each :hover rule this world can read, keyed+ // to the attribute (`.menu[data-webbrain-hover] .submenu`), is adopted by+ // the document or shadow root the rule is in. The pointer moving elsewhere+ // moves the attribute; a person's own pointer moving ends the emulation, as+ // the real :hover then follows them. Another origin's style sheets cannot+ // be read, so their :hover rules stay unmatched.+ const _VISION_HOVER_ATTRIBUTE = 'data-webbrain-hover';+ let _visionHoverChain = [];+ const _visionHoverSheets = new Map();+ let _visionHoverListening = false;++ // The rules of `rules` that :hover decides, as CSS text keyed to the+ // attribute, keeping the conditions around them (@media, @supports,+ // @layer, @container, an @import's media, nesting).+ function _visionHoverCopies(rules, depth = 0) {+ const copies = [];+ if (!rules || depth > 12) return copies;+ const hover = /:hover(?![\w-])/gi;+ for (const rule of Array.from(rules)) {+ try {+ const text = String(rule.cssText || '');+ if (typeof rule.selectorText === 'string') {+ const selector = rule.selectorText;+ if (new RegExp(hover.source, 'i').test(selector)) {+ const open = text.indexOf('{');+ if (open > 0) copies.push(`${selector.replace(hover, `[${_VISION_HOVER_ATTRIBUTE}]`)} ${text.slice(open)}`);+ } else if (rule.cssRules?.length) {+ const inner = _visionHoverCopies(rule.cssRules, depth + 1);+ if (inner.length) copies.push(`${selector} { ${inner.join(' ')} }`);+ }+ } else if (rule.styleSheet) {+ const inner = _visionHoverCopies(rule.styleSheet.cssRules, depth + 1);+ const media = String(rule.media?.mediaText || '');+ if (inner.length) copies.push(media && media !== 'all' ? `@media ${media} { ${inner.join(' ')} }` : inner.join(' '));+ } else if (rule.cssRules?.length) {+ const inner = _visionHoverCopies(rule.cssRules, depth + 1);+ const open = text.indexOf('{');+ if (inner.length && open > 0) copies.push(`${text.slice(0, open).trim()} { ${inner.join(' ')} }`);+ }+ } catch { /* a rule this world cannot read */ }+ }+ return copies;+ }++ // Adopt (or refresh) the copy of `root`'s :hover rules.+ function _visionHoverAdopt(root) {+ const view = (root.ownerDocument || root).defaultView;+ if (!view || typeof view.CSSStyleSheet !== 'function' || !root.adoptedStyleSheets) return;+ const ours = _visionHoverSheets.get(root);+ const copies = [];+ for (const sheet of [...Array.from(root.styleSheets || []), ...Array.from(root.adoptedStyleSheets || [])]) {+ if (sheet === ours) continue;+ let rules = null;+ try { rules = sheet.cssRules; } catch { rules = null; }+ copies.push(..._visionHoverCopies(rules));+ }+ if (!copies.length) { _visionHoverDrop(root); return; }+ const sheet = ours || new view.CSSStyleSheet();+ try { sheet.replaceSync(copies.join('\n')); } catch { return; }+ if (!Array.from(root.adoptedStyleSheets).includes(sheet)) root.adoptedStyleSheets = [...Array.from(root.adoptedStyleSheets), sheet];+ _visionHoverSheets.set(root, sheet);+ }++ function _visionHoverDrop(root) {+ const sheet = _visionHoverSheets.get(root);+ _visionHoverSheets.delete(root);+ try {+ if (sheet && root.adoptedStyleSheets) root.adoptedStyleSheets = Array.from(root.adoptedStyleSheets).filter(item => item !== sheet);+ } catch { /* the page replaced its sheets */ }+ }++ // Rest the emulated :hover on `target` (null ends it). `adopted` collects+ // the roots already refreshed by this input, so a drag's moves read the+ // style sheets once.+ function _visionHoverAt(target, adopted = new Set()) {+ const next = [];+ for (let node = target, depth = 0; node?.nodeType === 1 && depth < 512; depth++) {+ next.push(node);+ let frame = null;+ if (!node.parentElement && !node.getRootNode?.()?.host) {+ try { frame = node.ownerDocument?.defaultView?.frameElement || null; } catch { frame = null; }+ }+ node = node.parentElement || node.getRootNode?.()?.host || frame;+ }+ // The rules that can see these elements hovered: their documents' and+ // shadow roots', and the :host rules of the shadow roots they host.+ const roots = new Set();+ for (const element of next) {+ for (const root of [element.getRootNode?.(), element.shadowRoot || element.openOrClosedShadowRoot]) {+ if (root?.nodeType === 9 || root?.nodeType === 11) roots.add(root);+ }+ }+ for (const root of [..._visionHoverSheets.keys()]) if (!roots.has(root)) _visionHoverDrop(root);+ for (const root of roots) {+ if (adopted.has(root)) continue;+ adopted.add(root);+ try { _visionHoverAdopt(root); } catch { /* left without the copy */ }+ }+ // Elements are marked only where some :hover rule could be read.+ const marked = _visionHoverSheets.size ? next : [];+ const kept = new Set(marked);+ for (const element of _visionHoverChain) {+ if (!kept.has(element)) { try { element.removeAttribute(_VISION_HOVER_ATTRIBUTE); } catch { /* gone */ } }+ }+ for (const element of marked) {+ try { if (!element.hasAttribute(_VISION_HOVER_ATTRIBUTE)) element.setAttribute(_VISION_HOVER_ATTRIBUTE, ''); } catch { /* not settable */ }+ }+ _visionHoverChain = marked;+ if (marked.length && !_visionHoverListening) {+ _visionHoverListening = true;+ window.addEventListener('pointermove', event => {+ if (event.isTrusted && (_visionHoverChain.length || _visionHoverSheets.size)) _visionHoverAt(null);+ }, true);+ }+ }+// The pointerId of the vision pointer's events (a mouse's in Chrome).const _VISION_POINTER_ID = 1;@@ -2169,6 +2290,15 @@error: 'Not dispatched: the point is inside an embedded frame drawn rotated, skewed, flipped or in 3D, so it cannot be placed on the frame\'s content. Use a control outside that frame.',};}+ // A frame this world cannot enter (another site's, a plugin, or one+ // still loading): a click or drag sent here would stop at its element+ // and reach nothing inside it.+ if ((type === 'click' || type === 'drag') && /^(IFRAME|FRAME|OBJECT|EMBED)$/i.test(String(hit.el.tagName || ''))) {+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'opaque_frame',+ error: `Not dispatched: the point is inside an embedded frame whose content this browser's vision input cannot reach (another site's frame, a plugin, or a frame still loading): the ${type} would stop at the frame and do nothing inside it. Use a control outside that frame, or take a screenshot and try again if it was still loading.`,+ };+ }if (actionDeadlineExpired()) return { success: false, dispatched: false, noDispatch: true, deadlineExpired: true, error: 'The action deadline expired before dispatch.' };const { el, view, cx, cy } = hit;const button = params.button === 'right' ? 2 : params.button === 'middle' ? 1 : 0;@@ -2187,7 +2317,12 @@// what it entered (shared ancestors get neither), pointer events before// mouse events, then the move. Arriving with no known previous target,// or again on the same one (`arrive`), over and enter go to the target.+ // Style sheets whose :hover copy this input refreshed (_visionHoverAt).+ const hoverAdopted = new Set();const moveOnto = (target, extra = {}, { arrive = false } = {}) => {+ // :hover follows the pointer before its events go out, as it does for+ // a mouse (_visionHoverAt).+ _visionHoverAt(target, hoverAdopted);const last = _visionPointerOver;if (last !== target || arrive) {const previous = last && last !== target && last.isConnected ? last : null;diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 6a08a841a..038e2fc3d 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -13926,8 +13926,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout denabled: this.browserBatchEnabled !== false, tier: this._resolvePromptTier(provider), allowedToolNames, toolSchemas: batchSchemas,});// Vision mode: click_at becomes an ordinary coordinate click here, before- // any gate or guard runs (vision-runtime.js).- toolCalls = this._visionTranslateToolCalls(tabId, toolCalls);+ // any gate or guard runs (vision-runtime.js). Points read from a frame the+ // window was resized or zoomed since are refused first.+ const staleLayout = await this._visionStaleLayoutRefusal(tabId, toolCalls);+ toolCalls = this._visionTranslateToolCalls(tabId, toolCalls, { staleLayout });const originalUpdate = onUpdate;const cancellationState = { active: null };// Begin wrapper rows at their execution position, including rejected batches.diff --git a/src/firefox/src/agent/page-feedback.js b/src/firefox/src/agent/page-feedback.jsindex 4f2370660..629f6d895 100644--- a/src/firefox/src/agent/page-feedback.js+++ b/src/firefox/src/agent/page-feedback.js@@ -1989,8 +1989,10 @@ export function installPageFeedback(Agent) {params: { ...safeAction(name, args), runToken: run.token, operationId, allowPassiveRebase: allowPassiveRebase === true,// The monitor rebases these over passive churn while the element// at the CSS point (or the focused field) stays the same; a wheel- // binds no element. A person's input still voids all three.- ...(policy.kind === 'vision_point' && vision?.cssPoint ? { visionPoint: true, x: vision.cssPoint.x, y: vision.cssPoint.y } : {}),+ // binds no element. A person's input still voids all three. A+ // native hover_at / drag_at gives its point as monitorPoint.+ ...(policy.kind === 'vision_point' && (vision?.cssPoint || vision?.monitorPoint)+ ? { visionPoint: true, x: (vision.cssPoint || vision.monitorPoint).x, y: (vision.cssPoint || vision.monitorPoint).y } : {}),...(focusFollowUp ? { visionFocusFollowUp: true } : {}),...(name === 'scroll_at' ? { visionWheel: true } : {}),...(batchId ? { batchId } : {}),diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.jsindex 7ffb988de..e8414e1d3 100644--- a/src/firefox/src/agent/vision-runtime.js+++ b/src/firefox/src/agent/vision-runtime.js@@ -61,6 +61,13 @@ const MICRO_CHANGE_MIN_CELLS = 6;// as does the coordinate click click_at becomes.const VISION_LOOP_SCREEN_TOOLS = new Set([...VISION_NATIVE_TOOLS, 'type_text', 'press_keys']);+// Does this vision call place something at points of a screenshot (a click,+// hover, drag, zoom region, or a wheel at a point)?+function visionCallReadsPoints(name, args = {}) {+ if (['click_at', 'hover_at', 'drag_at', 'zoom'].includes(name)) return true;+ return name === 'scroll_at' && args?.x != null && args?.y != null;+}+export function isMicroScreenChange(diff) {return diff?.comparable === true && Number.isFinite(diff.changedCells)&& diff.changedCells <= Math.max(MICRO_CHANGE_MIN_CELLS, Math.round((Number(diff.totalCells) || 0) * MICRO_CHANGE_CELL_RATIO));@@ -715,7 +722,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {* coordinate calls follow an earlier action in the same batch (those are* re-checked against the live screen right before dispatch).*/- proto._visionTranslateToolCalls = function _visionTranslateToolCalls(tabId, toolCalls) {+ proto._visionTranslateToolCalls = function _visionTranslateToolCalls(tabId, toolCalls, { staleLayout = null } = {}) {const state = this._visionRunState(tabId);if (!state.active || !Array.isArray(toolCalls) || !toolCalls.some(call => VISION_TOOL_NAMES.has(call?.function?.name))) return toolCalls;const session = this._visionSessionFor(tabId, state);@@ -734,7 +741,12 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {let args = {};try { args = typeof call.function.arguments === 'string' ? JSON.parse(call.function.arguments || '{}') : (call.function.arguments || {}); }catch { args = null; }- if (args && VISION_TRANSLATED_TOOLS.has(name)) {+ if (args && staleLayout && visionCallReadsPoints(name, args)) {+ // Read from a frame the window was resized or zoomed since+ // (_visionStaleLayoutRefusal): refused, and the rest of the batch+ // with it, before any gate runs.+ translated = { ...call, _vision: { tool: name, error: staleLayout } };+ } else if (args && VISION_TRANSLATED_TOOLS.has(name)) {const result = translateVisionClick(args, session);translated = result.ok? { ...call, function: { ...call.function, name: result.name, arguments: JSON.stringify(result.args) },@@ -742,7 +754,15 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {: { ...call, _vision: { tool: name, error: result.result } };clickedAt = result.ok && result.vision.button === 'left' ? { ...result.vision.cssPoint } : null;} else if (args) {- translated = { ...call, _vision: { tool: name, needsStabilityCheck: recheck } };+ // hover_at and drag_at bind the page monitor to the element at+ // their point (a drag's start; its end is held by the pixel check),+ // as a translated click does, so unrelated churn before dispatch+ // does not void them (page-feedback.js, rebaseVisionPreparation).+ const at = name === 'hover_at' ? { x: args.x, y: args.y, image: args.image }+ : name === 'drag_at' ? { x: args.start_x, y: args.start_y, image: args.image } : null;+ const point = at ? session.resolvePoint(at) : null;+ translated = { ...call, _vision: { tool: name, needsStabilityCheck: recheck,+ ...(point?.ok ? { monitorPoint: { x: point.cssX, y: point.cssY } } : {}) } };}} else if ((name === 'type_text' || name === 'press_keys') && clickedAt) {translated = { ...call, _vision: { tool: name, focusFollowUp: { ...clickedAt } } };@@ -755,6 +775,45 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {});};+ /**+ * Before a batch that uses screenshot points: has the window been resized+ * or zoomed (CSS viewport size or device pixel ratio) since the frame they+ * were read from? Page feedback sets a decision aside for a resize it+ * reported while the model decided; this catches one it had not reported+ * yet, which would otherwise send the points through the old geometry.+ * Returns the refusal for those points (a full frame follows), or null.+ */+ proto._visionStaleLayoutRefusal = async function _visionStaleLayoutRefusal(tabId, toolCalls) {+ const state = this._visionRunState(tabId);+ const session = state.active ? this._visionSessions?.get(tabId) : null;+ const keyframe = session?.keyframe;+ if (!keyframe || !Array.isArray(toolCalls)) return null;+ const reads = toolCalls.some(call => {+ const name = call?.function?.name || '';+ if (!VISION_TOOL_NAMES.has(name)) return false;+ try { return visionCallReadsPoints(name, typeof call.function.arguments === 'string' ? JSON.parse(call.function.arguments || '{}') : (call.function.arguments || {})); }+ catch { return false; }+ });+ if (!reads) return null;+ let probe = null;+ try { probe = await this._visionPlatformProbe(tabId); }+ catch (error) {+ if (isControlFlowError(error)) throw error;+ return null;+ }+ if (!probe || !session.layoutChanged({ cssWidth: probe.cssWidth, cssHeight: probe.cssHeight, devicePixelRatio: probe.devicePixelRatio })) return null;+ session.forceKeyframe = true;+ const size = (width, height) => `${Math.round(Number(width) || 0)}x${Math.round(Number(height) || 0)}`;+ this._visionTraceNote(tabId, 'vision_layout_changed', {+ frame: keyframe.id, was: size(keyframe.cssWidth, keyframe.cssHeight), now: size(probe.cssWidth, probe.cssHeight),+ dpr: [keyframe.devicePixelRatio, probe.devicePixelRatio],+ });+ return {+ success: false, dispatched: false, noDispatch: true, visionCoordinateError: 'viewport_changed',+ error: `Not dispatched: the browser window was resized or zoomed after ${keyframe.id} was taken (viewport ${size(keyframe.cssWidth, keyframe.cssHeight)} CSS px then, ${size(probe.cssWidth, probe.cssHeight)} now), so points read from it no longer land where you saw them. A fresh full frame follows; read the point again there.`,+ };+ };+/*** Keyboard input right after a vision click: does the focused element sit* at the clicked point and take text? Then the click's own page effectsdiff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex aa1e56453..7ba817553 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -2362,6 +2362,127 @@return chain;}+ // ββ :hover for the synthetic pointer ββ+ // Synthetic events run a page's hover listeners but give nothing :hover,+ // so a menu or tooltip only CSS reveals (`.menu:hover .submenu`) would stay+ // hidden. While the vision pointer rests on an element, that element and+ // its ancestors (through shadow roots and out of same-origin frames) carry+ // an attribute, and a copy of each :hover rule this world can read, keyed+ // to the attribute (`.menu[data-webbrain-hover] .submenu`), is adopted by+ // the document or shadow root the rule is in. The pointer moving elsewhere+ // moves the attribute; a person's own pointer moving ends the emulation, as+ // the real :hover then follows them. Another origin's style sheets cannot+ // be read, so their :hover rules stay unmatched.+ const _VISION_HOVER_ATTRIBUTE = 'data-webbrain-hover';+ let _visionHoverChain = [];+ const _visionHoverSheets = new Map();+ let _visionHoverListening = false;++ // The rules of `rules` that :hover decides, as CSS text keyed to the+ // attribute, keeping the conditions around them (@media, @supports,+ // @layer, @container, an @import's media, nesting).+ function _visionHoverCopies(rules, depth = 0) {+ const copies = [];+ if (!rules || depth > 12) return copies;+ const hover = /:hover(?![\w-])/gi;+ for (const rule of Array.from(rules)) {+ try {+ const text = String(rule.cssText || '');+ if (typeof rule.selectorText === 'string') {+ const selector = rule.selectorText;+ if (new RegExp(hover.source, 'i').test(selector)) {+ const open = text.indexOf('{');+ if (open > 0) copies.push(`${selector.replace(hover, `[${_VISION_HOVER_ATTRIBUTE}]`)} ${text.slice(open)}`);+ } else if (rule.cssRules?.length) {+ const inner = _visionHoverCopies(rule.cssRules, depth + 1);+ if (inner.length) copies.push(`${selector} { ${inner.join(' ')} }`);+ }+ } else if (rule.styleSheet) {+ const inner = _visionHoverCopies(rule.styleSheet.cssRules, depth + 1);+ const media = String(rule.media?.mediaText || '');+ if (inner.length) copies.push(media && media !== 'all' ? `@media ${media} { ${inner.join(' ')} }` : inner.join(' '));+ } else if (rule.cssRules?.length) {+ const inner = _visionHoverCopies(rule.cssRules, depth + 1);+ const open = text.indexOf('{');+ if (inner.length && open > 0) copies.push(`${text.slice(0, open).trim()} { ${inner.join(' ')} }`);+ }+ } catch { /* a rule this world cannot read */ }+ }+ return copies;+ }++ // Adopt (or refresh) the copy of `root`'s :hover rules.+ function _visionHoverAdopt(root) {+ const view = (root.ownerDocument || root).defaultView;+ if (!view || typeof view.CSSStyleSheet !== 'function' || !root.adoptedStyleSheets) return;+ const ours = _visionHoverSheets.get(root);+ const copies = [];+ for (const sheet of [...Array.from(root.styleSheets || []), ...Array.from(root.adoptedStyleSheets || [])]) {+ if (sheet === ours) continue;+ let rules = null;+ try { rules = sheet.cssRules; } catch { rules = null; }+ copies.push(..._visionHoverCopies(rules));+ }+ if (!copies.length) { _visionHoverDrop(root); return; }+ const sheet = ours || new view.CSSStyleSheet();+ try { sheet.replaceSync(copies.join('\n')); } catch { return; }+ if (!Array.from(root.adoptedStyleSheets).includes(sheet)) root.adoptedStyleSheets = [...Array.from(root.adoptedStyleSheets), sheet];+ _visionHoverSheets.set(root, sheet);+ }++ function _visionHoverDrop(root) {+ const sheet = _visionHoverSheets.get(root);+ _visionHoverSheets.delete(root);+ try {+ if (sheet && root.adoptedStyleSheets) root.adoptedStyleSheets = Array.from(root.adoptedStyleSheets).filter(item => item !== sheet);+ } catch { /* the page replaced its sheets */ }+ }++ // Rest the emulated :hover on `target` (null ends it). `adopted` collects+ // the roots already refreshed by this input, so a drag's moves read the+ // style sheets once.+ function _visionHoverAt(target, adopted = new Set()) {+ const next = [];+ for (let node = target, depth = 0; node?.nodeType === 1 && depth < 512; depth++) {+ next.push(node);+ let frame = null;+ if (!node.parentElement && !node.getRootNode?.()?.host) {+ try { frame = node.ownerDocument?.defaultView?.frameElement || null; } catch { frame = null; }+ }+ node = node.parentElement || node.getRootNode?.()?.host || frame;+ }+ // The rules that can see these elements hovered: their documents' and+ // shadow roots', and the :host rules of the shadow roots they host.+ const roots = new Set();+ for (const element of next) {+ for (const root of [element.getRootNode?.(), element.shadowRoot || element.openOrClosedShadowRoot]) {+ if (root?.nodeType === 9 || root?.nodeType === 11) roots.add(root);+ }+ }+ for (const root of [..._visionHoverSheets.keys()]) if (!roots.has(root)) _visionHoverDrop(root);+ for (const root of roots) {+ if (adopted.has(root)) continue;+ adopted.add(root);+ try { _visionHoverAdopt(root); } catch { /* left without the copy */ }+ }+ // Elements are marked only where some :hover rule could be read.+ const marked = _visionHoverSheets.size ? next : [];+ const kept = new Set(marked);+ for (const element of _visionHoverChain) {+ if (!kept.has(element)) { try { element.removeAttribute(_VISION_HOVER_ATTRIBUTE); } catch { /* gone */ } }+ }+ for (const element of marked) {+ try { if (!element.hasAttribute(_VISION_HOVER_ATTRIBUTE)) element.setAttribute(_VISION_HOVER_ATTRIBUTE, ''); } catch { /* not settable */ }+ }+ _visionHoverChain = marked;+ if (marked.length && !_visionHoverListening) {+ _visionHoverListening = true;+ window.addEventListener('pointermove', event => {+ if (event.isTrusted && (_visionHoverChain.length || _visionHoverSheets.size)) _visionHoverAt(null);+ }, true);+ }+ }+// The pointerId of the vision pointer's events (a mouse's in Chrome).const _VISION_POINTER_ID = 1;@@ -2463,6 +2584,15 @@error: 'Not dispatched: the point is inside an embedded frame drawn rotated, skewed, flipped or in 3D, so it cannot be placed on the frame\'s content. Use a control outside that frame.',};}+ // A frame this world cannot enter (another site's, a plugin, or one+ // still loading): a click or drag sent here would stop at its element+ // and reach nothing inside it.+ if ((type === 'click' || type === 'drag') && /^(IFRAME|FRAME|OBJECT|EMBED)$/i.test(String(hit.el.tagName || ''))) {+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'opaque_frame',+ error: `Not dispatched: the point is inside an embedded frame whose content this browser's vision input cannot reach (another site's frame, a plugin, or a frame still loading): the ${type} would stop at the frame and do nothing inside it. Use a control outside that frame, or take a screenshot and try again if it was still loading.`,+ };+ }if (actionDeadlineExpired()) return { success: false, dispatched: false, noDispatch: true, deadlineExpired: true, error: 'The action deadline expired before dispatch.' };const { el, view, cx, cy } = hit;const button = params.button === 'right' ? 2 : params.button === 'middle' ? 1 : 0;@@ -2481,7 +2611,12 @@// what it entered (shared ancestors get neither), pointer events before// mouse events, then the move. Arriving with no known previous target,// or again on the same one (`arrive`), over and enter go to the target.+ // Style sheets whose :hover copy this input refreshed (_visionHoverAt).+ const hoverAdopted = new Set();const moveOnto = (target, extra = {}, { arrive = false } = {}) => {+ // :hover follows the pointer before its events go out, as it does for+ // a mouse (_visionHoverAt).+ _visionHoverAt(target, hoverAdopted);const last = _visionPointerOver;if (last !== target || arrive) {const previous = last && last !== target && last.isConnected ? last : null;diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjsindex 8d031c6d6..4c94118dd 100644--- a/test/vision-mode-agent.mjs+++ b/test/vision-mode-agent.mjs@@ -320,7 +320,20 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))return { content: 'done' };}, { maxSteps: 10 });const tab = nextTab++;- await runLoop(agent, tab, streaming);+ // The page monitor is prepared with each point tool's CSS point, so+ // passive churn before dispatch does not void it (rebaseVisionPreparation).+ const prepared = [];+ const sendMessage = api.tabs.sendMessage;+ api.tabs.sendMessage = async (tabId, message, ...rest) => {+ if (tabId === tab && message?.action === 'page_monitor_prepare') prepared.push(message.params);+ return sendMessage(tabId, message, ...rest);+ };+ try { await runLoop(agent, tab, streaming); } finally { api.tabs.sendMessage = sendMessage; }+ const preparedFor = tool => prepared.filter(params => params.tool === tool)+ .map(params => ({ visionPoint: params.visionPoint, visionWheel: params.visionWheel, x: params.x, y: params.y }));+ assert.deepEqual(preparedFor('hover_at'), [{ visionPoint: true, visionWheel: undefined, x: 360, y: 90 }], JSON.stringify(prepared));+ assert.deepEqual(preparedFor('drag_at'), [{ visionPoint: true, visionWheel: undefined, x: 0, y: 0 }], 'a drag binds its start point');+ assert.deepEqual(preparedFor('scroll_at'), [{ visionPoint: undefined, visionWheel: true, x: undefined, y: undefined }], 'a wheel binds no element');const wheel = platform.find(entry => entry.kind === 'wheel');assert.deepEqual({ x: wheel.x, y: wheel.y, deltaX: wheel.deltaX, deltaY: wheel.deltaY }, { x: 720, y: 450, deltaX: 0, deltaY: 300 });const hover = platform.find(entry => entry.kind === 'hover');@@ -782,6 +795,63 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))}});+ test(`${build}: a window resized while the model decides gets a fresh full frame before any point is used`, async () => {+ let tab;+ const ctx = runAgent(step => {+ if (step === 1) {+ // The window is resized while the model reads f1 and picks a point on it.+ ctx.scene.cssWidth = 1200;+ ctx.scene.cssHeight = 760;+ ctx.agent._queuePageFeedback(tab, { frameId: 0, revision: 1, kind: 'resize', source: 'unknown', target: 'viewport',+ viewport: { width: 1200, height: 760, visualWidth: 1200, visualHeight: 760, scale: 1, offsetX: 0, offsetY: 0 } });+ return { toolCalls: [call('click_at', { x: 300, y: 200 })] };+ }+ if (step === 2) return { toolCalls: [call('click_at', { x: 300, y: 200 })] };+ if (step === 3) return { toolCalls: [call('done', { summary: 'Clicked.', outcome: 'success' })] };+ return { content: 'done' };+ }, { maxSteps: 8 });+ tab = nextTab++;+ await runLoop(ctx.agent, tab, false);+ const clicks = ctx.dispatched.filter(entry => entry.name === 'click');+ assert.equal(clicks.length, 1, `only the click chosen on the resized screen went out: ${JSON.stringify(clicks.map(entry => entry.args))}`);+ // The second request shows the resized viewport as a new full frame, and+ // its point maps through that frame.+ const frames = visionMessages(ctx.requests[1].messages).map(textOf);+ const last = frames.at(-1) || '';+ assert.match(last, /full viewport/i, last.slice(0, 300));+ assert.match(last, /1200\s*[xΓ]\s*760/, last.slice(0, 400));+ const session = ctx.agent._visionSessions.get(tab);+ assert.equal(session.keyframe.cssWidth, 1200);+ const { x, y } = clicks[0].args;+ assert.ok(Math.abs(x - 300 * 1200 / session.keyframe.width) < 1 && Math.abs(y - 200 * 760 / session.keyframe.height) < 1, JSON.stringify({ x, y, frame: [session.keyframe.width, session.keyframe.height] }));+ });++ test(`${build}: a resize the page has not reported yet still keeps stale points from going out`, async () => {+ let tab;+ const ctx = runAgent(step => {+ if (step === 1) {+ // Resized as the batch starts: no page feedback reports it in time.+ ctx.scene.cssWidth = 1200;+ ctx.scene.cssHeight = 760;+ return { toolCalls: [call('click_at', { x: 300, y: 200 }), call('type_text', { text: 'typed on the old screen' })] };+ }+ if (step === 2) return { toolCalls: [call('click_at', { x: 300, y: 200 })] };+ if (step === 3) return { toolCalls: [call('done', { summary: 'Clicked.', outcome: 'success' })] };+ return { content: 'done' };+ }, { maxSteps: 8 });+ tab = nextTab++;+ await runLoop(ctx.agent, tab, false);+ // Neither the click planned on f1 nor the typing planned after it went out.+ assert.deepEqual(ctx.dispatched.map(entry => entry.name), ['click', 'done'], JSON.stringify(ctx.dispatched.map(entry => [entry.name, entry.args])));+ assert.ok(toolTexts(ctx.requests[1]).some(text => /viewport_changed/.test(text) && /resized or zoomed after f1/.test(text)), toolTexts(ctx.requests[1]).join('\n'));+ const last = visionMessages(ctx.requests[1].messages).map(textOf).at(-1) || '';+ assert.match(last, /full viewport/i, last.slice(0, 300));+ // The click chosen on the new frame maps through it.+ const session = ctx.agent._visionSessions.get(tab);+ const { x, y } = ctx.dispatched[0].args;+ assert.ok(Math.abs(x - 300 * 1200 / session.keyframe.width) < 1 && Math.abs(y - 200 * 760 / session.keyframe.height) < 1, JSON.stringify({ x, y }));+ });+test(`${build}: page churn before a decision costs no tree read, and no message or image while the screen stays as seen`, async () => {const treeReads = [];let tab;diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex 616ab619b..fde794fac 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -480,6 +480,8 @@ test('content-script vision input (Firefox path) moves the pointer off one elemeconst take = () => page.evaluate(() => window.log.splice(0).filter(entry => entry[1].startsWith('mouse')).map(entry => entry.slice(0, 3)));await run({ type: 'hover', x: 130, y: 120 });assert.deepEqual(await take(), [['item', 'mouseover', null], ['item', 'mouseenter', null]], 'the first hover arrives on the item');+ assert.equal(await page.evaluate(() => document.querySelectorAll('[data-webbrain-hover]').length + document.adoptedStyleSheets.length), 0,+ 'a page without :hover rules gets no emulated :hover');// Item -> its menu: out of the item, over the menu; no leave or enter on the menu.await run({ type: 'hover', x: 250, y: 180 });assert.deepEqual(await take(), [['item', 'mouseout', 'menu'], ['item', 'mouseleave', 'menu'], ['menu', 'mouseover', 'item']]);@@ -500,6 +502,69 @@ test('content-script vision input (Firefox path) moves the pointer off one eleme}});+test('content-script vision hover (Firefox path) reveals what CSS :hover reveals, until the pointer or a person moves on', async () => {+ const browser = await chromium.launch({ channel: 'chromium' });+ try {+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });+ // A CSS-only menu, a rule inside @media, a web component styled by+ // :host(:hover) in its shadow root, and a rule from another origin's+ // style sheet, which cannot be read.+ await page.setContent(`<!doctype html><style>body{margin:0}+ #menu{position:absolute;left:100px;top:100px;width:200px;height:40px;background:#ddd}+ #menu .submenu{display:none;position:absolute;left:0;top:40px;width:200px;height:120px;background:#bbb}+ #menu:hover .submenu{display:block}+ #wide{position:absolute;left:400px;top:100px;width:200px;height:40px;background:#ddd}+ @media (min-width: 600px) { #wide:hover { outline: 3px solid rgb(255, 0, 0); } }+ #away{position:absolute;left:100px;top:400px;width:200px;height:40px}</style>+ <div id="menu">Menu<div class="submenu"><a id="item" href="#item">Item</a></div></div>+ <div id="wide">Wide</div><x-tip style="position:absolute;left:700px;top:100px;width:200px;height:40px;display:block"></x-tip>+ <div id="away">Away</div>+ <script>customElements.define('x-tip', class extends HTMLElement {+ constructor() { super(); this.attachShadow({ mode: 'open' }).innerHTML = '<style>.tip{display:none}:host(:hover) .tip{display:block}</style><span>Tip</span><span class="tip">Shown</span>'; }+ });</script>`);+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));+ await page.evaluate(({ code, guardBlock }) => {+ // eslint-disable-next-line no-new-func+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }+ ${guardBlock}\n${code}\nreturn visionInput;`)();+ }, { code: block, guardBlock });+ const run = params => page.evaluate(p => window.__visionInput(p), params);+ const state = () => page.evaluate(() => ({+ submenu: getComputedStyle(document.querySelector('#menu .submenu')).display,+ outline: getComputedStyle(document.getElementById('wide')).outlineColor,+ tip: getComputedStyle(document.querySelector('x-tip').shadowRoot.querySelector('.tip')).display,+ marked: document.querySelectorAll('[data-webbrain-hover]').length,+ adopted: document.adoptedStyleSheets.length,+ }));+ assert.deepEqual(await state(), { submenu: 'none', outline: 'rgb(0, 0, 0)', tip: 'none', marked: 0, adopted: 0 });+ // Hovering the menu opens it; the pointer can then go down into it.+ let result = await run({ type: 'hover', x: 150, y: 120 });+ assert.equal(result.success, true, JSON.stringify(result));+ assert.equal((await state()).submenu, 'block');+ await run({ type: 'hover', x: 150, y: 170 });+ assert.equal((await state()).submenu, 'block', 'still open over its own submenu');+ // The rule inside @media applies; the menu closes behind the pointer.+ await run({ type: 'hover', x: 450, y: 120 });+ assert.deepEqual({ ...(await state()), marked: undefined }, { submenu: 'none', outline: 'rgb(255, 0, 0)', tip: 'none', marked: undefined, adopted: 1 });+ // A shadow root's :host(:hover) rule applies inside it.+ await run({ type: 'hover', x: 750, y: 120 });+ assert.equal((await state()).tip, 'block');+ // A click leaves the pointer, and :hover, where it clicked.+ await run({ type: 'click', x: 150, y: 120 });+ assert.equal((await state()).submenu, 'block');+ // A person's own pointer moving ends the emulation, leaving nothing behind.+ await page.mouse.move(150, 420);+ await page.waitForFunction(() => !document.querySelector('[data-webbrain-hover]'));+ assert.deepEqual(await state(), { submenu: 'none', outline: 'rgb(0, 0, 0)', tip: 'none', marked: 0, adopted: 0 });+ } finally {+ await browser.close();+ }+});+test('content-script vision input (Firefox path) honours the pointer capture a page takes on the press', async () => {const browser = await chromium.launch({ channel: 'chromium' });try {@@ -864,7 +929,7 @@ test('the DOM probe behind the dispatch checks reads controls, text fields, focu}});-test('vision input (Firefox path) and the DOM probe place points inside scaled same-origin frames', async () => {+test('vision input (Firefox path) and the DOM probe place points inside scaled same-origin frames, and refuse frames they cannot enter', async () => {const browser = await chromium.launch({ channel: 'chromium' });try {const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });@@ -880,8 +945,9 @@ test('vision input (Firefox path) and the DOM probe place points inside scaled sawait page.setContent(`<!doctype html><style>body{margin:0}iframe{position:absolute;width:400px;height:300px;border:3px solid #000}</style>${frame('scaled', 'left:100px;top:50px;transform:scale(0.5);transform-origin:0 0')}${frame('tilted', 'left:600px;top:50px;transform:rotate(10deg)')}+ <iframe id="cross" style="left:600px;top:420px;width:200px;height:100px" src="data:text/html,<button>Elsewhere</button>"></iframe><div style="position:absolute;left:100px;top:400px;transform:scale(1.5);transform-origin:0 0">${frame('nested', 'position:static;width:200px;height:100px;border:1px solid #000', inner(20, 10))}</div>`);- await page.waitForFunction(() => [...document.querySelectorAll('iframe')].every(item => item.contentWindow?.log));+ await page.waitForFunction(() => ['scaled', 'tilted', 'nested'].every(id => document.getElementById(id).contentWindow?.log));const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));@@ -918,6 +984,10 @@ test('vision input (Firefox path) and the DOM probe place points inside scaled sassert.equal(result.visionInputRefused, 'frame_unmappable', JSON.stringify(result));assert.equal(result.noDispatch, true);assert.deepEqual(await take('tilted'), []);+ // A frame from another origin is never entered: a click there would stop at the frame element.+ result = await run({ type: 'click', x: 700, y: 470 });+ assert.equal(result.visionInputRefused, 'opaque_frame', JSON.stringify(result));+ assert.equal(result.noDispatch, true);// The DOM probe behind the dispatch checks reads the same elements, in both builds.for (const build of ['chrome', 'firefox']) {const { visionDomProbeExpression } = await import(`../src/${build}/src/agent/vision-dom-probe.js`);