Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode (Firefox): emulate :hover in place, keeping each rule's cascade
Emre Sokulluf2f73a4e9d2e
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex 0b2bd9c9e..40790d5bd 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -119,7 +119,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. A guard that binds a consequential click to the control and payload it checked (a deletion, a commit after a correction) refuses coordinate clicks on the tree path, where a ref or selector click exists; in a vision run, where the point is all there is, it binds the click to the control the submit probe read at that point (`_resourceMutationActionKey`), with the same record, stage, sibling and confirmation checks and the same card, and `_visionDispatchClick` reads the point again right before input goes out (`_visionMutationBindingBlock`): when the authorized control is no longer there, nothing is sent (`resource_changed_before_dispatch`) and the model chooses again on the updated screen. A guard's advice names what the run offers (`click_at`), never a ref or selector. Only the final dispatch differs:- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively.-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute and each `:hover` rule the content script can read is copied, keyed to that attribute, into a style sheet the document or shadow root adopts, which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute, and every `:hover` in the selectors of the style rules the content script can read also accepts it, in place (`:hover` reads `:is(:hover, [data-webbrain-hover])`, keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (`recipient_bound_click`) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.@@ -170,7 +170,7 @@ npm run test:vision-mode- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard, CSS `:hover` revealed under the pointer (a CSS-only menu, a rule inside `@media`, a shadow root's `:host(:hover)`) and ended by a person's own pointer, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame and another origin's frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard, CSS `:hover` revealed under the pointer (a CSS-only menu, a selector listed beside a `:hover` keeping its own cascade, a later rule of equal specificity still winning, a rule inside `@media`, a shadow root's `:host(:hover)`, a rule added later) and ended by a person's own pointer with the page's selectors put back, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame and another origin's frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.@@ -182,4 +182,4 @@ The older MV3 quality fixtures script an accessibility-tree model (they answer w- HTML5 drag and drop: in Firefox a drag that starts on a `draggable="true"` item runs the drag-and-drop event sequence itself with a constructed `DataTransfer`; in Chrome the CDP pointer drag is left to the browser, as for `drag_drop`, so a page that relies only on `dataTransfer` may not react there.- A drag is not shown to the submit, money or publishing checks; it is refused when it would click a control, but a custom "slide to confirm" widget is only as safe as the page makes it.- The 45-second window that holds back a second click on the same committing caption covers text clicks (`click({text})`) only. Like a ref click on the tree path, a repeated vision click relies on the submit confirmation and the deletion and publication guards instead.-- Firefox input is synthetic (`isTrusted: false`); pages that require trusted events need Chrome. Its emulated `:hover` covers the style sheets the content script can read: a `:hover` rule in another origin's style sheet does not apply, and page scripts that test `:hover` themselves see no hover.+- Firefox input is synthetic (`isTrusted: false`); pages that require trusted events need Chrome. Its emulated `:hover` covers the style sheets the content script can read: a `:hover` rule in another origin's style sheet does not apply, page scripts that test `:hover` themselves see no hover, and while it lasts a script reading a rule's `selectorText` sees the added `:is()`.diff --git a/src/chrome/src/chrome-protected-pages.js b/src/chrome/src/chrome-protected-pages.jsindex 78e17120b..02d298af6 100644--- a/src/chrome/src/chrome-protected-pages.js+++ b/src/chrome/src/chrome-protected-pages.js@@ -12,6 +12,9 @@ const PROTECTED_DOM_TOOLS = new Set(['press_keys', 'scroll', 'extract_data', 'inspect_element_styles','patch_element', 'revert_patch', 'highlight_element', 'hover', 'drag_drop','wait_for_element', 'wait_for_stable', 'get_selection', 'find_text',+ // Vision mode's debugger input and region capture (click_at runs as click;+ // take_screenshot only asks for the frame the next update captures).+ 'hover_at', 'drag_at', 'scroll_at', 'zoom',]);export function isChromeProtectedPageDomTool(toolName) {diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex 447d71c0b..3166a6895 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -2073,95 +2073,69 @@// so a menu or tooltip only CSS reveals (`.menu:hover .submenu`) would stay// hidden. While the vision pointer rests on an element, that element and// its ancestors (through shadow roots and out of same-origin frames) carry- // an attribute, and a copy of each :hover rule this world can read, keyed- // to the attribute (`.menu[data-webbrain-hover] .submenu`), is adopted by- // the document or shadow root the rule is in. The pointer moving elsewhere- // moves the attribute; a person's own pointer moving ends the emulation, as- // the real :hover then follows them. Another origin's style sheets cannot- // be read, so their :hover rules stay unmatched.+ // an attribute, and every :hover in the selectors of the style rules this+ // world can read also accepts that attribute, in place:+ // `.menu:hover .submenu, .save` reads+ // `.menu:is(:hover, [data-webbrain-hover]) .submenu, .save`. Each rule keeps+ // its place in the cascade, its specificity (:is takes its argument's),+ // its conditions (@media, @supports, @layer, @container, nesting) and the+ // other selectors of its list; `:not(:hover)` stays its negation. The+ // pointer moving on moves the attribute; a person's own pointer moving+ // ends the emulation (the real :hover then follows them) and puts the+ // page's selectors back. Another origin's style sheets cannot be read, so+ // their :hover rules stay unmatched.const _VISION_HOVER_ATTRIBUTE = 'data-webbrain-hover';+ const _VISION_HOVER_MATCH = `:is(:hover, [${_VISION_HOVER_ATTRIBUTE}])`;let _visionHoverChain = [];- const _visionHoverSheets = new Map();+ // rule -> { original, rewritten } selector text, to put back.+ const _visionHoverRewritten = new Map();+ // A sheet is read again only when its rule count changed since.+ let _visionHoverScanned = new WeakMap();let _visionHoverListening = false;- // The rules of `rules` that :hover decides, as CSS text keyed to the- // attribute, keeping the conditions around them (@media, @supports,- // @layer, @container, an @import's media, nesting).- function _visionHoverCopies(rules, depth = 0) {- const copies = [];- if (!rules || depth > 12) return copies;- const hover = /:hover(?![\w-])/gi;+ function _visionHoverRewrite(rules, depth = 0) {+ if (!rules || depth > 12) return;for (const rule of Array.from(rules)) {try {- const text = String(rule.cssText || '');- if (typeof rule.selectorText === 'string') {- const selector = rule.selectorText;- if (new RegExp(hover.source, 'i').test(selector)) {- const open = text.indexOf('{');- if (open > 0) copies.push(`${selector.replace(hover, `[${_VISION_HOVER_ATTRIBUTE}]`)} ${text.slice(open)}`);- } else if (rule.cssRules?.length) {- const inner = _visionHoverCopies(rule.cssRules, depth + 1);- if (inner.length) copies.push(`${selector} { ${inner.join(' ')} }`);- }- } else if (rule.styleSheet) {- const inner = _visionHoverCopies(rule.styleSheet.cssRules, depth + 1);- const media = String(rule.media?.mediaText || '');- if (inner.length) copies.push(media && media !== 'all' ? `@media ${media} { ${inner.join(' ')} }` : inner.join(' '));+ const original = typeof rule.selectorText === 'string' ? rule.selectorText : '';+ if (original && !_visionHoverRewritten.has(rule) && /:hover(?![\w-])/i.test(original) && !original.includes(_VISION_HOVER_ATTRIBUTE)) {+ // A selector the result would not parse in is left as it was.+ rule.selectorText = original.replace(/:hover(?![\w-])/gi, _VISION_HOVER_MATCH);+ const rewritten = rule.selectorText;+ if (rewritten !== original) _visionHoverRewritten.set(rule, { original, rewritten });+ }+ if (rule.styleSheet) {+ let imported = null;+ try { imported = rule.styleSheet.cssRules; } catch { imported = null; }+ _visionHoverRewrite(imported, depth + 1);} else if (rule.cssRules?.length) {- const inner = _visionHoverCopies(rule.cssRules, depth + 1);- const open = text.indexOf('{');- if (inner.length && open > 0) copies.push(`${text.slice(0, open).trim()} { ${inner.join(' ')} }`);+ _visionHoverRewrite(rule.cssRules, depth + 1);}- } catch { /* a rule this world cannot read */ }+ } catch { /* a rule this world cannot read or change */ }}- return copies;}- // Adopt (or refresh) the copy of `root`'s :hover rules.- // A sheet's copies are read again only when its rule count changes (a- // page adding or removing rules), so a large site is read once.- const _visionHoverSheetCopies = new WeakMap();- const _visionHoverSheetTexts = new WeakMap();-- function _visionHoverAdopt(root) {- const view = (root.ownerDocument || root).defaultView;- if (!view || typeof view.CSSStyleSheet !== 'function' || !root.adoptedStyleSheets) return;- const ours = _visionHoverSheets.get(root);- const copies = [];+ function _visionHoverPrepare(root) {for (const sheet of [...Array.from(root.styleSheets || []), ...Array.from(root.adoptedStyleSheets || [])]) {- if (sheet === ours) continue;let rules = null;try { rules = sheet.cssRules; } catch { rules = null; }- if (!rules) continue;- const cached = _visionHoverSheetCopies.get(sheet);- if (cached && cached.length === rules.length) { copies.push(...cached.copies); continue; }- const own = _visionHoverCopies(rules);- _visionHoverSheetCopies.set(sheet, { length: rules.length, copies: own });- copies.push(...own);- }- if (!copies.length) { _visionHoverDrop(root); return; }- const sheet = ours || new view.CSSStyleSheet();- const text = copies.join('\n');- if (_visionHoverSheetTexts.get(sheet) !== text) {- try { sheet.replaceSync(text); } catch { return; }- _visionHoverSheetTexts.set(sheet, text);- }- if (!Array.from(root.adoptedStyleSheets).includes(sheet)) root.adoptedStyleSheets = [...Array.from(root.adoptedStyleSheets), sheet];- _visionHoverSheets.set(root, sheet);- }-- function _visionHoverDrop(root) {- const sheet = _visionHoverSheets.get(root);- _visionHoverSheets.delete(root);- try {- if (sheet && root.adoptedStyleSheets) root.adoptedStyleSheets = Array.from(root.adoptedStyleSheets).filter(item => item !== sheet);- } catch { /* the page replaced its sheets */ }+ if (!rules || _visionHoverScanned.get(sheet) === rules.length) continue;+ _visionHoverRewrite(rules);+ _visionHoverScanned.set(sheet, rules.length);+ }}- // Rest the emulated :hover on `target` (null ends it). `adopted` collects- // the roots already refreshed by this input, so a drag's moves read the- // style sheets once.- function _visionHoverAt(target, adopted = new Set()) {+ function _visionHoverRestore() {+ for (const [rule, { original, rewritten }] of _visionHoverRewritten) {+ try { if (rule.selectorText === rewritten) rule.selectorText = original; } catch { /* gone */ }+ }+ _visionHoverRewritten.clear();+ _visionHoverScanned = new WeakMap();+ }++ // Rest the emulated :hover on `target` (null ends it). `prepared` collects+ // the roots this input already read, so a drag's moves read them once.+ function _visionHoverAt(target, prepared = new Set()) {const next = [];for (let node = target, depth = 0; node?.nodeType === 1 && depth < 512; depth++) {next.push(node);@@ -2173,20 +2147,16 @@}// The rules that can see these elements hovered: their documents' and// shadow roots', and the :host rules of the shadow roots they host.- const roots = new Set();for (const element of next) {for (const root of [element.getRootNode?.(), element.shadowRoot || element.openOrClosedShadowRoot]) {- if (root?.nodeType === 9 || root?.nodeType === 11) roots.add(root);+ if ((root?.nodeType === 9 || root?.nodeType === 11) && !prepared.has(root)) {+ prepared.add(root);+ try { _visionHoverPrepare(root); } catch { /* left as the page wrote it */ }+ }}}- for (const root of [..._visionHoverSheets.keys()]) if (!roots.has(root)) _visionHoverDrop(root);- for (const root of roots) {- if (adopted.has(root)) continue;- adopted.add(root);- try { _visionHoverAdopt(root); } catch { /* left without the copy */ }- }// Elements are marked only where some :hover rule could be read.- const marked = _visionHoverSheets.size ? next : [];+ const marked = _visionHoverRewritten.size ? next : [];const kept = new Set(marked);for (const element of _visionHoverChain) {if (!kept.has(element)) { try { element.removeAttribute(_VISION_HOVER_ATTRIBUTE); } catch { /* gone */ } }@@ -2195,10 +2165,11 @@try { if (!element.hasAttribute(_VISION_HOVER_ATTRIBUTE)) element.setAttribute(_VISION_HOVER_ATTRIBUTE, ''); } catch { /* not settable */ }}_visionHoverChain = marked;+ if (!marked.length) _visionHoverRestore();if (marked.length && !_visionHoverListening) {_visionHoverListening = true;window.addEventListener('pointermove', event => {- if (event.isTrusted && (_visionHoverChain.length || _visionHoverSheets.size)) _visionHoverAt(null);+ if (event.isTrusted && (_visionHoverChain.length || _visionHoverRewritten.size)) _visionHoverAt(null);}, true);}}@@ -2331,12 +2302,12 @@// what it entered (shared ancestors get neither), pointer events before// mouse events, then the move. Arriving with no known previous target,// or again on the same one (`arrive`), over and enter go to the target.- // Style sheets whose :hover copy this input refreshed (_visionHoverAt).- const hoverAdopted = new Set();+ // Documents and shadow roots whose :hover rules this input read (_visionHoverAt).+ const hoverPrepared = new Set();const moveOnto = (target, extra = {}, { arrive = false } = {}) => {// :hover follows the pointer before its events go out, as it does for// a mouse (_visionHoverAt).- _visionHoverAt(target, hoverAdopted);+ _visionHoverAt(target, hoverPrepared);const last = _visionPointerOver;if (last !== target || arrive) {const previous = last && last !== target && last.isConnected ? last : null;diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex c62700e8d..a98dc3db5 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -2367,95 +2367,69 @@// so a menu or tooltip only CSS reveals (`.menu:hover .submenu`) would stay// hidden. While the vision pointer rests on an element, that element and// its ancestors (through shadow roots and out of same-origin frames) carry- // an attribute, and a copy of each :hover rule this world can read, keyed- // to the attribute (`.menu[data-webbrain-hover] .submenu`), is adopted by- // the document or shadow root the rule is in. The pointer moving elsewhere- // moves the attribute; a person's own pointer moving ends the emulation, as- // the real :hover then follows them. Another origin's style sheets cannot- // be read, so their :hover rules stay unmatched.+ // an attribute, and every :hover in the selectors of the style rules this+ // world can read also accepts that attribute, in place:+ // `.menu:hover .submenu, .save` reads+ // `.menu:is(:hover, [data-webbrain-hover]) .submenu, .save`. Each rule keeps+ // its place in the cascade, its specificity (:is takes its argument's),+ // its conditions (@media, @supports, @layer, @container, nesting) and the+ // other selectors of its list; `:not(:hover)` stays its negation. The+ // pointer moving on moves the attribute; a person's own pointer moving+ // ends the emulation (the real :hover then follows them) and puts the+ // page's selectors back. Another origin's style sheets cannot be read, so+ // their :hover rules stay unmatched.const _VISION_HOVER_ATTRIBUTE = 'data-webbrain-hover';+ const _VISION_HOVER_MATCH = `:is(:hover, [${_VISION_HOVER_ATTRIBUTE}])`;let _visionHoverChain = [];- const _visionHoverSheets = new Map();+ // rule -> { original, rewritten } selector text, to put back.+ const _visionHoverRewritten = new Map();+ // A sheet is read again only when its rule count changed since.+ let _visionHoverScanned = new WeakMap();let _visionHoverListening = false;- // The rules of `rules` that :hover decides, as CSS text keyed to the- // attribute, keeping the conditions around them (@media, @supports,- // @layer, @container, an @import's media, nesting).- function _visionHoverCopies(rules, depth = 0) {- const copies = [];- if (!rules || depth > 12) return copies;- const hover = /:hover(?![\w-])/gi;+ function _visionHoverRewrite(rules, depth = 0) {+ if (!rules || depth > 12) return;for (const rule of Array.from(rules)) {try {- const text = String(rule.cssText || '');- if (typeof rule.selectorText === 'string') {- const selector = rule.selectorText;- if (new RegExp(hover.source, 'i').test(selector)) {- const open = text.indexOf('{');- if (open > 0) copies.push(`${selector.replace(hover, `[${_VISION_HOVER_ATTRIBUTE}]`)} ${text.slice(open)}`);- } else if (rule.cssRules?.length) {- const inner = _visionHoverCopies(rule.cssRules, depth + 1);- if (inner.length) copies.push(`${selector} { ${inner.join(' ')} }`);- }- } else if (rule.styleSheet) {- const inner = _visionHoverCopies(rule.styleSheet.cssRules, depth + 1);- const media = String(rule.media?.mediaText || '');- if (inner.length) copies.push(media && media !== 'all' ? `@media ${media} { ${inner.join(' ')} }` : inner.join(' '));+ const original = typeof rule.selectorText === 'string' ? rule.selectorText : '';+ if (original && !_visionHoverRewritten.has(rule) && /:hover(?![\w-])/i.test(original) && !original.includes(_VISION_HOVER_ATTRIBUTE)) {+ // A selector the result would not parse in is left as it was.+ rule.selectorText = original.replace(/:hover(?![\w-])/gi, _VISION_HOVER_MATCH);+ const rewritten = rule.selectorText;+ if (rewritten !== original) _visionHoverRewritten.set(rule, { original, rewritten });+ }+ if (rule.styleSheet) {+ let imported = null;+ try { imported = rule.styleSheet.cssRules; } catch { imported = null; }+ _visionHoverRewrite(imported, depth + 1);} else if (rule.cssRules?.length) {- const inner = _visionHoverCopies(rule.cssRules, depth + 1);- const open = text.indexOf('{');- if (inner.length && open > 0) copies.push(`${text.slice(0, open).trim()} { ${inner.join(' ')} }`);+ _visionHoverRewrite(rule.cssRules, depth + 1);}- } catch { /* a rule this world cannot read */ }+ } catch { /* a rule this world cannot read or change */ }}- return copies;}- // Adopt (or refresh) the copy of `root`'s :hover rules.- // A sheet's copies are read again only when its rule count changes (a- // page adding or removing rules), so a large site is read once.- const _visionHoverSheetCopies = new WeakMap();- const _visionHoverSheetTexts = new WeakMap();-- function _visionHoverAdopt(root) {- const view = (root.ownerDocument || root).defaultView;- if (!view || typeof view.CSSStyleSheet !== 'function' || !root.adoptedStyleSheets) return;- const ours = _visionHoverSheets.get(root);- const copies = [];+ function _visionHoverPrepare(root) {for (const sheet of [...Array.from(root.styleSheets || []), ...Array.from(root.adoptedStyleSheets || [])]) {- if (sheet === ours) continue;let rules = null;try { rules = sheet.cssRules; } catch { rules = null; }- if (!rules) continue;- const cached = _visionHoverSheetCopies.get(sheet);- if (cached && cached.length === rules.length) { copies.push(...cached.copies); continue; }- const own = _visionHoverCopies(rules);- _visionHoverSheetCopies.set(sheet, { length: rules.length, copies: own });- copies.push(...own);- }- if (!copies.length) { _visionHoverDrop(root); return; }- const sheet = ours || new view.CSSStyleSheet();- const text = copies.join('\n');- if (_visionHoverSheetTexts.get(sheet) !== text) {- try { sheet.replaceSync(text); } catch { return; }- _visionHoverSheetTexts.set(sheet, text);- }- if (!Array.from(root.adoptedStyleSheets).includes(sheet)) root.adoptedStyleSheets = [...Array.from(root.adoptedStyleSheets), sheet];- _visionHoverSheets.set(root, sheet);- }-- function _visionHoverDrop(root) {- const sheet = _visionHoverSheets.get(root);- _visionHoverSheets.delete(root);- try {- if (sheet && root.adoptedStyleSheets) root.adoptedStyleSheets = Array.from(root.adoptedStyleSheets).filter(item => item !== sheet);- } catch { /* the page replaced its sheets */ }+ if (!rules || _visionHoverScanned.get(sheet) === rules.length) continue;+ _visionHoverRewrite(rules);+ _visionHoverScanned.set(sheet, rules.length);+ }}- // Rest the emulated :hover on `target` (null ends it). `adopted` collects- // the roots already refreshed by this input, so a drag's moves read the- // style sheets once.- function _visionHoverAt(target, adopted = new Set()) {+ function _visionHoverRestore() {+ for (const [rule, { original, rewritten }] of _visionHoverRewritten) {+ try { if (rule.selectorText === rewritten) rule.selectorText = original; } catch { /* gone */ }+ }+ _visionHoverRewritten.clear();+ _visionHoverScanned = new WeakMap();+ }++ // Rest the emulated :hover on `target` (null ends it). `prepared` collects+ // the roots this input already read, so a drag's moves read them once.+ function _visionHoverAt(target, prepared = new Set()) {const next = [];for (let node = target, depth = 0; node?.nodeType === 1 && depth < 512; depth++) {next.push(node);@@ -2467,20 +2441,16 @@}// The rules that can see these elements hovered: their documents' and// shadow roots', and the :host rules of the shadow roots they host.- const roots = new Set();for (const element of next) {for (const root of [element.getRootNode?.(), element.shadowRoot || element.openOrClosedShadowRoot]) {- if (root?.nodeType === 9 || root?.nodeType === 11) roots.add(root);+ if ((root?.nodeType === 9 || root?.nodeType === 11) && !prepared.has(root)) {+ prepared.add(root);+ try { _visionHoverPrepare(root); } catch { /* left as the page wrote it */ }+ }}}- for (const root of [..._visionHoverSheets.keys()]) if (!roots.has(root)) _visionHoverDrop(root);- for (const root of roots) {- if (adopted.has(root)) continue;- adopted.add(root);- try { _visionHoverAdopt(root); } catch { /* left without the copy */ }- }// Elements are marked only where some :hover rule could be read.- const marked = _visionHoverSheets.size ? next : [];+ const marked = _visionHoverRewritten.size ? next : [];const kept = new Set(marked);for (const element of _visionHoverChain) {if (!kept.has(element)) { try { element.removeAttribute(_VISION_HOVER_ATTRIBUTE); } catch { /* gone */ } }@@ -2489,10 +2459,11 @@try { if (!element.hasAttribute(_VISION_HOVER_ATTRIBUTE)) element.setAttribute(_VISION_HOVER_ATTRIBUTE, ''); } catch { /* not settable */ }}_visionHoverChain = marked;+ if (!marked.length) _visionHoverRestore();if (marked.length && !_visionHoverListening) {_visionHoverListening = true;window.addEventListener('pointermove', event => {- if (event.isTrusted && (_visionHoverChain.length || _visionHoverSheets.size)) _visionHoverAt(null);+ if (event.isTrusted && (_visionHoverChain.length || _visionHoverRewritten.size)) _visionHoverAt(null);}, true);}}@@ -2625,12 +2596,12 @@// what it entered (shared ancestors get neither), pointer events before// mouse events, then the move. Arriving with no known previous target,// or again on the same one (`arrive`), over and enter go to the target.- // Style sheets whose :hover copy this input refreshed (_visionHoverAt).- const hoverAdopted = new Set();+ // Documents and shadow roots whose :hover rules this input read (_visionHoverAt).+ const hoverPrepared = new Set();const moveOnto = (target, extra = {}, { arrive = false } = {}) => {// :hover follows the pointer before its events go out, as it does for// a mouse (_visionHoverAt).- _visionHoverAt(target, hoverAdopted);+ _visionHoverAt(target, hoverPrepared);const last = _visionPointerOver;if (last !== target || arrive) {const previous = last && last !== target && last.isConnected ? last : null;diff --git a/test/vision-mode-guards.mjs b/test/vision-mode-guards.mjsindex 56b31d16b..89e298586 100644--- a/test/vision-mode-guards.mjs+++ b/test/vision-mode-guards.mjs@@ -7,6 +7,7 @@ import assert from 'node:assert/strict';import { test } from 'node:test';import { defaultDecide, installModelStub } from './semantic-decision-stub.mjs';import { cdpClient } from '../src/chrome/src/cdp/cdp-client.js';+import { isChromeProtectedPageDomTool, chromeProtectedPageFailure } from '../src/chrome/src/chrome-protected-pages.js';const pageUrl = 'https://social.example/@me';const documentIdentity = `${pageUrl}|1700000000000.5`;@@ -473,3 +474,12 @@ test('chrome: a vision click result is not judged by a DOM text snapshot', asyncif (savedTabs) globalThis.chrome.tabs = savedTabs;}});++test('chrome: on a protected Chrome Web Store page vision input and zooms stop with the protected-page result', () => {+ for (const name of ['hover_at', 'drag_at', 'scroll_at', 'zoom', 'click']) assert.equal(isChromeProtectedPageDomTool(name), true, name);+ // take_screenshot sends nothing to the page, and wait reads nothing.+ for (const name of ['take_screenshot', 'wait']) assert.equal(isChromeProtectedPageDomTool(name), false, name);+ const failure = chromeProtectedPageFailure('https://chrome.google.com/webstore/devconsole/abc', 'drag_at');+ assert.equal(failure?.errorCode, 'chrome_protected_page');+ assert.equal(failure.noDispatch, true);+});diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex 2acb10e17..25045a230 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -506,17 +506,23 @@ test('content-script vision hover (Firefox path) reveals what CSS :hover revealsconst browser = await chromium.launch({ channel: 'chromium' });try {const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });- // A CSS-only menu, a rule inside @media, a web component styled by- // :host(:hover) in its shadow root, and a rule from another origin's- // style sheet, which cannot be read.+ // A CSS-only menu; a :hover selector listed with another (`, .save`) that+ // a later rule overrides; a later rule of equal specificity that wins+ // over :hover; a rule inside @media; a web component styled by+ // :host(:hover) in its shadow root.await page.setContent(`<!doctype html><style>body{margin:0}#menu{position:absolute;left:100px;top:100px;width:200px;height:40px;background:#ddd}#menu .submenu{display:none;position:absolute;left:0;top:40px;width:200px;height:120px;background:#bbb}- #menu:hover .submenu{display:block}+ #menu:hover .submenu, .save{display:block}+ .save{position:absolute;left:400px;top:300px;width:100px;height:30px;display:none}+ .btn{position:absolute;left:400px;top:400px;width:100px;height:30px;color:rgb(0, 0, 0)}+ .btn:hover{color:rgb(255, 0, 0)}+ .btn.on{color:rgb(0, 0, 255)}#wide{position:absolute;left:400px;top:100px;width:200px;height:40px;background:#ddd}@media (min-width: 600px) { #wide:hover { outline: 3px solid rgb(255, 0, 0); } }#away{position:absolute;left:100px;top:400px;width:200px;height:40px}</style><div id="menu">Menu<div class="submenu"><a id="item" href="#item">Item</a></div></div>+ <div class="save" id="save">Save</div><div class="btn on" id="btn">Button</div><div id="wide">Wide</div><x-tip style="position:absolute;left:700px;top:100px;width:200px;height:40px;display:block"></x-tip><div id="away">Away</div><script>customElements.define('x-tip', class extends HTMLElement {@@ -531,25 +537,33 @@ test('content-script vision hover (Firefox path) reveals what CSS :hover revealsfunction _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }${guardBlock}\n${code}\nreturn visionInput;`)();+ window.__selectors = () => [...document.styleSheets[0].cssRules].flatMap(rule => [rule.selectorText, ...[...(rule.cssRules || [])].map(inner => inner.selectorText)]).filter(Boolean);}, { code: block, guardBlock });const run = params => page.evaluate(p => window.__visionInput(p), params);const state = () => page.evaluate(() => ({submenu: getComputedStyle(document.querySelector('#menu .submenu')).display,+ save: getComputedStyle(document.getElementById('save')).display,outline: getComputedStyle(document.getElementById('wide')).outlineColor,tip: getComputedStyle(document.querySelector('x-tip').shadowRoot.querySelector('.tip')).display,marked: document.querySelectorAll('[data-webbrain-hover]').length,- adopted: document.adoptedStyleSheets.length,+ rewritten: window.__selectors().filter(selector => selector.includes('data-webbrain-hover')).length,}));- assert.deepEqual(await state(), { submenu: 'none', outline: 'rgb(0, 0, 0)', tip: 'none', marked: 0, adopted: 0 });- // Hovering the menu opens it; the pointer can then go down into it.+ const original = await page.evaluate(() => window.__selectors());+ assert.deepEqual(await state(), { submenu: 'none', save: 'none', outline: 'rgb(0, 0, 0)', tip: 'none', marked: 0, rewritten: 0 });+ // Hovering the menu opens it; the pointer can then go down into it. The+ // selector listed beside its :hover keeps its own cascade: Save stays hidden.let result = await run({ type: 'hover', x: 150, y: 120 });assert.equal(result.success, true, JSON.stringify(result));- assert.equal((await state()).submenu, 'block');+ assert.deepEqual({ ...(await state()), marked: undefined }, { submenu: 'block', save: 'none', outline: 'rgb(0, 0, 0)', tip: 'none', marked: undefined, rewritten: 3 });await run({ type: 'hover', x: 150, y: 170 });assert.equal((await state()).submenu, 'block', 'still open over its own submenu');- // The rule inside @media applies; the menu closes behind the pointer.+ // A later rule of equal specificity still wins over the hover, as it does for a mouse.+ await run({ type: 'hover', x: 450, y: 415 });+ assert.equal(await page.evaluate(() => getComputedStyle(document.getElementById('btn')).color), 'rgb(0, 0, 255)');+ assert.equal((await state()).submenu, 'none', 'the menu closed behind the pointer');+ // The rule inside @media applies.await run({ type: 'hover', x: 450, y: 120 });- assert.deepEqual({ ...(await state()), marked: undefined }, { submenu: 'none', outline: 'rgb(255, 0, 0)', tip: 'none', marked: undefined, adopted: 1 });+ assert.equal((await state()).outline, 'rgb(255, 0, 0)');// A shadow root's :host(:hover) rule applies inside it.await run({ type: 'hover', x: 750, y: 120 });assert.equal((await state()).tip, 'block');@@ -560,10 +574,12 @@ test('content-script vision hover (Firefox path) reveals what CSS :hover reveals// A click leaves the pointer, and :hover, where it clicked.await run({ type: 'click', x: 150, y: 120 });assert.equal((await state()).submenu, 'block');- // A person's own pointer moving ends the emulation, leaving nothing behind.- await page.mouse.move(150, 420);+ // A person's own pointer moving ends the emulation and puts the page's selectors back.+ await page.mouse.move(150, 600);await page.waitForFunction(() => !document.querySelector('[data-webbrain-hover]'));- assert.deepEqual(await state(), { submenu: 'none', outline: 'rgb(0, 0, 0)', tip: 'none', marked: 0, adopted: 0 });+ assert.deepEqual(await state(), { submenu: 'none', save: 'none', outline: 'rgb(0, 0, 0)', tip: 'none', marked: 0, rewritten: 0 });+ assert.deepEqual((await page.evaluate(() => window.__selectors())).slice(0, original.length), original);+ assert.equal(await page.evaluate(() => document.querySelector('x-tip').shadowRoot.styleSheets[0].cssRules[1].selectorText), ':host(:hover) .tip');} finally {await browser.close();}