Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Honour a pointerdown the page cancels in Firefox vision input

Emre Sokullu321e36726ee6
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index d24763c27..3352fa3e0 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -119,7 +119,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll
`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. A guard that binds a consequential click to the control and payload it checked (a deletion, a commit after a correction) refuses coordinate clicks on the tree path, where a ref or selector click exists; in a vision run, where the point is all there is, it binds the click to the control the submit probe read at that point (`_resourceMutationActionKey`), with the same record, stage, sibling and confirmation checks and the same card, and `_visionDispatchClick` reads the point again right before input goes out (`_visionMutationBindingBlock`): when the authorized control is no longer there, nothing is sent (`resource_changed_before_dispatch`) and the model chooses again on the updated screen. A guard's advice names what the run offers (`click_at`), never a ref or selector. Only the final dispatch differs:
- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively.
-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute, and every `:hover` in the selectors of the style rules the content script can read also accepts it, in place (`:hover` reads `:is(:hover, [data-webbrain-hover])`, keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release; a click's primary press on the track moves the thumb to its point the same way), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.
+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute, and every `:hover` in the selectors of the style rules the content script can read also accepts it, in place (`:hover` reads `:is(:hover, [data-webbrain-hover])`, keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A press whose `pointerdown` the page cancels gets what a browser gives it: no compatibility `mousedown`, `mousemove` or `mouseup` until its release (the boundary events still go out) and none of the press's defaults below (focus, the caret, a range thumb, a selection, drag and drop, a double click's word), while its `click` still fires. A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release; a click's primary press on the track moves the thumb to its point the same way), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.
In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (`recipient_bound_click`) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.
diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.js
index f0eb39ba0..0836d1707 100644
--- a/src/chrome/src/content/content.js
+++ b/src/chrome/src/content/content.js
@@ -2304,7 +2304,10 @@
// or again on the same one (`arrive`), over and enter go to the target.
// Documents and shadow roots whose :hover rules this input read (_visionHoverAt).
const hoverPrepared = new Set();
- const moveOnto = (target, extra = {}, { arrive = false } = {}) => {
+ // A press whose pointerdown the page cancelled gets no compatibility
+ // mousemove (`compatMouse` off) until its release; the boundary events
+ // (over, out, enter, leave) still go out, as the Pointer Events spec has it.
+ const moveOnto = (target, extra = {}, { arrive = false, compatMouse = true } = {}) => {
// :hover follows the pointer before its events go out, as it does for
// a mouse (_visionHoverAt).
_visionHoverAt(target, hoverPrepared);
@@ -2328,7 +2331,8 @@
for (const node of entering) send(node, `${prefix}enter`, { ...extra, bubbles: false, cancelable: false, relatedTarget: related(previous, node) });
}
}
- pointer(target, 'pointermove', extra); mouse(target, 'mousemove', extra);
+ pointer(target, 'pointermove', extra);
+ if (compatMouse) mouse(target, 'mousemove', extra);
_visionPointerOver = target;
};
const hover = target => moveOnto(target, {}, { arrive: true });
@@ -2384,8 +2388,11 @@
const steps = Math.min(40, Math.max(2, Math.floor(Number(params.steps) || 12)));
hover(el);
capture.press();
- pointer(el, 'pointerdown', { button: 0, buttons: 1 });
- const pressAllowed = mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });
+ // A page that cancels the pointerdown gets no compatibility mousedown,
+ // mousemove or mouseup for this press, and none of the press's
+ // defaults below, as in a browser.
+ const pointerAllowed = pointer(el, 'pointerdown', { button: 0, buttons: 1 });
+ const pressAllowed = pointerAllowed && mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });
// The press focuses what it lands on, as a click's does, and starts
// what a person's press would: on a native range input it takes the
// thumb (_visionRangeDrag), on selectable text or in a text field a
@@ -2445,7 +2452,7 @@
}
if (htmlStarted) { html.move(last, point); continue; }
const holder = capture.target({ button: 0, buttons: 1, ...heldAt(point, px, py) });
- moveOnto(holder || last, { button: 0, buttons: 1, ...(holder ? heldAt(point, px, py) : point) });
+ moveOnto(holder || last, { button: 0, buttons: 1, ...(holder ? heldAt(point, px, py) : point) }, { compatMouse: pointerAllowed });
const local = hit.toDoc(px, py);
range?.to(local.x, local.y);
selection?.to(local.x, local.y);
@@ -2460,10 +2467,10 @@
_visionPointerOver = releaseTarget;
return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe, html5Drag };
}
- if (releaseTarget !== _visionPointerOver) moveOnto(releaseTarget, { ...endInit, buttons: 1 });
+ if (releaseTarget !== _visionPointerOver) moveOnto(releaseTarget, { ...endInit, buttons: 1 }, { compatMouse: pointerAllowed });
pointer(releaseTarget, 'pointerup', endInit);
capture.release(endInit);
- mouse(releaseTarget, 'mouseup', { ...endInit, detail: 1 });
+ if (pointerAllowed) mouse(releaseTarget, 'mouseup', { ...endInit, detail: 1 });
const rangeValue = range?.release();
return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe,
...(range ? { rangeValue } : {}), ...(selection ? { selectedChars: selection.selectedChars() } : {}) };
@@ -2473,11 +2480,18 @@
// chooser (a file input, its label, an upload button that clicks one):
// the guard blocks it, and the agent settles the guard afterwards.
let rangeValue;
+ // Whether the last press went through to its mousedown: a double or
+ // triple click's selection is that press's default.
+ let lastPressAllowed = true;
const filePickerGuard = clickWithoutNativeFilePicker(() => {
for (let count = 1; count <= clicks; count++) {
capture.press();
- pointer(el, 'pointerdown', { button, buttons: buttonsMask });
- const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
+ // A page that cancels the pointerdown gets no compatibility
+ // mousedown or mouseup for the press, and none of its defaults
+ // (focus, the caret, a range thumb); its click still fires.
+ const pointerAllowed = pointer(el, 'pointerdown', { button, buttons: buttonsMask });
+ const pressAllowed = pointerAllowed && mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
+ lastPressAllowed = pressAllowed;
// A primary press on a native range input moves its thumb to the
// point, as a drag's does (_visionRangeDrag): `input` now, `change`
// on release.
@@ -2494,7 +2508,7 @@
const holder = capture.target({ button, buttons: buttonsMask }) || el;
pointer(holder, 'pointerup', { button, buttons: 0 });
capture.release({ button, buttons: 0 });
- mouse(holder, 'mouseup', { button, buttons: 0, detail: count });
+ if (pointerAllowed) mouse(holder, 'mouseup', { button, buttons: 0, detail: count });
if (range) rangeValue = range.release();
if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });
else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });
@@ -2502,10 +2516,11 @@
if (button === 0 && clicks >= 2) {
mouse(el, 'dblclick', { button: 0, buttons: 0, detail: 2 });
// Browsers select the word under the pointer (two clicks), or its
- // line or the whole field (three), as the default action; synthetic
- // events need it done. A double click never takes a whole field:
- // the typing that usually follows would replace all of it.
- try {
+ // line or the whole field (three), as the default action of the
+ // last press; synthetic events need it done. A double click never
+ // takes a whole field: the typing that usually follows would
+ // replace all of it.
+ if (lastPressAllowed) try {
const caret = _visionCaretAt(el.ownerDocument, cx, cy);
if (typeof el.setSelectionRange === 'function' && /^(input|textarea)$/.test(tag)) {
if (clicks >= 3 && tag === 'input') el.select();
diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.js
index 641ce6286..d9be52316 100644
--- a/src/firefox/src/content/content.js
+++ b/src/firefox/src/content/content.js
@@ -2598,7 +2598,10 @@
// or again on the same one (`arrive`), over and enter go to the target.
// Documents and shadow roots whose :hover rules this input read (_visionHoverAt).
const hoverPrepared = new Set();
- const moveOnto = (target, extra = {}, { arrive = false } = {}) => {
+ // A press whose pointerdown the page cancelled gets no compatibility
+ // mousemove (`compatMouse` off) until its release; the boundary events
+ // (over, out, enter, leave) still go out, as the Pointer Events spec has it.
+ const moveOnto = (target, extra = {}, { arrive = false, compatMouse = true } = {}) => {
// :hover follows the pointer before its events go out, as it does for
// a mouse (_visionHoverAt).
_visionHoverAt(target, hoverPrepared);
@@ -2622,7 +2625,8 @@
for (const node of entering) send(node, `${prefix}enter`, { ...extra, bubbles: false, cancelable: false, relatedTarget: related(previous, node) });
}
}
- pointer(target, 'pointermove', extra); mouse(target, 'mousemove', extra);
+ pointer(target, 'pointermove', extra);
+ if (compatMouse) mouse(target, 'mousemove', extra);
_visionPointerOver = target;
};
const hover = target => moveOnto(target, {}, { arrive: true });
@@ -2678,8 +2682,11 @@
const steps = Math.min(40, Math.max(2, Math.floor(Number(params.steps) || 12)));
hover(el);
capture.press();
- pointer(el, 'pointerdown', { button: 0, buttons: 1 });
- const pressAllowed = mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });
+ // A page that cancels the pointerdown gets no compatibility mousedown,
+ // mousemove or mouseup for this press, and none of the press's
+ // defaults below, as in a browser.
+ const pointerAllowed = pointer(el, 'pointerdown', { button: 0, buttons: 1 });
+ const pressAllowed = pointerAllowed && mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });
// The press focuses what it lands on, as a click's does, and starts
// what a person's press would: on a native range input it takes the
// thumb (_visionRangeDrag), on selectable text or in a text field a
@@ -2739,7 +2746,7 @@
}
if (htmlStarted) { html.move(last, point); continue; }
const holder = capture.target({ button: 0, buttons: 1, ...heldAt(point, px, py) });
- moveOnto(holder || last, { button: 0, buttons: 1, ...(holder ? heldAt(point, px, py) : point) });
+ moveOnto(holder || last, { button: 0, buttons: 1, ...(holder ? heldAt(point, px, py) : point) }, { compatMouse: pointerAllowed });
const local = hit.toDoc(px, py);
range?.to(local.x, local.y);
selection?.to(local.x, local.y);
@@ -2754,10 +2761,10 @@
_visionPointerOver = releaseTarget;
return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe, html5Drag };
}
- if (releaseTarget !== _visionPointerOver) moveOnto(releaseTarget, { ...endInit, buttons: 1 });
+ if (releaseTarget !== _visionPointerOver) moveOnto(releaseTarget, { ...endInit, buttons: 1 }, { compatMouse: pointerAllowed });
pointer(releaseTarget, 'pointerup', endInit);
capture.release(endInit);
- mouse(releaseTarget, 'mouseup', { ...endInit, detail: 1 });
+ if (pointerAllowed) mouse(releaseTarget, 'mouseup', { ...endInit, detail: 1 });
const rangeValue = range?.release();
return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe,
...(range ? { rangeValue } : {}), ...(selection ? { selectedChars: selection.selectedChars() } : {}) };
@@ -2767,11 +2774,18 @@
// chooser (a file input, its label, an upload button that clicks one):
// the guard blocks it, and the agent settles the guard afterwards.
let rangeValue;
+ // Whether the last press went through to its mousedown: a double or
+ // triple click's selection is that press's default.
+ let lastPressAllowed = true;
const filePickerGuard = clickWithoutNativeFilePicker(() => {
for (let count = 1; count <= clicks; count++) {
capture.press();
- pointer(el, 'pointerdown', { button, buttons: buttonsMask });
- const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
+ // A page that cancels the pointerdown gets no compatibility
+ // mousedown or mouseup for the press, and none of its defaults
+ // (focus, the caret, a range thumb); its click still fires.
+ const pointerAllowed = pointer(el, 'pointerdown', { button, buttons: buttonsMask });
+ const pressAllowed = pointerAllowed && mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
+ lastPressAllowed = pressAllowed;
// A primary press on a native range input moves its thumb to the
// point, as a drag's does (_visionRangeDrag): `input` now, `change`
// on release.
@@ -2788,7 +2802,7 @@
const holder = capture.target({ button, buttons: buttonsMask }) || el;
pointer(holder, 'pointerup', { button, buttons: 0 });
capture.release({ button, buttons: 0 });
- mouse(holder, 'mouseup', { button, buttons: 0, detail: count });
+ if (pointerAllowed) mouse(holder, 'mouseup', { button, buttons: 0, detail: count });
if (range) rangeValue = range.release();
if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });
else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });
@@ -2796,10 +2810,11 @@
if (button === 0 && clicks >= 2) {
mouse(el, 'dblclick', { button: 0, buttons: 0, detail: 2 });
// Browsers select the word under the pointer (two clicks), or its
- // line or the whole field (three), as the default action; synthetic
- // events need it done. A double click never takes a whole field:
- // the typing that usually follows would replace all of it.
- try {
+ // line or the whole field (three), as the default action of the
+ // last press; synthetic events need it done. A double click never
+ // takes a whole field: the typing that usually follows would
+ // replace all of it.
+ if (lastPressAllowed) try {
const caret = _visionCaretAt(el.ownerDocument, cx, cy);
if (typeof el.setSelectionRange === 'function' && /^(input|textarea)$/.test(tag)) {
if (clicks >= 3 && tag === 'input') el.select();
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index 4225b9d71..0d5ce6124 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -226,6 +226,80 @@ test('content-script vision input (Firefox path) clicks, double-clicks, focuses,
}
});
+test('content-script vision input (Firefox path) honours a pointerdown the page cancels, as a browser does', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ await page.setContent(`<!doctype html><style>body{margin:0;font:20px monospace}
+ #range{position:absolute;left:20px;top:20px;width:300px;height:20px;margin:0}
+ #field{position:absolute;left:20px;top:80px;width:300px;height:30px}
+ #words{position:absolute;left:20px;top:150px;margin:0;white-space:nowrap}
+ #pad{position:absolute;left:20px;top:220px;width:400px;height:60px;background:#ddd}</style>
+ <input id="range" type="range" min="0" max="100" step="5" value="10"><input id="field" value="hello">
+ <p id="words">alpha bravo charlie</p><div id="pad"></div>
+ <script>window.log=[];window.cancel=new Set(['range','field','words','pad']);
+ document.addEventListener('pointerdown',e=>{if(cancel.has(e.target.id))e.preventDefault();},true);
+ for(const t of ['pointerdown','mousedown','pointermove','mousemove','mouseover','pointerup','mouseup','click','dblclick','input','change'])
+ document.addEventListener(t,e=>{if(e.target.id)log.push(t+':'+e.target.id)},true);</script>`);
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn visionInput;`)();
+ }, { code: block, guardBlock });
+ const run = params => page.evaluate(p => window.__visionInput(p), params);
+ const take = () => page.evaluate(() => window.log.splice(0));
+ const bravo = await page.evaluate(() => { const text = document.getElementById('words').firstChild; const r = document.createRange();
+ r.setStart(text, 6); r.setEnd(text, 11); const b = r.getBoundingClientRect(); return { x: (b.left + b.right) / 2, y: (b.top + b.bottom) / 2 }; });
+
+ // A click on a range track: no thumb moves and no compatibility mouse
+ // events, but the click still fires.
+ let result = await run({ type: 'click', x: 170, y: 30 });
+ assert.equal(result.success, true);
+ assert.equal(result.rangeValue, undefined);
+ assert.equal(await page.evaluate(() => document.getElementById('range').value), '10');
+ let log = await take();
+ assert.ok(log.includes('pointerdown:range') && log.includes('pointerup:range') && log.includes('click:range'), log.join());
+ assert.ok(!log.some(entry => /^(mousedown|mouseup|input|change):/.test(entry)), log.join());
+ // A click in a text field: no focus.
+ await run({ type: 'click', x: 100, y: 95 });
+ assert.notEqual(await page.evaluate(() => document.activeElement.id), 'field');
+ await take();
+ // A double click on a word: dblclick fires, but no word is selected.
+ await page.evaluate(() => getSelection().removeAllRanges());
+ result = await run({ type: 'click', x: bravo.x, y: bravo.y, clickCount: 2 });
+ log = await take();
+ assert.ok(log.includes('dblclick:words'), log.join());
+ assert.equal(await page.evaluate(() => String(getSelection())), '');
+ // A drag: pointer moves and the release go out, no mousemove or mouseup,
+ // the boundary mouseover still does, and no text is selected.
+ result = await run({ type: 'drag', x: 30, y: 250, toX: 380, toY: 250, steps: 4 });
+ log = await take();
+ assert.equal(result.success, true);
+ const pressed = log.slice(log.indexOf('pointerdown:pad'));
+ assert.ok(pressed.filter(entry => entry === 'pointermove:pad').length >= 4, log.join());
+ assert.ok(pressed.includes('pointerup:pad'), log.join());
+ assert.ok(!pressed.some(entry => /^(mousedown|mousemove|mouseup):pad$/.test(entry)), log.join());
+ // The same sequences on presses the page lets through are unchanged.
+ await page.evaluate(() => window.cancel.clear());
+ result = await run({ type: 'click', x: 170, y: 30 });
+ assert.equal(result.rangeValue, '50');
+ log = await take();
+ assert.ok(log.includes('mousedown:range') && log.includes('mouseup:range'), log.join());
+ result = await run({ type: 'drag', x: 30, y: 250, toX: 380, toY: 250, steps: 4 });
+ log = await take();
+ assert.ok(log.includes('mousemove:pad') && log.includes('mouseup:pad'), log.join());
+ await run({ type: 'click', x: bravo.x, y: bravo.y, clickCount: 2 });
+ assert.equal(await page.evaluate(() => String(getSelection())), 'bravo');
+ } finally {
+ await browser.close();
+ }
+});
+
test('content-script vision drag (Firefox path) moves and releases in a same-origin frame\'s own coordinates', async () => {
const browser = await chromium.launch({ channel: 'chromium' });
try {