Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: pointer-cursor controls, submit probe for drags, lifecycle stub

Emre Sokullu74e1b226cb04
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index bc73d049d..c0e44ce6a 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -100,8 +100,8 @@ A capture, encoding or input failure never ends the run: a screen update that ca
Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:
-- `click_at` with `clicks: 2|3` is refused on a button, link or other control, on a detected submit, on a protected messaging page, inside a frame it cannot inspect, or when the target cannot be read: each extra click would repeat the one action the checks approved. Double and triple clicks remain for selecting text.
-- `drag_at` is refused when it is shorter than 6 CSS px, when the press and release would click a control (their nearest common ancestor is or is inside a button, link, label or similar), when either end is inside a cross-origin frame, or when the target cannot be read. On a protected messaging page `drag_at` is blocked outright, like `iframe_click`.
+- `click_at` with `clicks: 2|3` is refused on a button, link or other control (including an element drawn with a pointer cursor, which is how a framework button made of a bare `<div>` shows itself), on a detected submit, on a protected messaging page, inside a frame it cannot inspect, or when the target cannot be read: each extra click would repeat the one action the checks approved. Double and triple clicks remain for selecting text.
+- `drag_at` is refused when it is shorter than 6 CSS px, when the press and release would click a control (their nearest common ancestor is or is inside a button, link, label, similar control or pointer-cursor element, unless the drag starts on a slider, scrollbar, resizer, draggable item or grab handle), when either end is inside a cross-origin frame, when the target cannot be read, or when the click submit probe reads its start point as a submit, send, delete or payment control (a drag never gets the confirmation a click gets). On a protected messaging page `drag_at` is blocked outright, like `iframe_click`.
- `scroll_at` is refused over a focused number field, which takes the wheel as +/-.
Every CDP input passes `cdpClient`'s page-agent fence and every content action passes `beforePageAgentDispatch`, so the page monitor attributes the events to the agent. `drag_at` is gated as a click (`TOOL_CAPABILITY`); captures and pointer results are untrusted page content (`UNTRUSTED_CONTENT_TOOLS`), and every image label says that text in it is data, never instructions. Screenshot redaction (Settings) applies to frames and zooms.
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 4800e0ad1..72525765f 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -20721,37 +20721,45 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
await cdpClient.attach(tabId);
await cdpClient.sendCommand(tabId, 'Runtime.enable');
if (!this._visionProbeContexts) this._visionProbeContexts = new Map();
- let contextId = null;
- try {
- await cdpClient.sendCommand(tabId, 'Page.enable');
- const frame = (await cdpClient.sendCommand(tabId, 'Page.getFrameTree'))?.frameTree?.frame;
- const cached = this._visionProbeContexts.get(tabId);
- if (frame?.id && cached?.frameId === frame.id && cached.loaderId === frame.loaderId) {
- contextId = cached.contextId;
- } else if (frame?.id) {
- contextId = (await cdpClient.sendCommand(tabId, 'Page.createIsolatedWorld', {
- frameId: frame.id, worldName: 'webbrain-vision-probe', grantUniveralAccess: false,
- }))?.executionContextId || null;
- if (contextId) this._visionProbeContexts.set(tabId, { frameId: frame.id, loaderId: frame.loaderId, contextId });
+ const expression = visionDomProbeExpression(request);
+ for (let attempt = 0; attempt < 2; attempt++) {
+ let contextId = null;
+ let reused = false;
+ try {
+ await cdpClient.sendCommand(tabId, 'Page.enable');
+ const frame = (await cdpClient.sendCommand(tabId, 'Page.getFrameTree'))?.frameTree?.frame;
+ const cached = this._visionProbeContexts.get(tabId);
+ if (frame?.id && cached?.frameId === frame.id && cached.loaderId === frame.loaderId) {
+ contextId = cached.contextId;
+ reused = true;
+ } else if (frame?.id) {
+ contextId = (await cdpClient.sendCommand(tabId, 'Page.createIsolatedWorld', {
+ frameId: frame.id, worldName: 'webbrain-vision-probe', grantUniveralAccess: false,
+ }))?.executionContextId || null;
+ if (contextId) this._visionProbeContexts.set(tabId, { frameId: frame.id, loaderId: frame.loaderId, contextId });
+ }
+ } catch (error) {
+ if (error?.code === 'page_feedback_pending') throw error;
+ contextId = null;
+ }
+ try {
+ const response = await cdpClient.sendCommand(tabId, 'Runtime.evaluate', {
+ expression,
+ returnByValue: true,
+ ...(contextId ? { contextId } : {}),
+ });
+ if (response?.exceptionDetails) return null;
+ return response?.result?.value || null;
+ } catch (error) {
+ if (error?.code === 'page_feedback_pending') throw error;
+ this._visionProbeContexts.delete(tabId);
+ // A cached world that did not survive (a debugger re-attach on the
+ // same document) is made again once.
+ if (reused && attempt === 0) continue;
+ throw error;
}
- } catch (error) {
- if (error?.code === 'page_feedback_pending') throw error;
- contextId = null;
- }
- let response;
- try {
- response = await cdpClient.sendCommand(tabId, 'Runtime.evaluate', {
- expression: visionDomProbeExpression(request),
- returnByValue: true,
- ...(contextId ? { contextId } : {}),
- });
- } catch (error) {
- // A context that went away with its document is made again next time.
- this._visionProbeContexts.delete(tabId);
- throw error;
}
- if (response?.exceptionDetails) return null;
- return response?.result?.value || null;
+ return null;
}
/**
diff --git a/src/chrome/src/agent/vision-dom-probe.js b/src/chrome/src/agent/vision-dom-probe.js
index 7e46c14c6..f16f4af68 100644
--- a/src/chrome/src/agent/vision-dom-probe.js
+++ b/src/chrome/src/agent/vision-dom-probe.js
@@ -33,6 +33,11 @@ export function visionDomProbe(request) {
'input[type="date"]', 'input[type="datetime-local"]', 'input[type="time"]', 'input[type="month"]',
'input[type="week"]', 'input[type="range"]',
].join(',');
+ // Where a press-move-release is the control's own gesture, a pointer
+ // cursor around it does not make the drag a click.
+ const DRAG_AFFORDANCE = '[draggable="true"],[role="slider"],[role="scrollbar"],[role="separator"],input[type="range"]';
+ const DRAG_CURSORS = ['grab', 'grabbing', 'move', 'all-scroll', 'col-resize', 'row-resize', 'ew-resize', 'ns-resize',
+ 'nesw-resize', 'nwse-resize', 'e-resize', 'w-resize', 'n-resize', 's-resize', 'ne-resize', 'nw-resize', 'se-resize', 'sw-resize'];
const req = request && typeof request === 'object' ? request : {};
// Composed parent: through shadow roots to their host and out of
// same-origin frames to the frame element.
@@ -75,23 +80,43 @@ export function visionDomProbe(request) {
}
return { el: null, opaque: false };
};
- const clickableOf = node => {
+ const cursorOf = el => {
+ try {
+ const view = el && el.ownerDocument && el.ownerDocument.defaultView;
+ return view ? String(view.getComputedStyle(el).cursor || '') : '';
+ } catch (e) { return ''; }
+ };
+ // The control a click at `node` activates: a semantic control on its
+ // ancestor chain, nothing inside a text field, otherwise an element drawn
+ // with a pointer cursor (framework buttons are often a bare <div> with a
+ // listener; the cursor is the one sign of that a page script cannot hide
+ // from this world).
+ const controlOf = node => {
for (let n = node, i = 0; n && i < 256; n = up(n), i++) {
if (n.nodeType === 1 && typeof n.matches === 'function') {
try {
- if (n.matches(CLICKABLE)) return n;
+ if (n.matches(CLICKABLE)) return { el: n, kind: String(n.tagName || '').toLowerCase() };
if (n.matches(TEXT_ENTRY)) return null;
} catch (e) { /* invalid in this document */ }
}
}
- return null;
+ return node && node.nodeType === 1 && cursorOf(node) === 'pointer' ? { el: node, kind: 'clickable element' } : null;
+ };
+ const dragAffordance = node => {
+ if (DRAG_CURSORS.includes(cursorOf(node))) return true;
+ for (let n = node, i = 0; n && i < 64; n = up(n), i++) {
+ if (n.nodeType === 1 && typeof n.matches === 'function') {
+ try { if (n.matches(DRAG_AFFORDANCE)) return true; } catch (e) { /* invalid in this document */ }
+ }
+ }
+ return false;
};
const points = Array.isArray(req.points) ? req.points.slice(0, 2) : [];
const found = points.map(point => hit(Number(point && point.x), Number(point && point.y)));
const infos = found.map((item, index) => {
const el = item.el;
if (!el) return { found: false };
- const control = clickableOf(el);
+ const control = controlOf(el);
const info = {
found: true,
tag: String(el.tagName || '').toLowerCase(),
@@ -100,7 +125,7 @@ export function visionDomProbe(request) {
};
const role = el.getAttribute ? el.getAttribute('role') : '';
if (role) info.role = String(role).slice(0, 32);
- if (control) info.control = String(control.tagName || '').toLowerCase();
+ if (control) info.control = control.kind;
if (typeof el.matches === 'function' && el.matches('input[type="number"]')) {
const root = el.getRootNode ? el.getRootNode() : el.ownerDocument;
info.focusedNumberInput = !!root && root.activeElement === el;
@@ -130,9 +155,11 @@ export function visionDomProbe(request) {
for (let n = found[0].el, i = 0; n && i < 256; n = up(n), i++) {
if (ancestors.has(n)) { common = n; break; }
}
- const control = common ? clickableOf(common) : null;
+ let control = common ? controlOf(common) : null;
+ // A pointer cursor alone does not turn a slider or handle drag into a click.
+ if (control && control.kind === 'clickable element' && dragAffordance(found[0].el)) control = null;
result.pair = { known: true, clickEquivalent: !!control };
- if (control) result.pair.control = String(control.tagName || '').toLowerCase();
+ if (control) result.pair.control = control.kind;
}
}
return result;
diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.js
index b6ecc119a..988f6d80d 100644
--- a/src/chrome/src/agent/vision-runtime.js
+++ b/src/chrome/src/agent/vision-runtime.js
@@ -206,6 +206,18 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
return session;
};
+ /** The click submit probe at a CSS point: its reading when it is a submit, else null. */
+ proto._visionSubmitAt = async function _visionSubmitAt(tabId, point) {
+ if (typeof this._detectLikelySubmitAction !== 'function') return null;
+ try {
+ const detected = await this._detectLikelySubmitAction(tabId, 'click', { x: point.x, y: point.y, coordinate_space: 'css' });
+ return detected?.isSubmit === true ? detected : null;
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ return null;
+ }
+ };
+
/** What is under CSS points (vision-dom-probe.js); null when it cannot be read. */
proto._visionDomProbe = async function _visionDomProbe(tabId, request) {
if (typeof this._visionPlatformDomProbe !== 'function') return null;
@@ -713,6 +725,12 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
const probe = longEnough ? await this._visionDomProbe(tabId, { points: [from, to], pair: true }) : null;
const refused = visionDragRefusal(probe, from, to);
if (refused) return { ...refused, from, to };
+ if (await this._visionSubmitAt(tabId, from)) {
+ return {
+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'drag_on_submit', from, to,
+ error: 'Not dispatched: the drag starts on a control WebBrain reads as a submit, send, delete or payment action, and a drag never gets the confirmation a click gets. Use click_at on it.',
+ };
+ }
dispatch.started = true;
const result = await this._visionPlatformDrag(tabId, { from, to, steps: 12 });
return { success: result?.success !== false, from, to, ...(result || {}) };
diff --git a/src/firefox/src/agent/vision-dom-probe.js b/src/firefox/src/agent/vision-dom-probe.js
index 7e46c14c6..f16f4af68 100644
--- a/src/firefox/src/agent/vision-dom-probe.js
+++ b/src/firefox/src/agent/vision-dom-probe.js
@@ -33,6 +33,11 @@ export function visionDomProbe(request) {
'input[type="date"]', 'input[type="datetime-local"]', 'input[type="time"]', 'input[type="month"]',
'input[type="week"]', 'input[type="range"]',
].join(',');
+ // Where a press-move-release is the control's own gesture, a pointer
+ // cursor around it does not make the drag a click.
+ const DRAG_AFFORDANCE = '[draggable="true"],[role="slider"],[role="scrollbar"],[role="separator"],input[type="range"]';
+ const DRAG_CURSORS = ['grab', 'grabbing', 'move', 'all-scroll', 'col-resize', 'row-resize', 'ew-resize', 'ns-resize',
+ 'nesw-resize', 'nwse-resize', 'e-resize', 'w-resize', 'n-resize', 's-resize', 'ne-resize', 'nw-resize', 'se-resize', 'sw-resize'];
const req = request && typeof request === 'object' ? request : {};
// Composed parent: through shadow roots to their host and out of
// same-origin frames to the frame element.
@@ -75,23 +80,43 @@ export function visionDomProbe(request) {
}
return { el: null, opaque: false };
};
- const clickableOf = node => {
+ const cursorOf = el => {
+ try {
+ const view = el && el.ownerDocument && el.ownerDocument.defaultView;
+ return view ? String(view.getComputedStyle(el).cursor || '') : '';
+ } catch (e) { return ''; }
+ };
+ // The control a click at `node` activates: a semantic control on its
+ // ancestor chain, nothing inside a text field, otherwise an element drawn
+ // with a pointer cursor (framework buttons are often a bare <div> with a
+ // listener; the cursor is the one sign of that a page script cannot hide
+ // from this world).
+ const controlOf = node => {
for (let n = node, i = 0; n && i < 256; n = up(n), i++) {
if (n.nodeType === 1 && typeof n.matches === 'function') {
try {
- if (n.matches(CLICKABLE)) return n;
+ if (n.matches(CLICKABLE)) return { el: n, kind: String(n.tagName || '').toLowerCase() };
if (n.matches(TEXT_ENTRY)) return null;
} catch (e) { /* invalid in this document */ }
}
}
- return null;
+ return node && node.nodeType === 1 && cursorOf(node) === 'pointer' ? { el: node, kind: 'clickable element' } : null;
+ };
+ const dragAffordance = node => {
+ if (DRAG_CURSORS.includes(cursorOf(node))) return true;
+ for (let n = node, i = 0; n && i < 64; n = up(n), i++) {
+ if (n.nodeType === 1 && typeof n.matches === 'function') {
+ try { if (n.matches(DRAG_AFFORDANCE)) return true; } catch (e) { /* invalid in this document */ }
+ }
+ }
+ return false;
};
const points = Array.isArray(req.points) ? req.points.slice(0, 2) : [];
const found = points.map(point => hit(Number(point && point.x), Number(point && point.y)));
const infos = found.map((item, index) => {
const el = item.el;
if (!el) return { found: false };
- const control = clickableOf(el);
+ const control = controlOf(el);
const info = {
found: true,
tag: String(el.tagName || '').toLowerCase(),
@@ -100,7 +125,7 @@ export function visionDomProbe(request) {
};
const role = el.getAttribute ? el.getAttribute('role') : '';
if (role) info.role = String(role).slice(0, 32);
- if (control) info.control = String(control.tagName || '').toLowerCase();
+ if (control) info.control = control.kind;
if (typeof el.matches === 'function' && el.matches('input[type="number"]')) {
const root = el.getRootNode ? el.getRootNode() : el.ownerDocument;
info.focusedNumberInput = !!root && root.activeElement === el;
@@ -130,9 +155,11 @@ export function visionDomProbe(request) {
for (let n = found[0].el, i = 0; n && i < 256; n = up(n), i++) {
if (ancestors.has(n)) { common = n; break; }
}
- const control = common ? clickableOf(common) : null;
+ let control = common ? controlOf(common) : null;
+ // A pointer cursor alone does not turn a slider or handle drag into a click.
+ if (control && control.kind === 'clickable element' && dragAffordance(found[0].el)) control = null;
result.pair = { known: true, clickEquivalent: !!control };
- if (control) result.pair.control = String(control.tagName || '').toLowerCase();
+ if (control) result.pair.control = control.kind;
}
}
return result;
diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.js
index b6ecc119a..988f6d80d 100644
--- a/src/firefox/src/agent/vision-runtime.js
+++ b/src/firefox/src/agent/vision-runtime.js
@@ -206,6 +206,18 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
return session;
};
+ /** The click submit probe at a CSS point: its reading when it is a submit, else null. */
+ proto._visionSubmitAt = async function _visionSubmitAt(tabId, point) {
+ if (typeof this._detectLikelySubmitAction !== 'function') return null;
+ try {
+ const detected = await this._detectLikelySubmitAction(tabId, 'click', { x: point.x, y: point.y, coordinate_space: 'css' });
+ return detected?.isSubmit === true ? detected : null;
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ return null;
+ }
+ };
+
/** What is under CSS points (vision-dom-probe.js); null when it cannot be read. */
proto._visionDomProbe = async function _visionDomProbe(tabId, request) {
if (typeof this._visionPlatformDomProbe !== 'function') return null;
@@ -713,6 +725,12 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
const probe = longEnough ? await this._visionDomProbe(tabId, { points: [from, to], pair: true }) : null;
const refused = visionDragRefusal(probe, from, to);
if (refused) return { ...refused, from, to };
+ if (await this._visionSubmitAt(tabId, from)) {
+ return {
+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'drag_on_submit', from, to,
+ error: 'Not dispatched: the drag starts on a control WebBrain reads as a submit, send, delete or payment action, and a drag never gets the confirmation a click gets. Use click_at on it.',
+ };
+ }
dispatch.started = true;
const result = await this._visionPlatformDrag(tabId, { from, to, steps: 12 });
return { success: result?.success !== false, from, to, ...(result || {}) };
diff --git a/test/runtime-lifecycle.mjs b/test/runtime-lifecycle.mjs
index 2c39d1e59..dd8edb2a8 100644
--- a/test/runtime-lifecycle.mjs
+++ b/test/runtime-lifecycle.mjs
@@ -397,7 +397,7 @@ async function chatInputFixture(browser) {
assertNoActiveTabRun:() => { effects.run++; throw reached; },
};
for (const key of ['planBeforeActReady','planReviewReady','customSkillsReady','userMemoryReady','imageGenConfigReady','browserBatchEnabledReady',
- 'alwaysAllowApiMutationsReady','strictSecretModeReady','webMcpEnabledReady','screenshotRedactionReady','imageBudgetReady','researchEscalationReady',
+ 'alwaysAllowApiMutationsReady','strictSecretModeReady','visionModeReady','webMcpEnabledReady','screenshotRedactionReady','imageBudgetReady','researchEscalationReady',
'clarifyTimeoutReady']) context[key] = Promise.resolve();
const functions = ['assertChatRequestInput', 'handleMessage'].map(name => chatInputFunction(source,name)).join('\n');
chatInputVM.runInNewContext(functions + '\nglobalThis.request = handleMessage;', context);
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index adc3d83f6..3504e2457 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -392,6 +392,20 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
assert.ok(ctx.scene.probes.some(request => request.pair === true && request.points.length === 2));
});
+ test(`${build}: a drag that starts on a submit control is refused`, async () => {
+ const ctx = runAgent(step => {
+ if (step === 1) return { toolCalls: [call('drag_at', { start_x: 300, start_y: 300, end_x: 500, end_y: 300 })] };
+ if (step === 2) return { toolCalls: [call('done', { summary: 'No drag.', outcome: 'partial' })] };
+ return { content: 'done' };
+ });
+ const probed = [];
+ ctx.agent._detectLikelySubmitAction = async (_tab, name, args) => { probed.push({ name, args }); return { isSubmit: true }; };
+ await runLoop(ctx.agent, nextTab++, false);
+ assert.equal(ctx.platform.filter(entry => entry.kind === 'drag').length, 0);
+ assert.ok(toolTexts(ctx.requests[1]).some(text => /never gets the confirmation a click gets/.test(text)), toolTexts(ctx.requests[1]).join('\n'));
+ assert.deepEqual(probed.at(-1), { name: 'click', args: { x: 333.3, y: 333.3, coordinate_space: 'css' } });
+ });
+
test(`${build}: scroll_at over a focused number field is refused`, async () => {
const ctx = runAgent(step => {
if (step === 1) return { toolCalls: [call('scroll_at', { direction: 'down', x: 648, y: 405 })] };
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index 762d5519a..46b7a4b02 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -229,7 +229,12 @@ test('the DOM probe behind the dispatch checks reads controls, text fields, focu
#card{position:absolute;left:100px;top:380px;width:250px;height:100px;display:block;background:#eee}
#slider{position:absolute;left:400px;top:100px;width:300px;height:30px;background:#ddd}
#same{position:absolute;left:400px;top:200px;width:300px;height:150px;border:0}
- #cross{position:absolute;left:400px;top:400px;width:200px;height:100px;border:0}</style>
+ #cross{position:absolute;left:400px;top:400px;width:200px;height:100px;border:0}
+ #divbtn{position:absolute;left:700px;top:100px;width:200px;height:60px;cursor:pointer;background:#cde}
+ #track{position:absolute;left:700px;top:200px;width:250px;height:30px;cursor:pointer;background:#ddd}
+ #knob{position:absolute;left:0;top:0;width:30px;height:30px;cursor:grab;background:#333}
+ #list{position:absolute;left:700px;top:260px;width:250px;cursor:pointer}
+ #list div{height:40px;border:1px solid #999}</style>
<button id="btn"><span>Send</span></button>
<label id="lab">Name <input id="txt" type="text" value="hello world"></label>
<input id="num" type="number" value="10">
@@ -237,7 +242,10 @@ test('the DOM probe behind the dispatch checks reads controls, text fields, focu
<a id="card" href="#card">Card <span>inner</span></a>
<div id="slider" role="slider" aria-valuenow="1"><div style="width:20px;height:30px;background:#333"></div></div>
<iframe id="same" srcdoc="<style>body{margin:0}</style><button style='position:absolute;left:20px;top:20px;width:100px;height:40px'>In frame</button>"></iframe>
- <iframe id="cross" src="data:text/html,<button>x</button>"></iframe>`);
+ <iframe id="cross" src="data:text/html,<button>x</button>"></iframe>
+ <div id="divbtn"><span>Framework button</span></div>
+ <div id="track"><div id="knob"></div></div>
+ <div id="list"><div draggable="true">One</div><div draggable="true">Two</div></div>`);
await page.waitForFunction(() => document.getElementById('same').contentDocument?.querySelector('button'));
const probe = request => page.evaluate(visionDomProbeExpression(request));
const one = async (x, y, extra = {}) => (await probe({ points: [{ x, y }], ...extra })).points[0];
@@ -268,6 +276,13 @@ test('the DOM probe behind the dispatch checks reads controls, text fields, focu
assert.deepEqual(await pair({ x: 405, y: 115 }, { x: 650, y: 115 }), { known: true, clickEquivalent: false }, 'a slider drag');
assert.equal((await pair({ x: 425, y: 225 }, { x: 515, y: 255 })).clickEquivalent, true, 'inside a button in a same-origin frame');
assert.deepEqual(await pair({ x: 150, y: 125 }, { x: 5000, y: 10 }), { known: false });
+ // A framework button is often a bare <div> with a listener: its pointer
+ // cursor marks it, for repeated clicks and for drags inside it.
+ assert.deepEqual(await one(720, 120), { found: true, tag: 'div', clickable: true, opaqueFrame: false, control: 'clickable element' });
+ assert.equal((await pair({ x: 710, y: 110 }, { x: 890, y: 150 })).clickEquivalent, true);
+ // ...but a handle or draggable item under a pointer cursor still drags.
+ assert.equal((await pair({ x: 715, y: 215 }, { x: 900, y: 215 })).clickEquivalent, false, 'a knob with a grab cursor');
+ assert.equal((await pair({ x: 720, y: 280 }, { x: 720, y: 320 })).clickEquivalent, false, 'draggable list items');
} finally {
await browser.close();
}