Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: keep the file-chooser guard on Firefox vision clicks

Emre Sokullu4589c48423a7
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 98da32532..118268a7c 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -118,7 +118,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll
`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. Only the final dispatch differs:
- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency) and click-progress annotation. The legacy coordinate path's semantic re-targeting and nearby-input redirect are skipped: a vision click lands where the model pointed. Iframes, including cross-origin ones, receive the events natively.
-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus and a click's activation, selecting text for double/triple clicks, and entering same-origin iframes. A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.
+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus and a click's activation, selecting text for double/triple clicks, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.
Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:
@@ -159,7 +159,7 @@ The older MV3 quality fixtures script an accessibility-tree model (they answer w
## Limits
-- File uploads (`upload_file`) and page-resource downloads by selector are not offered in vision mode; switch it off for those tasks.
+- File uploads (`upload_file`) and page-resource downloads by selector are not offered in vision mode; switch it off for those tasks. A click that would open a native file chooser is blocked in both builds and reported in those terms (`visionFilePickerBlockedResult`).
- Closed shadow roots and canvas apps are reachable by pointer, but a native `<select>` popup is never visible in a screenshot: such a field is focused and `type_text` picks the option.
- HTML5 drag and drop driven only by `dataTransfer` may not react to synthetic or CDP pointer drags.
- A drag is not shown to the submit, money or publishing checks; it is refused when it would click a control, but a custom "slide to confirm" widget is only as safe as the page makes it.
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 00b2d6eca..17bb8618a 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -21,7 +21,7 @@ import { SOCIAL_PLATFORMS, socialPublicationApiPlatform, normalizePublicationCon
import { mastodonPublicationIntentMessages, normalizeMastodonPublicationIntent, captureNativeMastodonPublication, mastodonPublicationBinding, mastodonPublicationReceipt, mastodonPublicationNavigation } from './mastodon-publication.js';
import { AGENT_TOOLS, AGENT_TOOL_NAMES, RESERVED_AGENT_TOOL_NAMES, RETIRED_AGENT_TOOL_NAMES, getToolsForMode, SYSTEM_PROMPT_BATCHING, SYSTEM_PROMPT_ASK, SYSTEM_PROMPT_ACT, SYSTEM_PROMPT_ACT_COMPACT, SYSTEM_PROMPT_ACT_MID, SYSTEM_PROMPT_LEAN_NOTE, SYSTEM_PROMPT_GENERATIVE_MEDIA, SYSTEM_PROMPT_DEV_APPENDIX, SYSTEM_PROMPT_WEBMCP_ASK, SYSTEM_PROMPT_WEBMCP_ACT, getVisionActPrompt } from './tools.js';
import { installVisionRuntime } from './vision-runtime.js';
-import { VISION_NATIVE_TOOLS, VISION_SITE_NOTES_PREFACE, visionUniversalPreamble } from './vision-tools.js';
+import { VISION_NATIVE_TOOLS, VISION_SITE_NOTES_PREFACE, visionFilePickerBlockedResult, visionUniversalPreamble } from './vision-tools.js';
import { visionDomProbeExpression } from './vision-dom-probe.js';
import { normalizeVisionModeSetting } from './vision-mode.js';
import { rejectedCompletionRecovery } from './completion-recovery.js';
@@ -20888,7 +20888,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}
const blockedFileInput = await cdpClient.consumeFileInputClickGuard(tabId);
if (blockedFileInput?.blocked) {
- return cdpClient.fileInputClickBlockedResult(blockedFileInput, `The attempted control was at (${x}, ${y}).`);
+ return visionFilePickerBlockedResult(cdpClient.fileInputClickBlockedResult(blockedFileInput, `The attempted control was at (${x}, ${y}).`));
}
// The file-chooser guard's settle time (above) is already the quiet
// window: a click that has opened no tab by now opened none.
diff --git a/src/chrome/src/agent/vision-tools.js b/src/chrome/src/agent/vision-tools.js
index 1619d420b..e88029f82 100644
--- a/src/chrome/src/agent/vision-tools.js
+++ b/src/chrome/src/agent/vision-tools.js
@@ -342,6 +342,19 @@ export function visionUniversalPreamble(preamble) {
*/
export const VISION_SITE_NOTES_PREFACE = '(Written for WebBrain\'s element tools: click_ax, set_field, type_ax, ref ids, selectors and the accessibility tree are not available in this run. Apply the facts about the site with click_at, type_text, press_keys and scroll_at on what the screenshots show.)';
+/**
+ * A vision click the native file-chooser guard stopped. The guard's own
+ * guidance points to upload_file, which a vision run does not offer.
+ */
+export function visionFilePickerBlockedResult(result) {
+ if (result?.filePickerBlocked !== true) return result;
+ return {
+ ...result,
+ success: false,
+ error: 'Blocked a native file chooser: it opens outside the page, where screenshots cannot show or operate it, and file uploads are not available in vision mode. Do not click this control again; finish the rest of the task and tell the user the upload needs Vision mode off (Settings β†’ Advanced).',
+ };
+}
+
/** Trusted label placed before a full frame. */
export function visionKeyframeNote(frame, { coordinates = 'pixel', reason = '' } = {}) {
const ratio = Math.round((frame.cssPerPxX || 1) * 100) / 100;
diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.js
index 82c07514b..444d008d1 100644
--- a/src/chrome/src/content/content.js
+++ b/src/chrome/src/content/content.js
@@ -1826,42 +1826,49 @@
return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe };
}
hover(el);
- for (let count = 1; count <= clicks; count++) {
- pointer(el, 'pointerdown', { button, buttons: buttonsMask });
- const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
- if (count === 1 && button === 0 && pressAllowed) {
- const focusable = _visionFocusable(el);
- if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });
- else if (!focusable) el.ownerDocument.activeElement?.blur?.();
- }
- pointer(el, 'pointerup', { button, buttons: 0 });
- mouse(el, 'mouseup', { button, buttons: 0, detail: count });
- if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });
- else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });
- }
- if (button === 0 && clicks >= 2) {
- mouse(el, 'dblclick', { button: 0, buttons: 0, detail: 2 });
- // Browsers select a word (two clicks) or the whole line/field
- // (three) as the default action; synthetic events need it done.
- try {
- if (typeof el.select === 'function' && /^(input|textarea)$/.test(tag)) {
- el.select();
- } else {
- const selection = el.ownerDocument.getSelection();
- if (clicks >= 3) selection.selectAllChildren(el.closest('p,li,td,th,div,section,article,h1,h2,h3,h4,h5,h6') || el);
- else if (typeof selection.modify === 'function') {
- const caret = el.ownerDocument.caretPositionFromPoint?.(cx, cy);
- if (caret?.offsetNode) {
- selection.collapse(caret.offsetNode, caret.offset);
- selection.modify('move', 'backward', 'word');
- selection.modify('extend', 'forward', 'word');
+ // Like any agent click, a vision click must not open a native file
+ // chooser (a file input, its label, an upload button that clicks one):
+ // the guard blocks it, and the agent settles the guard afterwards.
+ const filePickerGuard = clickWithoutNativeFilePicker(() => {
+ for (let count = 1; count <= clicks; count++) {
+ pointer(el, 'pointerdown', { button, buttons: buttonsMask });
+ const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
+ if (count === 1 && button === 0 && pressAllowed) {
+ const focusable = _visionFocusable(el);
+ if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });
+ else if (!focusable) el.ownerDocument.activeElement?.blur?.();
+ }
+ pointer(el, 'pointerup', { button, buttons: 0 });
+ mouse(el, 'mouseup', { button, buttons: 0, detail: count });
+ if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });
+ else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });
+ }
+ if (button === 0 && clicks >= 2) {
+ mouse(el, 'dblclick', { button: 0, buttons: 0, detail: 2 });
+ // Browsers select a word (two clicks) or the whole line/field
+ // (three) as the default action; synthetic events need it done.
+ try {
+ if (typeof el.select === 'function' && /^(input|textarea)$/.test(tag)) {
+ el.select();
+ } else {
+ const selection = el.ownerDocument.getSelection();
+ if (clicks >= 3) selection.selectAllChildren(el.closest('p,li,td,th,div,section,article,h1,h2,h3,h4,h5,h6') || el);
+ else if (typeof selection.modify === 'function') {
+ const caret = el.ownerDocument.caretPositionFromPoint?.(cx, cy);
+ if (caret?.offsetNode) {
+ selection.collapse(caret.offsetNode, caret.offset);
+ selection.modify('move', 'backward', 'word');
+ selection.modify('extend', 'forward', 'word');
+ }
}
}
- }
- } catch { /* selection is best effort */ }
- }
- if (button === 2) mouse(el, 'contextmenu', { button: 2, buttons: 0 });
- return { success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks, target: describe };
+ } catch { /* selection is best effort */ }
+ }
+ if (button === 2) mouse(el, 'contextmenu', { button: 2, buttons: 0 });
+ });
+ if (filePickerGuard.blocked) return { ...filePickerBlockedResponse(filePickerGuard.blocked), target: describe };
+ return { success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks, target: describe,
+ ...(filePickerGuard.guardId ? { _filePickerGuardId: filePickerGuard.guardId } : {}) };
});
}
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 14969b033..ab947f825 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -20,7 +20,7 @@ import { SOCIAL_PLATFORMS, socialPublicationApiPlatform, normalizePublicationCon
import { mastodonPublicationIntentMessages, normalizeMastodonPublicationIntent, captureNativeMastodonPublication, mastodonPublicationBinding, mastodonPublicationReceipt, mastodonPublicationNavigation } from './mastodon-publication.js';
import { AGENT_TOOLS, AGENT_TOOL_NAMES, RESERVED_AGENT_TOOL_NAMES, RETIRED_AGENT_TOOL_NAMES, getToolsForMode, SYSTEM_PROMPT_BATCHING, SYSTEM_PROMPT_ASK, SYSTEM_PROMPT_ACT, SYSTEM_PROMPT_ACT_COMPACT, SYSTEM_PROMPT_ACT_MID, SYSTEM_PROMPT_LEAN_NOTE, SYSTEM_PROMPT_GENERATIVE_MEDIA, SYSTEM_PROMPT_DEV_APPENDIX, getVisionActPrompt } from './tools.js';
import { installVisionRuntime } from './vision-runtime.js';
-import { VISION_NATIVE_TOOLS, VISION_SITE_NOTES_PREFACE, visionUniversalPreamble } from './vision-tools.js';
+import { VISION_NATIVE_TOOLS, VISION_SITE_NOTES_PREFACE, visionFilePickerBlockedResult, visionUniversalPreamble } from './vision-tools.js';
import { visionDomProbeExpression } from './vision-dom-probe.js';
import { normalizeVisionModeSetting } from './vision-mode.js';
import { rejectedCompletionRecovery } from './completion-recovery.js';
@@ -17111,9 +17111,12 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
if (preflight.refusal) return preflight.refusal;
this._throwIfAborted(abortSignal);
dispatchState.started = true;
- const response = await this._visionContentInput(tabId, {
+ let response = await this._visionContentInput(tabId, {
type: 'click', x, y, button: vision?.button || 'left', clickCount: vision?.clickCount || 1,
}, 'click');
+ // The content click ran inside the native file-chooser guard, as an
+ // ordinary click does; settle it the same way (a deferred chooser too).
+ response = visionFilePickerBlockedResult(await this._settleContentFilePickerGuard(tabId, response));
return {
...response,
x,
diff --git a/src/firefox/src/agent/vision-tools.js b/src/firefox/src/agent/vision-tools.js
index 1619d420b..e88029f82 100644
--- a/src/firefox/src/agent/vision-tools.js
+++ b/src/firefox/src/agent/vision-tools.js
@@ -342,6 +342,19 @@ export function visionUniversalPreamble(preamble) {
*/
export const VISION_SITE_NOTES_PREFACE = '(Written for WebBrain\'s element tools: click_ax, set_field, type_ax, ref ids, selectors and the accessibility tree are not available in this run. Apply the facts about the site with click_at, type_text, press_keys and scroll_at on what the screenshots show.)';
+/**
+ * A vision click the native file-chooser guard stopped. The guard's own
+ * guidance points to upload_file, which a vision run does not offer.
+ */
+export function visionFilePickerBlockedResult(result) {
+ if (result?.filePickerBlocked !== true) return result;
+ return {
+ ...result,
+ success: false,
+ error: 'Blocked a native file chooser: it opens outside the page, where screenshots cannot show or operate it, and file uploads are not available in vision mode. Do not click this control again; finish the rest of the task and tell the user the upload needs Vision mode off (Settings β†’ Advanced).',
+ };
+}
+
/** Trusted label placed before a full frame. */
export function visionKeyframeNote(frame, { coordinates = 'pixel', reason = '' } = {}) {
const ratio = Math.round((frame.cssPerPxX || 1) * 100) / 100;
diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.js
index 29e68849f..90b681361 100644
--- a/src/firefox/src/content/content.js
+++ b/src/firefox/src/content/content.js
@@ -2120,42 +2120,49 @@
return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe };
}
hover(el);
- for (let count = 1; count <= clicks; count++) {
- pointer(el, 'pointerdown', { button, buttons: buttonsMask });
- const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
- if (count === 1 && button === 0 && pressAllowed) {
- const focusable = _visionFocusable(el);
- if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });
- else if (!focusable) el.ownerDocument.activeElement?.blur?.();
- }
- pointer(el, 'pointerup', { button, buttons: 0 });
- mouse(el, 'mouseup', { button, buttons: 0, detail: count });
- if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });
- else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });
- }
- if (button === 0 && clicks >= 2) {
- mouse(el, 'dblclick', { button: 0, buttons: 0, detail: 2 });
- // Browsers select a word (two clicks) or the whole line/field
- // (three) as the default action; synthetic events need it done.
- try {
- if (typeof el.select === 'function' && /^(input|textarea)$/.test(tag)) {
- el.select();
- } else {
- const selection = el.ownerDocument.getSelection();
- if (clicks >= 3) selection.selectAllChildren(el.closest('p,li,td,th,div,section,article,h1,h2,h3,h4,h5,h6') || el);
- else if (typeof selection.modify === 'function') {
- const caret = el.ownerDocument.caretPositionFromPoint?.(cx, cy);
- if (caret?.offsetNode) {
- selection.collapse(caret.offsetNode, caret.offset);
- selection.modify('move', 'backward', 'word');
- selection.modify('extend', 'forward', 'word');
+ // Like any agent click, a vision click must not open a native file
+ // chooser (a file input, its label, an upload button that clicks one):
+ // the guard blocks it, and the agent settles the guard afterwards.
+ const filePickerGuard = clickWithoutNativeFilePicker(() => {
+ for (let count = 1; count <= clicks; count++) {
+ pointer(el, 'pointerdown', { button, buttons: buttonsMask });
+ const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
+ if (count === 1 && button === 0 && pressAllowed) {
+ const focusable = _visionFocusable(el);
+ if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });
+ else if (!focusable) el.ownerDocument.activeElement?.blur?.();
+ }
+ pointer(el, 'pointerup', { button, buttons: 0 });
+ mouse(el, 'mouseup', { button, buttons: 0, detail: count });
+ if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });
+ else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });
+ }
+ if (button === 0 && clicks >= 2) {
+ mouse(el, 'dblclick', { button: 0, buttons: 0, detail: 2 });
+ // Browsers select a word (two clicks) or the whole line/field
+ // (three) as the default action; synthetic events need it done.
+ try {
+ if (typeof el.select === 'function' && /^(input|textarea)$/.test(tag)) {
+ el.select();
+ } else {
+ const selection = el.ownerDocument.getSelection();
+ if (clicks >= 3) selection.selectAllChildren(el.closest('p,li,td,th,div,section,article,h1,h2,h3,h4,h5,h6') || el);
+ else if (typeof selection.modify === 'function') {
+ const caret = el.ownerDocument.caretPositionFromPoint?.(cx, cy);
+ if (caret?.offsetNode) {
+ selection.collapse(caret.offsetNode, caret.offset);
+ selection.modify('move', 'backward', 'word');
+ selection.modify('extend', 'forward', 'word');
+ }
}
}
- }
- } catch { /* selection is best effort */ }
- }
- if (button === 2) mouse(el, 'contextmenu', { button: 2, buttons: 0 });
- return { success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks, target: describe };
+ } catch { /* selection is best effort */ }
+ }
+ if (button === 2) mouse(el, 'contextmenu', { button: 2, buttons: 0 });
+ });
+ if (filePickerGuard.blocked) return { ...filePickerBlockedResponse(filePickerGuard.blocked), target: describe };
+ return { success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks, target: describe,
+ ...(filePickerGuard.guardId ? { _filePickerGuardId: filePickerGuard.guardId } : {}) };
});
}
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index 269fac880..fb2b35381 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -562,6 +562,40 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
assert.equal(reasonAtStep2, 'workflow_needs_element_inventory');
});
+ if (build === 'firefox') {
+ test('firefox: a vision click settles the file-chooser guard and reports a blocked chooser in vision terms', async () => {
+ const agent = new Agent({});
+ agent._visionClickPreflight = async () => ({ target: { tag: 'button' } });
+ agent._visionContentInput = async () => ({ success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks: 1, target: { tag: 'button' }, _filePickerGuardId: 'fpg_test' });
+ const api = globalThis.browser;
+ const sendMessage = api.tabs.sendMessage;
+ const sent = [];
+ api.tabs.sendMessage = async (_tab, message) => {
+ sent.push(message);
+ return message.action === 'consume_file_picker_guard'
+ ? { success: false, dispatched: true, filePickerBlocked: true, selector: '#upload', error: 'Blocked a native file chooser. Call upload_file({selector: "#upload"}) directly.', settled: true }
+ : { ready: true };
+ };
+ try {
+ const blocked = await agent._visionDispatchClick(1, { x: 5, y: 5 }, { vision: { tool: 'click_at', imageId: 'f1' } });
+ assert.equal(blocked.filePickerBlocked, true, JSON.stringify(blocked));
+ assert.equal(blocked.success, false);
+ assert.equal(blocked.selector, '#upload');
+ assert.equal('_filePickerGuardId' in blocked, false);
+ assert.match(blocked.error, /not available in vision mode/);
+ assert.doesNotMatch(blocked.error, /upload_file/, 'never points to a tool a vision run does not have');
+ assert.deepEqual(sent.filter(message => message.action === 'consume_file_picker_guard').map(message => message.params.guardId), ['fpg_test']);
+ api.tabs.sendMessage = async (_tab, message) => (message.action === 'consume_file_picker_guard'
+ ? { success: true, settled: true, filePickerBlocked: false } : { ready: true });
+ const clean = await agent._visionDispatchClick(1, { x: 5, y: 5 }, { vision: { tool: 'click_at' } });
+ assert.equal(clean.success, true, JSON.stringify(clean));
+ assert.equal('_filePickerGuardId' in clean, false);
+ } finally {
+ api.tabs.sendMessage = sendMessage;
+ }
+ });
+ }
+
if (build === 'chrome') {
test('chrome: a vision click that opened no tab is known at once; a tab opened now or later is rerouted', async () => {
const agent = new Agent({});
diff --git a/test/vision-mode.mjs b/test/vision-mode.mjs
index 4aebb732d..bdd8da9ed 100644
--- a/test/vision-mode.mjs
+++ b/test/vision-mode.mjs
@@ -591,6 +591,25 @@ test('page feedback: vision points survive page churn, keyboard follow-ups ride
assert.match(feedback, /\.\.\.\(visionRun \? \{\} : \{ page: /);
});
+test('a blocked file chooser is reported in vision terms in both builds', () => {
+ const { visionFilePickerBlockedResult } = visionTools;
+ const blocked = visionFilePickerBlockedResult({ success: false, dispatched: true, filePickerBlocked: true, selector: '#f', error: 'Blocked a native file chooser. Call upload_file with selector "#f".' });
+ assert.equal(blocked.filePickerBlocked, true);
+ assert.equal(blocked.dispatched, true);
+ assert.equal(blocked.selector, '#f');
+ assert.match(blocked.error, /not available in vision mode/);
+ assert.doesNotMatch(blocked.error, /upload_file/);
+ const ok = { success: true, dispatched: true };
+ assert.equal(visionFilePickerBlockedResult(ok), ok);
+ // Chrome's CDP guard result and Firefox's settled content guard both pass through it.
+ assert.match(read('src/chrome/src/agent/agent.js'), /return visionFilePickerBlockedResult\(cdpClient\.fileInputClickBlockedResult\(blockedFileInput/);
+ assert.match(read('src/firefox/src/agent/agent.js'), /response = visionFilePickerBlockedResult\(await this\._settleContentFilePickerGuard\(tabId, response\)\);/);
+ // The content scripts' vision click runs inside the guard, identically in both builds.
+ for (const build of ['chrome', 'firefox']) {
+ assert.match(read(`src/${build}/src/content/content.js`), /const filePickerGuard = clickWithoutNativeFilePicker\(\(\) => \{\s*for \(let count = 1; count <= clicks; count\+\+\)/);
+ }
+});
+
test('micro screen changes: a caret, a clock or a spinner settle; a menu or a toast does not', () => {
const { isMicroScreenChange } = runtimeModule;
const width = 1296, height = 810;
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index 46b7a4b02..3fa60eeb3 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -181,10 +181,20 @@ test('content-script vision input (Firefox path) clicks, double-clicks, focuses,
const end = source.indexOf(' let _lastClickIdent = null;');
assert.ok(start > 0 && end > start, 'vision input block present in the Firefox content script');
const block = source.slice(start, end);
- await page.evaluate(code => {
+ // The native file-chooser guard every agent click runs inside.
+ const guardStart = source.indexOf(' function uniqueFileInputSelector(input) {');
+ const guardEnd = source.indexOf(' /**\n * Click an element by selector or coordinates.');
+ assert.ok(guardStart > 0 && guardEnd > guardStart, 'file-chooser guard block present in the Firefox content script');
+ const guardBlock = source.slice(guardStart, guardEnd);
+ await page.evaluate(({ code, guardBlock }) => {
// eslint-disable-next-line no-new-func
- window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }\n${code}\nreturn visionInput;`)();
- }, block);
+ const content = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn { visionInput, consumeFilePickerGuard };`)();
+ window.__visionInput = content.visionInput;
+ window.__consumeFilePickerGuard = content.consumeFilePickerGuard;
+ }, { code: block, guardBlock });
const run = params => page.evaluate(p => window.__visionInput(p), params);
let result = await run({ type: 'click', x: 150, y: 130 });
assert.equal(result.success, true);
@@ -216,6 +226,58 @@ test('content-script vision input (Firefox path) clicks, double-clicks, focuses,
}
});
+test('content-script vision clicks (Firefox path) never open a native file chooser', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ await page.setContent(`<!doctype html><style>body{margin:0}*{position:absolute}#upload{left:20px;top:20px;width:200px;height:30px}
+ #label{left:20px;top:80px;width:200px;height:30px;background:#eee}#pick{left:20px;top:140px;width:200px;height:30px}
+ #later{left:20px;top:200px;width:200px;height:30px}#plain{left:20px;top:260px;width:200px;height:30px}#hidden1,#hidden2,#hidden3{display:none}</style>
+ <input id="upload" type="file"><label id="label" for="hidden1">Attach</label><input id="hidden1" type="file">
+ <button id="pick">Upload</button><input id="hidden2" type="file"><button id="later">Upload later</button><input id="hidden3" type="file">
+ <button id="plain">Plain</button>
+ <script>window.choosers=[];for(const id of ['upload','hidden1','hidden2','hidden3'])document.getElementById(id).addEventListener('click',e=>{if(!e.defaultPrevented)choosers.push(id)});
+ document.getElementById('pick').addEventListener('click',()=>document.getElementById('hidden2').click());
+ document.getElementById('later').addEventListener('click',()=>setTimeout(()=>document.getElementById('hidden3').click(),100));</script>`);
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ const content = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn { visionInput, consumeFilePickerGuard };`)();
+ window.__visionInput = content.visionInput;
+ window.__consumeFilePickerGuard = content.consumeFilePickerGuard;
+ }, { code: block, guardBlock });
+ const run = params => page.evaluate(p => window.__visionInput(p), params);
+ const settle = async guardId => { await page.waitForTimeout(600); return page.evaluate(id => window.__consumeFilePickerGuard(id), guardId); };
+ // A file input, a label for one, and a button that clicks one: blocked at once.
+ for (const [y, selector] of [[35, '#upload'], [95, '#hidden1'], [155, '#hidden2']]) {
+ const result = await run({ type: 'click', x: 100, y });
+ assert.equal(result.filePickerBlocked, true, JSON.stringify(result));
+ assert.equal(result.dispatched, true);
+ assert.equal(result.selector, selector);
+ assert.equal(result._filePickerGuardId, undefined);
+ }
+ // A chooser opened a moment later is caught when the agent settles the guard.
+ const later = await run({ type: 'click', x: 100, y: 215 });
+ assert.equal(later.success, true, JSON.stringify(later));
+ assert.match(later._filePickerGuardId, /^fpg_/);
+ const settledLater = await settle(later._filePickerGuardId);
+ assert.equal(settledLater.filePickerBlocked, true, JSON.stringify(settledLater));
+ assert.equal(settledLater.selector, '#hidden3');
+ // An ordinary control settles clean.
+ const plain = await run({ type: 'click', x: 100, y: 275 });
+ assert.equal(plain.success, true);
+ assert.deepEqual(await settle(plain._filePickerGuardId), { success: true, settled: true, filePickerBlocked: false });
+ assert.deepEqual(await page.evaluate(() => window.choosers), [], 'no file input received an unblocked click');
+ } finally {
+ await browser.close();
+ }
+});
+
test('the DOM probe behind the dispatch checks reads controls, text fields, focus, selects and frames', async () => {
const { visionDomProbeExpression } = await import('../src/chrome/src/agent/vision-dom-probe.js');
const browser = await chromium.launch({ channel: 'chromium' });