Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: hand a blocked file chooser to upload_file for a few steps

Emre Sokullu4d7a42c5e91f
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 6838c6489..fb68b7998 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -13,7 +13,13 @@ Settings β†’ Advanced:
A run decides once, when it starts (`_beginVisionRun` snapshots both settings), like lean mode. The resolved state is recorded in the trace's runtime config as `vision_mode_setting` and `vision_mode`. The `Agent` class default is `off`, so embedders and unit tests opt in explicitly; `background.js` applies the stored setting, whose default is `on`.
-Three things end vision mode inside a run, all proofs that only the accessibility tree can give. A site workflow whose completion is proved against the app-owned element inventory (`requiresLedger` in `adapter-workflow.js`, form and publish jobs of some adapters) cannot be certified from screenshots: once the planner or a live adoption selects one, `resolveVisionMode` answers `workflow_needs_element_inventory`. A request for a whole conversation (a thread to summarize in full, Gmail's `read-complete-thread` job) is answered only after trusted coverage through its last page with every message expanded (`read-completeness.js`): once that requirement is armed, it answers `complete_thread_read`. WhatsApp Web and Telegram's web app confirm who an open chat is with, and bind the chat they read and send in, through the element inventory (a contact check made with `click_ax`, chat reads and sends bound to refs): on their pages it answers `native_chat_needs_element_inventory` (`_visionNativeChatPage`, from the tab's last address). Either way the next step gets the tree tools and the tree prompt (the loop rebuilds the system prompt when the vision state flips).
+Three things end vision mode inside a run for good, all proofs that only the accessibility tree can give. A site workflow whose completion is proved against the app-owned element inventory (`requiresLedger` in `adapter-workflow.js`, form and publish jobs of some adapters) cannot be certified from screenshots: once the planner or a live adoption selects one, `resolveVisionMode` answers `workflow_needs_element_inventory`. A request for a whole conversation (a thread to summarize in full, Gmail's `read-complete-thread` job) is answered only after trusted coverage through its last page with every message expanded (`read-completeness.js`): once that requirement is armed, it answers `complete_thread_read`. WhatsApp Web and Telegram's web app confirm who an open chat is with, and bind the chat they read and send in, through the element inventory (a contact check made with `click_ax`, chat reads and sends bound to refs): on their pages it answers `native_chat_needs_element_inventory` (`_visionNativeChatPage`, from the tab's last address). Either way the next step gets the tree tools and the tree prompt (the loop rebuilds the system prompt when the vision state flips).
+
+A file upload leaves vision mode for a few steps only. A native file chooser opens outside the page, where screenshots cannot show or operate it, so a `click_at` that lands on a file input is stopped by the file-chooser guard (Chrome's CDP guard, Firefox's content-script guard, on every dispatch path). The batch pipeline then arms an upload handoff (`_visionNoteUploadHandoff` in `vision-runtime.js`, kept in the run's snapshot) and rewrites the result: the next steps use the element tools, `upload_file` with the blocked input's `selector` (the compact tier's discovery call instead), and screenshots come back after the upload. Changes take effect only at step boundaries (`_visionUploadHandoffTick`, at the top of each step, before adapter notes, cookie banners and the step's tools), so the rest of the click's batch and its screen update stay in vision mode, and the upload's batch stays on the tree path. While the handoff is active `resolveVisionMode` answers `file_upload_needs_element_inventory`. It belongs to the page the tab shows when it starts (the click that armed it may itself push a route), and ends after a successful `upload_file` (the file is in the input in any build's success shape, not a discovery listing or an unconfirmed outcome), when the tab's address changes (the fragment aside), or after five decisions without an upload. A run hands off at most twice; a third chooser click is reported as before (uploads are not available in vision mode). Nested runs keep their own handoff with their snapshot.
+
+No handoff is armed when the upload could not succeed on the tree path either, and the click result says why instead: when only Chrome's protocol guard saw the chooser (`protocolOnly`: an input in a frame or a closed shadow root, which `upload_file` cannot reach), and on Chrome's compact tier when the run's message has no attachment (`_visionUploadSourceAvailable`; compact `upload_file` attaches only those).
+
+Whenever the run returns to vision after tree-path steps (an upload handoff, a ledger workflow let go, a WhatsApp page left), the step starts on a fresh frame (`_visionResumeKeyframe`, at the flip in the loop): the old frame is retired first, then a settled full frame is captured with the reason `vision resumed`. A batch planned on the tree path sends no screen update of its own even when vision is back on by its end (`_visionNoteStepTools` marks the step), so the return costs one frame. Points on images from before are `stale_image`, and the older screenshots leave the request. If the capture fails, a `retire` message tells the model so, and coordinates are refused (`no_frame`) until `take_screenshot` succeeds.
## Tools
@@ -181,7 +187,7 @@ The older MV3 quality fixtures script an accessibility-tree model (they answer w
## Limits
-- File uploads (`upload_file`) and page-resource downloads by selector are not offered in vision mode; switch it off for those tasks. A click that would open a native file chooser is blocked in both builds and reported in those terms (`visionFilePickerBlockedResult`).
+- `upload_file` is not among the vision tools: a click that would open a native file chooser is blocked in both builds and hands the next steps to the element tools for the upload (see above). The guard and `upload_file` reach the top document and its open shadow roots only, so a file input inside an iframe or a closed shadow root still cannot be uploaded to (Chrome still stops its chooser, through the protocol guard). Page-resource downloads by selector are not offered in vision mode; switch it off for those tasks.
- Closed shadow roots and canvas apps are reachable by pointer, but a native `<select>` popup is never visible in a screenshot: such a field is focused and `type_text` picks the option.
- HTML5 drag and drop: in Firefox a drag that starts on a `draggable="true"` item runs the drag-and-drop event sequence itself with a constructed `DataTransfer`; in Chrome the CDP pointer drag is left to the browser, as for `drag_drop`, so a page that relies only on `dataTransfer` may not react there.
- A drag is not shown to the submit, money or publishing checks; it is refused when it would click a control, but a custom "slide to confirm" widget is only as safe as the page makes it.
diff --git a/package.json b/package.json
index f2ebe48f9..f538fbc22 100644
--- a/package.json
+++ b/package.json
@@ -43,7 +43,7 @@
"test:webmcp": "node test/webmcp-e2e.mjs",
"test:vision": "node test/vision/run.mjs",
"test:vision:validate": "node --test test/vision/lib/score.test.mjs && node test/vision/validate.mjs",
- "test:vision-mode": "node --test test/vision-mode.mjs test/vision-mode-agent.mjs test/vision-mode-guards.mjs test/vision-pipeline-browser.mjs test/monitor-vision-wheel.mjs && node test/quality/vision-mode-mv3.mjs && node test/quality/vision-mode-mv3.mjs --dpr=2 && node test/quality/vision-mode-mv3.mjs --dpr=2 --zoom=1.5 && node test/quality/vision-mode-mv3.mjs --zoom=1.25 --scroll=100 && node test/quality/vision-mode-delete-mv3.mjs && node test/quality/vision-mode-delete-mv3.mjs --vague && node test/quality/vision-mode-repeat-mv3.mjs",
+ "test:vision-mode": "node --test test/vision-mode.mjs test/vision-mode-agent.mjs test/vision-mode-guards.mjs test/vision-pipeline-browser.mjs test/monitor-vision-wheel.mjs && node test/quality/vision-mode-mv3.mjs && node test/quality/vision-mode-mv3.mjs --dpr=2 && node test/quality/vision-mode-mv3.mjs --dpr=2 --zoom=1.5 && node test/quality/vision-mode-mv3.mjs --zoom=1.25 --scroll=100 && node test/quality/vision-mode-delete-mv3.mjs && node test/quality/vision-mode-delete-mv3.mjs --vague && node test/quality/vision-mode-repeat-mv3.mjs && node test/quality/vision-mode-upload-mv3.mjs",
"test:llm-tiny:validate": "node --test test/llm-tiny/cases.test.mjs test/llm-tiny/lib/adapters.test.mjs && node --check test/llm-tiny/run.mjs && node test/llm-tiny/run.mjs --help",
"test:anonymous": "node test/anonymous/run.mjs",
"test:ci": "node ci/test.mjs && node ci/cloud-capture.test.mjs",
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index fbb9ca348..4dccbc13f 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -13486,6 +13486,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
} catch (e) { return ''; }
}
+ // Compact upload_file attaches only a file the user attached to this run's
+ // message: without one, a chooser blocked in vision mode is not handed to
+ // the element tools (vision-runtime.js _visionNoteUploadHandoff).
+ _visionUploadSourceAvailable(tabId, promptTier) {
+ return promptTier !== 'compact' || (this._userAttachmentHandles.get(tabId)?.size || 0) > 0;
+ }
+
// WhatsApp Web and Telegram keep the tree path (vision-mode.js).
_visionNativeChatPage(tabId) {
const url = this._lastTabUrls?.get(tabId) || '';
@@ -17779,6 +17786,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
return this._pageFeedbackRecoveryResult(tabId, messages, onUpdate) || { action: 'continue' };
}
if (toolResult?.success !== false && toolResult?.noDispatch !== true && toolResult?.dispatched !== false) this._resetPageFeedbackProgress(tabId);
+ // A vision click on a file input hands the next steps to upload_file,
+ // and a successful upload hands them back (vision-runtime.js).
+ toolResult = this._visionNoteUploadHandoff(tabId, fnName, toolResult, { vision: tc._vision, promptTier });
if (jevPending) {
this._jevPendingCalls.delete(tc.id);
if (fnName !== 'get_accessibility_tree') jevPending.session.dispatched(toolResult);
@@ -57243,6 +57253,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
break;
}
+ // An upload handed to the element tools starts or ends here, before
+ // anything this step does depends on whether it runs on vision
+ // (vision-runtime.js).
+ await this._visionUploadHandoffTick(tabId, steps);
// Re-inject adapter notes if the user navigated to a different
// high-traffic site mid-conversation (no-op on the first iteration
// because _enrichUserMessageWithCurrentPage already seeded lastSeenAdapter).
@@ -57275,10 +57289,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
vision: this._visionToolOptions(tabId, mode),
});
tools = this._botMemoryRunTools(tabId, tools);
- // Vision mode can end mid-run (a site workflow that needs the element
+ // Vision mode can end and resume mid-run (a step that needs the element
// inventory, vision-mode.js); the system prompt follows the tools.
const visionPromptNow = !!this._visionToolOptions(tabId, mode);
if (visionPromptNow !== visionPromptActive && messages[0]?.role === 'system') messages[0].content = this._buildSystemPrompt(mode, tabId);
+ // Back on vision after tree-path steps: the step starts on a fresh frame,
+ // never on the one from before them.
+ if (visionPromptNow && !visionPromptActive && !selectionOnly && !standaloneChatRun) {
+ await this._visionResumeKeyframe(tabId, messages, onUpdate, { mode });
+ }
+ this._visionNoteStepTools(tabId, visionPromptNow);
visionPromptActive = visionPromptNow;
if (selectionOnly || standaloneChatRun) tools = [];
if (forceCompletionVerificationTurn) {
@@ -58784,6 +58804,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
return finish(steeringOutcome.content, steeringOutcome.status);
}
+ // An upload handed to the element tools starts or ends here, before
+ // anything this step does depends on whether it runs on vision
+ // (vision-runtime.js).
+ await this._visionUploadHandoffTick(tabId, steps);
if (steps > 0 && !selectionOnly && !standaloneChatRun) {
await this._maybeReinjectAdapter(tabId, messages);
}
@@ -58812,10 +58836,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
vision: this._visionToolOptions(tabId, mode),
});
tools = this._botMemoryRunTools(tabId, tools);
- // Vision mode can end mid-run (a site workflow that needs the element
+ // Vision mode can end and resume mid-run (a step that needs the element
// inventory, vision-mode.js); the system prompt follows the tools.
const visionPromptNow = !!this._visionToolOptions(tabId, mode);
if (visionPromptNow !== visionPromptActive && messages[0]?.role === 'system') messages[0].content = this._buildSystemPrompt(mode, tabId);
+ // Back on vision after tree-path steps: the step starts on a fresh frame,
+ // never on the one from before them.
+ if (visionPromptNow && !visionPromptActive && !selectionOnly && !standaloneChatRun) {
+ await this._visionResumeKeyframe(tabId, messages, onUpdate, { mode });
+ }
+ this._visionNoteStepTools(tabId, visionPromptNow);
visionPromptActive = visionPromptNow;
if (selectionOnly || standaloneChatRun) tools = [];
if (forceCompletionVerificationTurn) {
diff --git a/src/chrome/src/agent/vision-mode.js b/src/chrome/src/agent/vision-mode.js
index 9ca329c08..5d8fdeaaf 100644
--- a/src/chrome/src/agent/vision-mode.js
+++ b/src/chrome/src/agent/vision-mode.js
@@ -7,7 +7,10 @@
// The option was called On before, so a stored 'on' (or true)
// reads as 'auto'.
// 'off' β€” the accessibility-tree path.
-// A run decides once, when it starts, and keeps that answer to the end.
+// A run reads the setting once, when it starts, and keeps it to the end.
+// Whether vision applies is decided again before every step: a step that
+// needs the element inventory (below) runs on the tree path, and the run
+// returns to vision, on a fresh full frame, when that need has passed.
import { detectVisionProfile, normalizeVisionQuality } from './vision-budget.js';
@@ -39,6 +42,7 @@ export function resolveVisionMode(setting, {
ledgerWorkflow = false,
completeThreadRead = false,
nativeChatPage = false,
+ uploadHandoff = false,
} = {}) {
const normalized = normalizeVisionModeSetting(setting);
const base = { active: false, setting: normalized };
@@ -60,6 +64,11 @@ export function resolveVisionMode(setting, {
// check made with click_ax, chat reads and sends bound to refs): a run on
// their pages keeps the tree path.
if (nativeChatPage) return { ...base, reason: 'native_chat_needs_element_inventory' };
+ // A vision click opened a native file chooser, which screenshots cannot
+ // show or operate: the next steps attach the file with upload_file, which
+ // names the input through the element inventory (vision-runtime.js
+ // _visionNoteUploadHandoff). The run returns to vision after the upload.
+ if (uploadHandoff) return { ...base, reason: 'file_upload_needs_element_inventory' };
if (!provider) return { ...base, reason: 'no_provider' };
const canSee = rawImage == null ? provider.supportsVision === true : rawImage === true;
if (!canSee) return { ...base, reason: 'model_without_images' };
diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.js
index e8414e1d3..a9c0537a7 100644
--- a/src/chrome/src/agent/vision-runtime.js
+++ b/src/chrome/src/agent/vision-runtime.js
@@ -26,6 +26,7 @@ import {
VISION_NATIVE_TOOLS, VISION_TRANSLATED_TOOLS, VISION_NO_OBSERVE_TOOLS, VISION_TOOL_NAMES,
VISION_SCROLL_STEP_CSS_PX, VISION_DEFAULT_SCROLL_STEPS, VISION_MAX_WAIT_SECONDS,
translateVisionClick, visionKeyframeNote, visionPatchNote, visionNoChangeNote, visionZoomNote,
+ visionFilePickerBlockedResult,
} from './vision-tools.js';
import {
VISION_MIN_DRAG_CSS_PX, visionMultiClickRefusal, visionDragRefusal, visionWheelRefusal,
@@ -51,6 +52,12 @@ const STABILITY_MARGIN_PX = 12;
const SETTLE_INITIAL_DELAY_MS = 120;
const SETTLE_POLL_MS = 220;
const SETTLE_MAX_EXTRA_CAPTURES = 3;
+// An upload handoff (_visionNoteUploadHandoff) lasts this many model
+// decisions without a successful upload_file, and a run hands off at most
+// this many times: a page whose upload never succeeds goes back to vision
+// and then reports the chooser instead of handing off again.
+const UPLOAD_HANDOFF_DECISIONS = 5;
+const UPLOAD_HANDOFFS_PER_RUN = 2;
// A change this small between two captures (a blinking caret, a ticking
// clock, a spinner: a few 16 px cells) is not the page still reacting to an
// action, and right after the model looked it is not worth a new image.
@@ -88,6 +95,18 @@ function visionUserMessage(content, tag) {
return { role: 'user', content, webbrainAppOwned: true, webbrainAppOwnedKind: 'vision_frame', webbrainVisionFrame: tag };
}
+// The page an upload handoff belongs to: its address without the fragment.
+function pageKey(url) {
+ return String(url || '').split('#')[0];
+}
+
+// upload_file put the file into its input, in any build's success shape
+// (Chrome CDP, Firefox synthetic or BiDi, recipient-bound).
+function uploadAttached(result) {
+ return result?.success === true && result.discoveryOnly !== true && result.outcomeUnknown !== true
+ && result.noDispatch !== true && ['input_attached', 'page_consumed'].includes(result.attachmentState);
+}
+
function regionLabel(region) {
return { x: Math.round(region.x), y: Math.round(region.y), width: Math.round(region.width), height: Math.round(region.height) };
}
@@ -174,6 +193,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
ledgerWorkflow: this._planExecutionGuards?.get?.(tabId)?.siteWorkflow?.job?.requiresLedger === true,
completeThreadRead: this.readCompletenessStates?.get?.(tabId)?.required === true,
nativeChatPage: this._visionNativeChatPage?.(tabId) === true,
+ uploadHandoff: snapshot?.uploadHandoff?.active === true,
});
};
@@ -193,6 +213,13 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
this._visionRunSnapshots.set(tabId, {
setting: normalizeVisionModeSetting(this.visionMode, 'off'),
quality: normalizeVisionQuality(this.visionQuality),
+ // A file upload handed to the element tools (_visionNoteUploadHandoff):
+ // per run, so an enclosing run gets its own back with its snapshot.
+ uploadHandoff: null,
+ uploadHandoffs: 0,
+ // Whether the step being run was planned with the tree tools
+ // (_visionNoteStepTools).
+ treeStep: false,
});
this._visionSessions?.get(tabId)?.retireKeyframe?.();
void this._visionPlatformLateTab?.(tabId, { cancel: true })?.catch?.(() => {});
@@ -205,6 +232,95 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
void this._visionPlatformLateTab?.(tabId, { cancel: true })?.catch?.(() => {});
};
+ /**
+ * A vision click_at the file-chooser guard stopped, in either build and on
+ * any dispatch path, hands the next steps to the element tools, upload_file
+ * among them; a successful upload_file hands them back. Called once per
+ * model tool call from the batch pipeline with the final result, which it
+ * returns as the model should read it. Changes take effect at the next step
+ * (_visionUploadHandoffTick): the rest of this batch, and its screen update,
+ * run in the mode they were planned in.
+ */
+ proto._visionNoteUploadHandoff = function _visionNoteUploadHandoff(tabId, name, result, { vision = null, promptTier = '' } = {}) {
+ const snapshot = this._visionRunSnapshots?.get(tabId);
+ if (vision?.tool === 'click_at' && result?.filePickerBlocked === true) {
+ // Only Chrome's protocol guard saw the chooser: its input is in a frame
+ // or a closed shadow root, which upload_file cannot reach either. A
+ // build whose upload_file has no file to use says so (agent
+ // _visionUploadSourceAvailable).
+ const reason = result.protocolOnly === true ? 'unreachable'
+ : this._visionUploadSourceAvailable?.(tabId, promptTier) === false ? 'no_source' : '';
+ const handoff = !!snapshot && !reason && !snapshot.uploadHandoff && snapshot.uploadHandoffs < UPLOAD_HANDOFFS_PER_RUN;
+ if (handoff) {
+ snapshot.uploadHandoffs += 1;
+ snapshot.uploadHandoff = { active: false, done: false, since: 0, url: '' };
+ this._visionTraceNote(tabId, 'vision_upload_handoff', { stage: 'armed', count: snapshot.uploadHandoffs });
+ }
+ return visionFilePickerBlockedResult(result, { handoff, compact: promptTier === 'compact', reason });
+ }
+ const handoff = snapshot?.uploadHandoff;
+ if (handoff?.active && name === 'upload_file' && uploadAttached(result)) handoff.done = true;
+ return result;
+ };
+
+ /**
+ * Once per loop iteration, before anything in the step depends on vision:
+ * an armed handoff starts on the page the tab shows now (the click that
+ * armed it may itself have changed the address), and an active one ends
+ * after its upload, when that page is left, or after
+ * UPLOAD_HANDOFF_DECISIONS decisions without an upload. `step` is the
+ * loop's count of model calls, which grows only when the model decides.
+ */
+ proto._visionUploadHandoffTick = async function _visionUploadHandoffTick(tabId, step) {
+ const snapshot = this._visionRunSnapshots?.get(tabId);
+ const handoff = snapshot?.uploadHandoff;
+ if (!handoff) return;
+ if (!handoff.active) {
+ handoff.active = true;
+ handoff.since = step;
+ handoff.url = pageKey(await this._currentUrl?.(tabId));
+ return;
+ }
+ let end = handoff.done ? 'uploaded' : '';
+ if (!end && handoff.url) {
+ const url = pageKey(await this._currentUrl?.(tabId));
+ if (url && url !== handoff.url) end = 'navigated';
+ }
+ if (!end && step - handoff.since >= UPLOAD_HANDOFF_DECISIONS) end = 'expired';
+ if (end) {
+ snapshot.uploadHandoff = null;
+ this._visionTraceNote(tabId, 'vision_upload_handoff', { stage: 'ended', reason: end });
+ }
+ };
+
+ /** Records whether the step about to run was planned with the vision tools. */
+ proto._visionNoteStepTools = function _visionNoteStepTools(tabId, vision) {
+ const snapshot = this._visionRunSnapshots?.get(tabId);
+ if (snapshot) snapshot.treeStep = !vision;
+ };
+
+ /**
+ * Back on vision after steps on the tree path (an upload handoff, a page
+ * that needed the element inventory): the last frame may no longer show
+ * the screen. It is retired first, so a failed capture leaves no frame to
+ * aim at, and a settled full frame starts the step.
+ */
+ proto._visionResumeKeyframe = async function _visionResumeKeyframe(tabId, messages, onUpdate, { mode = 'act' } = {}) {
+ const state = this._visionRunState(tabId, mode);
+ if (!state.active) return null;
+ this._visionSessions?.get(tabId)?.retireKeyframe?.();
+ try {
+ const frame = await this._visionCaptureSettled(tabId, state);
+ return await this._visionPushKeyframe(tabId, messages, onUpdate, { state, frame, reason: 'vision resumed' });
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ const message = `[SCREEN: the screen could not be captured on returning to screenshots (${errorText(error).slice(0, 160)}). Earlier screenshots show an older screen and cannot be used for coordinates. Call take_screenshot to try again.]`;
+ messages.push(visionUserMessage(message, { epoch: 0, kind: 'retire' }));
+ this._visionTraceNote(tabId, 'vision_frame_error', { stage: 'resume', error: String(error?.message || error).slice(0, 200) });
+ return null;
+ }
+ };
+
/** A closed tab or cleared conversation drops its screen state. */
proto._visionForgetTab = function _visionForgetTab(tabId) {
this._visionSessions?.delete(tabId);
@@ -565,6 +681,10 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
proto._visionObserveAfterBatch = async function _visionObserveAfterBatch(tabId, messages, onUpdate, { toolCalls = [], force = false } = {}) {
const state = this._visionRunState(tabId);
if (!state.active) return null;
+ // A batch planned on the tree path gets no screen update even when it
+ // ends with vision back on (a page that needed the inventory left): the
+ // next step starts on a fresh frame instead (_visionResumeKeyframe).
+ if (!force && this._visionRunSnapshots?.get(tabId)?.treeStep === true) return null;
// Once per batch: the batch's own end and the wrapper that catches the
// batches leaving early (agent _executeToolBatch) both ask.
if (toolCalls.length) {
diff --git a/src/chrome/src/agent/vision-tools.js b/src/chrome/src/agent/vision-tools.js
index b73210903..0929228a2 100644
--- a/src/chrome/src/agent/vision-tools.js
+++ b/src/chrome/src/agent/vision-tools.js
@@ -344,14 +344,38 @@ export const VISION_SITE_NOTES_PREFACE = '(Written for WebBrain\'s element tools
/**
* A vision click the native file-chooser guard stopped. The guard's own
- * guidance points to upload_file, which a vision run does not offer.
+ * guidance points to upload_file, which the vision tools do not include.
+ * With `handoff`, the next steps run on the element tools, upload_file
+ * among them (vision-runtime.js _visionNoteUploadHandoff); `compact` is the
+ * compact tier, whose upload_file finds its own target. The input's selector
+ * stays in `selector`, never in the guidance: it is page data. Without a
+ * handoff, `reason` says why: 'unreachable' (an input upload_file cannot
+ * reach) or 'no_source' (no file this run's upload_file could attach).
*/
-export function visionFilePickerBlockedResult(result) {
+export function visionFilePickerBlockedResult(result, { handoff = false, compact = false, reason = '' } = {}) {
if (result?.filePickerBlocked !== true) return result;
+ if (!handoff) {
+ const why = reason === 'unreachable'
+ ? 'this file input is inside a frame or a closed shadow root that upload_file cannot reach. Do not click this control again; finish the rest of the task and tell the user this file has to be attached by hand.'
+ : reason === 'no_source'
+ ? 'there is no file to attach in this run (upload_file attaches only a file attached to the user\'s message). Do not click this control again; finish the rest of the task and ask the user to attach the file to their message.'
+ : 'file uploads are not available in vision mode. Do not click this control again; finish the rest of the task and tell the user the upload needs Vision mode off (Settings β†’ Advanced).';
+ return {
+ ...result,
+ success: false,
+ error: `Blocked a native file chooser: it opens outside the page, where screenshots cannot show or operate it, and ${why}`,
+ };
+ }
+ const target = compact
+ ? 'Call upload_file without targetId to list the file inputs, then again with the targetId of this one.'
+ : result.selector
+ ? 'Call upload_file with the selector in this result (`selector`).'
+ : 'Find this file input\'s exact selector with the element tools, then call upload_file with it.';
return {
...result,
success: false,
- error: 'Blocked a native file chooser: it opens outside the page, where screenshots cannot show or operate it, and file uploads are not available in vision mode. Do not click this control again; finish the rest of the task and tell the user the upload needs Vision mode off (Settings β†’ Advanced).',
+ uploadHandoff: true,
+ error: `Blocked a native file chooser: it opens outside the page, where screenshots cannot show or operate it. The next steps use the element tools instead of screenshots so the file can be attached: ${target} Do not click this control again. Screenshots and the coordinate tools come back after the upload.`,
};
}
diff --git a/src/chrome/src/cdp/cdp-client.js b/src/chrome/src/cdp/cdp-client.js
index 4e06afa13..5814d7fff 100644
--- a/src/chrome/src/cdp/cdp-client.js
+++ b/src/chrome/src/cdp/cdp-client.js
@@ -3287,9 +3287,13 @@ export class CDPClient {
timer: null,
};
state.handler = (params = {}) => {
+ // Only the protocol saw this chooser: the page-world guard covers the
+ // top document and its open shadow roots, so the input is in a frame
+ // or a closed shadow root, where upload_file cannot reach it either.
state.blocked = {
blocked: true,
selector: null,
+ protocolOnly: true,
ts: Date.now(),
...(params.backendNodeId ? { backendNodeId: params.backendNodeId } : {}),
};
@@ -3555,6 +3559,7 @@ export class CDPClient {
dispatched: true,
filePickerBlocked: true,
...(selector ? { selector } : {}),
+ ...(blocked?.protocolOnly === true ? { protocolOnly: true } : {}),
error: ['Blocked a native file chooser.', context, guidance].filter(Boolean).join(' '),
};
}
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index a8de6877b..8ba266c1e 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -15910,6 +15910,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
return this._pageFeedbackRecoveryResult(tabId, messages, onUpdate) || { action: 'continue' };
}
if (toolResult?.success !== false && toolResult?.noDispatch !== true && toolResult?.dispatched !== false) this._resetPageFeedbackProgress(tabId);
+ // A vision click on a file input hands the next steps to upload_file,
+ // and a successful upload hands them back (vision-runtime.js).
+ toolResult = this._visionNoteUploadHandoff(tabId, fnName, toolResult, { vision: tc._vision, promptTier });
if (jevPending) {
this._jevPendingCalls.delete(tc.id);
if (fnName !== 'get_accessibility_tree') jevPending.session.dispatched(toolResult);
@@ -48411,6 +48414,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
break;
}
+ // An upload handed to the element tools starts or ends here, before
+ // anything this step does depends on whether it runs on vision
+ // (vision-runtime.js).
+ await this._visionUploadHandoffTick(tabId, steps);
if (steps > 0 && !selectionOnly && !standaloneChatRun) {
await this._maybeReinjectAdapter(tabId, messages);
}
@@ -48439,10 +48446,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
vision: this._visionToolOptions(tabId, mode),
});
tools = this._botMemoryRunTools(tabId, tools);
- // Vision mode can end mid-run (a site workflow that needs the element
+ // Vision mode can end and resume mid-run (a step that needs the element
// inventory, vision-mode.js); the system prompt follows the tools.
const visionPromptNow = !!this._visionToolOptions(tabId, mode);
if (visionPromptNow !== visionPromptActive && messages[0]?.role === 'system') messages[0].content = this._buildSystemPrompt(mode, tabId);
+ // Back on vision after tree-path steps: the step starts on a fresh frame,
+ // never on the one from before them.
+ if (visionPromptNow && !visionPromptActive && !selectionOnly && !standaloneChatRun) {
+ await this._visionResumeKeyframe(tabId, messages, onUpdate, { mode });
+ }
+ this._visionNoteStepTools(tabId, visionPromptNow);
visionPromptActive = visionPromptNow;
if (selectionOnly || standaloneChatRun) tools = [];
if (forceCompletionVerificationTurn) {
@@ -49764,6 +49777,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
return finish(steeringOutcome.content, steeringOutcome.status);
}
+ // An upload handed to the element tools starts or ends here, before
+ // anything this step does depends on whether it runs on vision
+ // (vision-runtime.js).
+ await this._visionUploadHandoffTick(tabId, steps);
if (steps > 0 && !selectionOnly && !standaloneChatRun) {
await this._maybeReinjectAdapter(tabId, messages);
}
@@ -49791,10 +49808,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
vision: this._visionToolOptions(tabId, mode),
});
tools = this._botMemoryRunTools(tabId, tools);
- // Vision mode can end mid-run (a site workflow that needs the element
+ // Vision mode can end and resume mid-run (a step that needs the element
// inventory, vision-mode.js); the system prompt follows the tools.
const visionPromptNow = !!this._visionToolOptions(tabId, mode);
if (visionPromptNow !== visionPromptActive && messages[0]?.role === 'system') messages[0].content = this._buildSystemPrompt(mode, tabId);
+ // Back on vision after tree-path steps: the step starts on a fresh frame,
+ // never on the one from before them.
+ if (visionPromptNow && !visionPromptActive && !selectionOnly && !standaloneChatRun) {
+ await this._visionResumeKeyframe(tabId, messages, onUpdate, { mode });
+ }
+ this._visionNoteStepTools(tabId, visionPromptNow);
visionPromptActive = visionPromptNow;
if (selectionOnly || standaloneChatRun) tools = [];
if (forceCompletionVerificationTurn) {
diff --git a/src/firefox/src/agent/vision-mode.js b/src/firefox/src/agent/vision-mode.js
index 9ca329c08..5d8fdeaaf 100644
--- a/src/firefox/src/agent/vision-mode.js
+++ b/src/firefox/src/agent/vision-mode.js
@@ -7,7 +7,10 @@
// The option was called On before, so a stored 'on' (or true)
// reads as 'auto'.
// 'off' β€” the accessibility-tree path.
-// A run decides once, when it starts, and keeps that answer to the end.
+// A run reads the setting once, when it starts, and keeps it to the end.
+// Whether vision applies is decided again before every step: a step that
+// needs the element inventory (below) runs on the tree path, and the run
+// returns to vision, on a fresh full frame, when that need has passed.
import { detectVisionProfile, normalizeVisionQuality } from './vision-budget.js';
@@ -39,6 +42,7 @@ export function resolveVisionMode(setting, {
ledgerWorkflow = false,
completeThreadRead = false,
nativeChatPage = false,
+ uploadHandoff = false,
} = {}) {
const normalized = normalizeVisionModeSetting(setting);
const base = { active: false, setting: normalized };
@@ -60,6 +64,11 @@ export function resolveVisionMode(setting, {
// check made with click_ax, chat reads and sends bound to refs): a run on
// their pages keeps the tree path.
if (nativeChatPage) return { ...base, reason: 'native_chat_needs_element_inventory' };
+ // A vision click opened a native file chooser, which screenshots cannot
+ // show or operate: the next steps attach the file with upload_file, which
+ // names the input through the element inventory (vision-runtime.js
+ // _visionNoteUploadHandoff). The run returns to vision after the upload.
+ if (uploadHandoff) return { ...base, reason: 'file_upload_needs_element_inventory' };
if (!provider) return { ...base, reason: 'no_provider' };
const canSee = rawImage == null ? provider.supportsVision === true : rawImage === true;
if (!canSee) return { ...base, reason: 'model_without_images' };
diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.js
index e8414e1d3..a9c0537a7 100644
--- a/src/firefox/src/agent/vision-runtime.js
+++ b/src/firefox/src/agent/vision-runtime.js
@@ -26,6 +26,7 @@ import {
VISION_NATIVE_TOOLS, VISION_TRANSLATED_TOOLS, VISION_NO_OBSERVE_TOOLS, VISION_TOOL_NAMES,
VISION_SCROLL_STEP_CSS_PX, VISION_DEFAULT_SCROLL_STEPS, VISION_MAX_WAIT_SECONDS,
translateVisionClick, visionKeyframeNote, visionPatchNote, visionNoChangeNote, visionZoomNote,
+ visionFilePickerBlockedResult,
} from './vision-tools.js';
import {
VISION_MIN_DRAG_CSS_PX, visionMultiClickRefusal, visionDragRefusal, visionWheelRefusal,
@@ -51,6 +52,12 @@ const STABILITY_MARGIN_PX = 12;
const SETTLE_INITIAL_DELAY_MS = 120;
const SETTLE_POLL_MS = 220;
const SETTLE_MAX_EXTRA_CAPTURES = 3;
+// An upload handoff (_visionNoteUploadHandoff) lasts this many model
+// decisions without a successful upload_file, and a run hands off at most
+// this many times: a page whose upload never succeeds goes back to vision
+// and then reports the chooser instead of handing off again.
+const UPLOAD_HANDOFF_DECISIONS = 5;
+const UPLOAD_HANDOFFS_PER_RUN = 2;
// A change this small between two captures (a blinking caret, a ticking
// clock, a spinner: a few 16 px cells) is not the page still reacting to an
// action, and right after the model looked it is not worth a new image.
@@ -88,6 +95,18 @@ function visionUserMessage(content, tag) {
return { role: 'user', content, webbrainAppOwned: true, webbrainAppOwnedKind: 'vision_frame', webbrainVisionFrame: tag };
}
+// The page an upload handoff belongs to: its address without the fragment.
+function pageKey(url) {
+ return String(url || '').split('#')[0];
+}
+
+// upload_file put the file into its input, in any build's success shape
+// (Chrome CDP, Firefox synthetic or BiDi, recipient-bound).
+function uploadAttached(result) {
+ return result?.success === true && result.discoveryOnly !== true && result.outcomeUnknown !== true
+ && result.noDispatch !== true && ['input_attached', 'page_consumed'].includes(result.attachmentState);
+}
+
function regionLabel(region) {
return { x: Math.round(region.x), y: Math.round(region.y), width: Math.round(region.width), height: Math.round(region.height) };
}
@@ -174,6 +193,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
ledgerWorkflow: this._planExecutionGuards?.get?.(tabId)?.siteWorkflow?.job?.requiresLedger === true,
completeThreadRead: this.readCompletenessStates?.get?.(tabId)?.required === true,
nativeChatPage: this._visionNativeChatPage?.(tabId) === true,
+ uploadHandoff: snapshot?.uploadHandoff?.active === true,
});
};
@@ -193,6 +213,13 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
this._visionRunSnapshots.set(tabId, {
setting: normalizeVisionModeSetting(this.visionMode, 'off'),
quality: normalizeVisionQuality(this.visionQuality),
+ // A file upload handed to the element tools (_visionNoteUploadHandoff):
+ // per run, so an enclosing run gets its own back with its snapshot.
+ uploadHandoff: null,
+ uploadHandoffs: 0,
+ // Whether the step being run was planned with the tree tools
+ // (_visionNoteStepTools).
+ treeStep: false,
});
this._visionSessions?.get(tabId)?.retireKeyframe?.();
void this._visionPlatformLateTab?.(tabId, { cancel: true })?.catch?.(() => {});
@@ -205,6 +232,95 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
void this._visionPlatformLateTab?.(tabId, { cancel: true })?.catch?.(() => {});
};
+ /**
+ * A vision click_at the file-chooser guard stopped, in either build and on
+ * any dispatch path, hands the next steps to the element tools, upload_file
+ * among them; a successful upload_file hands them back. Called once per
+ * model tool call from the batch pipeline with the final result, which it
+ * returns as the model should read it. Changes take effect at the next step
+ * (_visionUploadHandoffTick): the rest of this batch, and its screen update,
+ * run in the mode they were planned in.
+ */
+ proto._visionNoteUploadHandoff = function _visionNoteUploadHandoff(tabId, name, result, { vision = null, promptTier = '' } = {}) {
+ const snapshot = this._visionRunSnapshots?.get(tabId);
+ if (vision?.tool === 'click_at' && result?.filePickerBlocked === true) {
+ // Only Chrome's protocol guard saw the chooser: its input is in a frame
+ // or a closed shadow root, which upload_file cannot reach either. A
+ // build whose upload_file has no file to use says so (agent
+ // _visionUploadSourceAvailable).
+ const reason = result.protocolOnly === true ? 'unreachable'
+ : this._visionUploadSourceAvailable?.(tabId, promptTier) === false ? 'no_source' : '';
+ const handoff = !!snapshot && !reason && !snapshot.uploadHandoff && snapshot.uploadHandoffs < UPLOAD_HANDOFFS_PER_RUN;
+ if (handoff) {
+ snapshot.uploadHandoffs += 1;
+ snapshot.uploadHandoff = { active: false, done: false, since: 0, url: '' };
+ this._visionTraceNote(tabId, 'vision_upload_handoff', { stage: 'armed', count: snapshot.uploadHandoffs });
+ }
+ return visionFilePickerBlockedResult(result, { handoff, compact: promptTier === 'compact', reason });
+ }
+ const handoff = snapshot?.uploadHandoff;
+ if (handoff?.active && name === 'upload_file' && uploadAttached(result)) handoff.done = true;
+ return result;
+ };
+
+ /**
+ * Once per loop iteration, before anything in the step depends on vision:
+ * an armed handoff starts on the page the tab shows now (the click that
+ * armed it may itself have changed the address), and an active one ends
+ * after its upload, when that page is left, or after
+ * UPLOAD_HANDOFF_DECISIONS decisions without an upload. `step` is the
+ * loop's count of model calls, which grows only when the model decides.
+ */
+ proto._visionUploadHandoffTick = async function _visionUploadHandoffTick(tabId, step) {
+ const snapshot = this._visionRunSnapshots?.get(tabId);
+ const handoff = snapshot?.uploadHandoff;
+ if (!handoff) return;
+ if (!handoff.active) {
+ handoff.active = true;
+ handoff.since = step;
+ handoff.url = pageKey(await this._currentUrl?.(tabId));
+ return;
+ }
+ let end = handoff.done ? 'uploaded' : '';
+ if (!end && handoff.url) {
+ const url = pageKey(await this._currentUrl?.(tabId));
+ if (url && url !== handoff.url) end = 'navigated';
+ }
+ if (!end && step - handoff.since >= UPLOAD_HANDOFF_DECISIONS) end = 'expired';
+ if (end) {
+ snapshot.uploadHandoff = null;
+ this._visionTraceNote(tabId, 'vision_upload_handoff', { stage: 'ended', reason: end });
+ }
+ };
+
+ /** Records whether the step about to run was planned with the vision tools. */
+ proto._visionNoteStepTools = function _visionNoteStepTools(tabId, vision) {
+ const snapshot = this._visionRunSnapshots?.get(tabId);
+ if (snapshot) snapshot.treeStep = !vision;
+ };
+
+ /**
+ * Back on vision after steps on the tree path (an upload handoff, a page
+ * that needed the element inventory): the last frame may no longer show
+ * the screen. It is retired first, so a failed capture leaves no frame to
+ * aim at, and a settled full frame starts the step.
+ */
+ proto._visionResumeKeyframe = async function _visionResumeKeyframe(tabId, messages, onUpdate, { mode = 'act' } = {}) {
+ const state = this._visionRunState(tabId, mode);
+ if (!state.active) return null;
+ this._visionSessions?.get(tabId)?.retireKeyframe?.();
+ try {
+ const frame = await this._visionCaptureSettled(tabId, state);
+ return await this._visionPushKeyframe(tabId, messages, onUpdate, { state, frame, reason: 'vision resumed' });
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ const message = `[SCREEN: the screen could not be captured on returning to screenshots (${errorText(error).slice(0, 160)}). Earlier screenshots show an older screen and cannot be used for coordinates. Call take_screenshot to try again.]`;
+ messages.push(visionUserMessage(message, { epoch: 0, kind: 'retire' }));
+ this._visionTraceNote(tabId, 'vision_frame_error', { stage: 'resume', error: String(error?.message || error).slice(0, 200) });
+ return null;
+ }
+ };
+
/** A closed tab or cleared conversation drops its screen state. */
proto._visionForgetTab = function _visionForgetTab(tabId) {
this._visionSessions?.delete(tabId);
@@ -565,6 +681,10 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
proto._visionObserveAfterBatch = async function _visionObserveAfterBatch(tabId, messages, onUpdate, { toolCalls = [], force = false } = {}) {
const state = this._visionRunState(tabId);
if (!state.active) return null;
+ // A batch planned on the tree path gets no screen update even when it
+ // ends with vision back on (a page that needed the inventory left): the
+ // next step starts on a fresh frame instead (_visionResumeKeyframe).
+ if (!force && this._visionRunSnapshots?.get(tabId)?.treeStep === true) return null;
// Once per batch: the batch's own end and the wrapper that catches the
// batches leaving early (agent _executeToolBatch) both ask.
if (toolCalls.length) {
diff --git a/src/firefox/src/agent/vision-tools.js b/src/firefox/src/agent/vision-tools.js
index b73210903..0929228a2 100644
--- a/src/firefox/src/agent/vision-tools.js
+++ b/src/firefox/src/agent/vision-tools.js
@@ -344,14 +344,38 @@ export const VISION_SITE_NOTES_PREFACE = '(Written for WebBrain\'s element tools
/**
* A vision click the native file-chooser guard stopped. The guard's own
- * guidance points to upload_file, which a vision run does not offer.
+ * guidance points to upload_file, which the vision tools do not include.
+ * With `handoff`, the next steps run on the element tools, upload_file
+ * among them (vision-runtime.js _visionNoteUploadHandoff); `compact` is the
+ * compact tier, whose upload_file finds its own target. The input's selector
+ * stays in `selector`, never in the guidance: it is page data. Without a
+ * handoff, `reason` says why: 'unreachable' (an input upload_file cannot
+ * reach) or 'no_source' (no file this run's upload_file could attach).
*/
-export function visionFilePickerBlockedResult(result) {
+export function visionFilePickerBlockedResult(result, { handoff = false, compact = false, reason = '' } = {}) {
if (result?.filePickerBlocked !== true) return result;
+ if (!handoff) {
+ const why = reason === 'unreachable'
+ ? 'this file input is inside a frame or a closed shadow root that upload_file cannot reach. Do not click this control again; finish the rest of the task and tell the user this file has to be attached by hand.'
+ : reason === 'no_source'
+ ? 'there is no file to attach in this run (upload_file attaches only a file attached to the user\'s message). Do not click this control again; finish the rest of the task and ask the user to attach the file to their message.'
+ : 'file uploads are not available in vision mode. Do not click this control again; finish the rest of the task and tell the user the upload needs Vision mode off (Settings β†’ Advanced).';
+ return {
+ ...result,
+ success: false,
+ error: `Blocked a native file chooser: it opens outside the page, where screenshots cannot show or operate it, and ${why}`,
+ };
+ }
+ const target = compact
+ ? 'Call upload_file without targetId to list the file inputs, then again with the targetId of this one.'
+ : result.selector
+ ? 'Call upload_file with the selector in this result (`selector`).'
+ : 'Find this file input\'s exact selector with the element tools, then call upload_file with it.';
return {
...result,
success: false,
- error: 'Blocked a native file chooser: it opens outside the page, where screenshots cannot show or operate it, and file uploads are not available in vision mode. Do not click this control again; finish the rest of the task and tell the user the upload needs Vision mode off (Settings β†’ Advanced).',
+ uploadHandoff: true,
+ error: `Blocked a native file chooser: it opens outside the page, where screenshots cannot show or operate it. The next steps use the element tools instead of screenshots so the file can be attached: ${target} Do not click this control again. Screenshots and the coordinate tools come back after the upload.`,
};
}
diff --git a/test/quality/vision-mode-upload-mv3.mjs b/test/quality/vision-mode-upload-mv3.mjs
new file mode 100644
index 000000000..6bc066040
--- /dev/null
+++ b/test/quality/vision-mode-upload-mv3.mjs
@@ -0,0 +1,163 @@
+import assert from 'node:assert/strict';
+import crypto from 'node:crypto';
+import fs from 'node:fs/promises';
+import http from 'node:http';
+import os from 'node:os';
+import path from 'node:path';
+import { chromium } from 'playwright';
+import { fixtureFixedChoiceReply } from '../semantic-decision-stub.mjs';
+
+// A file upload in a vision run, end to end in the installed MV3 extension:
+// background, Agent, CDP capture, trusted input and the file-chooser guard are
+// real; a loopback provider plays the model. Its click_at lands on a native
+// file input, the guard stops the chooser, the next step gets the element
+// tools and attaches the user's video with upload_file, and the step after
+// that is back on screenshots, on a fresh full frame.
+// node test/quality/vision-mode-upload-mv3.mjs
+const out = path.resolve(process.env.WEBBRAIN_QUALITY_ARTIFACTS || 'ci/artifacts/vision-mode-upload-mv3');
+await fs.mkdir(out, { recursive: true });
+
+const mp4 = Buffer.concat([Buffer.from([0, 0, 0, 24]), Buffer.from('ftypmp42'), Buffer.alloc(4), Buffer.from('mp42isom')]);
+const sha = bytes => crypto.createHash('sha256').update(bytes).digest('hex');
+const LAYOUT = { upload: { x: 60, y: 140, w: 360, h: 48 } };
+const body = `<!doctype html><meta charset="utf-8"><title>Vision upload fixture</title>
+<style>body{margin:0;font:16px sans-serif}h1{position:absolute;left:60px;top:40px;margin:0}label{position:absolute;left:60px;top:112px}
+#upload{position:absolute;left:${LAYOUT.upload.x}px;top:${LAYOUT.upload.y}px;width:${LAYOUT.upload.w}px;height:${LAYOUT.upload.h}px;box-sizing:border-box;font:16px sans-serif;background:#eef}
+#status{position:absolute;left:60px;top:230px;width:500px;height:120px;font:20px sans-serif}</style>
+<h1>Video upload</h1><label for="upload">QA video</label><input id="upload" name="video" type="file" accept="video/mp4,.mp4"><div id="status" role="status"></div>
+<script>window.nativeAttachments = [];
+document.getElementById('upload').addEventListener('change', async event => {
+ const file = event.target.files[0];
+ const digest = await crypto.subtle.digest('SHA-256', await file.arrayBuffer());
+ window.nativeAttachments.push({ name: file.name, size: file.size, mime: file.type, sha256: Array.from(new Uint8Array(digest), x => x.toString(16).padStart(2, '0')).join('') });
+ const status = document.getElementById('status');
+ status.textContent = 'Attached ' + file.name;
+ status.style.background = 'rgb(0,160,0)';
+});</script>`;
+
+const providerCalls = [];
+let actionCalls = 0;
+const labelOf = message => (Array.isArray(message.content) ? message.content.filter(block => block.type === 'text').map(block => block.text).join('\n') : '');
+const centre = (rect, frame) => ({ x: Math.round((rect.x + rect.w / 2) / frame.ratio), y: Math.round((rect.y + rect.h / 2) / frame.ratio) });
+
+const server = http.createServer(async (request, response) => {
+ if (request.url === '/v1/chat/completions') {
+ let text = ''; for await (const chunk of request) text += chunk;
+ const input = JSON.parse(text); providerCalls.push(input);
+ const isAction = Array.isArray(input.tools) && input.tools.length > 0;
+ const judge = String(input.messages?.[0]?.content || '').startsWith('Evaluate only the observable requirements of the genuine user task');
+ const fixedChoice = fixtureFixedChoiceReply(input.messages);
+ let completion;
+ if (fixedChoice) completion = { role: 'assistant', content: fixedChoice };
+ else if (judge) completion = { role: 'assistant', content: JSON.stringify({ outcome: 'uncertain', reason: 'Scripted fixture.' }) };
+ else if (!isAction) {
+ completion = { role: 'assistant', content: JSON.stringify({
+ request_kind: 'execute', scope_relation: 'new', deliverables: ['Attach the uploaded video to the page input'],
+ requires_state_change: true, requires_submission: false, messaging: null,
+ completion_requirements: { download: false }, allows_planner_shaped_result: false, allows_app_state_tool_evidence: false,
+ read_scope: 'visible_page', summary: 'Attach the uploaded video clip to the QA video input.',
+ steps: [{ id: '1', action: 'Attach the video to the QA video input.' }],
+ memory: { use_progress_ledger: false, progress_action: null }, scheduling: null, risks: [],
+ }) };
+ } else {
+ const offered = input.tools.map(tool => tool.function.name);
+ // The current full frame, as a model reading its label would size it.
+ const frameMessage = input.messages.findLast(message => /UNTRUSTED SCREENSHOT f\d+/.test(labelOf(message)) && message.content.some(block => block.type === 'image_url'));
+ const declared = frameMessage && /(\d+)x(\d+) image; 1 image px = ([\d.]+) CSS px/.exec(labelOf(frameMessage));
+ const frame = declared ? { width: Number(declared[1]), height: Number(declared[2]), ratio: Number(declared[3]) } : null;
+ const results = new Map(input.messages.filter(message => message.role === 'tool').map(message => [message.tool_call_id, String(message.content)]));
+ const pairs = input.messages.filter(message => message.role === 'assistant').flatMap(message => message.tool_calls || [])
+ .map(item => ({ name: item.function.name, text: results.get(item.id) || '' }));
+ const blocked = pairs.find(pair => pair.name === 'click_at' && /"filePickerBlocked":true/.test(pair.text));
+ const uploaded = pairs.some(pair => pair.name === 'upload_file' && /"success":true/.test(pair.text));
+ const attachmentId = JSON.stringify(input.messages).match(/attachment_[A-Za-z0-9]+_\d+(?= \(clip\.mp4\))/)?.[0];
+ let call;
+ if (offered.includes('upload_file') && blocked && !uploaded) {
+ call = { name: 'upload_file', args: { selector: /"selector":"((?:[^"\\]|\\.)+)"/.exec(blocked.text)?.[1]?.replace(/\\"/g, '"') || '#upload', attachmentId } };
+ } else if (offered.includes('click_at') && !blocked && frame) call = { name: 'click_at', args: centre(LAYOUT.upload, frame) };
+ else if (offered.includes('take_screenshot') && !frame) call = { name: 'take_screenshot', args: {} };
+ else call = { name: 'done', args: { summary: 'The clip was attached to the QA video input.', outcome: 'success' } };
+ const step = actionCalls++;
+ completion = { role: 'assistant', content: '', tool_calls: [{ id: `vision-upload-${step}`, type: 'function', function: { name: call.name, arguments: JSON.stringify(call.args) } }] };
+ }
+ response.writeHead(200, { 'Content-Type': 'application/json' });
+ response.end(JSON.stringify({ id: `completion-${providerCalls.length}`, choices: [{ message: completion, finish_reason: isAction ? 'tool_calls' : 'stop' }], usage: { prompt_tokens: 100, completion_tokens: 20 } }));
+ return;
+ }
+ response.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
+ response.end(body);
+});
+await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
+const base = `http://127.0.0.1:${server.address().port}`;
+const url = `${base}/upload`;
+const profile = await fs.mkdtemp(path.join(os.tmpdir(), 'webbrain-vision-upload-mv3-'));
+let context;
+try {
+ const extension = path.resolve('src/chrome');
+ context = await chromium.launchPersistentContext(profile, {
+ headless: true, channel: 'chromium', viewport: null,
+ args: [`--disable-extensions-except=${extension}`, `--load-extension=${extension}`, '--window-size=1280,900'],
+ });
+ const worker = context.serviceWorkers()[0] || await context.waitForEvent('serviceworker', { timeout: 30000 });
+ const id = new URL(worker.url()).host;
+ const harness = await context.newPage(); await harness.goto(`chrome-extension://${id}/src/ui/settings.html`);
+ const message = (action, data = {}) => harness.evaluate(({ action, data }) => chrome.runtime.sendMessage({ target: 'background', action, ...data }), { action, data });
+ await harness.evaluate(async () => {
+ window.events = [];
+ chrome.runtime.onMessage.addListener(event => { if (event.action === 'agent_update') events.push({ type: event.type, data: event.data }); });
+ // visionMode is deliberately left unset: vision mode is the default.
+ await chrome.storage.local.set({ helpImproveWebBrain: false, planBeforeActMode: 'off', planBeforeAct: false, planReviewMode: 'never', screenshotFallback: false, autoScreenshot: 'navigation', maxAgentSteps: 12, systemOneEnabled: false, tracingEnabled: false, agentAllowLocalNetwork: true, askBeforeConsequentialActions: false, openaiAskStreamingEnabled: false, leanMode: 'on' });
+ });
+ assert.equal((await message('update_provider', { providerId: 'openrouter', config: { apiKey: 'owned-fixture-fake-key', baseUrl: `${base}/v1`, model: 'anthropic/claude-sonnet-4.5', contextWindow: 200000, enabled: true, supportsAskStreaming: false, supportsVision: true } })).ok, true);
+ assert.equal((await message('set_active_provider', { providerId: 'openrouter' })).ok, true);
+ const page = await context.newPage(); await page.goto(url);
+ const tabId = await harness.evaluate(async url => (await chrome.tabs.query({})).find(tab => tab.url === url)?.id, url);
+ assert.ok(Number.isInteger(tabId), 'the fixture tab is known');
+ const result = await message('chat', {
+ tabId, text: 'Attach the clip I uploaded to the QA video input.', mode: 'act', requestId: 'vision-upload-mv3', locale: 'en',
+ attachments: [{ kind: 'video', name: 'clip.mp4', dataUrl: `data:video/mp4;base64,${mp4.toString('base64')}`, mimeType: 'video/mp4', size: mp4.length }],
+ });
+ const events = await harness.evaluate(() => events);
+ const nativeAttachments = await page.evaluate(() => window.nativeAttachments);
+ await fs.writeFile(path.join(out, 'evidence.json'), JSON.stringify({ result, events, nativeAttachments, providerCalls },
+ (_key, item) => typeof item === 'string' && item.startsWith('data:image/') ? `[image ${item.length} chars]` : item, 2));
+
+ const actionRequests = providerCalls.filter(call => Array.isArray(call.tools) && call.tools.length);
+ const offered = actionRequests.map(call => call.tools.map(tool => tool.function.name));
+ const toolTexts = request => request.messages.filter(item => item.role === 'tool').map(item => String(item.content));
+ const summary = JSON.stringify({ offered: offered.map(names => (names.includes('click_at') ? 'vision' : names.includes('upload_file') ? 'tree' : names.join(','))), tools: (result.updates || []).filter(event => event.type === 'tool_result').map(event => ({ name: event.data?.name, result: JSON.stringify(event.data?.result).slice(0, 300) })) }, null, 1);
+ assert.ok(offered[0]?.includes('click_at') && !offered[0].includes('upload_file'), `the run starts on screenshots: ${summary}`);
+ const handoffAt = offered.findIndex(names => names.includes('upload_file'));
+ assert.ok(handoffAt > 0, `a later step offers upload_file: ${summary}`);
+ assert.ok(!offered[handoffAt].includes('click_at'), 'the handoff step is on the element tools');
+ // The real CDP click reached the file input and the guard stopped its chooser.
+ const blocked = toolTexts(actionRequests[handoffAt]).find(text => /"filePickerBlocked":true/.test(text)) || '';
+ assert.match(blocked, /"uploadHandoff":true/, blocked);
+ assert.match(blocked, /"dispatched":true/, blocked);
+ assert.match(blocked, /"selector":"[^"]+"/, `the guard named the input: ${blocked}`);
+ const uploads = (result.updates || []).filter(event => event.type === 'tool_result' && event.data?.name === 'upload_file');
+ assert.equal(uploads.length, 1, summary);
+ assert.equal(uploads[0].data.result.success, true, JSON.stringify(uploads[0].data.result));
+ assert.deepEqual(nativeAttachments, [{ name: 'clip.mp4', size: mp4.length, mime: 'video/mp4', sha256: sha(mp4) }], 'the page input received the exact bytes once');
+ // The step after the upload is back on screenshots, on a fresh full frame.
+ const backAt = offered.findIndex((names, index) => index > handoffAt && names.includes('click_at'));
+ assert.ok(backAt > handoffAt, `vision resumed after the upload: ${summary}`);
+ assert.ok(!offered[backAt].includes('upload_file'));
+ const back = actionRequests[backAt];
+ const frameAt = back.messages.findLastIndex(item => /UNTRUSTED SCREENSHOT f\d+/.test(labelOf(item)) && item.content.some(block => block.type === 'image_url'));
+ assert.ok(frameAt >= 0, 'the resumed step has a full frame');
+ const frameLabel = labelOf(back.messages[frameAt]);
+ assert.match(frameLabel, /\(vision resumed\)/, frameLabel);
+ assert.notEqual(/SCREENSHOT (f\d+)/.exec(frameLabel)[1], 'f1');
+ assert.ok(back.messages.slice(frameAt + 1).every(item => !/UNTRUSTED SCREEN/.test(labelOf(item))), 'the fresh frame is the latest screen');
+ const olderImages = back.messages.slice(0, frameAt).flatMap(item => (Array.isArray(item.content) ? item.content : [])).filter(block => block.type === 'image_url').length;
+ assert.equal(olderImages, 0, 'screenshots from before the handoff left the request');
+ assert.equal(events.findLast(event => event.type === 'run_complete')?.data?.status, 'completed', summary);
+ assert.equal(JSON.stringify(providerCalls).includes(mp4.toString('base64')), false, 'the video bytes never reach the model');
+ console.log(JSON.stringify({ passed: true, offered: offered.map(names => (names.includes('click_at') ? 'vision' : 'tree')), frame: /SCREENSHOT (f\d+)/.exec(frameLabel)[1], attachment: nativeAttachments[0] }));
+} finally {
+ await context?.close(); await new Promise(resolve => server.close(resolve));
+ const resolved = path.resolve(profile);
+ if (!resolved.startsWith(path.resolve(os.tmpdir()) + path.sep + 'webbrain-vision-upload-mv3-')) throw new Error('Unexpected temporary profile path.');
+ await fs.rm(resolved, { recursive: true, force: true });
+}
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index 47c34c3ab..f72f3af64 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -7,6 +7,7 @@
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { withoutFixedChoiceAnswers } from './semantic-decision-stub.mjs';
+import { cdpClient } from '../src/chrome/src/cdp/cdp-client.js';
const PAGE = 'https://owned.example.test/form';
const area = { get: async () => ({}), set: async () => {}, remove: async () => {} };
@@ -658,6 +659,186 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
assert.equal(reasonAtStep2, 'workflow_needs_element_inventory');
});
+ test(`${build}: back on vision after tree-path steps, the next decision starts on a fresh full frame`, async () => {
+ let tab;
+ const ledger = { adapterName: 'Forms', revision: 1, job: { id: 'fill-form', template: 'form', stages: ['fill'], successEvidence: ['saved'], partialEvidence: ['not saved'], requiresLedger: true } };
+ const ctx = runAgent(step => {
+ if (step === 1) {
+ ctx.agent._planExecutionGuards.set(tab, { ...(ctx.agent._planExecutionGuards.get(tab) || {}), siteWorkflow: ledger });
+ return { toolCalls: [call('take_screenshot', {})] };
+ }
+ if (step === 2) {
+ // The tree step changes the page, and the workflow is let go.
+ ctx.scene.rects.push({ x: 500, y: 400, w: 200, h: 80, color: [0, 160, 0] });
+ ctx.agent._planExecutionGuards.set(tab, { ...ctx.agent._planExecutionGuards.get(tab), siteWorkflow: null });
+ return { toolCalls: [call('find_text', { text: 'Saved' })] };
+ }
+ if (step === 3) return { toolCalls: [call('done', { summary: 'Done.', outcome: 'success' })] };
+ return { content: 'done' };
+ });
+ tab = nextTab++;
+ await runLoop(ctx.agent, tab, false);
+ const [, tree, back] = ctx.requests;
+ assert.ok(!tree.tools.includes('click_at'));
+ assert.ok(back.tools.includes('click_at'), back.tools.join(','));
+ assert.match(back.system, /VISION CONTROL/);
+ const latest = visionMessages(back.messages).at(-1);
+ assert.equal(latest.webbrainVisionFrame.kind, 'key', textOf(latest));
+ assert.match(textOf(latest), /\(vision resumed\)/);
+ // The tree batch that let the workflow go sent no screen update of its
+ // own: one frame for the return, not a patch and then a frame.
+ assert.deepEqual(visionMessages(back.messages).map(message => message.webbrainVisionFrame.kind), ['key', 'key']);
+ const session = ctx.agent._visionSessions.get(tab);
+ assert.equal(session.keyframe.id, latest.webbrainVisionFrame.ids[0]);
+ for (const old of visionMessages(tree.messages).flatMap(message => message.webbrainVisionFrame.ids || [])) {
+ assert.equal(session.resolvePoint({ x: 1, y: 1, image: old }).code, 'stale_image', old);
+ }
+ assert.equal(totalImages(back), 1, 'only the fresh frame is sent');
+ });
+
+ // A vision click_at that lands on a native file input: the real per-build
+ // dispatch runs with the file-chooser guard stubbed to report a block.
+ function blockChooser(ctx) {
+ const { agent } = ctx;
+ agent._visionClickPreflight = async () => ({ target: { tag: 'input' } });
+ agent._visionMutationBindingBlock = async () => null;
+ agent._redirectTargetBlankClick = async () => null;
+ agent._visionPlatformLateTab = async () => null;
+ agent._showAgentTarget = () => {};
+ let restore = () => {};
+ if (build === 'chrome') {
+ const saved = { attach: cdpClient.attach, sendCommand: cdpClient.sendCommand, armFileInputClickGuard: cdpClient.armFileInputClickGuard, consumeFileInputClickGuard: cdpClient.consumeFileInputClickGuard };
+ Object.assign(cdpClient, { attach: async () => {}, sendCommand: async () => ({}), armFileInputClickGuard: async () => {},
+ consumeFileInputClickGuard: async () => ({ blocked: true, selector: '#upload', ts: 1 }) });
+ restore = () => Object.assign(cdpClient, saved);
+ } else {
+ agent._visionContentInput = async () => ({ success: true, dispatched: true, method: 'firefox-bidi', target: { tag: 'input' }, _filePickerGuardId: 'fpg_test' });
+ agent._settleContentFilePickerGuard = async () => ({ success: false, dispatched: true, filePickerBlocked: true, selector: '#upload', error: 'Blocked a native file chooser. Call upload_file({selector: "#upload"}) directly.' });
+ }
+ const base = agent._executeToolWithLifecycle;
+ agent._executeToolWithLifecycle = async function (tab, name, args, onUpdate, context) {
+ if (name === 'click' && context?._vision?.tool === 'click_at') {
+ ctx.dispatched.push({ name, args, vision: context._vision });
+ return this._visionDispatchClick(tab, args, { vision: context._vision });
+ }
+ if (name === 'upload_file') {
+ ctx.dispatched.push({ name, args, vision: null });
+ return ctx.upload(args);
+ }
+ return base.call(this, tab, name, args, onUpdate, context);
+ };
+ return restore;
+ }
+ const attached = args => ({ success: true, attached: { name: 'clip.mp4', size: 4 }, file: 'clip.mp4', verified: false,
+ attachmentState: 'input_attached', remoteStateVerified: false, uploadTarget: { selector: args.selector } });
+ const blockedText = request => toolTexts(request).find(text => /filePickerBlocked/.test(text)) || '';
+
+ for (const streaming of [false, true]) {
+ const label = `${build}${streaming ? ' (streaming)' : ''}`;
+
+ test(`${label}: a click_at on a file input hands the next step to upload_file, then vision resumes on a fresh full frame`, async () => {
+ let tab;
+ const reasons = [];
+ const ctx = runAgent(step => {
+ reasons.push(ctx.agent._visionRunState(tab, 'act').reason);
+ if (step === 1) return { toolCalls: [call('click_at', { x: 300, y: 200 })] };
+ if (step === 2) {
+ // The upload shows the file on the page.
+ ctx.scene.rects.push({ x: 600, y: 420, w: 200, h: 40, color: [0, 160, 0] });
+ return { toolCalls: [call('upload_file', { selector: '#upload', attachmentId: 'attachment_t_1' })] };
+ }
+ if (step === 3) return { toolCalls: [call('done', { summary: 'Attached.', outcome: 'success' })] };
+ return { content: 'done' };
+ });
+ ctx.upload = attached;
+ const restore = blockChooser(ctx);
+ tab = nextTab++;
+ try { await runLoop(ctx.agent, tab, streaming); } finally { restore(); }
+ const [first, handoff, back] = ctx.requests;
+ assert.ok(first.tools.includes('click_at') && !first.tools.includes('upload_file'));
+ assert.ok(handoff.tools.includes('upload_file'), handoff.tools.join(','));
+ assert.ok(!handoff.tools.includes('click_at'));
+ assert.doesNotMatch(handoff.system, /VISION CONTROL/);
+ assert.deepEqual(reasons, ['vision_capable_model', 'file_upload_needs_element_inventory', 'vision_capable_model']);
+ const blocked = blockedText(handoff);
+ assert.match(blocked, /"uploadHandoff":true/, blocked);
+ assert.match(blocked, /The next steps use the element tools/);
+ assert.match(blocked, /selector in this result/);
+ assert.match(blocked, /"selector":"#upload"/);
+ assert.doesNotMatch(blocked, /not available in vision mode/);
+ assert.deepEqual(ctx.dispatched.filter(entry => entry.name === 'upload_file').map(entry => entry.args.selector), ['#upload']);
+ assert.ok(back.tools.includes('click_at') && !back.tools.includes('upload_file'), back.tools.join(','));
+ assert.match(back.system, /VISION CONTROL/);
+ const latest = visionMessages(back.messages).at(-1);
+ assert.equal(latest.webbrainVisionFrame.kind, 'key', textOf(latest));
+ assert.match(textOf(latest), /\(vision resumed\)/);
+ assert.notEqual(latest.webbrainVisionFrame.ids[0], visionMessages(first.messages).at(-1).webbrainVisionFrame.ids[0]);
+ assert.equal(totalImages(back), 1, 'only the fresh frame is sent');
+ const session = ctx.agent._visionSessions.get(tab);
+ assert.equal(session.resolvePoint({ x: 1, y: 1, image: visionMessages(first.messages)[0].webbrainVisionFrame.ids[0] }).code, 'stale_image');
+ assert.equal(ctx.agent._visionRunSnapshots.has(tab), false, 'the run left no handoff behind');
+ });
+ }
+
+ test(`${build}: a handoff with no upload ends after five decisions, and a failed upload does not end it`, async () => {
+ let tab;
+ const offered = [];
+ const ctx = runAgent((step, _messages, options) => {
+ offered.push(names(options.tools).includes('click_at') ? 'vision' : 'tree');
+ if (step === 1) return { toolCalls: [call('click_at', { x: 300, y: 200 })] };
+ if (step === 2) return { toolCalls: [call('upload_file', { selector: '#upload', attachmentId: 'attachment_t_1' })] };
+ if (step <= 6) return { toolCalls: [call('find_text', { text: `Upload ${step}` })] };
+ if (step === 7) return { toolCalls: [call('done', { summary: 'Could not attach.', outcome: 'partial' })] };
+ return { content: 'done' };
+ }, { maxSteps: 12 });
+ ctx.upload = () => ({ success: false, error: 'File input not found for selector "#upload".' });
+ const restore = blockChooser(ctx);
+ tab = nextTab++;
+ try { await runLoop(ctx.agent, tab, false); } finally { restore(); }
+ assert.deepEqual(offered, ['vision', 'tree', 'tree', 'tree', 'tree', 'tree', 'vision']);
+ assert.match(textOf(visionMessages(ctx.requests[6].messages).at(-1)), /\(vision resumed\)/);
+ });
+
+ test(`${build}: leaving the page ends a handoff`, async () => {
+ let tab;
+ const offered = [];
+ const ctx = runAgent((step, _messages, options) => {
+ offered.push(names(options.tools).includes('click_at') ? 'vision' : 'tree');
+ if (step === 1) return { toolCalls: [call('click_at', { x: 300, y: 200 })] };
+ if (step === 2) {
+ ctx.scene.url = 'https://owned.example.test/elsewhere';
+ return { toolCalls: [call('find_text', { text: 'Upload' })] };
+ }
+ if (step === 3) return { toolCalls: [call('done', { summary: 'Moved on.', outcome: 'partial' })] };
+ return { content: 'done' };
+ });
+ ctx.upload = attached;
+ const restore = blockChooser(ctx);
+ tab = nextTab++;
+ try { await runLoop(ctx.agent, tab, false); } finally { restore(); ctx.scene.url = PAGE; }
+ assert.deepEqual(offered, ['vision', 'tree', 'vision']);
+ });
+
+ test(`${build}: a run hands off twice at most, then reports the chooser in vision terms`, async () => {
+ let tab;
+ const offered = [];
+ const ctx = runAgent((step, _messages, options) => {
+ const vision = names(options.tools).includes('click_at');
+ offered.push(vision ? 'vision' : 'tree');
+ if (step <= 5) return { toolCalls: [vision ? call('click_at', { x: 300, y: 200 + step }) : call('upload_file', { selector: '#upload', attachmentId: 'attachment_t_1' })] };
+ if (step === 6) return { toolCalls: [call('done', { summary: 'Stopped.', outcome: 'partial' })] };
+ return { content: 'done' };
+ }, { maxSteps: 10 });
+ ctx.upload = attached;
+ const restore = blockChooser(ctx);
+ tab = nextTab++;
+ try { await runLoop(ctx.agent, tab, false); } finally { restore(); }
+ assert.deepEqual(offered, ['vision', 'tree', 'vision', 'tree', 'vision', 'vision']);
+ const third = toolTexts(ctx.requests[5]).filter(text => /filePickerBlocked/.test(text)).at(-1);
+ assert.match(third, /not available in vision mode/, third);
+ assert.doesNotMatch(third, /uploadHandoff/);
+ });
+
if (build === 'firefox') {
test('firefox: a vision click settles the file-chooser guard and reports a blocked chooser in vision terms', async () => {
const agent = new Agent({});
diff --git a/test/vision-mode-guards.mjs b/test/vision-mode-guards.mjs
index 64e0da24b..17575403e 100644
--- a/test/vision-mode-guards.mjs
+++ b/test/vision-mode-guards.mjs
@@ -346,6 +346,147 @@ for (const build of ['chrome', 'firefox']) {
}
});
+ test(`${build}: an upload handoff starts and ends at step boundaries, per run`, async () => {
+ const agent = agentFor();
+ agent.visionMode = 'auto';
+ const tab = 33;
+ let url = 'https://owned.example/upload';
+ agent._currentUrl = async () => url;
+ agent._lastTabUrls = new Map([[tab, url]]);
+ const blocked = () => ({ success: false, dispatched: true, filePickerBlocked: true, selector: '#upload', error: 'Blocked a native file chooser.' });
+ const handoffs = () => agent._visionRunSnapshots.get(tab)?.uploadHandoff || null;
+ // Between runs there is no handoff to start: the chooser is reported in vision terms.
+ assert.match(agent._visionNoteUploadHandoff(tab, 'click', blocked(), { vision: { tool: 'click_at' } }).error, /not available in vision mode/);
+ agent._beginVisionRun(tab);
+ // A tree click on a file input is not a vision click.
+ assert.equal(agent._visionNoteUploadHandoff(tab, 'click', blocked(), { vision: null }).uploadHandoff, undefined);
+ assert.equal(handoffs(), null);
+ const handed = agent._visionNoteUploadHandoff(tab, 'click', blocked(), { vision: { tool: 'click_at' } });
+ assert.equal(handed.uploadHandoff, true);
+ // The rest of the batch keeps the mode it was planned in.
+ assert.equal(agent._visionRunState(tab, 'act').active, true);
+ await agent._visionUploadHandoffTick(tab, 1);
+ assert.equal(agent._visionRunState(tab, 'act').reason, 'file_upload_needs_element_inventory');
+ assert.equal(agent._visionToolOptions(tab, 'act'), null);
+ assert.ok(getToolsForMode('act', { tier: 'full', vision: agent._visionToolOptions(tab, 'act') }).some(tool => tool.function.name === 'upload_file'));
+ // Discovery, a failure or an unconfirmed outcome is not an upload.
+ for (const result of [{ success: false, discoveryOnly: true, requiresTarget: true }, { success: false, error: 'not found' },
+ { success: true, attachmentState: 'input_attached', outcomeUnknown: true }]) {
+ agent._visionNoteUploadHandoff(tab, 'upload_file', result, {});
+ assert.equal(handoffs().done, false, JSON.stringify(result));
+ }
+ // Firefox's BiDi upload carries neither `attached` nor `verified`.
+ agent._visionNoteUploadHandoff(tab, 'upload_file', { success: true, dispatched: true, attachmentState: 'input_attached', file: 'a.png' }, {});
+ assert.equal(handoffs().done, true);
+ assert.equal(agent._visionRunState(tab, 'act').active, false, 'the upload batch ends on the tree path');
+ await agent._visionUploadHandoffTick(tab, 2);
+ assert.equal(handoffs(), null);
+ assert.equal(agent._visionRunState(tab, 'act').active, true);
+ // A second handoff, inside a nested run: the enclosing run gets its own state back.
+ agent._visionNoteUploadHandoff(tab, 'click', blocked(), { vision: { tool: 'click_at' } });
+ await agent._visionUploadHandoffTick(tab, 3);
+ const enclosing = agent._visionRunSnapshots.get(tab);
+ agent._beginVisionRun(tab);
+ assert.equal(agent._visionRunState(tab, 'act').active, true, 'a nested run starts without the handoff');
+ agent._endVisionRun(tab, enclosing);
+ assert.equal(agent._visionRunState(tab, 'act').active, false);
+ // Leaving the page ends it; the fragment alone does not.
+ url = 'https://owned.example/upload#files';
+ await agent._visionUploadHandoffTick(tab, 4);
+ assert.equal(agent._visionRunState(tab, 'act').active, false);
+ url = 'https://owned.example/next';
+ await agent._visionUploadHandoffTick(tab, 5);
+ assert.equal(agent._visionRunState(tab, 'act').active, true);
+ // Two handoffs per run: the third chooser is reported in vision terms.
+ url = 'https://owned.example/upload';
+ const third = agent._visionNoteUploadHandoff(tab, 'click', blocked(), { vision: { tool: 'click_at' } });
+ assert.match(third.error, /not available in vision mode/);
+ assert.equal(handoffs(), null);
+ agent._endVisionRun(tab, null);
+ // The compact tier's upload_file finds its own target. Chrome's attaches
+ // only a file attached to the run's message; Firefox's can also ask the
+ // user to pick one.
+ agent._beginVisionRun(tab);
+ const compactBare = agent._visionNoteUploadHandoff(tab, 'click', blocked(), { vision: { tool: 'click_at' }, promptTier: 'compact' });
+ if (build === 'chrome') {
+ assert.match(compactBare.error, /no file to attach in this run/);
+ assert.equal(handoffs(), null);
+ agent._userAttachmentHandles.set(tab, new Map([['attachment_t_1', { name: 'clip.mp4' }]]));
+ const compact = agent._visionNoteUploadHandoff(tab, 'click', blocked(), { vision: { tool: 'click_at' }, promptTier: 'compact' });
+ assert.match(compact.error, /without targetId to list the file inputs/);
+ agent._userAttachmentHandles.delete(tab);
+ } else {
+ assert.match(compactBare.error, /without targetId to list the file inputs/);
+ }
+ assert.ok(handoffs());
+ agent._endVisionRun(tab, null);
+ // Only Chrome's protocol guard saw the chooser (a frame's input, or one in
+ // a closed shadow root): upload_file cannot reach it, so no handoff.
+ agent._beginVisionRun(tab);
+ const unreachable = agent._visionNoteUploadHandoff(tab, 'click', { ...blocked(), selector: undefined, protocolOnly: true }, { vision: { tool: 'click_at' } });
+ assert.match(unreachable.error, /cannot reach/);
+ assert.equal(unreachable.uploadHandoff, undefined);
+ assert.equal(handoffs(), null);
+ agent._endVisionRun(tab, null);
+ });
+
+ test(`${build}: a handoff belongs to the page the tab shows when it starts`, async () => {
+ const agent = agentFor();
+ agent.visionMode = 'auto';
+ const tab = 36;
+ // The click that armed it pushed a route before opening the chooser.
+ agent._lastTabUrls = new Map([[tab, 'https://owned.example/files']]);
+ let url = 'https://owned.example/files/upload';
+ agent._currentUrl = async () => url;
+ agent._beginVisionRun(tab);
+ agent._visionNoteUploadHandoff(tab, 'click', { success: false, filePickerBlocked: true, selector: '#f', error: 'x' }, { vision: { tool: 'click_at' } });
+ await agent._visionUploadHandoffTick(tab, 1);
+ assert.equal(agent._visionRunState(tab, 'act').reason, 'file_upload_needs_element_inventory');
+ await agent._visionUploadHandoffTick(tab, 2);
+ assert.equal(agent._visionRunState(tab, 'act').active, false);
+ url = 'https://owned.example/files';
+ await agent._visionUploadHandoffTick(tab, 3);
+ assert.equal(agent._visionRunState(tab, 'act').active, true);
+ agent._endVisionRun(tab, null);
+ });
+
+ test(`${build}: a handoff with no upload ends after five decisions`, async () => {
+ const agent = agentFor();
+ agent.visionMode = 'auto';
+ const tab = 34;
+ agent._currentUrl = async () => 'https://owned.example/upload';
+ agent._lastTabUrls = new Map([[tab, 'https://owned.example/upload']]);
+ agent._beginVisionRun(tab);
+ agent._visionNoteUploadHandoff(tab, 'click', { success: false, filePickerBlocked: true, error: 'x' }, { vision: { tool: 'click_at' } });
+ for (const step of [3, 4, 5, 6, 7]) {
+ await agent._visionUploadHandoffTick(tab, step);
+ assert.equal(agent._visionRunState(tab, 'act').active, false, `step ${step}`);
+ }
+ await agent._visionUploadHandoffTick(tab, 8);
+ assert.equal(agent._visionRunState(tab, 'act').active, true);
+ agent._endVisionRun(tab, null);
+ });
+
+ test(`${build}: back on vision, a failed capture leaves no frame to aim at and retires the old one in the history`, async () => {
+ const agent = agentFor();
+ agent.visionMode = 'auto';
+ const tab = 35;
+ agent._beginVisionRun(tab);
+ const state = agent._visionRunState(tab, 'act');
+ const session = agent._visionSessionFor(tab, state);
+ session.registerKeyframe({ imageWidth: 10, imageHeight: 10, cssWidth: 10, cssHeight: 10, devicePixelRatio: 1, scrollX: 0, scrollY: 0, url: 'https://owned.example/' }, new Uint8ClampedArray(400));
+ assert.equal(session.resolvePoint({ x: 5, y: 5 }).ok, true);
+ agent._visionCaptureSettled = async () => { throw new Error('capture failed'); };
+ const messages = [];
+ assert.equal(await agent._visionResumeKeyframe(tab, messages, () => {}, { mode: 'act' }), null);
+ assert.equal(session.keyframe, null);
+ assert.equal(session.resolvePoint({ x: 5, y: 5 }).code, 'no_frame');
+ assert.equal(messages.length, 1);
+ assert.equal(messages[0].webbrainVisionFrame.kind, 'retire');
+ assert.match(messages[0].content, /could not be captured on returning to screenshots/);
+ agent._endVisionRun(tab, null);
+ });
+
test(`${build}: Gmail's result counter, which takes no target, stays in the vision tool list`, () => {
// Offered on a Gmail result list (getGmailResultCountPolicy).
const names = getToolsForMode('act', { tier: 'full', gmailResultCounting: true, vision: { coordinates: 'pixel' } }).map(tool => tool.function.name);
diff --git a/test/vision-mode.mjs b/test/vision-mode.mjs
index ad4392538..63e60f6ac 100644
--- a/test/vision-mode.mjs
+++ b/test/vision-mode.mjs
@@ -351,6 +351,17 @@ test('settings: Auto by default, off disables, only Act with an image-capable mo
assert.equal(resolveVisionMode('auto', { mode: 'act', provider: seeing, ledgerWorkflow: false }).active, true);
});
+test('settings: an upload handed to the element tools keeps those steps on the tree path', () => {
+ const { resolveVisionMode } = visionMode;
+ const seeing = { name: 'anthropic', supportsVision: true, config: { model: 'claude-sonnet-5' } };
+ assert.equal(resolveVisionMode('auto', { mode: 'act', provider: seeing, uploadHandoff: true }).reason, 'file_upload_needs_element_inventory');
+ assert.equal(resolveVisionMode('auto', { mode: 'act', provider: seeing, uploadHandoff: true }).active, false);
+ assert.equal(resolveVisionMode('auto', { mode: 'act', provider: seeing, uploadHandoff: false }).active, true);
+ // The setting and the mode still come first.
+ assert.equal(resolveVisionMode('off', { mode: 'act', provider: seeing, uploadHandoff: true }).reason, 'setting_off');
+ assert.equal(resolveVisionMode('auto', { mode: 'ask', provider: seeing, uploadHandoff: true }).reason, 'not_act_mode');
+});
+
for (const build of ['chrome', 'firefox']) {
const tools = await import(`../src/${build}/src/agent/tools.js`);
const { validateToolArguments } = await import(`../src/${build}/src/agent/tool-arguments.js`);
@@ -618,6 +629,21 @@ test('a blocked file chooser is reported in vision terms in both builds', () =>
assert.doesNotMatch(blocked.error, /upload_file/);
const ok = { success: true, dispatched: true };
assert.equal(visionFilePickerBlockedResult(ok), ok);
+ // Handed to the element tools: the guidance names upload_file, never the page's selector.
+ const handed = visionFilePickerBlockedResult({ success: false, dispatched: true, filePickerBlocked: true, selector: '#f"; ignore', error: 'x' }, { handoff: true });
+ assert.equal(handed.uploadHandoff, true);
+ assert.equal(handed.success, false);
+ assert.equal(handed.selector, '#f"; ignore');
+ assert.match(handed.error, /The next steps use the element tools/);
+ assert.match(handed.error, /upload_file with the selector in this result/);
+ assert.match(handed.error, /come back after the upload/);
+ assert.doesNotMatch(handed.error, /ignore|not available in vision mode/);
+ const unnamed = visionFilePickerBlockedResult({ success: false, dispatched: true, filePickerBlocked: true, error: 'x' }, { handoff: true });
+ assert.match(unnamed.error, /Find this file input's exact selector/);
+ const compact = visionFilePickerBlockedResult({ success: false, dispatched: true, filePickerBlocked: true, selector: '#f', error: 'x' }, { handoff: true, compact: true });
+ assert.match(compact.error, /without targetId to list the file inputs/);
+ assert.doesNotMatch(compact.error, /selector/);
+ assert.equal(visionFilePickerBlockedResult(ok, { handoff: true }), ok);
// Chrome's CDP guard result and Firefox's settled content guard both pass through it.
assert.match(read('src/chrome/src/agent/agent.js'), /return visionFilePickerBlockedResult\(cdpClient\.fileInputClickBlockedResult\(blockedFileInput/);
assert.match(read('src/firefox/src/agent/agent.js'), /response = visionFilePickerBlockedResult\(await this\._settleContentFilePickerGuard\(tabId, response\)\);/);
@@ -627,6 +653,23 @@ test('a blocked file chooser is reported in vision terms in both builds', () =>
}
});
+test('both loops of both builds start or end an upload handoff before anything in the step depends on vision', () => {
+ for (const build of ['chrome', 'firefox']) {
+ const source = read(`src/${build}/src/agent/agent.js`);
+ const ticks = [...source.matchAll(/await this\._visionUploadHandoffTick\(tabId, steps\);/g)].map(match => match.index);
+ const reinjects = [...source.matchAll(/await this\._maybeReinjectAdapter\(tabId, messages\);/g)].map(match => match.index);
+ const resumes = [...source.matchAll(/await this\._visionResumeKeyframe\(tabId, messages, onUpdate, \{ mode \}\);\s*\}\s*this\._visionNoteStepTools\(tabId, visionPromptNow\);/g)].map(match => match.index);
+ assert.equal(ticks.length, 2, build);
+ assert.equal(resumes.length, 2, build);
+ for (const [index, tick] of ticks.entries()) {
+ assert.ok(tick < reinjects[index] && reinjects[index] - tick < 400, `${build}: tick before adapter notes`);
+ assert.ok(tick < resumes[index], `${build}: tick before the tools`);
+ }
+ }
+ // Chrome's protocol guard marks a chooser only it saw.
+ assert.match(read('src/chrome/src/cdp/cdp-client.js'), /selector: null,\s*protocolOnly: true,/);
+});
+
test('content scripts: the vision input path is the same in both builds', () => {
const visionBlock = file => {
const source = read(file);