Open-source AI browser agent for Chrome and Firefox (monorepo) π§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Bind the upload advisory to the node the inventory listed
Claudef78c2ac13658
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 96f9966b6..b3ef39249 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -16,7 +16,7 @@ This changelog was generated from the repository Git history and release tags. V- The spinning ring next to "Thinkingβ¦" in the side panel is now a small WebBrain brain drawn as a neural circuit: a signal keeps tracing the W, and the brain's outline lights up as it passes. It moves only while WebBrain is working and the panel is visible, follows the light and dark themes, and holds still when your system asks to reduce motion.### Fixed-- Act can send a photo or video in an Instagram direct message. Instagram's message box keeps its file input hidden, outside any form, deep in a long conversation, where WebBrain's check that an action still targets what the model saw cannot reach it. Every attachment was refused, even with the exact selector WebBrain itself listed for that input, often right after you had confirmed the recipient on WebBrain's card; the model then sent a link to the picture, or said Instagram blocks attachments. With Strict checks off, an upload to an input named by the selector WebBrain listed on that page now goes through with a note to the model, and the recipient checks still apply; a selector WebBrain did not list is still refused. When clicking "Add Photo or Video" opens a file chooser, WebBrain now names the input with that same selector, and the Instagram site notes say to attach the file to that input instead of clicking the button, and that a link to a picture is not the picture.+- Act can send a photo or video in an Instagram direct message. Instagram's message box keeps its file input hidden, outside any form, deep in a long conversation, where WebBrain's check that an action still targets what the model saw cannot reach it. Every attachment was refused, even with the exact selector WebBrain itself listed for that input, often right after you had confirmed the recipient on WebBrain's card; the model then sent a link to the picture, or said Instagram blocks attachments. With Strict checks off, an upload to the very input WebBrain listed on that page, named by the selector it gave, now goes through with a note to the model, and the recipient checks still apply. A selector WebBrain did not list is still refused, and so is an input the page re-created after the list under the same selector, until WebBrain lists the page's controls again. When clicking "Add Photo or Video" opens a file chooser, WebBrain now names the input with that same selector, and the Instagram site notes say to attach the file to that input instead of clicking the button, and that a link to a picture is not the picture.- Act can attach images and videos to a post on X again. While the post composer is open at x.com/compose/post, the timeline behind it keeps a second, identical file input, and WebBrain's list of the page's controls gave neither input a selector. After the model's first selector matched both, WebBrain refused every later upload until it saw a selector it could not get, so the run ended at its cost limit with nothing attached. The list now gives every file input a selector that matches only that input, scoped to the open dialog where that is what tells them apart, and marks the dialog's input `inDialog`. The X site notes name the dialog's input, and when no selector can single out an input, WebBrain tells the model to report the upload as blocked instead of retrying.## [42.3.1] - 2026-10-20diff --git a/docs/page-feedback-monitor.md b/docs/page-feedback-monitor.mdindex f68327765..341d26d30 100644--- a/docs/page-feedback-monitor.md+++ b/docs/page-feedback-monitor.md@@ -297,14 +297,18 @@ Steer or grant authorization. Enter and Alt+Enter retain their existing behaviorrejected call, opens a chooser, or grants upload permission. A file input nocertificate covers (`target_uncovered`; Instagram's direct-message composerkeeps a formless, `display:none` input deep in a thread too large for the- owner bounds) has no ref, so an `upload_file` whose selector is exactly one- that `get_interactive_elements` returned for the current top document and URL- names the input the model observed: with nothing pending and Strict checks- off it is sent with the ordinary unbound semantics and a- `page-feedback-target-binding` note, and the recipient checks of a protected- conversation still bind the attachment. Any other selector keeps its refusal.- A blocked native chooser names its input with the selector the inventory- uses.+ owner bounds) has no ref; the model observes it through+ `get_interactive_elements`. Each such inventory carries a token, and the+ monitor keeps the very nodes it listed with the selector it gave each. An+ `upload_file` whose selector is exactly one that inventory returned for the+ current top document and URL, and whose target the monitor reports as that+ same node (`inventoried`), is sent with nothing pending and Strict checks off,+ with the ordinary unbound semantics and a `page-feedback-target-binding`+ note; the recipient checks of a protected conversation still bind the+ attachment. A node the page re-mounts later under the same selector was+ never listed and keeps the refusal until a fresh inventory lists it, as does+ any other selector. A blocked native chooser names its input with the+ selector the inventory uses.- Native `summary` controls and authored click/tab-focus targets use the sameobserved owner and exact action footprint as native and ARIA buttons. Changedhandler identity, disclosure state, target/owner identity and user interventiondiff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex d759755c6..64a9f2e1d 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -52594,6 +52594,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d_nativeChatDraftBinding:dispatchContext.nativeChatDraftBinding || null,_nativeChatDraftPreparation:dispatchContext.nativeChatDraftPreparation === true };if (dispatchContext.jevBinding) contentArgs = { ...contentArgs, _jevBinding: dispatchContext.jevBinding };+ // Names the model's file-input inventory, so the page monitor keeps the+ // nodes it listed (see page-feedback.js, page-feedback-target-binding).+ if (name === 'get_interactive_elements' && typeof dispatchContext.fileInventoryToken === 'string' && dispatchContext.fileInventoryToken) {+ contentArgs = { ...contentArgs, _fileInventoryToken: dispatchContext.fileInventoryToken };+ }const sendContentAction = async stageAbortSignal => {if (Agent.STATE_CHANGE_TOOLS.has(name)) {await beforePageAgentDispatch(globalThis.browser || globalThis.chrome, tabId, {diff --git a/src/chrome/src/agent/page-feedback.js b/src/chrome/src/agent/page-feedback.jsindex eae5e612d..c492890d6 100644--- a/src/chrome/src/agent/page-feedback.js+++ b/src/chrome/src/agent/page-feedback.js@@ -252,8 +252,8 @@ function actionBindingUnavailableResult(run, name) {// Whether a selector is one that WebBrain's own file-input inventory// (get_interactive_elements) returned on the run's current top document and-// URL: the selectors that inventory recorded are this run's observed upload-// targets, and an exact copy of one names the input the model observed.+// URL. The page monitor separately confirms that the selector still names the+// very node that inventory listed (validation's `inventoried`).function uploadSelectorObserved(run, selector) {const scope = run?.uploadObservationScope;return typeof selector === 'string' && selector.length > 0 && !!scope@@ -1366,7 +1366,8 @@ export const pageFeedbackMethods = {}const run = this._pageFeedbackRuns?.get(tabId);const binding = state?.actionBinding;- if (run) { run.actionBindingUnavailable = null; run.actionBindingUnavailableReason = null; run.actionContextUnverified = false; }+ if (run) { run.actionBindingUnavailable = null; run.actionBindingUnavailableReason = null; run.actionContextUnverified = false;+ run.actionBindingInventoried = false; }if (!run || !binding || run.modelState !== state || !this._pageFeedbackStateCurrent(tabId, run, state)|| (run.events.size && !isAutomaticPageFeedback([...run.events.values()]))) return false;try {@@ -1374,6 +1375,7 @@ export const pageFeedbackMethods = {const relaxedMonitorGrantToken = retry && retry.grantToken === binding.relaxedMonitorGrantToken ? retry.grantToken : null;const response = await apiFor().tabs.sendMessage(tabId, { target: 'content', action: 'page_monitor_validate_model',params: { ...safeAction(name, args), runToken: run.token, snapshotToken: binding.snapshotToken,+ ...(name === 'upload_file' && run.uploadObservationScope?.token ? { fileInventoryToken: run.uploadObservationScope.token } : {}),...(this._pageFeedbackModelHeldMs(state) > 0 ? { modelHeldMs: this._pageFeedbackModelHeldMs(state) } : {}),...(batchId ? { batchId } : {}),...(relaxedMonitorGrantToken ? { relaxedMonitorGrantToken } : {}) } }, { frameId: 0 });@@ -1415,6 +1417,7 @@ export const pageFeedbackMethods = {run.actionCaptureTarget = safeAction(name, args);run.actionBindingUnavailable = `${name}:${JSON.stringify(args)}`;run.actionBindingUnavailableReason = response.reason;+ run.actionBindingInventoried = name === 'upload_file' && response.inventoried === true;// The monitor may reject ambiguity before the upload resolver runs.// Preserve that resolver's required inventory step on this path too.if (name === 'upload_file' && response.reason === 'target_ambiguous') {@@ -1454,21 +1457,24 @@ export const pageFeedbackMethods = {// attribution, and no correction), Strict checks off sends the call with the// ordinary unbound semantics and a note; the dispatch fences at the// transport still run. A ref is what the model read. A hidden file input has- // no ref, so upload_file names it by selector: one that WebBrain's own- // file-input inventory gave for this document and URL is the input the- // model observed. Instagram's direct-message composer keeps such an input,- // formless and display:none, deep in a large thread, where no certificate- // can reach it. Any other selector may name an element the model never- // observed (an upload slot it guessed, say), and a selector naming several- // elements or none has no single element to send to: those keep their- // answer, as does anything pending.+ // no ref, so upload_file names it by selector, and the model observed it+ // through WebBrain's own file-input inventory: the advisory holds when the+ // selector is one that inventory gave on this document and URL and the page+ // monitor reports the target is the very node it listed with it, not one+ // the page rendered later under the same selector (inventoried). Instagram's+ // direct-message composer keeps such an input, formless and display:none,+ // deep in a large thread, where no certificate can reach it. Any other+ // selector may name an element the model never observed (an upload slot it+ // guessed, say), and a selector naming several elements or none has no+ // single element to send to: those keep their answer, as does anything+ // pending._pageFeedbackBindingAdvises(tabId, run, args = {}, advisories = null, name = '') {if (!run || run.events.size || run.actionBindingUnavailableReason !== 'target_uncovered'|| this._hasPendingSteering?.(tabId) || typeof this._guardAdvisesFor !== 'function') return false;const hasSelector = typeof args?.selector === 'string' && !!args.selector.trim();const observedRef = typeof args?.ref_id === 'string' && /^ref_[A-Za-z0-9_-]+$/.test(args.ref_id) && !hasSelector;const observedUpload = name === 'upload_file' && args?.ref_id == null && hasSelector- && uploadSelectorObserved(run, args.selector);+ && run.actionBindingInventoried === true && uploadSelectorObserved(run, args.selector);if (!observedRef && !observedUpload) return false;return this._guardAdvisesFor(advisories, 'page-feedback-target-binding', observedUpload? 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input your file-input inventory named without that check. Read the page to confirm the file landed where you intended.'@@ -1872,7 +1878,11 @@ export function installPageFeedback(Agent) {&& !!run && this._pageFeedbackStateCurrent(tabId, run, run.modelState, run.url, { rootOnly: policy.kind === 'scroll' }));if (run && mutation && !modelBound && this._pageFeedbackBlocksCall(tabId, name, args)) return pageFeedbackPendingResult();const dispatchState = executionContext?._contentActionDispatchState || { started: false };+ // The model's file-input inventory gets a token the page monitor keeps+ // with the very nodes it lists (uploadObservationScope below).+ const fileInventoryToken = run && !executionContext?.pageFeedbackRead && name === 'get_interactive_elements' ? token() : '';executionContext = { ...executionContext, _contentActionDispatchState: dispatchState,+ ...(fileInventoryToken ? { fileInventoryToken } : {}),...(policy.kind === 'media' ? { _expectedMediaBinding: policy.binding } : {}) };if (name === 'download_social_media') {args = { ...args }; delete args.expectedMediaBinding;@@ -1930,7 +1940,7 @@ export function installPageFeedback(Agent) {if (run && !executionContext.pageFeedbackRead && name === 'get_interactive_elements' && Array.isArray(result)&& this._pageFeedbackRuns?.get(tabId) === run && observationDocument&& run.frames.get(0)?.token === observationDocument && run.url === observationUrl) {- run.uploadObservationScope = { documentToken: observationDocument, url: observationUrl,+ run.uploadObservationScope = { documentToken: observationDocument, url: observationUrl, token: fileInventoryToken,selectors: [...new Set(result.filter(item => item?.tag === 'input' && item.type === 'file'&& typeof item.selector === 'string' && item.selector.length > 0 && item.selector.length <= 4096).map(item => item.selector))].slice(0, 8) };diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex eaf559ca5..0b1cad0df 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -4357,8 +4357,9 @@return targets;}- function getInteractiveElementsFull() {- return queryInteractiveFull().map((c, i) => {+ function getInteractiveElementsFull(params = {}) {+ const fileInputs = [];+ const records = queryInteractiveFull().map((c, i) => {const el = c.el;const result = {index: i,@@ -4379,10 +4380,21 @@// Says which of two otherwise identical inputs belongs to the open// dialog, such as the composer the model just opened.if (_fileInputDialogScopes(el).length) result.inDialog = true;- if (selector) result.selector = selector;+ if (selector) {+ result.selector = selector;+ fileInputs.push([el, selector]);+ }}return result;});+ // The page monitor keeps the very nodes this inventory named. An upload+ // to an input it cannot certify relies on being one of them; another node+ // the page renders later under the same selector is not (see+ // page-feedback-target-binding).+ if (typeof params._fileInventoryToken === 'string' && params._fileInventoryToken) {+ try { window.__wbPageMonitor?.recordFileInventory?.(params._fileInventoryToken, fileInputs); } catch {}+ }+ return records;}// ββ Dev-mode reversible page patches and targeting helpers ββββββββββββ@@ -9174,7 +9186,7 @@'get_page_info': () => getPageInfo(),'get_page_info_cdp': () => ({ ...getPageInfoFull(msg.params || {}), tableCount: document.querySelectorAll('table').length }),'get_interactive_elements': () => getInteractiveElements(),- 'get_interactive_elements_cdp': () => getInteractiveElementsFull(),+ 'get_interactive_elements_cdp': () => getInteractiveElementsFull(msg.params || {}),'get_file_input_targets': () => getFileInputTargets(),'click': () => clickElement(msg.params || {}, actionDeadlineExpired),'consume_file_picker_guard': () => consumeFilePickerGuard(msg.params?.guardId),diff --git a/src/chrome/src/content/page-monitor.js b/src/chrome/src/content/page-monitor.jsindex 11023adbd..2bbc09a15 100644--- a/src/chrome/src/content/page-monitor.js+++ b/src/chrome/src/content/page-monitor.js@@ -56,6 +56,10 @@// Only the authenticated background state response can establish this grant.// Model/page metadata and ordinary action parameters cannot create one.let relaxedMonitorGrant = null;+ // The file inputs the latest get_interactive_elements result named, each+ // with the selector it gave. A hidden input has no ref: this inventory is+ // how the model observed it (page-feedback-target-binding).+ let fileInventory = null;const revokedRelaxedGrants = new Set();const relaxedObservedBindings = new Set();function revokeRelaxedGrant() {@@ -1995,6 +1999,24 @@if (point) return nativeReceiverBelongsTo(target, receiver);return !!receiver && (receiver === target || target.contains(receiver));}+ function recordFileInventory(token, pairs) {+ if (!active || typeof token !== 'string' || !token || token.length > 100 || !Array.isArray(pairs)) return false;+ const entries = new WeakMap();+ for (const [element, selector] of pairs.slice(0, 64)) {+ if (element instanceof HTMLInputElement && element.type === 'file'+ && typeof selector === 'string' && selector && selector.length <= 4096) entries.set(element, selector);+ }+ fileInventory = { token, runToken, entries };+ return true;+ }+ // Whether an upload's target is the very node that inventory named with+ // this selector. A node the page renders later under the same selector, as+ // a reopened composer does, was never observed.+ function inventoriedFileInput(target, params) {+ return params.tool === 'upload_file' && typeof params.fileInventoryToken === 'string' && !!params.fileInventoryToken+ && fileInventory?.token === params.fileInventoryToken && fileInventory.runToken === runToken+ && typeof params.selector === 'string' && fileInventory.entries.get(target) === params.selector;+ }function modelBindingResult(params = {}) {const snapshot = modelSnapshots.get(params.snapshotToken || params.expectedModelSnapshot);if (!active || params.runToken !== runToken || snapshot?.runToken !== runToken) return { reason: 'model_snapshot_invalid' };@@ -2022,7 +2044,8 @@const original = snapshot.targets.get(target);const before = snapshot.batchInputs?.get(params.batchId)?.get(original) || original;const previous = snapshot.uncertified.get(target);- if (!relaxed && !before && !previous) return { reason: 'target_uncovered', certificationFailure: snapshot.captureFailures.get(target) || 'identity_unobserved' };+ if (!relaxed && !before && !previous) return { reason: 'target_uncovered', certificationFailure: snapshot.captureFailures.get(target) || 'identity_unobserved',+ ...(inventoriedFileInput(target, params) ? { inventoried: true } : {}) };if (params.ref_id && snapshot.hasRefs && snapshot.refs.get(params.ref_id) !== target) {if (relaxed) revokeRelaxedGrant();return { reason: 'target_changed' };@@ -3917,6 +3940,7 @@respond({ ready: !!result.binding, ...(result.uncertified ? { uncertified: true } : {}),...(result.relaxed ? { relaxed: true, bindingKind: 'relaxed_target' } : {}),...(result.certificationFailure ? { certificationFailure: result.certificationFailure } : {}),+ ...(result.inventoried ? { inventoried: true } : {}),...(result.reason ? { reason: result.reason } : {}), runToken, documentToken, snapshotToken: params.snapshotToken });}else if (msg.action === 'page_monitor_traversal') {@@ -4051,7 +4075,7 @@navigationApi?.addEventListener('navigate', onNavigate, true);navigationApi?.addEventListener('navigateerror', onNavigateError, true);navigationApi?.addEventListener('currententrychange', onCurrentEntryChange, true);- window.__wbPageMonitor = { beginContentAction, beforeLocalDispatch, withLocalPreparation, withLocalDispatch, captureLocalInputObservation, dispatch: params => dispatch(params), activatePreparedDispatch, withPreparedDispatch, registerDecoration, isReadonlyPreparationDecoration, currentIdentityPreparationCapture,+ window.__wbPageMonitor = { beginContentAction, beforeLocalDispatch, withLocalPreparation, withLocalDispatch, captureLocalInputObservation, dispatch: params => dispatch(params), activatePreparedDispatch, withPreparedDispatch, registerDecoration, isReadonlyPreparationDecoration, currentIdentityPreparationCapture, recordFileInventory,get active() { return active; },get disposed() { return disposed; },dispose() {diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex c41109168..cd2e0b106 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -45778,6 +45778,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d_nativeChatDraftBinding:dispatchContext.nativeChatDraftBinding || null,_nativeChatDraftPreparation:dispatchContext.nativeChatDraftPreparation === true };if (dispatchContext.jevBinding) contentArgs = { ...contentArgs, _jevBinding: dispatchContext.jevBinding };+ // Names the model's file-input inventory, so the page monitor keeps the+ // nodes it listed (see page-feedback.js, page-feedback-target-binding).+ if (name === 'get_interactive_elements' && typeof dispatchContext.fileInventoryToken === 'string' && dispatchContext.fileInventoryToken) {+ contentArgs = { ...contentArgs, _fileInventoryToken: dispatchContext.fileInventoryToken };+ }const sendContentAction = async stageAbortSignal => {if (Agent.STATE_CHANGE_TOOLS.has(name)) {await beforePageAgentDispatch(globalThis.browser || globalThis.chrome, tabId, {diff --git a/src/firefox/src/agent/page-feedback.js b/src/firefox/src/agent/page-feedback.jsindex eae5e612d..c492890d6 100644--- a/src/firefox/src/agent/page-feedback.js+++ b/src/firefox/src/agent/page-feedback.js@@ -252,8 +252,8 @@ function actionBindingUnavailableResult(run, name) {// Whether a selector is one that WebBrain's own file-input inventory// (get_interactive_elements) returned on the run's current top document and-// URL: the selectors that inventory recorded are this run's observed upload-// targets, and an exact copy of one names the input the model observed.+// URL. The page monitor separately confirms that the selector still names the+// very node that inventory listed (validation's `inventoried`).function uploadSelectorObserved(run, selector) {const scope = run?.uploadObservationScope;return typeof selector === 'string' && selector.length > 0 && !!scope@@ -1366,7 +1366,8 @@ export const pageFeedbackMethods = {}const run = this._pageFeedbackRuns?.get(tabId);const binding = state?.actionBinding;- if (run) { run.actionBindingUnavailable = null; run.actionBindingUnavailableReason = null; run.actionContextUnverified = false; }+ if (run) { run.actionBindingUnavailable = null; run.actionBindingUnavailableReason = null; run.actionContextUnverified = false;+ run.actionBindingInventoried = false; }if (!run || !binding || run.modelState !== state || !this._pageFeedbackStateCurrent(tabId, run, state)|| (run.events.size && !isAutomaticPageFeedback([...run.events.values()]))) return false;try {@@ -1374,6 +1375,7 @@ export const pageFeedbackMethods = {const relaxedMonitorGrantToken = retry && retry.grantToken === binding.relaxedMonitorGrantToken ? retry.grantToken : null;const response = await apiFor().tabs.sendMessage(tabId, { target: 'content', action: 'page_monitor_validate_model',params: { ...safeAction(name, args), runToken: run.token, snapshotToken: binding.snapshotToken,+ ...(name === 'upload_file' && run.uploadObservationScope?.token ? { fileInventoryToken: run.uploadObservationScope.token } : {}),...(this._pageFeedbackModelHeldMs(state) > 0 ? { modelHeldMs: this._pageFeedbackModelHeldMs(state) } : {}),...(batchId ? { batchId } : {}),...(relaxedMonitorGrantToken ? { relaxedMonitorGrantToken } : {}) } }, { frameId: 0 });@@ -1415,6 +1417,7 @@ export const pageFeedbackMethods = {run.actionCaptureTarget = safeAction(name, args);run.actionBindingUnavailable = `${name}:${JSON.stringify(args)}`;run.actionBindingUnavailableReason = response.reason;+ run.actionBindingInventoried = name === 'upload_file' && response.inventoried === true;// The monitor may reject ambiguity before the upload resolver runs.// Preserve that resolver's required inventory step on this path too.if (name === 'upload_file' && response.reason === 'target_ambiguous') {@@ -1454,21 +1457,24 @@ export const pageFeedbackMethods = {// attribution, and no correction), Strict checks off sends the call with the// ordinary unbound semantics and a note; the dispatch fences at the// transport still run. A ref is what the model read. A hidden file input has- // no ref, so upload_file names it by selector: one that WebBrain's own- // file-input inventory gave for this document and URL is the input the- // model observed. Instagram's direct-message composer keeps such an input,- // formless and display:none, deep in a large thread, where no certificate- // can reach it. Any other selector may name an element the model never- // observed (an upload slot it guessed, say), and a selector naming several- // elements or none has no single element to send to: those keep their- // answer, as does anything pending.+ // no ref, so upload_file names it by selector, and the model observed it+ // through WebBrain's own file-input inventory: the advisory holds when the+ // selector is one that inventory gave on this document and URL and the page+ // monitor reports the target is the very node it listed with it, not one+ // the page rendered later under the same selector (inventoried). Instagram's+ // direct-message composer keeps such an input, formless and display:none,+ // deep in a large thread, where no certificate can reach it. Any other+ // selector may name an element the model never observed (an upload slot it+ // guessed, say), and a selector naming several elements or none has no+ // single element to send to: those keep their answer, as does anything+ // pending._pageFeedbackBindingAdvises(tabId, run, args = {}, advisories = null, name = '') {if (!run || run.events.size || run.actionBindingUnavailableReason !== 'target_uncovered'|| this._hasPendingSteering?.(tabId) || typeof this._guardAdvisesFor !== 'function') return false;const hasSelector = typeof args?.selector === 'string' && !!args.selector.trim();const observedRef = typeof args?.ref_id === 'string' && /^ref_[A-Za-z0-9_-]+$/.test(args.ref_id) && !hasSelector;const observedUpload = name === 'upload_file' && args?.ref_id == null && hasSelector- && uploadSelectorObserved(run, args.selector);+ && run.actionBindingInventoried === true && uploadSelectorObserved(run, args.selector);if (!observedRef && !observedUpload) return false;return this._guardAdvisesFor(advisories, 'page-feedback-target-binding', observedUpload? 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input your file-input inventory named without that check. Read the page to confirm the file landed where you intended.'@@ -1872,7 +1878,11 @@ export function installPageFeedback(Agent) {&& !!run && this._pageFeedbackStateCurrent(tabId, run, run.modelState, run.url, { rootOnly: policy.kind === 'scroll' }));if (run && mutation && !modelBound && this._pageFeedbackBlocksCall(tabId, name, args)) return pageFeedbackPendingResult();const dispatchState = executionContext?._contentActionDispatchState || { started: false };+ // The model's file-input inventory gets a token the page monitor keeps+ // with the very nodes it lists (uploadObservationScope below).+ const fileInventoryToken = run && !executionContext?.pageFeedbackRead && name === 'get_interactive_elements' ? token() : '';executionContext = { ...executionContext, _contentActionDispatchState: dispatchState,+ ...(fileInventoryToken ? { fileInventoryToken } : {}),...(policy.kind === 'media' ? { _expectedMediaBinding: policy.binding } : {}) };if (name === 'download_social_media') {args = { ...args }; delete args.expectedMediaBinding;@@ -1930,7 +1940,7 @@ export function installPageFeedback(Agent) {if (run && !executionContext.pageFeedbackRead && name === 'get_interactive_elements' && Array.isArray(result)&& this._pageFeedbackRuns?.get(tabId) === run && observationDocument&& run.frames.get(0)?.token === observationDocument && run.url === observationUrl) {- run.uploadObservationScope = { documentToken: observationDocument, url: observationUrl,+ run.uploadObservationScope = { documentToken: observationDocument, url: observationUrl, token: fileInventoryToken,selectors: [...new Set(result.filter(item => item?.tag === 'input' && item.type === 'file'&& typeof item.selector === 'string' && item.selector.length > 0 && item.selector.length <= 4096).map(item => item.selector))].slice(0, 8) };diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex 5b5a7d6d9..7ee319006 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -1534,8 +1534,9 @@return Number.isFinite(params.x) && Number.isFinite(params.y) ? _shadowAwareElementFromPoint(params.x, params.y) : null;}- function getInteractiveElementsFull() {- return queryInteractiveFull().map((c, i) => {+ function getInteractiveElementsFull(params = {}) {+ const fileInputs = [];+ const records = queryInteractiveFull().map((c, i) => {const el = c.el;const result = {index: i,@@ -1556,10 +1557,21 @@// Says which of two otherwise identical inputs belongs to the open// dialog, such as the composer the model just opened.if (_fileInputDialogScopes(el).length) result.inDialog = true;- if (selector) result.selector = selector;+ if (selector) {+ result.selector = selector;+ fileInputs.push([el, selector]);+ }}return result;});+ // The page monitor keeps the very nodes this inventory named. An upload+ // to an input it cannot certify relies on being one of them; another node+ // the page renders later under the same selector is not (see+ // page-feedback-target-binding).+ if (typeof params._fileInventoryToken === 'string' && params._fileInventoryToken) {+ try { window.__wbPageMonitor?.recordFileInventory?.(params._fileInventoryToken, fileInputs); } catch {}+ }+ return records;}// When click({index}) or type_text({index}) gets an out-of-range index,@@ -8299,7 +8311,7 @@'get_page_info': () => getPageInfo(msg.params || {}),'get_page_info_cdp': () => ({ ...getPageInfoFull(msg.params || {}), tableCount: document.querySelectorAll('table').length }),'get_interactive_elements': () => getInteractiveElements(),- 'get_interactive_elements_cdp': () => getInteractiveElementsFull(),+ 'get_interactive_elements_cdp': () => getInteractiveElementsFull(msg.params || {}),'get_file_input_targets': () => getFileInputTargets(),'click': () => clickElement(msg.params || {}, actionDeadlineExpired),'consume_file_picker_guard': () => consumeFilePickerGuard(msg.params?.guardId),diff --git a/src/firefox/src/content/page-monitor.js b/src/firefox/src/content/page-monitor.jsindex 11023adbd..2bbc09a15 100644--- a/src/firefox/src/content/page-monitor.js+++ b/src/firefox/src/content/page-monitor.js@@ -56,6 +56,10 @@// Only the authenticated background state response can establish this grant.// Model/page metadata and ordinary action parameters cannot create one.let relaxedMonitorGrant = null;+ // The file inputs the latest get_interactive_elements result named, each+ // with the selector it gave. A hidden input has no ref: this inventory is+ // how the model observed it (page-feedback-target-binding).+ let fileInventory = null;const revokedRelaxedGrants = new Set();const relaxedObservedBindings = new Set();function revokeRelaxedGrant() {@@ -1995,6 +1999,24 @@if (point) return nativeReceiverBelongsTo(target, receiver);return !!receiver && (receiver === target || target.contains(receiver));}+ function recordFileInventory(token, pairs) {+ if (!active || typeof token !== 'string' || !token || token.length > 100 || !Array.isArray(pairs)) return false;+ const entries = new WeakMap();+ for (const [element, selector] of pairs.slice(0, 64)) {+ if (element instanceof HTMLInputElement && element.type === 'file'+ && typeof selector === 'string' && selector && selector.length <= 4096) entries.set(element, selector);+ }+ fileInventory = { token, runToken, entries };+ return true;+ }+ // Whether an upload's target is the very node that inventory named with+ // this selector. A node the page renders later under the same selector, as+ // a reopened composer does, was never observed.+ function inventoriedFileInput(target, params) {+ return params.tool === 'upload_file' && typeof params.fileInventoryToken === 'string' && !!params.fileInventoryToken+ && fileInventory?.token === params.fileInventoryToken && fileInventory.runToken === runToken+ && typeof params.selector === 'string' && fileInventory.entries.get(target) === params.selector;+ }function modelBindingResult(params = {}) {const snapshot = modelSnapshots.get(params.snapshotToken || params.expectedModelSnapshot);if (!active || params.runToken !== runToken || snapshot?.runToken !== runToken) return { reason: 'model_snapshot_invalid' };@@ -2022,7 +2044,8 @@const original = snapshot.targets.get(target);const before = snapshot.batchInputs?.get(params.batchId)?.get(original) || original;const previous = snapshot.uncertified.get(target);- if (!relaxed && !before && !previous) return { reason: 'target_uncovered', certificationFailure: snapshot.captureFailures.get(target) || 'identity_unobserved' };+ if (!relaxed && !before && !previous) return { reason: 'target_uncovered', certificationFailure: snapshot.captureFailures.get(target) || 'identity_unobserved',+ ...(inventoriedFileInput(target, params) ? { inventoried: true } : {}) };if (params.ref_id && snapshot.hasRefs && snapshot.refs.get(params.ref_id) !== target) {if (relaxed) revokeRelaxedGrant();return { reason: 'target_changed' };@@ -3917,6 +3940,7 @@respond({ ready: !!result.binding, ...(result.uncertified ? { uncertified: true } : {}),...(result.relaxed ? { relaxed: true, bindingKind: 'relaxed_target' } : {}),...(result.certificationFailure ? { certificationFailure: result.certificationFailure } : {}),+ ...(result.inventoried ? { inventoried: true } : {}),...(result.reason ? { reason: result.reason } : {}), runToken, documentToken, snapshotToken: params.snapshotToken });}else if (msg.action === 'page_monitor_traversal') {@@ -4051,7 +4075,7 @@navigationApi?.addEventListener('navigate', onNavigate, true);navigationApi?.addEventListener('navigateerror', onNavigateError, true);navigationApi?.addEventListener('currententrychange', onCurrentEntryChange, true);- window.__wbPageMonitor = { beginContentAction, beforeLocalDispatch, withLocalPreparation, withLocalDispatch, captureLocalInputObservation, dispatch: params => dispatch(params), activatePreparedDispatch, withPreparedDispatch, registerDecoration, isReadonlyPreparationDecoration, currentIdentityPreparationCapture,+ window.__wbPageMonitor = { beginContentAction, beforeLocalDispatch, withLocalPreparation, withLocalDispatch, captureLocalInputObservation, dispatch: params => dispatch(params), activatePreparedDispatch, withPreparedDispatch, registerDecoration, isReadonlyPreparationDecoration, currentIdentityPreparationCapture, recordFileInventory,get active() { return active; },get disposed() { return disposed; },dispose() {diff --git a/test/guard-advisory-actions.mjs b/test/guard-advisory-actions.mjsindex 4cac0a3da..6cc9f73e5 100644--- a/test/guard-advisory-actions.mjs+++ b/test/guard-advisory-actions.mjs@@ -710,7 +710,8 @@ for (const build of ['chrome', 'firefox']) {agent._pageFeedbackRuns = new Map([[tab, run]]);return run;}- async function bindingCall(strict, reason, events = [], { name = 'click_ax', args = { ref_id: 'ref_uncovered' }, prepare = null, lean = false } = {}) {+ async function bindingCall(strict, reason, events = [], { name = 'click_ax', args = { ref_id: 'ref_uncovered' }, prepare = null, lean = false,+ listedNode = true } = {}) {const agent = new Agent({ getVisionProvider: async () => null });agent.strictGuards = strict;const tab = nextTab++;@@ -722,12 +723,20 @@ for (const build of ['chrome', 'firefox']) {const impl = [];agent._executeToolImpl = async (_tab, name) => { impl.push(name); return { success: true, dispatched: true }; };const oldSend = api.tabs.sendMessage;- api.tabs.sendMessage = async (_id, message) => message.action === 'page_monitor_validate_model'- ? { ready: false, reason, runToken: message.params.runToken, documentToken: 'content-1', snapshotToken: message.params.snapshotToken }- : { ready: true };+ const validations = [];+ // The page monitor says whether the upload's node is the one the+ // inventory with this token listed under this selector.+ api.tabs.sendMessage = async (_id, message) => {+ if (message.action !== 'page_monitor_validate_model') return { ready: true };+ validations.push(message.params);+ const inventoried = listedNode && reason === 'target_uncovered' && message.params.tool === 'upload_file'+ && message.params.fileInventoryToken === INVENTORY_TOKEN;+ return { ready: false, reason, ...(inventoried ? { inventoried: true } : {}),+ runToken: message.params.runToken, documentToken: 'content-1', snapshotToken: message.params.snapshotToken };+ };try {const result = await Agent.prototype.executeTool.call(agent, tab, name, args, null, { guardAdvisories: [] });- return { result, impl };+ return { result, impl, validations };} finally { api.tabs.sendMessage = oldSend; }}// Instagram's direct-message composer (trace b71fd1dc): the one file input@@ -736,9 +745,10 @@ for (const build of ['chrome', 'firefox']) {// get_interactive_elements returned. That inventory is the observation an// input without a ref has.const INSTAGRAM_UPLOAD = 'input[type="file"][accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg"][multiple]';+ const INVENTORY_TOKEN = 'file-inventory-1';const inventoried = (selectors = [INSTAGRAM_UPLOAD], { documentToken = 'content-1', url = PAGE_URL } = {}) => run => {run.frames.set(0, { token: 'content-1', id: 'document-1' });- run.uploadObservationScope = { documentToken, url, selectors };+ run.uploadObservationScope = { documentToken, url, selectors, token: INVENTORY_TOKEN };};const upload = (prepare, selector = INSTAGRAM_UPLOAD) => ({ name: 'upload_file', args: { selector, downloadId: 2157 }, prepare });for (const [mode, strict] of MODES) {@@ -764,7 +774,8 @@ for (const build of ['chrome', 'firefox']) {});test(`${build}: page-feedback-target-binding ${mode} mode ${strict ? 'refuses' : 'sends'} an upload to the file input the inventory named`, async () => {- const { result, impl } = await bindingCall(strict, 'target_uncovered', [], upload(inventoried()));+ const { result, impl, validations } = await bindingCall(strict, 'target_uncovered', [], upload(inventoried()));+ assert.equal(validations[0]?.fileInventoryToken, INVENTORY_TOKEN, 'the monitor is asked about that inventory\'s node');if (strict) {assert.deepEqual(impl, []);assert.equal(result.errorCode, 'action_binding_unavailable');@@ -795,6 +806,9 @@ for (const build of ['chrome', 'firefox']) {['another URL\'s inventory', 'target_uncovered', [], upload(inventoried(undefined, { url: 'https://example.com/other' }))],['an ambiguous inventory selector', 'target_ambiguous', [], upload(inventoried())],['pending page activity', 'target_uncovered', [{ kind: 'dom', source: 'unknown', frameId: 0, target: 'div#thread' }], upload(inventoried())],+ // Codex review of #502: a node the page rendered after the inventory+ // under the same selector (a re-mounted composer's input).+ ['a node the inventory did not list', 'target_uncovered', [], { ...upload(inventoried()), listedNode: false }],];for (const [label, reason, events, options] of cases) {const { result, impl } = await bindingCall(strict, reason, events, options);diff --git a/test/lib/upload-release-fixture.mjs b/test/lib/upload-release-fixture.mjsindex b25460226..50fe0b40f 100644--- a/test/lib/upload-release-fixture.mjs+++ b/test/lib/upload-release-fixture.mjs@@ -49,6 +49,36 @@ export const xComposeUploadHtml = `<!doctype html><meta charset="utf-8"><title>C// A page may replace its hidden transport wrapper with a directly clipped// native slot after consuming the first attachment. Asset editors live next// to the exclusive dropzone and remain part of the same release form.+// Instagram's direct-message thread as trace b71fd1dc read it (2026-10): no+// form, header or footer; the one file input, display:none by class between+// Voice Clip and Add Photo or Video, sits several wrappers deep below a+// thread too large for the page monitor's owner bounds, so no certificate+// covers it. rerenderComposerInput() swaps in an identical new input, as a+// re-mounted composer does. Owned names only.+export function instagramDirectThreadHtml(recipient = 'Owned Recipient', handle = 'owned.recipient') {+ const icon = label => `<div role="button" tabindex="0"><svg role="img" aria-label="${label}"><title>${label}</title><path d="M0 0h24v24H0z"/></svg></div>`;+ const nest = (html, depth) => Array.from({ length: depth }).reduce(inner => `<div>${inner}</div>`, html);+ const threads = ['Ungvb', recipient, 'Owned Contact One', 'Owned Contact Two', 'Owned Contact Three']+ .map(name => `<div role="button" tabindex="0"><img alt="user-profile-picture" src="data:,"><span>${name}</span></div>`).join('');+ const history = Array.from({ length: 60 }, (_, index) => nest(`<div role="group"><div role="article" data-message-direction="incoming">Message ${index + 1}</div></div>`, 5)).join('');+ const composer = `<div class="composer">${icon('Choose an emoji')}<div role="textbox" contenteditable="true" aria-placeholder="Message..." id="body"><p><br></p></div>`+ + `${icon('Voice Clip')}<input accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg" class="x1s85apg" multiple type="file">${icon('Add Photo or Video')}${icon('Choose a GIF or sticker')}</div>`;+ return `<!doctype html><meta charset="utf-8"><title>Instagram β’ Messages</title>+<style>body{margin:0;font:14px sans-serif}.x1s85apg{display:none}#threads{position:fixed;left:72px;top:0;width:400px;height:700px;overflow:auto}+#pane{position:fixed;left:480px;top:0;width:640px;height:700px;display:flex;flex-direction:column}#head{height:75px}+#thread{flex:1;overflow:auto}.composer{display:flex;align-items:center;height:44px;margin:12px;border:1px solid #888;border-radius:22px}+[contenteditable]{flex:1;min-height:20px}[role=button]{display:inline-flex;width:40px;height:40px}svg{width:24px;height:24px}</style>+<div role="main"><div id="threads"><nav aria-label="Thread list"><h1>owned.account</h1>${threads}</nav></div>+<div id="pane"><div id="head"><a href="/${handle}/" aria-label="Open the profile page of ${handle}"><img alt="user-profile-picture" src="data:,"><h2>${recipient}</h2><span>${handle}</span></a>+${icon('Audio call')}${icon('Video call')}${icon('Conversation information')}</div>+<div id="thread" role="log">${history}</div>${nest(composer, 6)}</div></div>+<script>window.composerInputChanges=[];+const onComposerInputChange=event=>{composerInputChanges.push([...event.target.files].map(file=>file.name));window.dispatchEvent(new CustomEvent('composer-input-change',{detail:event.target}));};+document.querySelector('input[type=file]').addEventListener('change',onComposerInputChange);+window.rerenderComposerInput=()=>{const current=document.querySelector('input[type=file]'),next=current.cloneNode(false);+ next.addEventListener('change',onComposerInputChange);current.replaceWith(next);};</script>`;+}+export function stageSequentialReleaseUploads(hiding = 'clipped') {document.querySelectorAll('h1,h2').forEach(node => node.remove());const decoration = document.createElement('div'); decoration.hidden = true;diff --git a/test/page-monitor-upload-release.mjs b/test/page-monitor-upload-release.mjsindex 2a70af83c..40a53ea19 100644--- a/test/page-monitor-upload-release.mjs+++ b/test/page-monitor-upload-release.mjs@@ -2,7 +2,7 @@ import assert from 'node:assert/strict';import fs from 'node:fs/promises';import { test } from 'node:test';import { chromium, firefox } from 'playwright';-import { uploadReleaseHtml, stageSequentialReleaseUploads, xComposeUploadHtml } from './lib/upload-release-fixture.mjs';+import { uploadReleaseHtml, stageSequentialReleaseUploads, xComposeUploadHtml, instagramDirectThreadHtml } from './lib/upload-release-fixture.mjs';const recipientUploadHtml = `<!doctype html><style>#binary-widget{display:block;width:400px;padding:10px}input[type=file]{width:1px;height:1px;opacity:0}@@ -285,6 +285,34 @@ for (const [build, engine] of [['chrome', chromium], ['firefox', firefox]]) {const validation = await validate(page, fresh, dialogInput.selector);assert.equal(validation.ready, true, JSON.stringify(validation));}, xComposeUploadHtml));+ // Trace b71fd1dc and the Codex review of #502: Instagram's message input+ // is formless, hidden and deep in a large thread, so it is never+ // certified. The monitor knows which node an inventory listed under its+ // selector; a node the page re-mounts under the same selector is not it.+ await t.test('Instagram composer input is known by the node its inventory listed, not by its selector', () => withFixture(async page => {+ const inventory = token => page.evaluate(token => deliver('get_interactive_elements_cdp', { _fileInventoryToken: token }), token)+ .then(items => items.filter(item => item.type === 'file'));+ const check = (snapshot, selector, fileInventoryToken) => page.evaluate(params => deliver('page_monitor_validate_model',+ { tool: 'upload_file', ...params }), { selector, snapshotToken: snapshot.snapshotToken, ...(fileInventoryToken ? { fileInventoryToken } : {}) });+ const [file] = await inventory('inventory-1');+ assert.equal(file?.selector, 'input[type="file"][accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg"][multiple]', JSON.stringify(file));+ const snapshot = await capture(page, { uploadTargets: [file.selector] });+ const listed = await check(snapshot, file.selector, 'inventory-1');+ assert.equal(listed.ready, false, JSON.stringify(listed));+ assert.equal(listed.reason, 'target_uncovered', JSON.stringify(listed));+ assert.equal(listed.inventoried, true, JSON.stringify(listed));+ assert.equal((await check(snapshot, file.selector, 'inventory-0')).inventoried, undefined, 'another inventory');+ assert.equal((await check(snapshot, file.selector)).inventoried, undefined, 'no inventory');+ await page.evaluate(() => window.rerenderComposerInput());+ const remounted = await check(await capture(page, { uploadTargets: [file.selector] }), file.selector, 'inventory-1');+ assert.equal(remounted.reason, 'target_uncovered', JSON.stringify(remounted));+ assert.equal(remounted.inventoried, undefined, `a re-mounted input was never listed: ${JSON.stringify(remounted)}`);+ const [fresh] = await inventory('inventory-2');+ assert.equal(fresh?.selector, file.selector);+ const current = await capture(page, { uploadTargets: [fresh.selector] });+ assert.equal((await check(current, fresh.selector, 'inventory-2')).inventoried, true, 'the fresh inventory lists the new node');+ assert.equal((await check(current, fresh.selector, 'inventory-1')).inventoried, undefined, 'the older inventory is spent');+ }, instagramDirectThreadHtml()));for (const authority of ['user', 'Stop', 'run', 'document']) await t.test(`${authority} revokes upload authority`, () => withFixture(async page => {const snapshot = await capture(page);const attached = await attach(page, snapshot, async page => {diff --git a/test/quality/recipient-clarify-mv3.mjs b/test/quality/recipient-clarify-mv3.mjsindex 2c0d91490..535f8837a 100644--- a/test/quality/recipient-clarify-mv3.mjs+++ b/test/quality/recipient-clarify-mv3.mjs@@ -6,6 +6,7 @@ import path from 'node:path';import { chromium } from 'playwright';import { conformFixtureCompletionCall } from './completion-provider-contract.mjs';import { fixedChoiceQuestion, fixtureFixedChoiceReply } from '../semantic-decision-stub.mjs';+import { instagramDirectThreadHtml } from '../lib/upload-release-fixture.mjs';// The questions a messaging page's own checks read its texts with.const MESSAGING_TEXT_KINDS = ['message_control', 'message_field', 'message_author', 'message_status'];@@ -31,33 +32,13 @@ const variants = [// Trace b71fd1dc: on Instagram's own composer every upload_file was refused// (action_binding_unavailable) after the user had confirmed the recipient.['instagram-composer', ['download', 'list', 'inventory', 'upload', 'tree', 'done'], null, null, 'once', 1, 'instagram'],+ // The page re-renders that input after the inventory. The new node was never+ // listed, so its upload is refused; the next inventory's upload goes out.+ ['instagram-composer-rerendered', ['download', 'list', 'inventory', 'upload', 'inventory', 'upload', 'tree', 'done'],+ null, null, 'once', 1, 'instagram-rerendered'],];for (const variant of variants) await runVariant(...variant);-// Instagram's composer as trace b71fd1dc read it (2026-10): no form, header or-// footer, and the one file input, display:none by class between Voice Clip and-// Add Photo or Video, sits several wrappers deep in a thread too large for the-// page monitor's owner bounds. No certificate covers that input, so the model-// names it by the selector get_interactive_elements returned.-function instagramThreadPage() {- const icon = label => `<div role="button" tabindex="0"><svg role="img" aria-label="${label}"><title>${label}</title><path d="M0 0h24v24H0z"/></svg></div>`;- const nest = (html, depth) => Array.from({ length: depth }).reduce(inner => `<div>${inner}</div>`, html);- const threads = ['Ungvb', recipient, 'Owned Contact One', 'Owned Contact Two', 'Owned Contact Three']- .map(name => `<div role="button" tabindex="0"><img alt="user-profile-picture" src="data:,"><span>${name}</span></div>`).join('');- const history = Array.from({ length: 60 }, (_, index) => nest(`<div role="group"><div role="article" data-message-direction="incoming">Message ${index + 1}</div></div>`, 5)).join('');- const composer = `<div class="composer">${icon('Choose an emoji')}<div role="textbox" contenteditable="true" aria-placeholder="Message..." id="body"><p><br></p></div>`- + `${icon('Voice Clip')}<input accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg" class="x1s85apg" multiple type="file">${icon('Add Photo or Video')}${icon('Choose a GIF or sticker')}</div>`;- return `<!doctype html><meta charset="utf-8"><title>Instagram β’ Messages</title>- <style>body{margin:0;font:14px sans-serif}.x1s85apg{display:none}#threads{position:fixed;left:72px;top:0;width:400px;height:700px;overflow:auto}- #pane{position:fixed;left:480px;top:0;width:640px;height:700px;display:flex;flex-direction:column}#head{height:75px}- #thread{flex:1;overflow:auto}.composer{display:flex;align-items:center;height:44px;margin:12px;border:1px solid #888;border-radius:22px}- [contenteditable]{flex:1;min-height:20px}[role=button]{display:inline-flex;width:40px;height:40px}svg{width:24px;height:24px}</style>- <div role="main"><div id="threads"><nav aria-label="Thread list"><h1>owned.account</h1>${threads}</nav></div>- <div id="pane"><div id="head"><a href="/dr.gamzec_/" aria-label="Open the profile page of dr.gamzec_"><img alt="user-profile-picture" src="data:,"><h2>${recipient}</h2><span>dr.gamzec_</span></a>- ${icon('Audio call')}${icon('Video call')}${icon('Conversation information')}</div>- <div id="thread" role="log">${history}</div>${nest(composer, 6)}</div></div>`;-}-async function runVariant(variant, steps, answer, source, cardAnswer, expectedSends, threadPage = 'owned') {const approved = expectedSends > 0;const out = path.resolve(process.env.WEBBRAIN_QUALITY_ARTIFACTS@@ -71,6 +52,7 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSesummary: 'Attach the downloaded photo in the Gamze conversation.', steps: [{ id: '1', action: 'Attach the photo and verify it was sent.' }],memory: { use_progress_ledger: false, progress_action: null }, scheduling: null, risks: [] };const sends = [], providerRequests = [], clarifications = [], cards = [];+ let threadTab = null, rerendered = false;let actionCalls = 0, judgeCalls = 0, blockedRemote = 0, base = '';const completionRecovery = { reads: 0 };const textOf = content => typeof content === 'string' ? content : Array.isArray(content)@@ -111,6 +93,11 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSeelse {const step = actionCalls++;const kind = steps[Math.min(step, steps.length - 1)];+ // After the model listed the file inputs, the page re-mounts its own.+ if (kind === 'upload' && threadPage === 'instagram-rerendered' && !rerendered) {+ rerendered = true;+ await threadTab.evaluate(() => window.rerenderComposerInput());+ }const call = kind === 'download' ? { name: 'download_files', args: { urls: [`${base}/photo.png`] } }: kind === 'list' ? { name: 'list_downloads', args: { limit: 5 } }: kind === 'inventory' ? { name: 'get_interactive_elements', args: {} }@@ -138,7 +125,20 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSe});await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));base = `http://127.0.0.1:${server.address().port}`;- const pageBody = (threadPage === 'instagram' ? instagramThreadPage() : `<!doctype html><meta charset="utf-8"><title>Owned direct thread</title>+ // The page sends a photo once its message box's file input changes.+ const sendPhoto = `async input => {+ const file = input.files[0]; if (!file) return;+ const body = new FormData(); body.append('photo', file);+ const result = await fetch('/api/v1/direct_v2/threads/17848282661759097/items/configure_photo/', { method: 'POST', body });+ if (result.ok) {+ const item = document.createElement('div'); item.dataset.messageDirection = 'outgoing'; item.textContent = 'You sent an attachment.';+ document.querySelector('#thread').append(item);+ }+ }`;+ const pageBody = threadPage.startsWith('instagram')+ ? instagramDirectThreadHtml(recipient, 'dr.gamzec_')+ + `<script>window.addEventListener('composer-input-change', event => (${sendPhoto})(event.detail));</script>`+ : `<!doctype html><meta charset="utf-8"><title>Owned direct thread</title><style>body{margin:0;font:16px sans-serif} nav{position:fixed;left:0;top:0;width:330px;height:700px}header{position:fixed;left:350px;top:10px;width:600px;height:70px} #thread{position:fixed;left:350px;top:90px;width:600px;height:420px}footer{position:fixed;left:350px;top:600px;width:600px;height:60px} [contenteditable]{display:inline-block;width:420px;height:40px;border:1px solid #888}@@ -148,18 +148,8 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSe<div id="thread" role="log"><div data-message-direction="incoming">Merhaba!</div></div><footer><form id="composer" onsubmit="return false"><div role="textbox" contenteditable="true" aria-label="Message" id="body"></div><div role="button" id="attach"><svg role="img" aria-label="Add Photo or Video"><title>Add Photo or Video</title><path d="M0 0h24v24H0z"/></svg></div>- <input type="file" accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg" multiple hidden></form></footer></main>`) + `- <script>- document.querySelector('input[type=file]').addEventListener('change', async event => {- const file = event.target.files[0]; if (!file) return;- const body = new FormData(); body.append('photo', file);- const result = await fetch('/api/v1/direct_v2/threads/17848282661759097/items/configure_photo/', { method: 'POST', body });- if (result.ok) {- const item = document.createElement('div'); item.dataset.messageDirection = 'outgoing'; item.textContent = 'You sent an attachment.';- document.querySelector('#thread').append(item);- }- });- </script>`;+ <input type="file" accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg" multiple hidden></form></footer></main>+ <script>document.querySelector('input[type=file]').addEventListener('change', event => (${sendPhoto})(event.target));</script>`;const profile = await fs.mkdtemp(path.join(os.tmpdir(), 'webbrain-recipient-clarify-mv3-'));let context;try {@@ -200,7 +190,7 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSecontextWindow: 1000000, supportsAskStreaming: false } })).ok, true);assert.equal((await message('set_active_provider', { providerId: 'openrouter' })).ok, true);await worker.evaluate(() => { recipientFixtureAgent.clarifyTimeoutSec = -1; });- const page = await context.newPage(); await page.goto(threadUrl);+ const page = await context.newPage(); await page.goto(threadUrl); threadTab = page;const tabId = await harness.evaluate(async url => (await chrome.tabs.query({})).find(tab => tab.url === url)?.id, threadUrl);// Answer the model's question the way the user did, and WebBrain's card// with the variant's choice.@@ -244,8 +234,12 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSeassert.equal(card.question, `Send to ${recipient}?`);}if (approved) {- assert.equal(uploads.length, 1, JSON.stringify(uploads));- assert.equal(uploads[0].success, true, JSON.stringify(uploads[0]));+ assert.equal(uploads.length, steps.filter(kind => kind === 'upload').length, JSON.stringify(uploads));+ assert.equal(uploads.at(-1).success, true, JSON.stringify(uploads.at(-1)));+ if (threadPage === 'instagram-rerendered') {+ assert.equal(uploads[0].success, false, JSON.stringify(uploads[0]));+ assert.match(uploads[0].error || '', /not covered by the observed action context/, JSON.stringify(uploads[0]));+ }assert.deepEqual(sends, [{ method: 'POST', path: '/api/v1/direct_v2/threads/17848282661759097/items/configure_photo/', containsPhoto: true }]);if (clarifications.length) {const clarify = toolResults.find(item => item.name === 'clarify');