Open-source AI browser agent for Chrome and Firefox (monorepo) π§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Let Act send photos and videos in Instagram direct messages
Claudec1bdfaecd3c4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 89b50ff98..96f9966b6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -16,6 +16,7 @@ This changelog was generated from the repository Git history and release tags. V- The spinning ring next to "Thinkingβ¦" in the side panel is now a small WebBrain brain drawn as a neural circuit: a signal keeps tracing the W, and the brain's outline lights up as it passes. It moves only while WebBrain is working and the panel is visible, follows the light and dark themes, and holds still when your system asks to reduce motion.### Fixed+- Act can send a photo or video in an Instagram direct message. Instagram's message box keeps its file input hidden, outside any form, deep in a long conversation, where WebBrain's check that an action still targets what the model saw cannot reach it. Every attachment was refused, even with the exact selector WebBrain itself listed for that input, often right after you had confirmed the recipient on WebBrain's card; the model then sent a link to the picture, or said Instagram blocks attachments. With Strict checks off, an upload to an input named by the selector WebBrain listed on that page now goes through with a note to the model, and the recipient checks still apply; a selector WebBrain did not list is still refused. When clicking "Add Photo or Video" opens a file chooser, WebBrain now names the input with that same selector, and the Instagram site notes say to attach the file to that input instead of clicking the button, and that a link to a picture is not the picture.- Act can attach images and videos to a post on X again. While the post composer is open at x.com/compose/post, the timeline behind it keeps a second, identical file input, and WebBrain's list of the page's controls gave neither input a selector. After the model's first selector matched both, WebBrain refused every later upload until it saw a selector it could not get, so the run ended at its cost limit with nothing attached. The list now gives every file input a selector that matches only that input, scoped to the open dialog where that is what tells them apart, and marks the dialog's input `inDialog`. The X site notes name the dialog's input, and when no selector can single out an input, WebBrain tells the model to report the upload as blocked instead of retrying.## [42.3.1] - 2026-10-20diff --git a/docs/page-feedback-monitor.md b/docs/page-feedback-monitor.mdindex 738ae6941..f68327765 100644--- a/docs/page-feedback-monitor.md+++ b/docs/page-feedback-monitor.md@@ -294,7 +294,17 @@ Steer or grant authorization. Enter and Alt+Enter retain their existing behaviorand one activation purpose; the actual input, widget, form, owner and documentremain bound. The current observation pairs the selector with its visiblewidget as untrusted page data. Discovery never repairs or dispatches an old- rejected call, opens a chooser, or grants upload permission.+ rejected call, opens a chooser, or grants upload permission. A file input no+ certificate covers (`target_uncovered`; Instagram's direct-message composer+ keeps a formless, `display:none` input deep in a thread too large for the+ owner bounds) has no ref, so an `upload_file` whose selector is exactly one+ that `get_interactive_elements` returned for the current top document and URL+ names the input the model observed: with nothing pending and Strict checks+ off it is sent with the ordinary unbound semantics and a+ `page-feedback-target-binding` note, and the recipient checks of a protected+ conversation still bind the attachment. Any other selector keeps its refusal.+ A blocked native chooser names its input with the selector the inventory+ uses.- Native `summary` controls and authored click/tab-focus targets use the sameobserved owner and exact action footprint as native and ARIA buttons. Changedhandler identity, disclosure state, target/owner identity and user interventiondiff --git a/src/chrome/src/agent/adapters.js b/src/chrome/src/agent/adapters.jsindex 96344e868..54ddd6efa 100644--- a/src/chrome/src/agent/adapters.js+++ b/src/chrome/src/agent/adapters.js@@ -17506,6 +17506,7 @@ const ADAPTERS = [- Story bar at top of profile / feed is keyboard-driven: left/right arrows advance, Esc closes. Clicking is unreliable.- Profile grid (/<user>) lazy-loads via IntersectionObserver β scroll the page (not a sub-container) to load more posts.- DMs at /direct/inbox β sign-in required.+- In a /direct/t/<thread> conversation, a photo or video attaches through the composer's hidden input[type="file"] beside "Add Photo or Video" (accept "audio/*,.mp4,.mov,.png,.jpg,.jpeg"): pass the selector get_interactive_elements returns for it to upload_file and never click that button, which opens an OS file dialog. A link to the image is not the image: when the user asks to send a picture, send the file (as of 2026-10).- Hashtag pages: /explore/tags/<tag>. Location pages: /explore/locations/<id>.- Post carousels at /p/<id>/ expose deterministic ?img_index=N routes. Use carousel_navigate({index:N}) to visit slides directly and monotonically; decrease only for a fresh user-requested reverse scan. Never use ArrowLeft/ArrowRight, coordinate clicks, or alternate Next/Go back while enumerating a carousel.- "Add to story / Add to post" actions require the mobile app for most content types β surface the limitation.diff --git a/src/chrome/src/agent/page-feedback.js b/src/chrome/src/agent/page-feedback.jsindex 7d3c1740f..eae5e612d 100644--- a/src/chrome/src/agent/page-feedback.js+++ b/src/chrome/src/agent/page-feedback.js@@ -250,6 +250,17 @@ function actionBindingUnavailableResult(run, name) {: { error: 'The exact target could not be bound to the observed action context. Fresh available target/page context will be observed before the next decision; inspect it or read the exact target inventory before trying the action again.' }) };}+// Whether a selector is one that WebBrain's own file-input inventory+// (get_interactive_elements) returned on the run's current top document and+// URL: the selectors that inventory recorded are this run's observed upload+// targets, and an exact copy of one names the input the model observed.+function uploadSelectorObserved(run, selector) {+ const scope = run?.uploadObservationScope;+ return typeof selector === 'string' && selector.length > 0 && !!scope+ && scope.documentToken === run.frames?.get(0)?.token && scope.url === run.url+ && Array.isArray(scope.selectors) && scope.selectors.includes(selector);+}+function preparationVetoDiagnostics(veto) {const facts = {};if (!veto || typeof veto !== 'object') return facts;@@ -1433,27 +1444,36 @@ export const pageFeedbackMethods = {|| pageFeedbackCallPolicy(name, args, run.modelState, run.latestPage).kind !== 'bound_target') return null;if (await this._validatePageFeedbackAction(tabId, name, args, run.modelState, null, { answerOnly: true }) !== false) return null;if (run.actionBindingUnavailable !== `${name}:${JSON.stringify(args)}`) return null;- return this._pageFeedbackBindingAdvises(tabId, run, args, advisories) ? null : actionBindingUnavailableResult(run, name);+ return this._pageFeedbackBindingAdvises(tabId, run, args, advisories, name) ? null : actionBindingUnavailableResult(run, name);},// page-feedback-target-binding (quality): the observed action certificate- // does not cover this call's ref (target_uncovered: the element the model- // read exists and is unique, but the certified observation did not include- // it). With nothing else pending (no page activity, so no human input or- // unknown attribution, and no correction), Strict checks off sends the call- // with the ordinary unbound semantics and a note; the dispatch fences at the- // transport still run. A selector may name an element the model never+ // does not cover this call's target (target_uncovered: the element exists+ // and is unique, but the certified observation did not include it). With+ // nothing else pending (no page activity, so no human input or unknown+ // attribution, and no correction), Strict checks off sends the call with the+ // ordinary unbound semantics and a note; the dispatch fences at the+ // transport still run. A ref is what the model read. A hidden file input has+ // no ref, so upload_file names it by selector: one that WebBrain's own+ // file-input inventory gave for this document and URL is the input the+ // model observed. Instagram's direct-message composer keeps such an input,+ // formless and display:none, deep in a large thread, where no certificate+ // can reach it. Any other selector may name an element the model never// observed (an upload slot it guessed, say), and a selector naming several// elements or none has no single element to send to: those keep their// answer, as does anything pending.- _pageFeedbackBindingAdvises(tabId, run, args = {}, advisories = null) {+ _pageFeedbackBindingAdvises(tabId, run, args = {}, advisories = null, name = '') {if (!run || run.events.size || run.actionBindingUnavailableReason !== 'target_uncovered'- || typeof args?.ref_id !== 'string' || !/^ref_[A-Za-z0-9_-]+$/.test(args.ref_id)- || (typeof args?.selector === 'string' && args.selector.trim())|| this._hasPendingSteering?.(tabId) || typeof this._guardAdvisesFor !== 'function') return false;- return this._guardAdvisesFor(advisories, 'page-feedback-target-binding',- 'This target was outside the page observation WebBrain certified for this action, so its binding to what you observed was not checked. It was sent anyway; read the page to confirm what it changed.',- { tabId, decided: false });+ const hasSelector = typeof args?.selector === 'string' && !!args.selector.trim();+ const observedRef = typeof args?.ref_id === 'string' && /^ref_[A-Za-z0-9_-]+$/.test(args.ref_id) && !hasSelector;+ const observedUpload = name === 'upload_file' && args?.ref_id == null && hasSelector+ && uploadSelectorObserved(run, args.selector);+ if (!observedRef && !observedUpload) return false;+ return this._guardAdvisesFor(advisories, 'page-feedback-target-binding', observedUpload+ ? 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input your file-input inventory named without that check. Read the page to confirm the file landed where you intended.'+ : 'This target was outside the page observation WebBrain certified for this action, so its binding to what you observed was not checked. It was sent anyway; read the page to confirm what it changed.',+ { tabId, decided: false });},_pageFeedbackBlocksCall(tabId, name, args = {}) {@@ -1839,7 +1859,7 @@ export function installPageFeedback(Agent) {// A pure binding failure with nothing else pending may advise instead// (it then dispatches with the ordinary, unbound semantics below).if (validated === false && !(unbindable- && this._pageFeedbackBindingAdvises(tabId, run, args, executionContext?.guardAdvisories))) return unbindable+ && this._pageFeedbackBindingAdvises(tabId, run, args, executionContext?.guardAdvisories, name))) return unbindable? actionBindingUnavailableResult(run, name): run.actionContextUnverified === true? { ...pageFeedbackPendingResult(), actionContextUnverified: true, errorCode: 'action_context_unverified',diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex 2f549a454..eaf559ca5 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -1411,8 +1411,12 @@const isFileInput = (input) =>input?.tagName === 'INPUT'&& String(input.getAttribute?.('type') || input.type || '').toLowerCase() === 'file';+ // Name the input as the file-input inventory does, so the selector the+ // model is told to use is one get_interactive_elements also returns.const blockFileInput = (input) => {- state.blocked = { selector: uniqueFileInputSelector(input) };+ let selector = '';+ try { selector = _uniqueFileInputSelector(input); } catch {}+ state.blocked = { selector: selector || uniqueFileInputSelector(input) };};const guard = (event) => {const path = typeof event.composedPath === 'function'@@ -1434,13 +1438,24 @@root.setAttribute(guardAttr, guardId);document.dispatchEvent(new Event('webbrain:file-picker-guard-arm'));};+ // The page world names the input by its path from the root. When that+ // path still singles out one file input, name it as the inventory does.+ const inventorySelector = (selector) => {+ try {+ const matches = _deepSelectorMatches(selector);+ if (matches.length === 1 && matches[0] instanceof HTMLInputElement && matches[0].type === 'file') {+ return _uniqueFileInputSelector(matches[0]) || selector;+ }+ } catch {}+ return selector;+ };const onBlocked = () => {try {const payload = JSON.parse(root.getAttribute(blockedAttr) || 'null');if (payload?.guardId !== guardId) return;state.blocked = {selector: typeof payload.selector === 'string' && payload.selector- ? payload.selector+ ? inventorySelector(payload.selector): null,};} catch {}diff --git a/src/firefox/src/agent/adapters.js b/src/firefox/src/agent/adapters.jsindex 25ef0e542..904792439 100644--- a/src/firefox/src/agent/adapters.js+++ b/src/firefox/src/agent/adapters.js@@ -17504,6 +17504,7 @@ const ADAPTERS = [- Story bar at top of profile / feed is keyboard-driven: left/right arrows advance, Esc closes. Clicking is unreliable.- Profile grid (/<user>) lazy-loads via IntersectionObserver β scroll the page (not a sub-container) to load more posts.- DMs at /direct/inbox β sign-in required.+- In a /direct/t/<thread> conversation, a photo or video attaches through the composer's hidden input[type="file"] beside "Add Photo or Video" (accept "audio/*,.mp4,.mov,.png,.jpg,.jpeg"): pass the selector get_interactive_elements returns for it to upload_file and never click that button, which opens an OS file dialog. A link to the image is not the image: when the user asks to send a picture, send the file (as of 2026-10).- Hashtag pages: /explore/tags/<tag>. Location pages: /explore/locations/<id>.- Post carousels at /p/<id>/ expose deterministic ?img_index=N routes. Use carousel_navigate({index:N}) to visit slides directly and monotonically; decrease only for a fresh user-requested reverse scan. Never use ArrowLeft/ArrowRight, coordinate clicks, or alternate Next/Go back while enumerating a carousel.- "Add to story / Add to post" actions require the mobile app for most content types β surface the limitation.diff --git a/src/firefox/src/agent/page-feedback.js b/src/firefox/src/agent/page-feedback.jsindex 7d3c1740f..eae5e612d 100644--- a/src/firefox/src/agent/page-feedback.js+++ b/src/firefox/src/agent/page-feedback.js@@ -250,6 +250,17 @@ function actionBindingUnavailableResult(run, name) {: { error: 'The exact target could not be bound to the observed action context. Fresh available target/page context will be observed before the next decision; inspect it or read the exact target inventory before trying the action again.' }) };}+// Whether a selector is one that WebBrain's own file-input inventory+// (get_interactive_elements) returned on the run's current top document and+// URL: the selectors that inventory recorded are this run's observed upload+// targets, and an exact copy of one names the input the model observed.+function uploadSelectorObserved(run, selector) {+ const scope = run?.uploadObservationScope;+ return typeof selector === 'string' && selector.length > 0 && !!scope+ && scope.documentToken === run.frames?.get(0)?.token && scope.url === run.url+ && Array.isArray(scope.selectors) && scope.selectors.includes(selector);+}+function preparationVetoDiagnostics(veto) {const facts = {};if (!veto || typeof veto !== 'object') return facts;@@ -1433,27 +1444,36 @@ export const pageFeedbackMethods = {|| pageFeedbackCallPolicy(name, args, run.modelState, run.latestPage).kind !== 'bound_target') return null;if (await this._validatePageFeedbackAction(tabId, name, args, run.modelState, null, { answerOnly: true }) !== false) return null;if (run.actionBindingUnavailable !== `${name}:${JSON.stringify(args)}`) return null;- return this._pageFeedbackBindingAdvises(tabId, run, args, advisories) ? null : actionBindingUnavailableResult(run, name);+ return this._pageFeedbackBindingAdvises(tabId, run, args, advisories, name) ? null : actionBindingUnavailableResult(run, name);},// page-feedback-target-binding (quality): the observed action certificate- // does not cover this call's ref (target_uncovered: the element the model- // read exists and is unique, but the certified observation did not include- // it). With nothing else pending (no page activity, so no human input or- // unknown attribution, and no correction), Strict checks off sends the call- // with the ordinary unbound semantics and a note; the dispatch fences at the- // transport still run. A selector may name an element the model never+ // does not cover this call's target (target_uncovered: the element exists+ // and is unique, but the certified observation did not include it). With+ // nothing else pending (no page activity, so no human input or unknown+ // attribution, and no correction), Strict checks off sends the call with the+ // ordinary unbound semantics and a note; the dispatch fences at the+ // transport still run. A ref is what the model read. A hidden file input has+ // no ref, so upload_file names it by selector: one that WebBrain's own+ // file-input inventory gave for this document and URL is the input the+ // model observed. Instagram's direct-message composer keeps such an input,+ // formless and display:none, deep in a large thread, where no certificate+ // can reach it. Any other selector may name an element the model never// observed (an upload slot it guessed, say), and a selector naming several// elements or none has no single element to send to: those keep their// answer, as does anything pending.- _pageFeedbackBindingAdvises(tabId, run, args = {}, advisories = null) {+ _pageFeedbackBindingAdvises(tabId, run, args = {}, advisories = null, name = '') {if (!run || run.events.size || run.actionBindingUnavailableReason !== 'target_uncovered'- || typeof args?.ref_id !== 'string' || !/^ref_[A-Za-z0-9_-]+$/.test(args.ref_id)- || (typeof args?.selector === 'string' && args.selector.trim())|| this._hasPendingSteering?.(tabId) || typeof this._guardAdvisesFor !== 'function') return false;- return this._guardAdvisesFor(advisories, 'page-feedback-target-binding',- 'This target was outside the page observation WebBrain certified for this action, so its binding to what you observed was not checked. It was sent anyway; read the page to confirm what it changed.',- { tabId, decided: false });+ const hasSelector = typeof args?.selector === 'string' && !!args.selector.trim();+ const observedRef = typeof args?.ref_id === 'string' && /^ref_[A-Za-z0-9_-]+$/.test(args.ref_id) && !hasSelector;+ const observedUpload = name === 'upload_file' && args?.ref_id == null && hasSelector+ && uploadSelectorObserved(run, args.selector);+ if (!observedRef && !observedUpload) return false;+ return this._guardAdvisesFor(advisories, 'page-feedback-target-binding', observedUpload+ ? 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input your file-input inventory named without that check. Read the page to confirm the file landed where you intended.'+ : 'This target was outside the page observation WebBrain certified for this action, so its binding to what you observed was not checked. It was sent anyway; read the page to confirm what it changed.',+ { tabId, decided: false });},_pageFeedbackBlocksCall(tabId, name, args = {}) {@@ -1839,7 +1859,7 @@ export function installPageFeedback(Agent) {// A pure binding failure with nothing else pending may advise instead// (it then dispatches with the ordinary, unbound semantics below).if (validated === false && !(unbindable- && this._pageFeedbackBindingAdvises(tabId, run, args, executionContext?.guardAdvisories))) return unbindable+ && this._pageFeedbackBindingAdvises(tabId, run, args, executionContext?.guardAdvisories, name))) return unbindable? actionBindingUnavailableResult(run, name): run.actionContextUnverified === true? { ...pageFeedbackPendingResult(), actionContextUnverified: true, errorCode: 'action_context_unverified',diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex ede18bb76..5b5a7d6d9 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -2144,8 +2144,12 @@const isFileInput = (input) =>input?.tagName === 'INPUT'&& String(input.getAttribute?.('type') || input.type || '').toLowerCase() === 'file';+ // Name the input as the file-input inventory does, so the selector the+ // model is told to use is one get_interactive_elements also returns.const blockFileInput = (input) => {- state.blocked = { selector: uniqueFileInputSelector(input) };+ let selector = '';+ try { selector = _uniqueFileInputSelector(input); } catch {}+ state.blocked = { selector: selector || uniqueFileInputSelector(input) };};const guard = (event) => {const path = typeof event.composedPath === 'function'@@ -2167,13 +2171,24 @@root.setAttribute(guardAttr, guardId);document.dispatchEvent(new Event('webbrain:file-picker-guard-arm'));};+ // The page world names the input by its path from the root. When that+ // path still singles out one file input, name it as the inventory does.+ const inventorySelector = (selector) => {+ try {+ const matches = _deepSelectorMatches(selector);+ if (matches.length === 1 && matches[0] instanceof HTMLInputElement && matches[0].type === 'file') {+ return _uniqueFileInputSelector(matches[0]) || selector;+ }+ } catch {}+ return selector;+ };const onBlocked = () => {try {const payload = JSON.parse(root.getAttribute(blockedAttr) || 'null');if (payload?.guardId !== guardId) return;state.blocked = {selector: typeof payload.selector === 'string' && payload.selector- ? payload.selector+ ? inventorySelector(payload.selector): null,};} catch {}diff --git a/test/fixtures/run.mjs b/test/fixtures/run.mjsindex cdb1aa1d9..726c3eca4 100644--- a/test/fixtures/run.mjs+++ b/test/fixtures/run.mjs@@ -4535,6 +4535,41 @@ for (const browserKind of ['chrome', 'firefox']) {}});+ // Trace b71fd1dc: Instagram's Add Photo or Video button opens a formless,+ // display:none input. The chooser guard named it by its path from the root+ // ("div:nth-of-type(3) > input"), the inventory by its attributes, so the+ // model held two selectors for one input. Both now name it the same way.+ for (const opener of ['click', 'showPicker']) {+ test(`file picker guard (${browserKind}): names a ${opener} chooser's input as the file-input inventory does`, async (page) => {+ const isolatedCall = await setupIsolatedContentHtml(page, `<!doctype html>+ <style>.x1s85apg{display:none}</style>+ <input type="text" name="searchInput" placeholder="Search">+ <div><div><div role="textbox" contenteditable="true" aria-placeholder="Message..."></div></div>+ <div><div role="button" tabindex="0" aria-label="Voice Clip">Voice</div></div>+ <div><input accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg" class="x1s85apg" multiple type="file">+ <div role="button" tabindex="0" id="add-photo" aria-label="Add Photo or Video">Photo</div></div></div>+ <script>+ document.querySelector('#add-photo').addEventListener('click', () => {+ document.querySelector('input[type=file]').${opener}();+ });+ </script>`, browserKind);++ let chooserOpened = false;+ page.once('filechooser', () => { chooserOpened = true; });+ const result = await isolatedCall('click', { text: 'Photo' });+ await page.waitForTimeout(20);+ if (chooserOpened) throw new Error(`${opener} native chooser was not suppressed`);+ const inventory = (await isolatedCall('get_interactive_elements_cdp', {})).filter(item => item.type === 'file');+ const expected = 'input[type="file"][accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg"][multiple]';+ if (inventory.length !== 1 || inventory[0].selector !== expected) {+ throw new Error(`expected one inventory record named ${expected}, got ${JSON.stringify(inventory)}`);+ }+ if (!result?.filePickerBlocked || result.success !== false || result.selector !== expected) {+ throw new Error(`the blocked chooser must name the inventory's selector, got ${JSON.stringify(result)}`);+ }+ });+ }+test(`file picker guard (${browserKind}): withholds selectors that collide across shadow roots`, async (page) => {await setupContentHtml(page, `<!doctype html><button id="choose">Select a shadow file...</button>diff --git a/test/guard-advisory-actions.mjs b/test/guard-advisory-actions.mjsindex 26bbc793d..4cac0a3da 100644--- a/test/guard-advisory-actions.mjs+++ b/test/guard-advisory-actions.mjs@@ -710,11 +710,13 @@ for (const build of ['chrome', 'firefox']) {agent._pageFeedbackRuns = new Map([[tab, run]]);return run;}- async function bindingCall(strict, reason, events = []) {+ async function bindingCall(strict, reason, events = [], { name = 'click_ax', args = { ref_id: 'ref_uncovered' }, prepare = null, lean = false } = {}) {const agent = new Agent({ getVisionProvider: async () => null });agent.strictGuards = strict;const tab = nextTab++;- boundRun(agent, tab, events);+ if (lean) agent._leanRuns.set(tab, { active: true, setting: 'on', reason: 'setting_on' });+ const run = boundRun(agent, tab, events);+ prepare?.(run);agent._uncertainTextMutationBlock = async () => null;agent._finalizeToolResultOnce = async (_tab, _name, _args, result) => result;const impl = [];@@ -724,10 +726,21 @@ for (const build of ['chrome', 'firefox']) {? { ready: false, reason, runToken: message.params.runToken, documentToken: 'content-1', snapshotToken: message.params.snapshotToken }: { ready: true };try {- const result = await Agent.prototype.executeTool.call(agent, tab, 'click_ax', { ref_id: 'ref_uncovered' }, null, { guardAdvisories: [] });+ const result = await Agent.prototype.executeTool.call(agent, tab, name, args, null, { guardAdvisories: [] });return { result, impl };} finally { api.tabs.sendMessage = oldSend; }}+ // Instagram's direct-message composer (trace b71fd1dc): the one file input+ // is formless and display:none deep in a large thread, so no certificate+ // covers it, and every upload_file was refused even with the exact selector+ // get_interactive_elements returned. That inventory is the observation an+ // input without a ref has.+ const INSTAGRAM_UPLOAD = 'input[type="file"][accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg"][multiple]';+ const inventoried = (selectors = [INSTAGRAM_UPLOAD], { documentToken = 'content-1', url = PAGE_URL } = {}) => run => {+ run.frames.set(0, { token: 'content-1', id: 'document-1' });+ run.uploadObservationScope = { documentToken, url, selectors };+ };+ const upload = (prepare, selector = INSTAGRAM_UPLOAD) => ({ name: 'upload_file', args: { selector, downloadId: 2157 }, prepare });for (const [mode, strict] of MODES) {test(`${build}: page-feedback-target-binding ${mode} mode ${strict ? 'refuses an uncovered target' : 'sends an uncovered target with a note'}`, async () => {const { result, impl } = await bindingCall(strict, 'target_uncovered');@@ -749,6 +762,47 @@ for (const build of ['chrome', 'firefox']) {assert.deepEqual(guardsOf(result), [], reason);}});++ test(`${build}: page-feedback-target-binding ${mode} mode ${strict ? 'refuses' : 'sends'} an upload to the file input the inventory named`, async () => {+ const { result, impl } = await bindingCall(strict, 'target_uncovered', [], upload(inventoried()));+ if (strict) {+ assert.deepEqual(impl, []);+ assert.equal(result.errorCode, 'action_binding_unavailable');+ assert.equal(result.bindingFailure, 'target_uncovered');+ } else {+ assert.deepEqual(impl, ['upload_file']);+ assert.deepEqual(guardsOf(result), ['page-feedback-target-binding']);+ assert.match(result.guardAdvisories[0].note, /file-input inventory/);+ }+ });++ // The trace's runs were lean: lean mode skips this quality guard, but the+ // selector rule refused before the guard was ever asked.+ test(`${build}: page-feedback-target-binding lean run with ${mode} checks sends the inventory's upload without a note, not a guessed one`, async () => {+ const sent = await bindingCall(strict, 'target_uncovered', [], { ...upload(inventoried()), lean: true });+ assert.deepEqual(sent.impl, ['upload_file']);+ assert.deepEqual(guardsOf(sent.result), []);+ const guessed = await bindingCall(strict, 'target_uncovered', [], { ...upload(inventoried(), 'input[type="file"]'), lean: true });+ assert.deepEqual(guessed.impl, []);+ assert.equal(guessed.result.errorCode, 'action_binding_unavailable');+ });++ test(`${build}: page-feedback-target-binding ${mode} mode keeps an upload selector the inventory did not name refused`, async () => {+ const cases = [+ ['a guessed selector', 'target_uncovered', [], upload(inventoried(), 'input[type="file"]')],+ ['no inventory', 'target_uncovered', [], upload(null)],+ ['another document\'s inventory', 'target_uncovered', [], upload(inventoried(undefined, { documentToken: 'content-0' }))],+ ['another URL\'s inventory', 'target_uncovered', [], upload(inventoried(undefined, { url: 'https://example.com/other' }))],+ ['an ambiguous inventory selector', 'target_ambiguous', [], upload(inventoried())],+ ['pending page activity', 'target_uncovered', [{ kind: 'dom', source: 'unknown', frameId: 0, target: 'div#thread' }], upload(inventoried())],+ ];+ for (const [label, reason, events, options] of cases) {+ const { result, impl } = await bindingCall(strict, reason, events, options);+ assert.deepEqual(impl, [], label);+ assert.equal(result.errorCode, 'action_binding_unavailable', label);+ assert.deepEqual(guardsOf(result), [], label);+ }+ });}// ββ rich-text toolbar guards ββββββββββββββββββββββββββββββββββββββββββββdiff --git a/test/quality/recipient-clarify-mv3.mjs b/test/quality/recipient-clarify-mv3.mjsindex 7f8840814..2c0d91490 100644--- a/test/quality/recipient-clarify-mv3.mjs+++ b/test/quality/recipient-clarify-mv3.mjs@@ -22,16 +22,43 @@ const question = `Send the photo to ${recipient} on Instagram?`;const selector = 'input[type="file"][accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg"][multiple]';const png = Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aS3kAAAAASUVORK5CYII=', 'base64');-// [variant, model steps, model-question answer, its source, card answer, sends expected]+// [variant, model steps, model-question answer, its source, card answer, sends expected, thread page]const variants = [['card-send', ['download', 'list', 'upload', 'tree', 'done'], null, null, 'once', 1],['card-dont-send', ['download', 'list', 'upload', 'done'], null, null, 'deny', 0],['model-asks-first', ['download', 'list', 'clarify', 'upload', 'tree', 'done'], recipient, 'option', 'once', 1],['typed-refusal', ['download', 'list', 'clarify', 'upload', 'done'], `No, not ${recipient}`, 'user', 'deny', 0],+ // Trace b71fd1dc: on Instagram's own composer every upload_file was refused+ // (action_binding_unavailable) after the user had confirmed the recipient.+ ['instagram-composer', ['download', 'list', 'inventory', 'upload', 'tree', 'done'], null, null, 'once', 1, 'instagram'],];for (const variant of variants) await runVariant(...variant);-async function runVariant(variant, steps, answer, source, cardAnswer, expectedSends) {+// Instagram's composer as trace b71fd1dc read it (2026-10): no form, header or+// footer, and the one file input, display:none by class between Voice Clip and+// Add Photo or Video, sits several wrappers deep in a thread too large for the+// page monitor's owner bounds. No certificate covers that input, so the model+// names it by the selector get_interactive_elements returned.+function instagramThreadPage() {+ const icon = label => `<div role="button" tabindex="0"><svg role="img" aria-label="${label}"><title>${label}</title><path d="M0 0h24v24H0z"/></svg></div>`;+ const nest = (html, depth) => Array.from({ length: depth }).reduce(inner => `<div>${inner}</div>`, html);+ const threads = ['Ungvb', recipient, 'Owned Contact One', 'Owned Contact Two', 'Owned Contact Three']+ .map(name => `<div role="button" tabindex="0"><img alt="user-profile-picture" src="data:,"><span>${name}</span></div>`).join('');+ const history = Array.from({ length: 60 }, (_, index) => nest(`<div role="group"><div role="article" data-message-direction="incoming">Message ${index + 1}</div></div>`, 5)).join('');+ const composer = `<div class="composer">${icon('Choose an emoji')}<div role="textbox" contenteditable="true" aria-placeholder="Message..." id="body"><p><br></p></div>`+ + `${icon('Voice Clip')}<input accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg" class="x1s85apg" multiple type="file">${icon('Add Photo or Video')}${icon('Choose a GIF or sticker')}</div>`;+ return `<!doctype html><meta charset="utf-8"><title>Instagram β’ Messages</title>+ <style>body{margin:0;font:14px sans-serif}.x1s85apg{display:none}#threads{position:fixed;left:72px;top:0;width:400px;height:700px;overflow:auto}+ #pane{position:fixed;left:480px;top:0;width:640px;height:700px;display:flex;flex-direction:column}#head{height:75px}+ #thread{flex:1;overflow:auto}.composer{display:flex;align-items:center;height:44px;margin:12px;border:1px solid #888;border-radius:22px}+ [contenteditable]{flex:1;min-height:20px}[role=button]{display:inline-flex;width:40px;height:40px}svg{width:24px;height:24px}</style>+ <div role="main"><div id="threads"><nav aria-label="Thread list"><h1>owned.account</h1>${threads}</nav></div>+ <div id="pane"><div id="head"><a href="/dr.gamzec_/" aria-label="Open the profile page of dr.gamzec_"><img alt="user-profile-picture" src="data:,"><h2>${recipient}</h2><span>dr.gamzec_</span></a>+ ${icon('Audio call')}${icon('Video call')}${icon('Conversation information')}</div>+ <div id="thread" role="log">${history}</div>${nest(composer, 6)}</div></div>`;+}++async function runVariant(variant, steps, answer, source, cardAnswer, expectedSends, threadPage = 'owned') {const approved = expectedSends > 0;const out = path.resolve(process.env.WEBBRAIN_QUALITY_ARTIFACTS? path.join(process.env.WEBBRAIN_QUALITY_ARTIFACTS, variant)@@ -86,6 +113,7 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSeconst kind = steps[Math.min(step, steps.length - 1)];const call = kind === 'download' ? { name: 'download_files', args: { urls: [`${base}/photo.png`] } }: kind === 'list' ? { name: 'list_downloads', args: { limit: 5 } }+ : kind === 'inventory' ? { name: 'get_interactive_elements', args: {} }: kind === 'upload' ? { name: 'upload_file', args: { selector, downloadId: downloadIdIn(input) } }: kind === 'clarify' ? { name: 'clarify', args: { options: [recipient, 'Cancel'], question } }: kind === 'tree' ? { name: 'get_accessibility_tree', args: { filter: 'all' } }@@ -110,7 +138,7 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSe});await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));base = `http://127.0.0.1:${server.address().port}`;- const pageBody = `<!doctype html><meta charset="utf-8"><title>Owned direct thread</title>+ const pageBody = (threadPage === 'instagram' ? instagramThreadPage() : `<!doctype html><meta charset="utf-8"><title>Owned direct thread</title><style>body{margin:0;font:16px sans-serif} nav{position:fixed;left:0;top:0;width:330px;height:700px}header{position:fixed;left:350px;top:10px;width:600px;height:70px} #thread{position:fixed;left:350px;top:90px;width:600px;height:420px}footer{position:fixed;left:350px;top:600px;width:600px;height:60px} [contenteditable]{display:inline-block;width:420px;height:40px;border:1px solid #888}@@ -120,7 +148,7 @@ async function runVariant(variant, steps, answer, source, cardAnswer, expectedSe<div id="thread" role="log"><div data-message-direction="incoming">Merhaba!</div></div><footer><form id="composer" onsubmit="return false"><div role="textbox" contenteditable="true" aria-label="Message" id="body"></div><div role="button" id="attach"><svg role="img" aria-label="Add Photo or Video"><title>Add Photo or Video</title><path d="M0 0h24v24H0z"/></svg></div>- <input type="file" accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg" multiple hidden></form></footer></main>+ <input type="file" accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg" multiple hidden></form></footer></main>`) + `<script>document.querySelector('input[type=file]').addEventListener('change', async event => {const file = event.target.files[0]; if (!file) return;diff --git a/test/run.js b/test/run.jsindex d8246fec2..1aa28db28 100644--- a/test/run.js+++ b/test/run.js@@ -4699,6 +4699,21 @@ test('matches Hugging Face upload pages with staged-upload guidance and rejects}});+// Trace b71fd1dc: the model clicked "Add Photo or Video", which only opens an+// OS file dialog, and finally sent a link to the picture instead of the file.+test('Instagram adapter attaches direct-message media through the composer file input', () => {+ const url = 'https://www.instagram.com/direct/t/17848282661759097/';+ const chromeAdapter = getActiveAdapter(url);+ const firefoxAdapter = getActiveAdapterFx(url);++ assert.equal(chromeAdapter?.name, 'instagram');+ assert.equal(firefoxAdapter?.name, 'instagram');+ assert.equal(firefoxAdapter?.notes, chromeAdapter?.notes);+ assert.match(chromeAdapter?.notes || '', /\/direct\/t\/<thread>.*hidden input\[type="file"\] beside "Add Photo or Video"/s);+ assert.match(chromeAdapter?.notes || '', /selector get_interactive_elements returns for it to upload_file and never click that button/);+ assert.match(chromeAdapter?.notes || '', /A link to the image is not the image/);+});+test('matches Mozilla Add-ons Developer Hub and guides version submission', () => {const sourceUrl = 'https://addons.mozilla.org/en-US/developers/addon/example-addon/versions/submit/6358210/source';const chromeAdapter = getActiveAdapter(sourceUrl);