Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
On X and Bluesky hosts WebBrain cannot read, an unread caption proves nothing; Enter in their post editor is no post
Claude9b8d4f9cf199
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a318a9ad4..ee94cbfad 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,7 +7,7 @@ This changelog was generated from the repository Git history and release tags. V## [Unreleased]### Fixed-- Posting on another site after a correction: after a Mastodon post, "ok now on X" no longer leaves WebBrain going back and forth between the two sites, typing the X post again each time. It counted X as the same platform as Mastodon, because it reads account names only on X and Bluesky, and asked to find the Mastodon post on X itself, which can never happen. The site a post was made on now names its platform when no account name was read, and a post on another site goes ahead once a page has been read since the correction, provided the task made no earlier post on that site. What the task attempted is saved with the conversation, and just before each action that may post or submit something, so this still holds when the task continues after WebBrain restarted, and a post already sent on X under the correction is not sent again by a later run of the task. When the task did post on that site before, or WebBrain has no full record of its earlier attempts, it stops once and says so, instead of searching for a post it cannot find. A search, a settings save or consent click on X, a click that sent nothing or a form the page rejected is not such a post; a click that Stop interrupted after it was sent counts. On the other hosts and clients of X and Bluesky, such as X Pro, whose Post buttons WebBrain does not recognize, anything sent that may have been a post counts as one, so a post from X Pro is not repeated on x.com. If that record cannot be saved just before a possible post on X or Bluesky, the post is not sent and WebBrain says why, as it already does when a run's recovery checkpoint cannot be saved. x.com and twitter.com are one platform.+- Posting on another site after a correction: after a Mastodon post, "ok now on X" no longer leaves WebBrain going back and forth between the two sites, typing the X post again each time. It counted X as the same platform as Mastodon, because it reads account names only on X and Bluesky, and asked to find the Mastodon post on X itself, which can never happen. The site a post was made on now names its platform when no account name was read, and a post on another site goes ahead once a page has been read since the correction, provided the task made no earlier post on that site. What the task attempted is saved with the conversation, and just before each action that may post or submit something, so this still holds when the task continues after WebBrain restarted, and a post already sent on X under the correction is not sent again by a later run of the task. When the task did post on that site before, or WebBrain has no full record of its earlier attempts, it stops once and says so, instead of searching for a post it cannot find. A search, a settings save or consent click on X, a click that sent nothing or a form the page rejected is not such a post; a click that Stop interrupted after it was sent counts. On the other hosts and clients of X and Bluesky, such as X Pro, whose Post buttons WebBrain does not recognize, anything sent that may have been a post counts as one, so a post from X Pro is not repeated on x.com; this includes a Post button whose caption no model read in time, which looked like a harmless button, while Enter in the post editor there edits the draft and is not a post. If that record cannot be saved just before a possible post on X or Bluesky, the post is not sent and WebBrain says why, as it already does when a run's recovery checkpoint cannot be saved. x.com and twitter.com are one platform.- Chrome: a page's "Leave site?" prompt no longer waits for you when WebBrain navigates away during a task that started on a page the debugger cannot attach to, such as chrome://extensions, or whose debugger connection dropped. WebBrain arms its dialog handling again at the next debugger attach and around each of its own navigations, and answers Leave for its own navigation as it already did on other runs. If you cancel the debugger, prompts stay yours to answer. Firefox offers extensions no way to answer these prompts.- Chat sites such as WhatsApp: a request that points at the open chat instead of naming the person, such as "send some test messages to this one" or "message this user", now sends there instead of asking "which contact, and what text?". The planner never sees the page, and its rules counted only words like "reply here" as the open chat, so it asked the same question every time without reading the page. WebBrain still reads the open chat's header itself and binds every send to that one contact, and stops if no single chat is open. The planner is also told it cannot see the page, no longer reads a note about a screenshot it is never given, and leaves the wording of described messages, such as test messages or a thank-you, to the agent. One request can now cover the open chat and someone named besides it, as in "test messages here, then thank Rustem Bey": the open chat needs no confirmation, the reviewed plan shows the named person, and that person is confirmed once on WebBrain's card before the first send to them. Words that refer back to someone named earlier still mean that person, and on mail sites the people named besides "reply here" are handled as before. On WhatsApp, where a contact is confirmed through its contact drawer, one run still confirms only one chat's contact, so the second chat of such a request needs a request of its own; the agent is now told so, instead of being sent to re-read the page and retry.- A subscription, registration or sign-in wall in front of a news article is now recognized in any language, by a model reading the wall's text once, instead of only when it used English phrases such as "Subscribe to continue reading". A Turkish "Okumaya devam etmek için abone olun" overlay now keeps WebBrain from reading the article hidden behind it, as an English one did, while an offer to subscribe beside text you can still read is not taken for a wall. A wall inside the article is found also after a long preview, and everything after the wall's text stays withheld; the preview before it stays readable, except the short passage just before the wall's text. If a page cannot be read again after its wall was recognized, the read fails rather than return what the wall hides. A wall whose number changes on every read, such as a countdown, stays a wall while its new text is read, and needs no more readings once two of its numbers were read as the wall, while a count that stops being a wall when its number changes reads as what it is once read; a wall whose other text keeps changing makes the read fail rather than leak, and the agent may read the page once more in case the wall went away.diff --git a/docs/architecture.md b/docs/architecture.mdindex 10c9793a5..5179dca8d 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -109,7 +109,10 @@ is not one. Their other hosts and clients, such as X Pro, post without thecomposer controls WebBrain recognizes, so there any action that may have beensent counts as a publication unless the page probe proved it cannot post (anordinary form, a deletion, or a control resolved as a link, field, menu item or-other non-submitting control).+other non-submitting control). There, a control that looks ordinary only+because no model read its caption in time proves nothing, and under a+correction it is not clicked; Enter in the app-marked post editor edits the+draft, as on x.com.The agent records, for the task from its first run on, the site of everydispatched attempt that may have changed something and of every publicationamong them (a search, a send the browser refused or a form the page rejecteddiff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 664db487a..d3cf1dd1b 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -5585,7 +5585,7 @@ export class Agent extends LoopDetector {const resolvedNonSubmit = detectedSubmit?.isSubmit !== true && (detectedSubmit?.resolvedNavigationTarget === true|| detectedSubmit?.resolvedEditableTarget === true|| detectedSubmit?.resolvedNonSubmitTarget === true- || detectedSubmit?.resolvedOrdinaryActivationTarget === true+ || (detectedSubmit?.resolvedOrdinaryActivationTarget === true && detectedSubmit.captionUnread !== true)|| detectedSubmit?.resolvedPreparatoryTarget === true);if (!isNetworkMutation(name, args) && name !== 'chrome_web_store_publish'&& !(detectedSubmit?.isSubmit || (!resolvedNonSubmit && this._isFormValidationCandidate(name, args)))) return null;@@ -5884,6 +5884,9 @@ export class Agent extends LoopDetector {_cannotPost(detected) {if (!detected || typeof detected !== 'object' || detected.publicationControl === true) return false;if (detected.publicationControl === false || detected.mutationKind === 'delete') return true;+ // An ordinary control on X or Bluesky whose caption no model read proves+ // nothing: it may be a Post button.+ if (detected.captionUnread === true) return false;return detected.isSubmit === false && (detected.resolvedNavigationTarget === true || detected.resolvedEditableTarget === true|| detected.resolvedNonSubmitTarget === true || detected.resolvedOrdinaryActivationTarget === true|| detected.resolvedPreparatoryTarget === true);@@ -6028,8 +6031,12 @@ export class Agent extends LoopDetector {const platforms = add(record.platforms, potential.keys);const publications = potential.publication ? add(record.publications, potential.keys): potential.keys.includes('*') ? add(record.publications, ['*']) : record.publications;+ // Where there is no session storage at all, nothing is restored after a+ // restart either, so there is nothing to keep in step.+ const session = globalThis.chrome?.storage?.session;+ if (typeof session?.set !== 'function') return potential;try {- await chrome.storage.session.set({ [this._taskAttemptCheckpointKey(tabId)]: { ...record,+ await session.set({ [this._taskAttemptCheckpointKey(tabId)]: { ...record,platforms: platforms?.length > 64 ? null : platforms, publications: publications?.length > 64 ? null : publications } });} catch {const matters = potential.keys.includes('*')@@ -6278,7 +6285,7 @@ export class Agent extends LoopDetector {// to look for the other site's post on this page.if (prior && publication && otherSite && !crossPlatform) {return blocked(earlierAttemptHere ? 'earlier_attempt_on_platform' : 'attempt_history_unknown', earlierAttemptHere- ? `Nothing was published. This task already made an attempt on this site, and WebBrain can check only the latest attempt from before the correction${priorHost ? `, on ${priorHost}` : ''}, so this post could repeat the earlier one or sit beside it. Do not publish it again. Tell the user what was not done and that the earlier attempt on this site may already be there; they can check it and ask again.`+ ? `Nothing was published. This task already sent something on this site that may have been a post, and WebBrain can check only the latest attempt from before the correction${priorHost ? `, on ${priorHost}` : ''}, so this post could repeat it or sit beside it. Do not publish it again. Tell the user what was not done and that what was sent earlier on this site may already be a post there; they can check it and ask again.`: `Nothing was published. WebBrain has no full record of what this task attempted (for example after WebBrain restarted), so it cannot rule out an earlier attempt on this site, and the latest attempt from before the correction${priorHost ? `, on ${priorHost},` : ''} cannot be checked here. Do not publish it again. Tell the user what was not done; they can ask again to post it.`,{ retryable: false });}@@ -28350,6 +28357,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (ordinaryActivations.length === 1 && ['click', 'click_ax', 'iframe_click'].includes(name)) {return { isSubmit: false, resolvedOrdinaryActivationTarget: true,...(ordinaryActivations[0].resolvedPreparatoryTarget === true ? { resolvedPreparatoryTarget: true } : {}),+ ...(ordinaryActivations[0].captionUnread === true ? { captionUnread: true } : {}),...(ordinaryActivations[0].mutationKind === 'delete' ? { mutationKind: 'delete' } : {}),...(ordinaryActivations[0].popupLauncherToken ? { popupLauncherToken: ordinaryActivations[0].popupLauncherToken } : {}),frameDocumentIdentity: ordinaryActivations[0].frameDocumentIdentity,@@ -29155,6 +29163,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (siteHost === 'bsky.app') return 'bluesky';return '';};+ // Any host of X or Bluesky, including those whose composer WebBrain does+ // not read, such as X Pro (pro.x.com).+ const socialSiteHost = () => /(?:^|\.)(?:x\.com|twitter\.com|bsky\.app)$/.test(String(host || '').toLowerCase().replace(/\.$/, ''));const socialPublishComposerFor = (candidate) => {if (!socialPublishAdapterName() || !candidate || candidate.nodeType !== 1) return null;let node = candidate;@@ -30350,8 +30361,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (form) return submitInfo(form, 'Enter key in a form field', null, null, 'strong', field);// X and Bluesky publish from the keyboard only with a modifier, which// press_keys cannot send. An unmodified Enter in the app-marked post- // editor edits the draft (a line break or a typeahead choice).- const postEditor = !!field && !!socialPublishAdapterName()+ // editor edits the draft (a line break or a typeahead choice), on X+ // Pro as on x.com.+ const postEditor = !!field && socialSiteHost()&& (field.isContentEditable || String(field.tagName || '').toUpperCase() === 'TEXTAREA')&& !!field.closest?.('[data-testid^="tweetTextarea_"],[data-testid="composerTextInput"]');return postEditor ? { isSubmit: false, host, url, tool: toolName,@@ -30495,6 +30507,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (popupTarget && !actionResource && recordOwnerFor(control)) rememberMenuLaunch(control);return { isSubmit: false, host, url, resolvedOrdinaryActivationTarget: true,resolvedPreparatoryTarget: popupTarget,+ // On an X or Bluesky host whose app markers WebBrain does not+ // read, a caption no model read leaves a Post button looking+ // ordinary.+ ...(captionsReading(labels) === null && socialSiteHost() && !socialPublishAdapterName() ? { captionUnread: true } : {}),frameDocumentIdentity: [url, performance.timeOrigin].join('|'), actionResource,popupLauncherToken: popupTarget ? rememberPopupLauncher(control, actionResource) : '' };}diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex eae4dafd4..987692490 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -5258,7 +5258,7 @@ export class Agent extends LoopDetector {const resolvedNonSubmit = detectedSubmit?.isSubmit !== true && (detectedSubmit?.resolvedNavigationTarget === true|| detectedSubmit?.resolvedEditableTarget === true|| detectedSubmit?.resolvedNonSubmitTarget === true- || detectedSubmit?.resolvedOrdinaryActivationTarget === true+ || (detectedSubmit?.resolvedOrdinaryActivationTarget === true && detectedSubmit.captionUnread !== true)|| detectedSubmit?.resolvedPreparatoryTarget === true);if (!isNetworkMutation(name, args) && name !== 'chrome_web_store_publish'&& !(detectedSubmit?.isSubmit || (!resolvedNonSubmit && this._isFormValidationCandidate(name, args)))) return null;@@ -5557,6 +5557,9 @@ export class Agent extends LoopDetector {_cannotPost(detected) {if (!detected || typeof detected !== 'object' || detected.publicationControl === true) return false;if (detected.publicationControl === false || detected.mutationKind === 'delete') return true;+ // An ordinary control on X or Bluesky whose caption no model read proves+ // nothing: it may be a Post button.+ if (detected.captionUnread === true) return false;return detected.isSubmit === false && (detected.resolvedNavigationTarget === true || detected.resolvedEditableTarget === true|| detected.resolvedNonSubmitTarget === true || detected.resolvedOrdinaryActivationTarget === true|| detected.resolvedPreparatoryTarget === true);@@ -5701,8 +5704,12 @@ export class Agent extends LoopDetector {const platforms = add(record.platforms, potential.keys);const publications = potential.publication ? add(record.publications, potential.keys): potential.keys.includes('*') ? add(record.publications, ['*']) : record.publications;+ // Where there is no session storage at all, nothing is restored after a+ // restart either, so there is nothing to keep in step.+ const session = globalThis.browser?.storage?.session;+ if (typeof session?.set !== 'function') return potential;try {- await browser.storage.session.set({ [this._taskAttemptCheckpointKey(tabId)]: { ...record,+ await session.set({ [this._taskAttemptCheckpointKey(tabId)]: { ...record,platforms: platforms?.length > 64 ? null : platforms, publications: publications?.length > 64 ? null : publications } });} catch {const matters = potential.keys.includes('*')@@ -5951,7 +5958,7 @@ export class Agent extends LoopDetector {// to look for the other site's post on this page.if (prior && publication && otherSite && !crossPlatform) {return blocked(earlierAttemptHere ? 'earlier_attempt_on_platform' : 'attempt_history_unknown', earlierAttemptHere- ? `Nothing was published. This task already made an attempt on this site, and WebBrain can check only the latest attempt from before the correction${priorHost ? `, on ${priorHost}` : ''}, so this post could repeat the earlier one or sit beside it. Do not publish it again. Tell the user what was not done and that the earlier attempt on this site may already be there; they can check it and ask again.`+ ? `Nothing was published. This task already sent something on this site that may have been a post, and WebBrain can check only the latest attempt from before the correction${priorHost ? `, on ${priorHost}` : ''}, so this post could repeat it or sit beside it. Do not publish it again. Tell the user what was not done and that what was sent earlier on this site may already be a post there; they can check it and ask again.`: `Nothing was published. WebBrain has no full record of what this task attempted (for example after WebBrain restarted), so it cannot rule out an earlier attempt on this site, and the latest attempt from before the correction${priorHost ? `, on ${priorHost},` : ''} cannot be checked here. Do not publish it again. Tell the user what was not done; they can ask again to post it.`,{ retryable: false });}@@ -25779,6 +25786,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (ordinaryActivations.length === 1 && ['click', 'click_ax', 'iframe_click'].includes(name)) {return { isSubmit: false, resolvedOrdinaryActivationTarget: true,...(ordinaryActivations[0].resolvedPreparatoryTarget === true ? { resolvedPreparatoryTarget: true } : {}),+ ...(ordinaryActivations[0].captionUnread === true ? { captionUnread: true } : {}),...(ordinaryActivations[0].mutationKind === 'delete' ? { mutationKind: 'delete' } : {}),...(ordinaryActivations[0].popupLauncherToken ? { popupLauncherToken: ordinaryActivations[0].popupLauncherToken } : {}),frameDocumentIdentity: ordinaryActivations[0].frameDocumentIdentity,@@ -26496,6 +26504,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (siteHost === 'bsky.app') return 'bluesky';return '';};+ // Any host of X or Bluesky, including those whose composer WebBrain does+ // not read, such as X Pro (pro.x.com).+ const socialSiteHost = () => /(?:^|\.)(?:x\.com|twitter\.com|bsky\.app)$/.test(String(host || '').toLowerCase().replace(/\.$/, ''));const socialPublishComposerFor = (candidate) => {if (!socialPublishAdapterName() || !candidate || candidate.nodeType !== 1) return null;let node = candidate;@@ -27656,8 +27667,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (form) return submitInfo(form, 'Enter key in a form field', null, null, 'strong', field);// X and Bluesky publish from the keyboard only with a modifier, which// press_keys cannot send. An unmodified Enter in the app-marked post- // editor edits the draft (a line break or a typeahead choice).- const postEditor = !!field && !!socialPublishAdapterName()+ // editor edits the draft (a line break or a typeahead choice), on X+ // Pro as on x.com.+ const postEditor = !!field && socialSiteHost()&& (field.isContentEditable || String(field.tagName || '').toUpperCase() === 'TEXTAREA')&& !!field.closest?.('[data-testid^="tweetTextarea_"],[data-testid="composerTextInput"]');return postEditor ? { isSubmit: false, host, url, tool: toolName,@@ -27801,6 +27813,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (popupTarget && !actionResource && recordOwnerFor(control)) rememberMenuLaunch(control);return { isSubmit: false, host, url, resolvedOrdinaryActivationTarget: true,resolvedPreparatoryTarget: popupTarget,+ // On an X or Bluesky host whose app markers WebBrain does not+ // read, a caption no model read leaves a Post button looking+ // ordinary.+ ...(captionsReading(labels) === null && socialSiteHost() && !socialPublishAdapterName() ? { captionUnread: true } : {}),frameDocumentIdentity: [url, performance.timeOrigin].join('|'), actionResource,popupLauncherToken: popupTarget ? rememberPopupLauncher(control, actionResource) : '' };}diff --git a/test/guards-control-classification.mjs b/test/guards-control-classification.mjsindex a4505606e..95d43e3d8 100644--- a/test/guards-control-classification.mjs+++ b/test/guards-control-classification.mjs@@ -242,6 +242,46 @@ for (const build of ['chrome', 'firefox']) {}});+ // X Pro (pro.x.com) has none of the app markers the X adapter reads.+ // A plain Post button whose caption no model read in time looks+ // ordinary to the probe, which says so; Enter in the app-marked post+ // editor edits the draft there as on x.com.+ await t.test('on X Pro an unread caption proves nothing, and Enter in the post editor edits the draft', async () => {+ const html = '<main><div><div data-testid="tweetTextarea_0" contenteditable="true" role="textbox" id="editor">Draft</div><button id="t" type="button">Post</button></div></main>';+ for (const reading of ['unread', 'read']) {+ const f = await setup('https://pro.x.com/i/decks/1', html, { steering: true });+ if (reading === 'unread') {+ const chat = f.agent._chatWithCostAllowance;+ f.agent._chatWithCostAllowance = async (provider, messages, options, cost, meta) => meta?.generationName === 'semantic_control_effect'+ ? { content: '' } : chat(provider, messages, options, cost, meta);+ }+ const { detected, block } = await gate(f, 'click_ax', { ref_id: await ref('t') });+ if (reading === 'unread') {+ assert.equal(detected?.resolvedOrdinaryActivationTarget, true, reading);+ assert.equal(detected.captionUnread, true, reading);+ assert.equal(f.agent._cannotPost(detected), false, reading);+ } else {+ assert.notEqual(detected?.resolvedOrdinaryActivationTarget, true, 'read as a post, it is no ordinary button');+ }+ // Under a correction it is not clicked either way.+ assert.equal(block?.actionGuard?.reason, 'target_unresolved', reading);+ }+ const f = await setup('https://pro.x.com/i/decks/1', html, { steering: true });+ await page.locator('#editor').focus();+ const result = await gate(f, 'press_keys', { key: 'Enter' });+ assert.equal(result.detected?.resolvedEditableTarget, true, 'Enter in the X Pro post editor');+ assert.equal(f.agent._cannotPost(result.detected), true);+ assert.equal(result.block, null);+ // On a site that is not X or Bluesky nothing changes.+ const other = await setup('https://deck.example/', html);+ const chat = other.agent._chatWithCostAllowance;+ other.agent._chatWithCostAllowance = async (provider, messages, options, cost, meta) => meta?.generationName === 'semantic_control_effect'+ ? { content: '' } : chat(provider, messages, options, cost, meta);+ const plain = await gate(other, 'click_ax', { ref_id: await ref('t') });+ assert.equal(plain.detected?.resolvedOrdinaryActivationTarget, true);+ assert.equal(plain.detected.captionUnread, undefined);+ });+await t.test('the duplicate-click window exempts draft tools and popups, not ordinary commits by caption', async () => {const f = await setup('https://x.com/home', '<nav><a data-testid="AppTabBar_Profile_Link" href="/alice">Profile</a></nav><div><div data-testid="tweetTextarea_0" contenteditable="true" role="textbox">Hello</div><nav aria-label="Composer tools"><button id="emoji" type="button" aria-label="Add emoji">Add emoji</button></nav><button type="button" data-testid="tweetButtonInline">Post</button></div>');const emoji = await f.agent._detectLikelySubmitAction(1, 'click', { text: 'Add emoji' });diff --git a/test/guards-steering-receipts.mjs b/test/guards-steering-receipts.mjsindex b732b604a..88b6afd72 100644--- a/test/guards-steering-receipts.mjs+++ b/test/guards-steering-receipts.mjs@@ -661,7 +661,11 @@ for (const build of ['chrome', 'firefox']) {'X Pro, caption read': { url: pro, caption: 'Post', now: xPublish('WebBrain test post', '') },'X Pro, caption unread': { url: pro, now: xPublish('WebBrain test post', '') },'X Pro, a script': { url: pro, name: 'execute_js', args: { code: 'document.querySelector("button").click()' }, now: xPublish('WebBrain test post', '') },- 'X Pro, Space on the button': { url: pro, name: 'press_keys', args: { key: 'Space' }, now: xPublish('WebBrain test post', '') },+ 'X Pro, Enter on the focused button': { url: pro, name: 'press_keys', args: { key: 'Enter' }, now: xPublish('WebBrain test post', '') },+ // A plain Post button whose caption no model read in time looks+ // ordinary to the probe; that proves nothing.+ 'X Pro, a plain button whose caption went unread': { url: pro, probe: { isSubmit: false, resolvedOrdinaryActivationTarget: true, captionUnread: true },+ now: xPublish('WebBrain test post', '') },'mobile Twitter': { url: 'https://mobile.twitter.com/compose/tweet', caption: 'Tweet', now: xPublish('WebBrain test post') },'staging Bluesky': { url: 'https://staging.bsky.app/', caption: 'Post', now: bskyPublish('WebBrain test post', '') },// A control the probe resolved as one that submits nothing is not a@@ -670,6 +674,12 @@ for (const build of ['chrome', 'firefox']) {now: xPublish('WebBrain test post', ''), allowed: true },// A click the browser refused sent nothing.'X Pro, refused': { url: pro, caption: 'Post', result: refused, now: xPublish('WebBrain test post', ''), allowed: true },+ // An ordinary button whose caption a model read, and Enter in the post+ // editor, which edits the draft, post nothing.+ 'X Pro, an ordinary button read as such': { url: pro, probe: { isSubmit: false, resolvedOrdinaryActivationTarget: true },+ now: xPublish('WebBrain test post', ''), allowed: true },+ 'X Pro, Enter in the draft': { url: pro, name: 'press_keys', args: { key: 'Enter' }, probe: { isSubmit: false, resolvedEditableTarget: true },+ now: xPublish('WebBrain test post', ''), allowed: true },};for (const [variant, spec] of Object.entries(cases)) {const h = await fixture('Post "WebBrain test post" on X and on mastoturk.org.', { pageUrl: spec.now.url });@@ -703,8 +713,12 @@ for (const build of ['chrome', 'firefox']) {attempt(h, mastodonCompose, deliveredUnobserved);await h.steer('ok now on X');// Under a correction a control the probe could not identify is not- // clicked at all; one it identified as a form goes ahead.+ // clicked at all, nor one that looks ordinary only because its caption+ // went unread; one it identified as a form goes ahead.assert.equal(h.steeringBlock(null)?.actionGuard.reason, 'target_unresolved');+ assert.equal(h.steeringBlock({ isSubmit: false, resolvedOrdinaryActivationTarget: true, captionUnread: true })?.actionGuard.reason,+ 'target_unresolved');+ assert.equal(h.steeringBlock({ isSubmit: false, resolvedOrdinaryActivationTarget: true }), null);attempt(h, { ...formOn(pro), actionUrl: 'https://pro.x.com/i/api/compose' });if (resume === 'Continue' || resume === 'typed continue') {h.agent._storeContinuationExecutionEvidence(h.tab);diff --git a/test/quality/correction-cross-site-mv3.mjs b/test/quality/correction-cross-site-mv3.mjsindex d239a1e6a..df2d2df84 100644--- a/test/quality/correction-cross-site-mv3.mjs+++ b/test/quality/correction-cross-site-mv3.mjs@@ -25,6 +25,11 @@ import { fixedChoiceQuestion, fixtureFixedChoiceReply, fixturePageTextReply } fr// xpro-then-x: the X post is made from X Pro (pro.x.com), whose Post// button WebBrain does not recognize, then the Mastodon post, then "ok// now on X": the X Pro post still counts, so x.com gets no second one.+// xpro-slow-plain-then-x: the same with a plain Post button, while the+// model takes six seconds to read captions, longer than WebBrain waits:+// the button looks ordinary, and its post still counts.+// xpro-enter-then-x: only Enter is pressed in an X Pro draft that is never+// posted: that is no post, so the X post after the correction goes out.// Pages, posts and the model are owned local fixtures; nothing leaves the host.const require = createRequire(import.meta.url);const library = (name, file) => fs.readFile(path.join(path.dirname(require.resolve(`${name}/package.json`)), file), 'utf8');@@ -49,14 +54,15 @@ document.querySelector('#compose').addEventListener('submit', async event => {const mastodonHome = posts => `<!doctype html><meta charset="utf-8"><title>Anasayfa - MastoTurk</title><h1>Anasayfa</h1><div role="feed" aria-label="Anasayfa">${posts.map((post, index) =>`<article aria-label="esokullu"><a href="/@esokullu/${1000 + index}">esokullu</a><p>${post.replace(/</g, '<')}</p></article>`).join('')}</div>`;-// X Pro: a deck with its own composer, whose Post button sits outside any form.-const xProHtml = `<!doctype html><meta charset="utf-8"><title>X Pro</title>+// X Pro: a deck with its own composer, whose Post button sits outside any+// form; on the second deck the button carries no app marker at all.+const xProHtml = ({ plain = false } = {}) => `<!doctype html><meta charset="utf-8"><title>X Pro</title><style>body{font:15px sans-serif;margin:24px} [role=textbox]{border:1px solid #888;min-height:80px;width:500px;padding:8px} button{padding:8px 16px}</style><main aria-label="Deck"><section aria-label="Compose a post">-<div role="textbox" contenteditable="true" aria-multiline="true" aria-label="Post text"></div>-<div><button type="button" data-testid="tweetButton">Post</button></div></section></main>+<div role="textbox" contenteditable="true" aria-multiline="true" aria-label="Post text" data-testid="tweetTextarea_0"></div>+<div><button type="button" id="post"${plain ? '' : ' data-testid="tweetButton"'}>Post</button></div></section></main><script>-document.querySelector('[data-testid="tweetButton"]').addEventListener('click', async () => {+document.querySelector('#post').addEventListener('click', async () => {const box = document.querySelector('[role="textbox"]'), body = box.textContent;if (!body.trim()) return;await fetch('/owned-fixture/post', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ body }) });@@ -197,6 +203,58 @@ const variants = {() => ({ name: 'done', args: { outcome: 'partial', summary: 'The post was already made on X from X Pro; it was not posted again.' } }),],},+ 'xpro-slow-plain-then-x': {+ task: 'try to post something on X from X Pro (pro.x.com), then on mastoturk.org',+ slowCaptionReadings: true,+ before: [+ () => ({ name: 'navigate', args: { url: 'https://pro.x.com/i/decks/2' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'type_ax', args: { ref_id: refFrom(tree, /textbox "Post text"/), text: 'WebBrain test post', clear: true } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'click_ax', args: { ref_id: refFrom(tree, /button "Post"/) } }),+ () => ({ name: 'navigate', args: { url: 'https://mastoturk.org/publish' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'set_field', args: { ref_id: refFrom(tree, /textbox "Aklında ne var\?"/), text: 'WebBrain test gönderisi', submit: false } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'click_ax', args: { ref_id: refFrom(tree, /button "Gönder"/) } }),+ async (_tree, run) => { await run.steer(); return { name: 'get_accessibility_tree', args: { filter: 'all' } }; },+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ ],+ after: [+ () => ({ name: 'navigate', args: { url: 'https://x.com/home' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'type_ax', args: { ref_id: refFrom(tree, /textbox "Post text"/), text: 'WebBrain test post', clear: true } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'click_ax', args: { ref_id: refFrom(tree, /button "Post"/) } }),+ () => ({ name: 'done', args: { outcome: 'partial', summary: 'The post was already made on X from X Pro; it was not posted again.' } }),+ ],+ },+ 'xpro-enter-then-x': {+ task: 'write a two-line draft on X Pro (pro.x.com) without posting it, then try to post something on mastoturk.org',+ before: [+ () => ({ name: 'navigate', args: { url: 'https://pro.x.com/i/decks/1' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'type_ax', args: { ref_id: refFrom(tree, /textbox "Post text"/), text: 'First line', clear: true } }),+ () => ({ name: 'press_keys', args: { key: 'Enter' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ () => ({ name: 'navigate', args: { url: 'https://mastoturk.org/publish' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'set_field', args: { ref_id: refFrom(tree, /textbox "Aklında ne var\?"/), text: 'WebBrain test gönderisi', submit: false } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'click_ax', args: { ref_id: refFrom(tree, /button "Gönder"/) } }),+ async (_tree, run) => { await run.steer(); return { name: 'get_accessibility_tree', args: { filter: 'all' } }; },+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ ],+ after: [+ () => ({ name: 'navigate', args: { url: 'https://x.com/home' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'type_ax', args: { ref_id: refFrom(tree, /textbox "Post text"/), text: 'WebBrain test post', clear: true } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'click_ax', args: { ref_id: refFrom(tree, /button "Post"/) } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ () => ({ name: 'done', args: { outcome: 'success', summary: 'Drafted on X Pro, posted on mastoturk.org and on X.' } }),+ ],+ },'leave-prompt': {before: [() => ({ name: 'navigate', args: { url: 'https://x.com/home' } }),@@ -220,6 +278,7 @@ async function runVariant(variant, script) {const task = script.task || (variant === 'leave-prompt'? 'open x.com, write a draft post without posting it, then go to mastoturk.org/home' : request);const mastodonPosts = [], xPosts = [], xProPosts = [], unknownPrompts = [], dialogs = [], settleDialogs = [];+ let slowReadings = 0;let steered = false, judgeCalls = 0, blockedRemote = 0, base = '', harnessAnswered = 0;const phase = { name: 'before', index: 0, lastId: null, emitted: 0 };const completionRecovery = { reads: 0 };@@ -260,6 +319,12 @@ async function runVariant(variant, script) {let raw = ''; for await (const part of incoming) raw += part;const input = JSON.parse(raw), system = textOf(input.messages?.[0]?.content);let completion;+ // A slow model: caption readings take longer than WebBrain waits for+ // them, until the X Pro post is out.+ if (script.slowCaptionReadings && !xProPosts.length && fixedChoiceQuestion(input.messages)?.kind === 'control_effect') {+ slowReadings++;+ await new Promise(resolve => setTimeout(resolve, 6000));+ }try {if (system.startsWith('Extract stable user preferences')) completion = { role: 'assistant', content: '{"memories":[]}' };// The provider check made when the provider is saved.@@ -349,7 +414,7 @@ async function runVariant(variant, script) {xProPosts.push(JSON.parse(routed.postData() || '{}').body);return route.fulfill({ status: 200, contentType: 'application/json', body: '{"ok":true}' });}- if (routed.isNavigationRequest()) return route.fulfill({ contentType: 'text/html', body: xProHtml });+ if (routed.isNavigationRequest()) return route.fulfill({ contentType: 'text/html', body: xProHtml({ plain: requested.pathname === '/i/decks/2' }) });return route.fulfill({ status: 404, body: '' });}if (/^https?:$/.test(requested.protocol)) { blockedRemote++; return route.abort('blockedbyclient'); }@@ -444,12 +509,19 @@ async function runVariant(variant, script) {assert.deepEqual(xPosts, ['WebBrain test post'], `No second X post after "continue": ${JSON.stringify(continuedResults)}`);const again = continuedResults.filter(item => item.name === 'click_ax');assert.deepEqual(again.map(item => item.reason), ['earlier_attempt_on_platform'], JSON.stringify(continuedResults));- } else if (variant === 'xpro-then-x') {+ } else if (variant === 'xpro-then-x' || variant === 'xpro-slow-plain-then-x') {+ if (script.slowCaptionReadings) assert.ok(slowReadings > 0, 'A caption reading was slower than WebBrain waits');assert.deepEqual(xProPosts, ['WebBrain test post'], `The X Pro post went out: ${JSON.stringify(toolResults)}`);assert.deepEqual(mastodonPosts, ['WebBrain test gönderisi'], 'One Mastodon post');assert.deepEqual(xPosts, [], `No second X post on x.com: ${JSON.stringify(toolResults)}`);const xClick = toolResults.filter(item => item.name === 'click_ax').at(-1);assert.equal(xClick?.reason, 'earlier_attempt_on_platform', JSON.stringify(toolResults));+ } else if (variant === 'xpro-enter-then-x') {+ assert.deepEqual(xProPosts, [], 'The X Pro draft was never posted');+ assert.deepEqual(mastodonPosts, ['WebBrain test gönderisi'], 'One Mastodon post');+ const stopped = toolResults.filter(item => ['earlier_attempt_on_platform', 'attempt_history_unknown'].includes(item.reason));+ assert.deepEqual(stopped, [], `Enter in a draft is no post: ${JSON.stringify(toolResults)}`);+ assert.deepEqual(xPosts, ['WebBrain test post'], `One X post: ${JSON.stringify(toolResults)}`);} else {assert.ok(dialogs.includes('beforeunload'), `The X page asked before it was left: ${JSON.stringify(dialogs)}`);assert.equal(harnessAnswered, 0, 'WebBrain answered the leave prompt for its own navigation; nobody else had to');