Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Count posts from X and Bluesky hosts WebBrain cannot read; stop a post whose record cannot be saved
Claude3ab0dd2bd16c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1e1f7dd93..a318a9ad4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,7 +7,7 @@ This changelog was generated from the repository Git history and release tags. V## [Unreleased]### Fixed-- Posting on another site after a correction: after a Mastodon post, "ok now on X" no longer leaves WebBrain going back and forth between the two sites, typing the X post again each time. It counted X as the same platform as Mastodon, because it reads account names only on X and Bluesky, and asked to find the Mastodon post on X itself, which can never happen. The site a post was made on now names its platform when no account name was read, and a post on another site goes ahead once a page has been read since the correction, provided the task made no earlier post on that site. What the task attempted is saved with the conversation, and just before each action that may post or submit something, so this still holds when the task continues after WebBrain restarted, and a post already sent on X under the correction is not sent again by a later run of the task. When the task did post on that site before, or WebBrain has no full record of its earlier attempts, it stops once and says so, instead of searching for a post it cannot find. A search, a settings save or consent click on X, a click that sent nothing or a form the page rejected is not such a post; a click that Stop interrupted after it was sent counts. x.com and twitter.com are one platform.+- Posting on another site after a correction: after a Mastodon post, "ok now on X" no longer leaves WebBrain going back and forth between the two sites, typing the X post again each time. It counted X as the same platform as Mastodon, because it reads account names only on X and Bluesky, and asked to find the Mastodon post on X itself, which can never happen. The site a post was made on now names its platform when no account name was read, and a post on another site goes ahead once a page has been read since the correction, provided the task made no earlier post on that site. What the task attempted is saved with the conversation, and just before each action that may post or submit something, so this still holds when the task continues after WebBrain restarted, and a post already sent on X under the correction is not sent again by a later run of the task. When the task did post on that site before, or WebBrain has no full record of its earlier attempts, it stops once and says so, instead of searching for a post it cannot find. A search, a settings save or consent click on X, a click that sent nothing or a form the page rejected is not such a post; a click that Stop interrupted after it was sent counts. On the other hosts and clients of X and Bluesky, such as X Pro, whose Post buttons WebBrain does not recognize, anything sent that may have been a post counts as one, so a post from X Pro is not repeated on x.com. If that record cannot be saved just before a possible post on X or Bluesky, the post is not sent and WebBrain says why, as it already does when a run's recovery checkpoint cannot be saved. x.com and twitter.com are one platform.- Chrome: a page's "Leave site?" prompt no longer waits for you when WebBrain navigates away during a task that started on a page the debugger cannot attach to, such as chrome://extensions, or whose debugger connection dropped. WebBrain arms its dialog handling again at the next debugger attach and around each of its own navigations, and answers Leave for its own navigation as it already did on other runs. If you cancel the debugger, prompts stay yours to answer. Firefox offers extensions no way to answer these prompts.- Chat sites such as WhatsApp: a request that points at the open chat instead of naming the person, such as "send some test messages to this one" or "message this user", now sends there instead of asking "which contact, and what text?". The planner never sees the page, and its rules counted only words like "reply here" as the open chat, so it asked the same question every time without reading the page. WebBrain still reads the open chat's header itself and binds every send to that one contact, and stops if no single chat is open. The planner is also told it cannot see the page, no longer reads a note about a screenshot it is never given, and leaves the wording of described messages, such as test messages or a thank-you, to the agent. One request can now cover the open chat and someone named besides it, as in "test messages here, then thank Rustem Bey": the open chat needs no confirmation, the reviewed plan shows the named person, and that person is confirmed once on WebBrain's card before the first send to them. Words that refer back to someone named earlier still mean that person, and on mail sites the people named besides "reply here" are handled as before. On WhatsApp, where a contact is confirmed through its contact drawer, one run still confirms only one chat's contact, so the second chat of such a request needs a request of its own; the agent is now told so, instead of being sent to re-read the page and retry.- A subscription, registration or sign-in wall in front of a news article is now recognized in any language, by a model reading the wall's text once, instead of only when it used English phrases such as "Subscribe to continue reading". A Turkish "Okumaya devam etmek için abone olun" overlay now keeps WebBrain from reading the article hidden behind it, as an English one did, while an offer to subscribe beside text you can still read is not taken for a wall. A wall inside the article is found also after a long preview, and everything after the wall's text stays withheld; the preview before it stays readable, except the short passage just before the wall's text. If a page cannot be read again after its wall was recognized, the read fails rather than return what the wall hides. A wall whose number changes on every read, such as a countdown, stays a wall while its new text is read, and needs no more readings once two of its numbers were read as the wall, while a count that stops being a wall when its number changes reads as what it is once read; a wall whose other text keeps changing makes the read fail rather than leak, and the agent may read the page once more in case the wall went away.diff --git a/docs/architecture.md b/docs/architecture.mdindex 965fd7b38..10c9793a5 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -103,17 +103,26 @@ Mastodon post or a form, is placed by the site it was made on. Hosts that sharetheir last two labels count as one site, and x.com and twitter.com as oneplatform. A post on another site than the earlier attempt counts as anotherplatform only when the correction's receipt lists every publication the task-had made and none was on the post's site; on X and Bluesky only a composer-publication posts, so a settings save or a consent click there is not one.+had made and none was on the post's site; on x.com, twitter.com and bsky.app+only a composer publication posts, so a settings save or a consent click there+is not one. Their other hosts and clients, such as X Pro, post without the+composer controls WebBrain recognizes, so there any action that may have been+sent counts as a publication unless the page probe proved it cannot post (an+ordinary form, a deletion, or a control resolved as a link, field, menu item or+other non-submitting control).The agent records, for the task from its first run on, the site of everydispatched attempt that may have changed something and of every publicationamong them (a search, a send the browser refused or a form the page rejectedis not one; an action Stop interrupted after it was sent is). The record issaved with the conversation, and on its own just before each action that maycommit something is sent, so a restart right after that action still counts-it; each correction copies the record into its receipt. A post made under-the correction is kept from repeating by the publication contract's outcomes-and dispatch records of the run that applied it; a run rebuilt from the saved+it. When that save fails before a possible post on X or Bluesky, or an action+of unknown site, the action is not sent and the run stops with persistence+degraded, as it does when the run's recovery checkpoint cannot be saved. Each+correction copies the record into its receipt. A post made under the+correction is kept from repeating by the publication contract's outcomes and+dispatch records of the run that applied it, and by that run's own list of+the possible posts they never see, such as one from X Pro; a run rebuilt from the savedconversation, as after a restart or a typed "continue", has neither, so therea publication anywhere in the task on the post's site holds it too. Without acomplete record, or with an earlier publication on the post's own site, thediff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex d3feac6bc..664db487a 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -503,6 +503,10 @@ const PUBLICATION_SITE_ALIASES = Object.freeze({'x.com': 'twitter', 'twitter.com': 'twitter', 'bsky.app': 'bluesky','threads.net': 'threads', 'threads.com': 'threads', 'fb.com': 'facebook', 'facebook.com': 'facebook',});+// X and Bluesky: WebBrain recognizes their posts only by the composer's own+// controls on x.com, twitter.com and bsky.app, while their other hosts and+// clients, such as X Pro, post too.+const SOCIAL_POST_SITES = new Set(['twitter', 'bluesky']);const COMPLETION_DOCUMENT_OBSERVATION_TOOLS = new Set(['chat_observe',@@ -1986,7 +1990,7 @@ export class Agent extends LoopDetector {_recordCompletionSubmitAttempt(tabId, detectedSubmit, name, args, beforeUrl, afterUrl, result, beforeDocument = '', afterDocument = '', executionContext = {}) {const state = this._recordCompletionSubmitState(tabId, detectedSubmit, name, args, beforeUrl, afterUrl, result,beforeDocument, afterDocument, executionContext);- this._recordTaskAttempt(tabId, name, args, detectedSubmit, result, state);+ this._recordTaskAttempt(tabId, name, args, detectedSubmit, result, state, beforeUrl);return state;}@@ -2091,6 +2095,8 @@ export class Agent extends LoopDetector {taskKey: this._progressTaskKeyHash(tabId),runToken: this.completionInvariants.get(tabId)?.runToken || '',mutationKind: detectedSubmit?.mutationKind || (detectedSubmit?.publicationControl === true ? 'publish' : 'submit'),+ // An ordinary form, a deletion or a control that submits nothing.+ ...(this._cannotPost(detectedSubmit) ? { cannotPost: true } : {}),actionDocument: detectedSubmit?.frameDocumentIdentity || '',actionResource: detectedSubmit?.actionResource ? structuredClone(detectedSubmit.actionResource) : null,// Identity only: the receipt is persisted, and its row text is not needed.@@ -5872,18 +5878,84 @@ export class Agent extends LoopDetector {return created;}+ // What the page probe proved an action cannot post: an ordinary HTML form,+ // a deletion, or a control it resolved as a link, a field, a menu item or+ // another control that submits nothing.+ _cannotPost(detected) {+ if (!detected || typeof detected !== 'object' || detected.publicationControl === true) return false;+ if (detected.publicationControl === false || detected.mutationKind === 'delete') return true;+ return detected.isSubmit === false && (detected.resolvedNavigationTarget === true || detected.resolvedEditableTarget === true+ || detected.resolvedNonSubmitTarget === true || detected.resolvedOrdinaryActivationTarget === true+ || detected.resolvedPreparatoryTarget === true);+ }++ // Whether an attempt that may have changed something counts as a post: one+ // made with a recognized publish control, or on X or Bluesky any other one+ // the probe did not prove cannot post. Those sites post from hosts and+ // clients whose controls WebBrain does not recognize, such as X Pro.+ _attemptMayPost(keys, recognized, cannotPost) {+ return recognized === true || (cannotPost !== true && keys.some(key => SOCIAL_POST_SITES.has(key)));+ }++ // The same for a receipt. One of unknown site may have been a post anywhere.+ _receiptMayPost(receipt, keys = this._receiptPlatformKeys(receipt)) {+ return keys.includes('*') || !!receipt?.publicationSnapshot || !!receipt?.workflowBinding?.socialPublication?.snapshot+ || this._attemptMayPost(keys, receipt?.mutationKind === 'publish', receipt?.cannotPost === true || receipt?.mutationKind === 'delete');+ }++ // Enter, Return or Space sent by press_keys activates the focused control.+ _pressesActivationKey(name, args) {+ if (name !== 'press_keys') return false;+ const keys = args?.key ?? args?.keys ?? '';+ return (Array.isArray(keys) ? keys : [keys]).some(key => typeof key === 'string' && /^(?:enter|return|space|spacebar| )$/i.test(key));+ }++ // Whether an action may commit anything at all: a request that sends a+ // change, a store publication, a detected submission, or an activation+ // (a click, a script, a submitted field, Enter or Space).+ _mayCommitAction(name, args, detected) {+ return isNetworkMutation(name, args) || name === 'chrome_web_store_publish' || detected?.isSubmit === true+ || this._isFormValidationCandidate(name, args) || this._pressesActivationKey(name, args);+ }++ // An activation on an X or Bluesky page that the probe did not prove cannot+ // post, such as a click on X Pro's Post button, may be a post.+ _socialActivationMayPost(name, args, detected, pageUrl = '') {+ if (!this._isFormValidationCandidate(name, args) && !this._pressesActivationKey(name, args)) return false;+ if (this._cannotPost(detected) || this._nonCommittingSubmission(detected)) return false;+ return this._attemptPlatformKeys(detected?.publicationSnapshot?.account, [detected?.url || pageUrl, detected?.frameUrl])+ .some(key => SOCIAL_POST_SITES.has(key));+ }++ // Whether an action's result leaves open that it was sent, judged as the+ // submit receipt judges it.+ _actionMayHaveBeenSent(result) {+ if (!result || typeof result !== 'object') return false;+ const explicitlyNotSent = (result.dispatched === false || result.noDispatch === true)+ && result.dispatched !== true && result.missingToolResponse !== true+ && result.outcomeUnknown !== true && result.nativeInputAcknowledged !== true;+ return !explicitlyNotSent && (result.dispatched === true || result.nativeInputAcknowledged === true+ || result.missingToolResponse === true || result.outcomeUnknown === true || (result.success !== false && !result.error));+ }+// Adds what an attempt may have changed to the task's record: a dispatched- // submission, publication or deletion that commits something, a fetch that- // sends a mutation, and a store publication. A search, a send the browser- // refused and a form the page rejected commit nothing.- _recordTaskAttempt(tabId, name, args, detected, result, submitState) {+ // submission, publication or deletion that commits something, an activation+ // on X or Bluesky that may have posted without leaving a receipt, a fetch+ // that sends a mutation, and a store publication. A search, a send the+ // browser refused and a form the page rejected commit nothing.+ _recordTaskAttempt(tabId, name, args, detected, result, submitState, pageUrl = '') {try {const keys = new Set();let publication = false;if (submitState?.dispatched === true && submitState.formValidationFailed !== true && submitState.nonCommitting !== true) {for (const key of this._attemptPlatformKeys(submitState.publicationSnapshot?.account,[submitState.originatingUrl, detected?.url, detected?.frameUrl, submitState.iframeSubmission?.currentUrl])) keys.add(key);- publication = detected?.publicationControl === true || submitState.mutationKind === 'publish';+ publication = this._attemptMayPost([...keys], detected?.publicationControl === true || submitState.mutationKind === 'publish',+ submitState.cannotPost === true || submitState.mutationKind === 'delete' || this._cannotPost(detected));+ } else if (!submitState && this._socialActivationMayPost(name, args, detected, pageUrl) && this._actionMayHaveBeenSent(result)) {+ for (const key of this._attemptPlatformKeys(detected?.publicationSnapshot?.account,+ [detected?.url || pageUrl, detected?.frameUrl])) keys.add(key);+ publication = true;}if (isNetworkMutation(name, args) || name === 'chrome_web_store_publish') {const notSent = result?.noDispatch === true || result?.denied || result?.skipped || result?.cancelled@@ -5896,7 +5968,7 @@ export class Agent extends LoopDetector {}}if (keys.size) {- this._addTaskAttemptKeys(tabId, [...keys], publication);+ this._addTaskAttemptKeys(tabId, [...keys], publication, detected?.publicationControl === true);this._taskAttemptCheckpoints?.delete(tabId);} else {this._resaveTaskAttemptCheckpoint(tabId);@@ -5907,18 +5979,21 @@ export class Agent extends LoopDetector {}// What an action may commit, judged before it is sent: the sites it may- // change and whether it may publish there, or null when it commits nothing.+ // change and whether it may post there, or null when it commits nothing.// A control without its own address acts on the page it is on._potentialTaskAttempt(name, args, detected, pageUrl = '') {const network = isNetworkMutation(name, args) || name === 'chrome_web_store_publish';const commit = (detected?.isSubmit === true && !this._nonCommittingSubmission(detected))- || name === 'execute_js' || this._formValidationActionLooksSubmit(name, args, null, detected);+ || name === 'execute_js' || this._formValidationActionLooksSubmit(name, args, null, detected)+ || this._socialActivationMayPost(name, args, detected, pageUrl);if (!commit && !network) return null;const keys = new Set(commit ? this._attemptPlatformKeys(detected?.publicationSnapshot?.account,[detected?.url || pageUrl, detected?.frameUrl]) : []);if (network) keys.add(this._publicationSiteOf(name === 'chrome_web_store_publish'? 'https://chromewebstore.google.com/' : args?.url) || '*');- return { keys: [...keys], publication: network || detected?.publicationControl === true || name === 'execute_js' };+ const list = [...keys];+ return { keys: list, publication: network || name === 'execute_js'+ || this._attemptMayPost(list, detected?.publicationControl === true, this._cannotPost(detected)) };}// An action that may have been sent but whose result was never recorded,@@ -5927,7 +6002,7 @@ export class Agent extends LoopDetector {_recordInterruptedTaskAttempt(tabId, name, args, detected, potential = undefined) {try {const attempt = potential === undefined ? this._potentialTaskAttempt(name, args, detected) : potential;- if (attempt) this._addTaskAttemptKeys(tabId, attempt.keys, attempt.publication);+ if (attempt) this._addTaskAttemptKeys(tabId, attempt.keys, attempt.publication, detected?.publicationControl === true);this._taskAttemptCheckpoints?.delete(tabId);} catch {this._markTaskAttemptsUnknown(tabId);@@ -5936,10 +6011,14 @@ export class Agent extends LoopDetector {// Before an action that may commit something is sent, the record is saved// with its sites as possibly attempted, so a restart right after it cannot- // forget it. The conversation itself is saved only every so often.+ // forget it. The conversation itself is saved only every so often. Only an+ // action that may commit anything needs its page's address. A possible post+ // on X or Bluesky, or an action of unknown site, whose record could not be+ // saved comes back marked notSaved, and is not sent.async _checkpointTaskAttempt(tabId, name, args, detected) {let potential = null;try {+ if (!this._mayCommitAction(name, args, detected)) return null;const pageUrl = detected?.url ? '' : await this._currentUrl(tabId).catch(() => '');potential = this._potentialTaskAttempt(name, args, detected, pageUrl);if (!potential) return potential;@@ -5947,10 +6026,17 @@ export class Agent extends LoopDetector {if (!record) return potential;const add = (list, values) => Array.isArray(list) ? [...new Set([...list, ...values])].slice(0, 65) : null;const platforms = add(record.platforms, potential.keys);- const publications = potential.publication ? add(record.publications, potential.keys) : record.publications;- await chrome.storage.session.set({ [this._taskAttemptCheckpointKey(tabId)]: { ...record,- platforms: platforms?.length > 64 ? null : platforms, publications: publications?.length > 64 ? null : publications } });- this._taskAttemptCheckpoints.add(tabId);+ const publications = potential.publication ? add(record.publications, potential.keys)+ : potential.keys.includes('*') ? add(record.publications, ['*']) : record.publications;+ try {+ await chrome.storage.session.set({ [this._taskAttemptCheckpointKey(tabId)]: { ...record,+ platforms: platforms?.length > 64 ? null : platforms, publications: publications?.length > 64 ? null : publications } });+ } catch {+ const matters = potential.keys.includes('*')+ || (potential.publication && potential.keys.some(key => SOCIAL_POST_SITES.has(key)));+ return matters ? { ...potential, notSaved: true } : potential;+ }+ (this._taskAttemptCheckpoints ||= new Set()).add(tabId);} catch {}return potential;}@@ -5967,7 +6053,7 @@ export class Agent extends LoopDetector {_taskAttemptCheckpointKey(tabId) { return `agentAttempts:${tabId}`; }- _addTaskAttemptKeys(tabId, keys, publication) {+ _addTaskAttemptKeys(tabId, keys, publication, contracted = false) {const ledger = this._taskAttemptLedger(tabId);const add = (list, values) => {if (!Array.isArray(list)) return null;@@ -5977,6 +6063,14 @@ export class Agent extends LoopDetector {ledger.platforms = add(ledger.platforms, keys);if (publication) ledger.publications = add(ledger.publications, keys);else if (keys.includes('*')) ledger.publications = add(ledger.publications, ['*']);+ // The run keeps its own list of the posts its publication contract and+ // dispatch records never see, such as one from X Pro, and of attempts of+ // unknown site.+ const guard = this._planExecutionGuards?.get(tabId);+ if (guard && ((publication && !contracted) || keys.includes('*'))) {+ guard.uncontractedPublications = add(guard.uncontractedPublications === null ? null : guard.uncontractedPublications || [],+ publication ? keys : ['*']);+ }}// A record that could not be updated is no longer complete; the action@@ -5985,6 +6079,8 @@ export class Agent extends LoopDetector {try {const entry = this._taskAttemptLedgers?.get(tabId);if (entry) { entry.platforms = null; entry.publications = null; }+ const guard = this._planExecutionGuards?.get(tabId);+ if (guard) guard.uncontractedPublications = null;} catch {}}@@ -6022,8 +6118,7 @@ export class Agent extends LoopDetector {const ledger = this._taskAttemptLedger(tabId, messages, binding);const committed = receipt && receipt.nonCommitting !== true && receipt.formValidationFailed !== true;const keys = committed ? this._receiptPlatformKeys(receipt) : [];- const published = committed && (receipt.mutationKind === 'publish' || !!receipt.publicationSnapshot- || !!receipt.workflowBinding?.socialPublication?.snapshot || keys.includes('*'));+ const published = committed && this._receiptMayPost(receipt, keys);const add = (list, values) => {if (!Array.isArray(list)) return null;const merged = [...new Set([...list, ...values])];@@ -6092,15 +6187,19 @@ export class Agent extends LoopDetector {// checked against the X and Bluesky outcomes and the receipts archived at// earlier corrections. A sent-nothing outcome is not an attempt; an// attempt of unknown site may have been anywhere.- const attempted = Array.isArray(prior?.taskAttemptPublications) ? prior.taskAttemptPublications : null;+ // A receipt that lists only the sites attempted covers its publications too.+ const attempted = Array.isArray(prior?.taskAttemptPublications) ? prior.taskAttemptPublications+ : Array.isArray(prior?.taskAttemptPlatforms) ? prior.taskAttemptPlatforms : null;const outcomePlatforms = archive => (archive?.contract?.actions || []).filter(action => action?.id && archive.outcomes?.[action.id] && archive.outcomes[action.id].status !== 'failed').map(action => action.platform || '*');// Posts made under the correction itself: in the run that applied it, or// a Continue that carries that run, the publication contract's outcomes- // and the dispatch records stop a repeat. A run rebuilt from the- // conversation has neither, so the task's record of publications does.- const sinceCorrection = guard.steeringHistoryInMemory === true ? []+ // and the dispatch records stop a repeat, and the run's own list holds the+ // posts they never see, such as one from X Pro. A run rebuilt from the+ // conversation has none of them, so the task's record of publications does.+ const sinceCorrection = guard.steeringHistoryInMemory === true+ ? (guard.uncontractedPublications === null ? null : guard.uncontractedPublications || []): this._taskAttemptLedger(tabId).publications;const earlierPlatforms = [...(attempted || []), ...(sinceCorrection || []),...[guard.steeringPriorPublication, guard.steeringPriorContext?.socialPublication].flatMap(outcomePlatforms),@@ -6108,8 +6207,7 @@ export class Agent extends LoopDetector {&& receipt.nonCommitting !== true && receipt.formValidationFailed !== true).flatMap(receipt => {const keys = this._receiptPlatformKeys(receipt);- return receipt.mutationKind === 'publish' || receipt.publicationSnapshot || receipt.workflowBinding?.socialPublication?.snapshot- || keys.includes('*') ? keys : [];+ return this._receiptMayPost(receipt, keys) ? keys : [];})];const earlierAttemptHere = earlierPlatforms.some(platform => currentPlatforms.includes(platform));const historyKnown = !!attempted && Array.isArray(sinceCorrection) && !earlierPlatforms.includes('*');@@ -14707,6 +14805,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dthis._throwIfAborted(this._runAbortSignal(tabId));const _toolStart = Date.now();let steeringBeforeDispatch = false;+ let attemptRecordNotSaved = false;let toolbarPreflight = { block: null };let rawToolResult = protectedPageFailure;let toolResult;@@ -14749,6 +14848,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// sent, ahead of the last correction check.callState.potentialAttempt = await this._checkpointTaskAttempt(tabId, fnName, fnArgs, detectedSubmitAction);this._throwIfAborted(abortSignal);+ // A possible post whose record could not be saved is not sent:+ // after a restart WebBrain would not know it was attempted.+ if (callState.potentialAttempt?.notSaved === true) return { attemptRecordNotSaved: true };if (this._hasPendingSteering(tabId) || await this._refreshPageFeedbackForCall(tabId, tc, onUpdate)) {this._resaveTaskAttemptCheckpoint(tabId);return { steered: true };@@ -14881,6 +14983,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dthis._runAbortSignal(tabId),);steeringBeforeDispatch = pipelineResult.steered === true;+ attemptRecordNotSaved = pipelineResult.attemptRecordNotSaved === true;toolbarPreflight = pipelineResult.toolbarPreflight;rawToolResult = pipelineResult.rawToolResult;toolResult = pipelineResult.toolResult;@@ -14901,11 +15004,22 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d} else {const pipelineResult = await runActionPipeline(null);steeringBeforeDispatch = pipelineResult.steered === true;+ attemptRecordNotSaved = pipelineResult.attemptRecordNotSaved === true;toolbarPreflight = pipelineResult.toolbarPreflight;rawToolResult = pipelineResult.rawToolResult;toolResult = pipelineResult.toolResult;}}+ if (attemptRecordNotSaved) {+ const value = 'Stopped before sending the page action because the record of what this task has posted could not be saved. Retry after extension storage is available.';+ this._appendSyntheticToolResults(tabId, toolCalls, toolIndex, messages, onUpdate, step, () => ({+ success: false, dispatched: false, noDispatch: true, outcomeUnknown: false,+ errorCode: 'checkpoint_not_durable', error: value,+ }));+ this._markPersistenceDegraded(tabId, 'attempt_record');+ onUpdate('warning', { message: value });+ return { action: 'return', value, status: 'persistence_degraded' };+ }if (steeringBeforeDispatch) {this._appendSyntheticToolResults(tabId, toolCalls, toolIndex, messages, onUpdate, step, () => ({success: false, skipped: true, dispatched: false, noDispatch: true,@@ -35027,6 +35141,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// keeps them; any other start rebuilds the guard from the conversation.steeringHistoryInMemory: carryMatches && !!carried.steeringPriorSubmission&& carried.steeringHistoryInMemory === true,+ // The posts of that run its contract and dispatch records never saw.+ uncontractedPublications: carryMatches && !!carried.steeringPriorSubmission && carried.steeringHistoryInMemory === true+ ? (Array.isArray(carried.uncontractedPublications) ? [...carried.uncontractedPublications] : null) : [],steeringMutationReceipts: carryMatches ? structuredClone(carried.steeringMutationReceipts || []) : [],steeringPriorContext: carryMatches ? structuredClone(carried.steeringPriorContext || null) : null,steeringPriorSubmissionReconciliation: carryMatches ? structuredClone(carried.steeringPriorSubmissionReconciliation || null) : null,@@ -35561,6 +35678,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dsteeringPriorSubmission: guard.steeringPriorSubmission ? structuredClone(guard.steeringPriorSubmission) : null,steeringPriorReadAfterCorrection: guard.steeringPriorReadAfterCorrection === true,steeringHistoryInMemory: guard.steeringHistoryInMemory === true,+ uncontractedPublications: Array.isArray(guard.uncontractedPublications) ? [...guard.uncontractedPublications]+ : guard.uncontractedPublications === null ? null : [],steeringMutationReceipts: structuredClone(guard.steeringMutationReceipts || []),steeringPriorContext: structuredClone(guard.steeringPriorContext || null),steeringPriorSubmissionReconciliation: structuredClone(guard.steeringPriorSubmissionReconciliation || null),diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 11e6c7cf2..eae4dafd4 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -342,6 +342,10 @@ const PUBLICATION_SITE_ALIASES = Object.freeze({'x.com': 'twitter', 'twitter.com': 'twitter', 'bsky.app': 'bluesky','threads.net': 'threads', 'threads.com': 'threads', 'fb.com': 'facebook', 'facebook.com': 'facebook',});+// X and Bluesky: WebBrain recognizes their posts only by the composer's own+// controls on x.com, twitter.com and bsky.app, while their other hosts and+// clients, such as X Pro, post too.+const SOCIAL_POST_SITES = new Set(['twitter', 'bluesky']);const COMPLETION_DOCUMENT_OBSERVATION_TOOLS = new Set(['chat_observe',@@ -1675,7 +1679,7 @@ export class Agent extends LoopDetector {_recordCompletionSubmitAttempt(tabId, detectedSubmit, name, args, beforeUrl, afterUrl, result, beforeDocument = '', afterDocument = '', executionContext = {}) {const state = this._recordCompletionSubmitState(tabId, detectedSubmit, name, args, beforeUrl, afterUrl, result,beforeDocument, afterDocument, executionContext);- this._recordTaskAttempt(tabId, name, args, detectedSubmit, result, state);+ this._recordTaskAttempt(tabId, name, args, detectedSubmit, result, state, beforeUrl);return state;}@@ -1780,6 +1784,8 @@ export class Agent extends LoopDetector {taskKey: this._progressTaskKeyHash(tabId),runToken: this.completionInvariants.get(tabId)?.runToken || '',mutationKind: detectedSubmit?.mutationKind || (detectedSubmit?.publicationControl === true ? 'publish' : 'submit'),+ // An ordinary form, a deletion or a control that submits nothing.+ ...(this._cannotPost(detectedSubmit) ? { cannotPost: true } : {}),actionDocument: detectedSubmit?.frameDocumentIdentity || '',actionResource: detectedSubmit?.actionResource ? structuredClone(detectedSubmit.actionResource) : null,// Identity only: the receipt is persisted, and its row text is not needed.@@ -5545,18 +5551,84 @@ export class Agent extends LoopDetector {return created;}+ // What the page probe proved an action cannot post: an ordinary HTML form,+ // a deletion, or a control it resolved as a link, a field, a menu item or+ // another control that submits nothing.+ _cannotPost(detected) {+ if (!detected || typeof detected !== 'object' || detected.publicationControl === true) return false;+ if (detected.publicationControl === false || detected.mutationKind === 'delete') return true;+ return detected.isSubmit === false && (detected.resolvedNavigationTarget === true || detected.resolvedEditableTarget === true+ || detected.resolvedNonSubmitTarget === true || detected.resolvedOrdinaryActivationTarget === true+ || detected.resolvedPreparatoryTarget === true);+ }++ // Whether an attempt that may have changed something counts as a post: one+ // made with a recognized publish control, or on X or Bluesky any other one+ // the probe did not prove cannot post. Those sites post from hosts and+ // clients whose controls WebBrain does not recognize, such as X Pro.+ _attemptMayPost(keys, recognized, cannotPost) {+ return recognized === true || (cannotPost !== true && keys.some(key => SOCIAL_POST_SITES.has(key)));+ }++ // The same for a receipt. One of unknown site may have been a post anywhere.+ _receiptMayPost(receipt, keys = this._receiptPlatformKeys(receipt)) {+ return keys.includes('*') || !!receipt?.publicationSnapshot || !!receipt?.workflowBinding?.socialPublication?.snapshot+ || this._attemptMayPost(keys, receipt?.mutationKind === 'publish', receipt?.cannotPost === true || receipt?.mutationKind === 'delete');+ }++ // Enter, Return or Space sent by press_keys activates the focused control.+ _pressesActivationKey(name, args) {+ if (name !== 'press_keys') return false;+ const keys = args?.key ?? args?.keys ?? '';+ return (Array.isArray(keys) ? keys : [keys]).some(key => typeof key === 'string' && /^(?:enter|return|space|spacebar| )$/i.test(key));+ }++ // Whether an action may commit anything at all: a request that sends a+ // change, a store publication, a detected submission, or an activation+ // (a click, a script, a submitted field, Enter or Space).+ _mayCommitAction(name, args, detected) {+ return isNetworkMutation(name, args) || name === 'chrome_web_store_publish' || detected?.isSubmit === true+ || this._isFormValidationCandidate(name, args) || this._pressesActivationKey(name, args);+ }++ // An activation on an X or Bluesky page that the probe did not prove cannot+ // post, such as a click on X Pro's Post button, may be a post.+ _socialActivationMayPost(name, args, detected, pageUrl = '') {+ if (!this._isFormValidationCandidate(name, args) && !this._pressesActivationKey(name, args)) return false;+ if (this._cannotPost(detected) || this._nonCommittingSubmission(detected)) return false;+ return this._attemptPlatformKeys(detected?.publicationSnapshot?.account, [detected?.url || pageUrl, detected?.frameUrl])+ .some(key => SOCIAL_POST_SITES.has(key));+ }++ // Whether an action's result leaves open that it was sent, judged as the+ // submit receipt judges it.+ _actionMayHaveBeenSent(result) {+ if (!result || typeof result !== 'object') return false;+ const explicitlyNotSent = (result.dispatched === false || result.noDispatch === true)+ && result.dispatched !== true && result.missingToolResponse !== true+ && result.outcomeUnknown !== true && result.nativeInputAcknowledged !== true;+ return !explicitlyNotSent && (result.dispatched === true || result.nativeInputAcknowledged === true+ || result.missingToolResponse === true || result.outcomeUnknown === true || (result.success !== false && !result.error));+ }+// Adds what an attempt may have changed to the task's record: a dispatched- // submission, publication or deletion that commits something, a fetch that- // sends a mutation, and a store publication. A search, a send the browser- // refused and a form the page rejected commit nothing.- _recordTaskAttempt(tabId, name, args, detected, result, submitState) {+ // submission, publication or deletion that commits something, an activation+ // on X or Bluesky that may have posted without leaving a receipt, a fetch+ // that sends a mutation, and a store publication. A search, a send the+ // browser refused and a form the page rejected commit nothing.+ _recordTaskAttempt(tabId, name, args, detected, result, submitState, pageUrl = '') {try {const keys = new Set();let publication = false;if (submitState?.dispatched === true && submitState.formValidationFailed !== true && submitState.nonCommitting !== true) {for (const key of this._attemptPlatformKeys(submitState.publicationSnapshot?.account,[submitState.originatingUrl, detected?.url, detected?.frameUrl, submitState.iframeSubmission?.currentUrl])) keys.add(key);- publication = detected?.publicationControl === true || submitState.mutationKind === 'publish';+ publication = this._attemptMayPost([...keys], detected?.publicationControl === true || submitState.mutationKind === 'publish',+ submitState.cannotPost === true || submitState.mutationKind === 'delete' || this._cannotPost(detected));+ } else if (!submitState && this._socialActivationMayPost(name, args, detected, pageUrl) && this._actionMayHaveBeenSent(result)) {+ for (const key of this._attemptPlatformKeys(detected?.publicationSnapshot?.account,+ [detected?.url || pageUrl, detected?.frameUrl])) keys.add(key);+ publication = true;}if (isNetworkMutation(name, args) || name === 'chrome_web_store_publish') {const notSent = result?.noDispatch === true || result?.denied || result?.skipped || result?.cancelled@@ -5569,7 +5641,7 @@ export class Agent extends LoopDetector {}}if (keys.size) {- this._addTaskAttemptKeys(tabId, [...keys], publication);+ this._addTaskAttemptKeys(tabId, [...keys], publication, detected?.publicationControl === true);this._taskAttemptCheckpoints?.delete(tabId);} else {this._resaveTaskAttemptCheckpoint(tabId);@@ -5580,18 +5652,21 @@ export class Agent extends LoopDetector {}// What an action may commit, judged before it is sent: the sites it may- // change and whether it may publish there, or null when it commits nothing.+ // change and whether it may post there, or null when it commits nothing.// A control without its own address acts on the page it is on._potentialTaskAttempt(name, args, detected, pageUrl = '') {const network = isNetworkMutation(name, args) || name === 'chrome_web_store_publish';const commit = (detected?.isSubmit === true && !this._nonCommittingSubmission(detected))- || name === 'execute_js' || this._formValidationActionLooksSubmit(name, args, null, detected);+ || name === 'execute_js' || this._formValidationActionLooksSubmit(name, args, null, detected)+ || this._socialActivationMayPost(name, args, detected, pageUrl);if (!commit && !network) return null;const keys = new Set(commit ? this._attemptPlatformKeys(detected?.publicationSnapshot?.account,[detected?.url || pageUrl, detected?.frameUrl]) : []);if (network) keys.add(this._publicationSiteOf(name === 'chrome_web_store_publish'? 'https://chromewebstore.google.com/' : args?.url) || '*');- return { keys: [...keys], publication: network || detected?.publicationControl === true || name === 'execute_js' };+ const list = [...keys];+ return { keys: list, publication: network || name === 'execute_js'+ || this._attemptMayPost(list, detected?.publicationControl === true, this._cannotPost(detected)) };}// An action that may have been sent but whose result was never recorded,@@ -5600,7 +5675,7 @@ export class Agent extends LoopDetector {_recordInterruptedTaskAttempt(tabId, name, args, detected, potential = undefined) {try {const attempt = potential === undefined ? this._potentialTaskAttempt(name, args, detected) : potential;- if (attempt) this._addTaskAttemptKeys(tabId, attempt.keys, attempt.publication);+ if (attempt) this._addTaskAttemptKeys(tabId, attempt.keys, attempt.publication, detected?.publicationControl === true);this._taskAttemptCheckpoints?.delete(tabId);} catch {this._markTaskAttemptsUnknown(tabId);@@ -5609,10 +5684,14 @@ export class Agent extends LoopDetector {// Before an action that may commit something is sent, the record is saved// with its sites as possibly attempted, so a restart right after it cannot- // forget it. The conversation itself is saved only every so often.+ // forget it. The conversation itself is saved only every so often. Only an+ // action that may commit anything needs its page's address. A possible post+ // on X or Bluesky, or an action of unknown site, whose record could not be+ // saved comes back marked notSaved, and is not sent.async _checkpointTaskAttempt(tabId, name, args, detected) {let potential = null;try {+ if (!this._mayCommitAction(name, args, detected)) return null;const pageUrl = detected?.url ? '' : await this._currentUrl(tabId).catch(() => '');potential = this._potentialTaskAttempt(name, args, detected, pageUrl);if (!potential) return potential;@@ -5620,10 +5699,17 @@ export class Agent extends LoopDetector {if (!record) return potential;const add = (list, values) => Array.isArray(list) ? [...new Set([...list, ...values])].slice(0, 65) : null;const platforms = add(record.platforms, potential.keys);- const publications = potential.publication ? add(record.publications, potential.keys) : record.publications;- await browser.storage.session.set({ [this._taskAttemptCheckpointKey(tabId)]: { ...record,- platforms: platforms?.length > 64 ? null : platforms, publications: publications?.length > 64 ? null : publications } });- this._taskAttemptCheckpoints.add(tabId);+ const publications = potential.publication ? add(record.publications, potential.keys)+ : potential.keys.includes('*') ? add(record.publications, ['*']) : record.publications;+ try {+ await browser.storage.session.set({ [this._taskAttemptCheckpointKey(tabId)]: { ...record,+ platforms: platforms?.length > 64 ? null : platforms, publications: publications?.length > 64 ? null : publications } });+ } catch {+ const matters = potential.keys.includes('*')+ || (potential.publication && potential.keys.some(key => SOCIAL_POST_SITES.has(key)));+ return matters ? { ...potential, notSaved: true } : potential;+ }+ (this._taskAttemptCheckpoints ||= new Set()).add(tabId);} catch {}return potential;}@@ -5640,7 +5726,7 @@ export class Agent extends LoopDetector {_taskAttemptCheckpointKey(tabId) { return `agentAttempts:${tabId}`; }- _addTaskAttemptKeys(tabId, keys, publication) {+ _addTaskAttemptKeys(tabId, keys, publication, contracted = false) {const ledger = this._taskAttemptLedger(tabId);const add = (list, values) => {if (!Array.isArray(list)) return null;@@ -5650,6 +5736,14 @@ export class Agent extends LoopDetector {ledger.platforms = add(ledger.platforms, keys);if (publication) ledger.publications = add(ledger.publications, keys);else if (keys.includes('*')) ledger.publications = add(ledger.publications, ['*']);+ // The run keeps its own list of the posts its publication contract and+ // dispatch records never see, such as one from X Pro, and of attempts of+ // unknown site.+ const guard = this._planExecutionGuards?.get(tabId);+ if (guard && ((publication && !contracted) || keys.includes('*'))) {+ guard.uncontractedPublications = add(guard.uncontractedPublications === null ? null : guard.uncontractedPublications || [],+ publication ? keys : ['*']);+ }}// A record that could not be updated is no longer complete; the action@@ -5658,6 +5752,8 @@ export class Agent extends LoopDetector {try {const entry = this._taskAttemptLedgers?.get(tabId);if (entry) { entry.platforms = null; entry.publications = null; }+ const guard = this._planExecutionGuards?.get(tabId);+ if (guard) guard.uncontractedPublications = null;} catch {}}@@ -5695,8 +5791,7 @@ export class Agent extends LoopDetector {const ledger = this._taskAttemptLedger(tabId, messages, binding);const committed = receipt && receipt.nonCommitting !== true && receipt.formValidationFailed !== true;const keys = committed ? this._receiptPlatformKeys(receipt) : [];- const published = committed && (receipt.mutationKind === 'publish' || !!receipt.publicationSnapshot- || !!receipt.workflowBinding?.socialPublication?.snapshot || keys.includes('*'));+ const published = committed && this._receiptMayPost(receipt, keys);const add = (list, values) => {if (!Array.isArray(list)) return null;const merged = [...new Set([...list, ...values])];@@ -5765,15 +5860,19 @@ export class Agent extends LoopDetector {// checked against the X and Bluesky outcomes and the receipts archived at// earlier corrections. A sent-nothing outcome is not an attempt; an// attempt of unknown site may have been anywhere.- const attempted = Array.isArray(prior?.taskAttemptPublications) ? prior.taskAttemptPublications : null;+ // A receipt that lists only the sites attempted covers its publications too.+ const attempted = Array.isArray(prior?.taskAttemptPublications) ? prior.taskAttemptPublications+ : Array.isArray(prior?.taskAttemptPlatforms) ? prior.taskAttemptPlatforms : null;const outcomePlatforms = archive => (archive?.contract?.actions || []).filter(action => action?.id && archive.outcomes?.[action.id] && archive.outcomes[action.id].status !== 'failed').map(action => action.platform || '*');// Posts made under the correction itself: in the run that applied it, or// a Continue that carries that run, the publication contract's outcomes- // and the dispatch records stop a repeat. A run rebuilt from the- // conversation has neither, so the task's record of publications does.- const sinceCorrection = guard.steeringHistoryInMemory === true ? []+ // and the dispatch records stop a repeat, and the run's own list holds the+ // posts they never see, such as one from X Pro. A run rebuilt from the+ // conversation has none of them, so the task's record of publications does.+ const sinceCorrection = guard.steeringHistoryInMemory === true+ ? (guard.uncontractedPublications === null ? null : guard.uncontractedPublications || []): this._taskAttemptLedger(tabId).publications;const earlierPlatforms = [...(attempted || []), ...(sinceCorrection || []),...[guard.steeringPriorPublication, guard.steeringPriorContext?.socialPublication].flatMap(outcomePlatforms),@@ -5781,8 +5880,7 @@ export class Agent extends LoopDetector {&& receipt.nonCommitting !== true && receipt.formValidationFailed !== true).flatMap(receipt => {const keys = this._receiptPlatformKeys(receipt);- return receipt.mutationKind === 'publish' || receipt.publicationSnapshot || receipt.workflowBinding?.socialPublication?.snapshot- || keys.includes('*') ? keys : [];+ return this._receiptMayPost(receipt, keys) ? keys : [];})];const earlierAttemptHere = earlierPlatforms.some(platform => currentPlatforms.includes(platform));const historyKnown = !!attempted && Array.isArray(sinceCorrection) && !earlierPlatforms.includes('*');@@ -13276,6 +13374,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dthis._throwIfAborted(this._runAbortSignal(tabId));const _toolStart = Date.now();let steeringBeforeDispatch = false;+ let attemptRecordNotSaved = false;let toolbarPreflight = { block: null };let rawToolResult;let toolResult;@@ -13308,6 +13407,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// sent, ahead of the last correction check.callState.potentialAttempt = await this._checkpointTaskAttempt(tabId, fnName, fnArgs, detectedSubmitAction);this._throwIfAborted(abortSignal);+ // A possible post whose record could not be saved is not sent:+ // after a restart WebBrain would not know it was attempted.+ if (callState.potentialAttempt?.notSaved === true) return { attemptRecordNotSaved: true };if (this._hasPendingSteering(tabId) || await this._refreshPageFeedbackForCall(tabId, tc, onUpdate)) {this._resaveTaskAttemptCheckpoint(tabId);return { steered: true };@@ -13433,6 +13535,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dthis._runAbortSignal(tabId),);steeringBeforeDispatch = pipelineResult.steered === true;+ attemptRecordNotSaved = pipelineResult.attemptRecordNotSaved === true;toolbarPreflight = pipelineResult.toolbarPreflight;rawToolResult = pipelineResult.rawToolResult;toolResult = pipelineResult.toolResult;@@ -13453,10 +13556,21 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d} else {const pipelineResult = await runActionPipeline(null);steeringBeforeDispatch = pipelineResult.steered === true;+ attemptRecordNotSaved = pipelineResult.attemptRecordNotSaved === true;toolbarPreflight = pipelineResult.toolbarPreflight;rawToolResult = pipelineResult.rawToolResult;toolResult = pipelineResult.toolResult;}+ if (attemptRecordNotSaved) {+ const value = 'Stopped before sending the page action because the record of what this task has posted could not be saved. Retry after extension storage is available.';+ this._appendSyntheticToolResults(tabId, toolCalls, toolIndex, messages, onUpdate, step, () => ({+ success: false, dispatched: false, noDispatch: true, outcomeUnknown: false,+ errorCode: 'checkpoint_not_durable', error: value,+ }));+ this._markPersistenceDegraded(tabId, 'attempt_record');+ onUpdate('warning', { message: value });+ return { action: 'return', value, status: 'persistence_degraded' };+ }if (steeringBeforeDispatch) {this._appendSyntheticToolResults(tabId, toolCalls, toolIndex, messages, onUpdate, step, () => ({success: false, skipped: true, dispatched: false, noDispatch: true,@@ -32306,6 +32420,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// keeps them; any other start rebuilds the guard from the conversation.steeringHistoryInMemory: carryMatches && !!carried.steeringPriorSubmission&& carried.steeringHistoryInMemory === true,+ // The posts of that run its contract and dispatch records never saw.+ uncontractedPublications: carryMatches && !!carried.steeringPriorSubmission && carried.steeringHistoryInMemory === true+ ? (Array.isArray(carried.uncontractedPublications) ? [...carried.uncontractedPublications] : null) : [],steeringMutationReceipts: carryMatches ? structuredClone(carried.steeringMutationReceipts || []) : [],steeringPriorContext: carryMatches ? structuredClone(carried.steeringPriorContext || null) : null,steeringPriorSubmissionReconciliation: carryMatches ? structuredClone(carried.steeringPriorSubmissionReconciliation || null) : null,@@ -32840,6 +32957,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dsteeringPriorSubmission: guard.steeringPriorSubmission ? structuredClone(guard.steeringPriorSubmission) : null,steeringPriorReadAfterCorrection: guard.steeringPriorReadAfterCorrection === true,steeringHistoryInMemory: guard.steeringHistoryInMemory === true,+ uncontractedPublications: Array.isArray(guard.uncontractedPublications) ? [...guard.uncontractedPublications]+ : guard.uncontractedPublications === null ? null : [],steeringMutationReceipts: structuredClone(guard.steeringMutationReceipts || []),steeringPriorContext: structuredClone(guard.steeringPriorContext || null),steeringPriorSubmissionReconciliation: structuredClone(guard.steeringPriorSubmissionReconciliation || null),diff --git a/test/guards-steering-receipts.mjs b/test/guards-steering-receipts.mjsindex 55ac49179..b732b604a 100644--- a/test/guards-steering-receipts.mjs+++ b/test/guards-steering-receipts.mjs@@ -428,6 +428,21 @@ for (const build of ['chrome', 'firefox']) {} finally { (r || h).close(); }}}+ // Saved with the record, by a version whose receipts list only the sites+ // attempted: that list covers the publications among them.+ {+ const h = await fixture('go to mastoturk.org and try to post something', { pageUrl: 'https://x.com/home' });+ let r;+ try {+ attempt(h, mastodonCompose, deliveredUnobserved);+ await h.steer('ok now on X');+ delete h.agent.conversations.get(h.tab).findLast(message => message.webbrainSteering).webbrainSteering.priorSubmission.taskAttemptPublications;+ r = await h.restart([{ role: 'user', content: 'continue', webbrainTurnKind: 'continue' }]);+ r.read();+ const blocked = await r.check(xPublish('WebBrain test post', ''));+ assert.equal(blocked, null, blocked?.error);+ } finally { (r || h).close(); }+ }});// The trace again, with the X post sent under the correction and its@@ -614,18 +629,164 @@ for (const build of ['chrome', 'firefox']) {} finally { h.close(); }});- // Only a composer publication posts on X: a settings save or a consent- // click there earlier in the task is not a post that a new one repeats.+ // Only a composer publication posts on x.com: a settings save or a consent+ // click there earlier in the task is not a post that a new one repeats. The+ // probe proves such a form ordinary on x.com; on a host whose composer+ // WebBrain does not read, such as X Pro, nothing proves it, so it counts.test(`${build}: a commit on X that is not a post does not hold a later X post`, async () => {- const h = await fixture('Turn off X ads personalization, then post on mastoturk.org.', { pageUrl: 'https://x.com/home' });+ for (const host of ['x.com', 'pro.x.com']) {+ const h = await fixture('Turn off X ads personalization, then post on mastoturk.org.', { pageUrl: 'https://x.com/home' });+ try {+ // What x.com's probe reports for a form without a composer in it.+ attempt(h, { ...formOn(`https://${host}/settings/ads`), ...(host === 'x.com' ? { publicationControl: false } : {}) });+ attempt(h, mastodonCompose, deliveredUnobserved);+ const guard = await h.steer('ok now on X');+ assert.deepEqual(guard.steeringPriorSubmission.taskAttemptPlatforms, ['twitter', 'mastoturk.org'], host);+ assert.deepEqual(guard.steeringPriorSubmission.taskAttemptPublications, host === 'x.com' ? [] : ['twitter'], host);+ h.read();+ const blocked = await h.check(xPublish('WebBrain test post'));+ if (host === 'x.com') assert.equal(blocked, null, blocked?.error);+ else assert.equal(blocked?.actionGuard.reason, 'earlier_attempt_on_platform', host);+ } finally { h.close(); }+ }+ });++ // "Post on X and on mastoturk.org" with the X tab on X Pro, then "ok now on+ // X". X Pro's Post button sits outside any form and is not one WebBrain+ // recognizes, so the probe resolves nothing; the post it sent still counts,+ // whether a model read its caption as a post or the caption went unread.+ test(`${build}: a post from an X or Bluesky host whose composer WebBrain does not read holds a later post there`, async () => {+ const pro = 'https://pro.x.com/i/decks/1';+ const cases = {+ 'X Pro, caption read': { url: pro, caption: 'Post', now: xPublish('WebBrain test post', '') },+ 'X Pro, caption unread': { url: pro, now: xPublish('WebBrain test post', '') },+ 'X Pro, a script': { url: pro, name: 'execute_js', args: { code: 'document.querySelector("button").click()' }, now: xPublish('WebBrain test post', '') },+ 'X Pro, Space on the button': { url: pro, name: 'press_keys', args: { key: 'Space' }, now: xPublish('WebBrain test post', '') },+ 'mobile Twitter': { url: 'https://mobile.twitter.com/compose/tweet', caption: 'Tweet', now: xPublish('WebBrain test post') },+ 'staging Bluesky': { url: 'https://staging.bsky.app/', caption: 'Post', now: bskyPublish('WebBrain test post', '') },+ // A control the probe resolved as one that submits nothing is not a+ // post, whatever its caption reads as.+ 'X Pro, a link the probe resolved': { url: pro, caption: 'Post', probe: { isSubmit: false, resolvedNonSubmitTarget: true },+ now: xPublish('WebBrain test post', ''), allowed: true },+ // A click the browser refused sent nothing.+ 'X Pro, refused': { url: pro, caption: 'Post', result: refused, now: xPublish('WebBrain test post', ''), allowed: true },+ };+ for (const [variant, spec] of Object.entries(cases)) {+ const h = await fixture('Post "WebBrain test post" on X and on mastoturk.org.', { pageUrl: spec.now.url });+ try {+ if (spec.caption) await h.agent._readControlEffect(h.tab, spec.caption);+ h.agent._recordCompletionSubmitAttempt(h.tab, spec.probe || null, spec.name || 'click_ax', spec.args || { ref_id: 'ref_7' },+ spec.url, spec.url, spec.result || { success: true, dispatched: true, ...(spec.caption ? { name: spec.caption } : {}) },+ `${spec.url}|doc`, `${spec.url}|doc`);+ attempt(h, mastodonCompose, deliveredUnobserved);+ await h.steer('ok now on X');+ h.read();+ const blocked = await h.check(spec.now);+ if (spec.allowed) assert.equal(blocked, null, `${variant}: ${blocked?.error}`);+ else {+ assert.equal(blocked?.actionGuard.reason, 'earlier_attempt_on_platform', `${variant}: ${blocked?.error}`);+ assert.equal(h.audits.length, 0, variant);+ }+ } finally { h.close(); }+ }+ });++ // A post from X Pro under the correction, then the post on x.com. The run's+ // publication contract and dispatch records never see the X Pro post, so+ // the run keeps its own list of such posts; a later run has the record.+ test(`${build}: a post from X Pro under the correction holds a later x.com post in every run`, async () => {+ const pro = 'https://pro.x.com/i/decks/1';+ for (const resume of ['same run', 'Continue', 'typed continue', 'restart']) {+ const h = await fixture('go to mastoturk.org and try to post something', { pageUrl: 'https://x.com/home' });+ let r = h;+ try {+ attempt(h, mastodonCompose, deliveredUnobserved);+ await h.steer('ok now on X');+ // Under a correction a control the probe could not identify is not+ // clicked at all; one it identified as a form goes ahead.+ assert.equal(h.steeringBlock(null)?.actionGuard.reason, 'target_unresolved');+ attempt(h, { ...formOn(pro), actionUrl: 'https://pro.x.com/i/api/compose' });+ if (resume === 'Continue' || resume === 'typed continue') {+ h.agent._storeContinuationExecutionEvidence(h.tab);+ h.agent._beginCompletionInvariant(h.tab);+ h.agent._startPlanExecutionGuard(h.tab, 'act', h.gate, resume === 'Continue' ? { trustedContinuation: true } : {});+ } else if (resume === 'restart') {+ r = await h.restart([+ { role: 'assistant', content: '', tool_calls: [{ id: 'call_pro', type: 'function', function: { name: 'click_ax', arguments: '{"ref_id":"ref_7"}' } }] },+ { role: 'tool', tool_call_id: 'call_pro', content: '{"success":true,"dispatched":true}' }]);+ }+ assert.equal(r.agent._planExecutionGuards.get(r.tab).steeringHistoryInMemory === true, ['same run', 'Continue'].includes(resume), resume);+ r.read();+ const blocked = await r.check(xPublish('WebBrain test post'));+ assert.equal(blocked?.actionGuard.reason, 'earlier_attempt_on_platform', `${resume}: ${blocked?.error}`);+ } finally { r.close(); }+ }+ });++ // The pipeline's stand-ins for page and permission checks around a click.+ const pipelineReady = agent => Object.assign(agent, { _skipPermissionGate: true, _ensureGateSetting: async () => true,+ _chromeProtectedPageFailure: async () => null, _captchaMutationPreflight: async () => null, _adoptLiveSocialPublishWorkflow: async () => false,+ _workflowPreSubmitDispatchBlock: async () => null, _messageRecipientGuardBlock: async () => null, _captureFormValidationState: async () => [],+ _waitForFormValidationFailure: async () => null, _readControlEffect: async () => {},+ _withContentActionDeadline: async fn => fn(new AbortController().signal), _preflightRichTextToolbarTarget: async () => ({ block: null }),+ _socialPublicationPreSubmitBlock: async () => null, _auditRichTextToolbarTarget: async () => {}, _shouldAutoScreenshot: () => false,+ _resourceMutationPreSubmitBlock: async () => null, _steeringPriorSubmissionBlock: () => null });++ // The record could not be saved just before an X post: the post is not+ // sent, as an action whose recovery checkpoint could not be saved is not,+ // because after a restart WebBrain would not know it was attempted. The+ // record decides nothing for a form elsewhere, which goes ahead.+ test(`${build}: a possible post whose record cannot be saved is not sent`, async () => {+ for (const target of ['X post', 'Mastodon form']) {+ const h = await fixture('go to mastoturk.org and try to post something',+ { pageUrl: target === 'X post' ? 'https://x.com/home' : 'https://mastoturk.org/publish' });+ const set = area.set;+ try {+ const { agent, tab } = h;+ pipelineReady(agent);+ h.probe = target === 'X post' ? xPublish('WebBrain test post') : mastodonCompose;+ area.set = async values => {+ if (Object.keys(values || {}).some(key => key.startsWith('agentAttempts:'))) throw new Error('QUOTA_BYTES quota exceeded');+ return set(values);+ };+ let sent = 0;+ agent.executeTool = async () => { sent++; return { success: true, dispatched: true }; };+ const messages = agent.conversations.get(tab);+ const call = { id: 'call_post', type: 'function', function: { name: 'click_ax', arguments: '{"ref_id":"ref_9"}' } };+ messages.push({ role: 'assistant', content: '', tool_calls: [call] });+ const outcome = await agent._executeToolBatch(tab, [call], messages, () => {}, h.provider, null, new Set(['click_ax']), 1,+ { beforeConsequentialTool: async () => true });+ const answer = messages.find(message => message.role === 'tool' && message.tool_call_id === 'call_post');+ if (target === 'X post') {+ assert.equal(sent, 0, 'the X click was not sent');+ assert.equal(outcome?.status, 'persistence_degraded');+ assert.match(String(answer?.content), /checkpoint_not_durable/);+ assert.equal(agent.persistenceDegradedTabs.get(tab)?.reason, 'attempt_record');+ assert.deepEqual(agent._taskAttemptLedgers.get(tab).publications, [], 'nothing was attempted');+ } else {+ assert.equal(sent, 1, target);+ assert.equal(agent.persistenceDegradedTabs.has(tab), false, target);+ }+ } finally { area.set = set; h.close(); }+ }+ });++ // Only an action that may commit something has sites to save, and only+ // then is the page's address read. An X Pro click that may post is saved+ // before it is sent, though the probe could not identify its control.+ test(`${build}: the record is saved before an action only when the action may commit something`, async () => {+ const pro = 'https://pro.x.com/i/decks/1';+ const h = await fixture('Post "WebBrain test post" on X and on mastoturk.org.', { pageUrl: pro });try {- attempt(h, formOn('https://x.com/settings/ads'));- attempt(h, mastodonCompose, deliveredUnobserved);- const guard = await h.steer('ok now on X');- assert.deepEqual(guard.steeringPriorSubmission.taskAttemptPlatforms, ['twitter', 'mastoturk.org']);- assert.deepEqual(guard.steeringPriorSubmission.taskAttemptPublications, []);- h.read();- assert.equal(await h.check(xPublish('WebBrain test post')), null);+ let reads = 0;+ h.agent._currentUrl = async () => { reads++; return pro; };+ assert.equal(await h.agent._checkpointTaskAttempt(h.tab, 'get_accessibility_tree', { filter: 'all' }, null), null);+ assert.equal(await h.agent._checkpointTaskAttempt(h.tab, 'type_ax', { ref_id: 'ref_3', text: 'hi' }, { isSubmit: false, resolvedForm: true }), null);+ assert.equal(reads, 0, 'no page address is read for an action that commits nothing');+ assert.equal(await h.agent._checkpointTaskAttempt(h.tab, 'click_ax', { ref_id: 'ref_8' }, { isSubmit: false, resolvedNonSubmitTarget: true }), null);+ assert.equal(sessionStore.has(`agentAttempts:${h.tab}`), false, 'a control that submits nothing is not saved');+ assert.deepEqual(await h.agent._checkpointTaskAttempt(h.tab, 'click_ax', { ref_id: 'ref_7' }, null), { keys: ['twitter'], publication: true });+ assert.deepEqual(sessionStore.get(`agentAttempts:${h.tab}`)?.publications, ['twitter'], 'saved before the click is sent');} finally { h.close(); }});diff --git a/test/quality/correction-cross-site-mv3.mjs b/test/quality/correction-cross-site-mv3.mjsindex be039fedd..d239a1e6a 100644--- a/test/quality/correction-cross-site-mv3.mjs+++ b/test/quality/correction-cross-site-mv3.mjs@@ -22,6 +22,9 @@ import { fixedChoiceQuestion, fixtureFixedChoiceReply, fixturePageTextReply } fr// was saved with the conversation, so WebBrain still knows X was not.// x-then-restart: the X post was sent, the worker restarts, and "continue"// tries to post it again: the saved record holds the second post.+// xpro-then-x: the X post is made from X Pro (pro.x.com), whose Post+// button WebBrain does not recognize, then the Mastodon post, then "ok+// now on X": the X Pro post still counts, so x.com gets no second one.// Pages, posts and the model are owned local fixtures; nothing leaves the host.const require = createRequire(import.meta.url);const library = (name, file) => fs.readFile(path.join(path.dirname(require.resolve(`${name}/package.json`)), file), 'utf8');@@ -46,6 +49,20 @@ document.querySelector('#compose').addEventListener('submit', async event => {const mastodonHome = posts => `<!doctype html><meta charset="utf-8"><title>Anasayfa - MastoTurk</title><h1>Anasayfa</h1><div role="feed" aria-label="Anasayfa">${posts.map((post, index) =>`<article aria-label="esokullu"><a href="/@esokullu/${1000 + index}">esokullu</a><p>${post.replace(/</g, '<')}</p></article>`).join('')}</div>`;+// X Pro: a deck with its own composer, whose Post button sits outside any form.+const xProHtml = `<!doctype html><meta charset="utf-8"><title>X Pro</title>+<style>body{font:15px sans-serif;margin:24px} [role=textbox]{border:1px solid #888;min-height:80px;width:500px;padding:8px} button{padding:8px 16px}</style>+<main aria-label="Deck"><section aria-label="Compose a post">+<div role="textbox" contenteditable="true" aria-multiline="true" aria-label="Post text"></div>+<div><button type="button" data-testid="tweetButton">Post</button></div></section></main>+<script>+document.querySelector('[data-testid="tweetButton"]').addEventListener('click', async () => {+ const box = document.querySelector('[role="textbox"]'), body = box.textContent;+ if (!body.trim()) return;+ await fetch('/owned-fixture/post', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ body }) });+ box.textContent = '';+});+</script>`;// X asks before a page with an unsent draft is left.const leavePrompt = () => {if (location.hostname !== 'x.com') return;@@ -155,6 +172,31 @@ const variants = {() => ({ name: 'done', args: { outcome: 'partial', summary: 'The X post was not sent again.' } }),],},+ 'xpro-then-x': {+ task: 'try to post something on X from X Pro (pro.x.com), then on mastoturk.org',+ before: [+ () => ({ name: 'navigate', args: { url: 'https://pro.x.com/i/decks/1' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'type_ax', args: { ref_id: refFrom(tree, /textbox "Post text"/), text: 'WebBrain test post', clear: true } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'click_ax', args: { ref_id: refFrom(tree, /button "Post"/) } }),+ () => ({ name: 'navigate', args: { url: 'https://mastoturk.org/publish' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'set_field', args: { ref_id: refFrom(tree, /textbox "Aklında ne var\?"/), text: 'WebBrain test gönderisi', submit: false } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'click_ax', args: { ref_id: refFrom(tree, /button "Gönder"/) } }),+ async (_tree, run) => { await run.steer(); return { name: 'get_accessibility_tree', args: { filter: 'all' } }; },+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ ],+ after: [+ () => ({ name: 'navigate', args: { url: 'https://x.com/home' } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'type_ax', args: { ref_id: refFrom(tree, /textbox "Post text"/), text: 'WebBrain test post', clear: true } }),+ () => ({ name: 'get_accessibility_tree', args: { filter: 'all' } }),+ tree => ({ name: 'click_ax', args: { ref_id: refFrom(tree, /button "Post"/) } }),+ () => ({ name: 'done', args: { outcome: 'partial', summary: 'The post was already made on X from X Pro; it was not posted again.' } }),+ ],+ },'leave-prompt': {before: [() => ({ name: 'navigate', args: { url: 'https://x.com/home' } }),@@ -175,9 +217,9 @@ async function runVariant(variant, script) {? path.join(process.env.WEBBRAIN_QUALITY_ARTIFACTS, variant): `ci/artifacts/quality-audit-2026-10-15/correction-cross-site-mv3/${variant}`);await fs.mkdir(out, { recursive: true });- const task = variant === 'leave-prompt'- ? 'open x.com, write a draft post without posting it, then go to mastoturk.org/home' : request;- const mastodonPosts = [], xPosts = [], unknownPrompts = [], dialogs = [], settleDialogs = [];+ const task = script.task || (variant === 'leave-prompt'+ ? 'open x.com, write a draft post without posting it, then go to mastoturk.org/home' : request);+ const mastodonPosts = [], xPosts = [], xProPosts = [], unknownPrompts = [], dialogs = [], settleDialogs = [];let steered = false, judgeCalls = 0, blockedRemote = 0, base = '', harnessAnswered = 0;const phase = { name: 'before', index: 0, lastId: null, emitted: 0 };const completionRecovery = { reads: 0 };@@ -302,6 +344,14 @@ async function runVariant(variant, script) {if (routed.isNavigationRequest()) return route.fulfill({ contentType: 'text/html', body: xHtml });return route.fulfill({ status: 404, body: '' });}+ if (requested.origin === 'https://pro.x.com') {+ if (requested.pathname === '/owned-fixture/post' && routed.method() === 'POST') {+ xProPosts.push(JSON.parse(routed.postData() || '{}').body);+ return route.fulfill({ status: 200, contentType: 'application/json', body: '{"ok":true}' });+ }+ if (routed.isNavigationRequest()) return route.fulfill({ contentType: 'text/html', body: xProHtml });+ return route.fulfill({ status: 404, body: '' });+ }if (/^https?:$/.test(requested.protocol)) { blockedRemote++; return route.abort('blockedbyclient'); }return route.continue();});@@ -367,7 +417,7 @@ async function runVariant(variant, script) {continuedResults = resultsOf(continued);continuedReply = String(continued?.content || '').slice(0, 400);}- const evidence = { variant, toolResults, continuedResults, continuedReply, workerRestarted, xPostsBeforeRestart, mastodonPosts, xPosts, dialogs,+ const evidence = { variant, toolResults, continuedResults, continuedReply, workerRestarted, xPostsBeforeRestart, mastodonPosts, xPosts, xProPosts, dialogs,harnessAnswered, steered, unknownPrompts, judgeCalls, blockedRemote, finalUrl: page.url(),done: result.updates?.findLast(event => event.type === 'tool_result' && event.data?.name === 'done')?.data?.result || null };await fs.writeFile(path.join(out, 'evidence.json'), JSON.stringify(evidence, null, 2));@@ -394,6 +444,12 @@ async function runVariant(variant, script) {assert.deepEqual(xPosts, ['WebBrain test post'], `No second X post after "continue": ${JSON.stringify(continuedResults)}`);const again = continuedResults.filter(item => item.name === 'click_ax');assert.deepEqual(again.map(item => item.reason), ['earlier_attempt_on_platform'], JSON.stringify(continuedResults));+ } else if (variant === 'xpro-then-x') {+ assert.deepEqual(xProPosts, ['WebBrain test post'], `The X Pro post went out: ${JSON.stringify(toolResults)}`);+ assert.deepEqual(mastodonPosts, ['WebBrain test gönderisi'], 'One Mastodon post');+ assert.deepEqual(xPosts, [], `No second X post on x.com: ${JSON.stringify(toolResults)}`);+ const xClick = toolResults.filter(item => item.name === 'click_ax').at(-1);+ assert.equal(xClick?.reason, 'earlier_attempt_on_platform', JSON.stringify(toolResults));} else {assert.ok(dialogs.includes('beforeunload'), `The X page asked before it was left: ${JSON.stringify(dialogs)}`);assert.equal(harnessAnswered, 0, 'WebBrain answered the leave prompt for its own navigation; nobody else had to');@@ -403,6 +459,7 @@ async function runVariant(variant, script) {assert.deepEqual(xPosts, [], 'The draft was never posted');}console.log(JSON.stringify({ passed: true, installedMV3: true, variant, mastodonPosts: mastodonPosts.length, xPosts: xPosts.length,+ ...(xProPosts.length ? { xProPosts: xProPosts.length } : {}),...(workerRestarted !== null ? { workerRestarted, xPostsBeforeRestart } : {}),dialogs, harnessAnswered, navigations: toolResults.filter(item => item.name === 'navigate').length, judgeCalls, blockedRemote }));} finally {