Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Lean mode: a site's own message job keeps its delivery receipt
Barack Sokullude11729b39a2
diff --git a/docs/lean-mode.md b/docs/lean-mode.mdindex ac2ea9138..a3592a3c0 100644--- a/docs/lean-mode.md+++ b/docs/lean-mode.md@@ -56,7 +56,7 @@ Every guard of class `quality` in the guard registry (`src/*/src/agent/guard-polThe page reads and model calls that only those guards use are skipped as well:-- **At `done`:** the verification screenshot and page probes, the completion judge, the page-warning and workflow-evidence checks, the done-summary reading of a failed or partial result, and the forced read-only and done-only turns. `done` is the model's own call.+- **At `done`:** the verification screenshot and page probes, the completion judge, the page-warning and workflow-evidence checks (except a message job's delivery, below), the done-summary reading of a failed or partial result, and the forced read-only and done-only turns. `done` is the model's own call.- **Typing:** the readback of text writes. A write the content script cannot read back exactly is reported as `success: true, verified: false, unverifiedWrite: true`, with no note, no lock and no text-write debt, so the next write to the field goes through.- **Around submits:** the form-validation snapshots before and after a submit and their caption readings, and the GitHub commit gates (the editor-proof binding and the split-write rule).- **Progress and reading:** the progress ledger (its classification and its checks), the read-coverage scope reading, the arithmetic-task reading, and the readings of upload lines, authored field labels and review-thread controls for workflow evidence.@@ -81,7 +81,7 @@ Some steps that look like quality work feed the hard checks, so they stay too:- the social publication contract;- the rich-text toolbar probe, which pins the typing target (only its classification is skipped);- automatic screenshots, as set in Settings;-- a chat message sent in the run: its delivery must be confirmed in the conversation and its receipt saved before `done` reports success, or a resumed run could send it again. A receipt that cannot be saved refuses success every time; a send the conversation cannot confirm refuses it once per send, and `partial` or `failed` always ends the run. A plain final answer gets the same check; after two refusals it ends the run as partial instead of looping.+- a chat message sent in the run: its delivery must be confirmed in the conversation and its receipt saved before `done` reports success, or a resumed run could send it again. A receipt that cannot be saved refuses success every time; a send the conversation cannot confirm refuses it once per send, and `partial` or `failed` always ends the run. A plain final answer gets the same check; after two refusals it ends the run as partial instead of looping. A site's own message job (a direct message on X, LinkedIn or Douyin, an email in Gmail) is checked the way a full run checks it, from the conversation and the page's sent status, and its receipt is the job's evidence, which a continued run checks before it would send again.These quality guards keep running, each because a hard check depends on it (`LEAN_KEPT_QUALITY_GUARDS` in `lean-mode.js`):diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 7efe602b6..913347319 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -1499,8 +1499,11 @@ export class Agent extends LoopDetector {// receipt that cannot be saved refuses every time, as in a full run. A send// the conversation cannot confirm (its read failed, the conversation// changed, a new message arrived) refuses once per send, as a full run's- // submission-evidence check does.+ // submission-evidence check does. A site's own message job has its own+ // receipt (_leanWorkflowMessageReceiptBlock).async _leanChatReceiptBlock(tabId) {+ const workflowSend = this._leanWorkflowMessageSend(tabId);+ if (workflowSend) return this._leanWorkflowMessageReceiptBlock(tabId, workflowSend);if (!this.completionInvariants.get(tabId)?.lastAction?.chatDeliveryBinding&& !this.chatSessions.get(tabId)?.completionReceipt) return null;let pageUrl = '';@@ -1521,9 +1524,9 @@ export class Agent extends LoopDetector {// done in a lean run. The completion is the model's own call: WebBrain// takes no verification screenshot, runs no completion or incomplete-task- // judge and checks no page warnings or workflow evidence. Only the chat- // receipt check stays (_leanChatReceiptBlock); partial and failed always- // end.+ // judge and checks no page warnings or other workflow evidence. Only the+ // message receipt checks stay (_leanChatReceiptBlock); partial and failed+ // always end.async _leanDoneResult(tabId, args = {}) {const outcome = normalizeDoneOutcome(args?.outcome);if (outcome === 'success') {@@ -1553,6 +1556,98 @@ export class Agent extends LoopDetector {return true;}+ // An adapter's own message job (send-message on X, LinkedIn or Douyin,+ // send-email in Gmail) whose send this task dispatched: its submit receipt,+ // bound to the job. Null for any other run.+ _leanWorkflowMessageSend(tabId) {+ const guard = this._planExecutionGuards.get(tabId);+ if (!guard?.enabled || !guard.siteWorkflow?.job+ || this._workflowVerificationKind(guard.siteWorkflow) !== 'message_sent') return null;+ const submit = this._currentCompletionSubmitState(tabId);+ const binding = submit?.workflowBinding;+ if (submit?.dispatched !== true || !binding+ || binding.bindingKey !== this._adapterWorkflowBindingKey(guard.siteWorkflow)) return null;+ return { guard, submit, binding };+ }++ // The receipt check of a lean run for that send. Such a job has no native+ // conversation receipt: its receipt is the job's terminal evidence, which+ // a full done records and a continued run checks before it would send+ // again. So the same verification runs here: the conversation is read+ // back against the dispatched body and recipients, and the page's live+ // messages are read in any language for the sent status. The evidence is+ // recorded and saved before the run ends as a success. As for a chat+ // send, evidence that cannot be saved refuses every time, and a send that+ // cannot be confirmed refuses once per send.+ async _leanWorkflowMessageReceiptBlock(tabId, { guard, submit, binding }) {+ const save = async () => {+ const saved = await this._persistNow(tabId);+ if (saved === true || saved?.ok === true) return null;+ return { receiptNotDurable: true,+ error: 'The verified message delivery receipt could not be saved. Retry completion after storage is available; do not send the message again.' };+ };+ if (this._workflowTerminalEvidenceMatchesState(guard, guard.workflowTerminalEvidence)) return save();+ let pageUrl = '';+ try { pageUrl = await this._currentUrl(tabId); } catch {}+ const adapterName = guard.siteWorkflow.adapterName;+ let probe = null;+ try {+ probe = await this._messageRecipientGuardProbe(tabId, {+ tool: 'observe_active_conversation',+ args: {},+ adapterName,+ expectedMessageBody: binding.messageBody || '',+ expectedRecipients: binding.recipientTargets || [],+ // Google Chat routes in Gmail have no recipient set of their own.+ supportsRecipientSets: adapterName === 'gmail' && !isGmailChatRoute(pageUrl),+ });+ } catch (error) {+ if (this._checkAbort(tabId)) throw error;+ }+ const pageState = { liveRegionMessages: await this._liveRegionMessages(tabId) };+ const submissionEvidence = { submit };+ await this._readWorkflowTerminalTexts(tabId, pageState, submissionEvidence, probe);+ const current = this._planExecutionGuards.get(tabId) === guard+ && !this._checkAbort(tabId) && !this._hasUnvalidatedSteering(tabId);+ const evidence = current && probe+ ? this._workflowTerminalEvidenceFromDone(tabId, pageState, pageUrl, submissionEvidence, probe) : null;+ if (evidence) {+ guard.workflowTerminalEvidence = evidence;+ guard.verifiedSubmissionEvidence = true;+ return save();+ }+ const key = `workflow:${binding.bindingKey}:${submit.runToken || ''}:${submit.actionSequence ?? ''}`;+ const refused = (this._guardRunState(tabId).leanUnconfirmedChatSends ||= new Set());+ if (refused.has(key)) return null;+ refused.add(key);+ return { chatDeliveryUnconfirmed: true,+ error: 'WebBrain could not confirm on the page that your message was delivered: the conversation does not show it sent to the intended recipient with its exact text. Do not send the message again. Read the conversation now, then call done again; if the delivery still cannot be confirmed, finish with outcome "partial" and say so.' };+ }++ // The page's visible live messages (status, alert and aria-live regions),+ // collected as the done probe collects them and returned as they are: a+ // model reads what they say before anything relies on one.+ async _liveRegionMessages(tabId) {+ try {+ const probe = await cdpClient.evaluate(tabId, `(() => {+ const visible = el => {+ try {+ const r = el.getBoundingClientRect();+ if (r.width < 1 || r.height < 1) return false;+ const s = window.getComputedStyle(el);+ return !(s.visibility === 'hidden' || s.display === 'none' || parseFloat(s.opacity) === 0);+ } catch (e) { return false; }+ };+ return Array.from(document.querySelectorAll('[role=status],[role=alert],[aria-live]'))+ .filter(visible).map(e => (e.innerText || '').trim().slice(0, 120)).filter(Boolean).slice(0, 6);+ })()`);+ const value = probe?.result?.value;+ return Array.isArray(value) ? value.filter(text => typeof text === 'string' && text) : [];+ } catch {+ return [];+ }+ }+_readCompletenessNeedsScopeClassification(tabId) {// Lean mode enforces no read coverage, so it needs no scope reading.if (this._leanRunActive(tabId)) return false;diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 69c91d31d..e19c772d6 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -1131,8 +1131,11 @@ export class Agent extends LoopDetector {// receipt that cannot be saved refuses every time, as in a full run. A send// the conversation cannot confirm (its read failed, the conversation// changed, a new message arrived) refuses once per send, as a full run's- // submission-evidence check does.+ // submission-evidence check does. A site's own message job has its own+ // receipt (_leanWorkflowMessageReceiptBlock).async _leanChatReceiptBlock(tabId) {+ const workflowSend = this._leanWorkflowMessageSend(tabId);+ if (workflowSend) return this._leanWorkflowMessageReceiptBlock(tabId, workflowSend);if (!this.completionInvariants.get(tabId)?.lastAction?.chatDeliveryBinding&& !this.chatSessions.get(tabId)?.completionReceipt) return null;let pageUrl = '';@@ -1153,9 +1156,9 @@ export class Agent extends LoopDetector {// done in a lean run. The completion is the model's own call: WebBrain// takes no verification screenshot, runs no completion or incomplete-task- // judge and checks no page warnings or workflow evidence. Only the chat- // receipt check stays (_leanChatReceiptBlock); partial and failed always- // end.+ // judge and checks no page warnings or other workflow evidence. Only the+ // message receipt checks stay (_leanChatReceiptBlock); partial and failed+ // always end.async _leanDoneResult(tabId, args = {}) {const outcome = normalizeDoneOutcome(args?.outcome);if (outcome === 'success') {@@ -1185,6 +1188,98 @@ export class Agent extends LoopDetector {return true;}+ // An adapter's own message job (send-message on X, LinkedIn or Douyin,+ // send-email in Gmail) whose send this task dispatched: its submit receipt,+ // bound to the job. Null for any other run.+ _leanWorkflowMessageSend(tabId) {+ const guard = this._planExecutionGuards.get(tabId);+ if (!guard?.enabled || !guard.siteWorkflow?.job+ || this._workflowVerificationKind(guard.siteWorkflow) !== 'message_sent') return null;+ const submit = this._currentCompletionSubmitState(tabId);+ const binding = submit?.workflowBinding;+ if (submit?.dispatched !== true || !binding+ || binding.bindingKey !== this._adapterWorkflowBindingKey(guard.siteWorkflow)) return null;+ return { guard, submit, binding };+ }++ // The receipt check of a lean run for that send. Such a job has no native+ // conversation receipt: its receipt is the job's terminal evidence, which+ // a full done records and a continued run checks before it would send+ // again. So the same verification runs here: the conversation is read+ // back against the dispatched body and recipients, and the page's live+ // messages are read in any language for the sent status. The evidence is+ // recorded and saved before the run ends as a success. As for a chat+ // send, evidence that cannot be saved refuses every time, and a send that+ // cannot be confirmed refuses once per send.+ async _leanWorkflowMessageReceiptBlock(tabId, { guard, submit, binding }) {+ const save = async () => {+ const saved = await this._persistNow(tabId);+ if (saved === true || saved?.ok === true) return null;+ return { receiptNotDurable: true,+ error: 'The verified message delivery receipt could not be saved. Retry completion after storage is available; do not send the message again.' };+ };+ if (this._workflowTerminalEvidenceMatchesState(guard, guard.workflowTerminalEvidence)) return save();+ let pageUrl = '';+ try { pageUrl = await this._currentUrl(tabId); } catch {}+ const adapterName = guard.siteWorkflow.adapterName;+ let probe = null;+ try {+ probe = await this._messageRecipientGuardProbe(tabId, {+ tool: 'observe_active_conversation',+ args: {},+ adapterName,+ expectedMessageBody: binding.messageBody || '',+ expectedRecipients: binding.recipientTargets || [],+ // Google Chat routes in Gmail have no recipient set of their own.+ supportsRecipientSets: adapterName === 'gmail' && !isGmailChatRoute(pageUrl),+ });+ } catch (error) {+ if (this._checkAbort(tabId)) throw error;+ }+ const pageState = { liveRegionMessages: await this._liveRegionMessages(tabId) };+ const submissionEvidence = { submit };+ await this._readWorkflowTerminalTexts(tabId, pageState, submissionEvidence, probe);+ const current = this._planExecutionGuards.get(tabId) === guard+ && !this._checkAbort(tabId) && !this._hasUnvalidatedSteering(tabId);+ const evidence = current && probe+ ? this._workflowTerminalEvidenceFromDone(tabId, pageState, pageUrl, submissionEvidence, probe) : null;+ if (evidence) {+ guard.workflowTerminalEvidence = evidence;+ guard.verifiedSubmissionEvidence = true;+ return save();+ }+ const key = `workflow:${binding.bindingKey}:${submit.runToken || ''}:${submit.actionSequence ?? ''}`;+ const refused = (this._guardRunState(tabId).leanUnconfirmedChatSends ||= new Set());+ if (refused.has(key)) return null;+ refused.add(key);+ return { chatDeliveryUnconfirmed: true,+ error: 'WebBrain could not confirm on the page that your message was delivered: the conversation does not show it sent to the intended recipient with its exact text. Do not send the message again. Read the conversation now, then call done again; if the delivery still cannot be confirmed, finish with outcome "partial" and say so.' };+ }++ // The page's visible live messages (status, alert and aria-live regions),+ // collected as the done probe collects them and returned as they are: a+ // model reads what they say before anything relies on one.+ async _liveRegionMessages(tabId) {+ try {+ const results = await browser.tabs.executeScript(tabId, { code: `(() => {+ const visible = el => {+ try {+ const r = el.getBoundingClientRect();+ if (r.width < 1 || r.height < 1) return false;+ const s = window.getComputedStyle(el);+ return !(s.visibility === 'hidden' || s.display === 'none' || parseFloat(s.opacity) === 0);+ } catch (e) { return false; }+ };+ return Array.from(document.querySelectorAll('[role=status],[role=alert],[aria-live]'))+ .filter(visible).map(e => (e.innerText || '').trim().slice(0, 120)).filter(Boolean).slice(0, 6);+ })()` });+ const value = results && results[0];+ return Array.isArray(value) ? value.filter(text => typeof text === 'string' && text) : [];+ } catch {+ return [];+ }+ }+_readCompletenessNeedsScopeClassification(tabId) {// Lean mode enforces no read coverage, so it needs no scope reading.if (this._leanRunActive(tabId)) return false;diff --git a/test/lean-mode.mjs b/test/lean-mode.mjsindex c181b0ab7..9bd49eace 100644--- a/test/lean-mode.mjs+++ b/test/lean-mode.mjs@@ -12,6 +12,7 @@ import path from 'node:path';import { fileURLToPath } from 'node:url';import { readFile } from 'node:fs/promises';import { test } from 'node:test';+import vm from 'node:vm';import { chromium } from 'playwright';import { defaultDecide, installModelStub, stubPageTextReadings, withoutFixedChoiceAnswers } from './semantic-decision-stub.mjs';@@ -570,6 +571,106 @@ for (const build of ['chrome', 'firefox']) {assert.ok(reads >= 3, 'the conversation is read again at every success');});+ test(`${build}: a lean done after a site's own message send records its delivery as the job's evidence`, async () => {+ // An X direct message sent through the adapter's send-message job has no+ // native conversation receipt; its receipt is the job's terminal+ // evidence, which a continued run checks before it would send again.+ const xUrl = 'https://x.com/i/chat/123-456';+ const fixture = ({ delivered = true, saves = true } = {}) => {+ const agent = new Agent({ getActive: () => STRONG });+ const tab = nextTab++;+ agent.conversationModes.set(tab, 'act');+ agent._runModeOverrides.set(tab, 'act');+ leanState(agent, tab);+ agent.conversations.set(tab, [{ role: 'system', content: 'sys' }, { role: 'user', content: 'Send Alex: Hello there' }]);+ const siteWorkflow = agent._resolvePlannerSiteWorkflow(xUrl, { request_kind: 'execute', requires_submission: true,+ messaging: { target_kind: 'named', recipients: ['@altryne'] } });+ assert.equal(siteWorkflow?.job?.id, 'send-message');+ const guard = agent._startPlanExecutionGuard(tab, 'act', { requestKind: 'execute', requiresStateChange: true,+ requiresSubmission: true, messaging: { target_kind: 'named', recipients: ['@altryne'] }, siteWorkflow });+ guard.workflowMetadataRequirements = [{ field: 'body', value: 'Hello there' }];+ guard.workflowMetadataRequirementsResolved = true;+ const workflowBinding = agent._workflowSubmitBindingForAttempt(tab, xUrl, {+ messageRecipientGuardRequired: true, messageRecipientDispatchBinding: { token: 'x-send' },+ messageRecipientBody: 'Hello there', messageRecipientBodyBaselineCount: 0,+ messageRecipientExistingMessageIds: ['message-prior'] });+ assert.ok(workflowBinding, 'the send is bound to the job');+ agent._completionSubmitStates.set(tab, { dispatched: true, observedAfterSubmit: true, originatingUrl: xUrl,+ currentUrl: xUrl, taskKey: agent._progressTaskKeyHash(tab), runToken: 'run-1', actionSequence: 2, workflowBinding });+ agent._currentUrl = async () => xUrl;+ agent._liveRegionMessages = async () => [];+ const reads = [];+ agent._messageRecipientGuardProbe = async (_tab, params) => {+ reads.push(params);+ return { success: true, conclusive: true, composerEmpty: true,+ strongRecipientCandidates: [{ identity: '@altryne', role: 'to' }],+ existingMessageIds: delivered ? ['message-prior', 'message-new'] : ['message-prior'],+ matchingOutgoingMessageIds: delivered ? ['message-new'] : [],+ matchingOutgoingMessageCount: delivered ? 1 : 0 };+ };+ let persisted = 0;+ agent._persistNow = async () => { persisted++; return saves ? { ok: true } : false; };+ return { agent, tab, guard, reads, persisted: () => persisted };+ };++ // Delivered: the job's evidence is recorded and saved, and the run ends.+ const sent = fixture();+ assert.deepEqual(await sent.agent.executeTool(sent.tab, 'done', { summary: 'Sent.', outcome: 'success' }),+ { done: true, summary: 'Sent.', outcome: 'success' });+ assert.equal(sent.reads.length, 1, 'the conversation was read');+ assert.equal(sent.reads[0].expectedMessageBody, 'Hello there');+ assert.equal(sent.guard.workflowTerminalEvidence?.verificationKind, 'message_sent');+ assert.equal(sent.guard.workflowTerminalEvidence?.job, 'send-message');+ assert.equal(sent.persisted(), 1, 'the receipt was saved before the run ended');+ assert.equal(sent.agent._workflowTerminalEvidenceMatchesState(sent.guard, sent.guard.workflowTerminalEvidence), true,+ 'a continued run sees the send as done');++ // Delivered but not saved: refused every time.+ const unsaved = fixture({ saves: false });+ for (let attempt = 0; attempt < 3; attempt++) {+ const refused = await unsaved.agent.executeTool(unsaved.tab, 'done', { summary: 'Sent.', outcome: 'success' });+ assert.equal(refused.blockedDone, true, JSON.stringify(refused));+ assert.match(refused.error, /delivery receipt could not be saved/);+ }++ // Not confirmed: refused once per send, and an honest ending is never refused.+ const unconfirmed = fixture({ delivered: false });+ const refused = await unconfirmed.agent.executeTool(unconfirmed.tab, 'done', { summary: 'Sent.', outcome: 'success' });+ assert.equal(refused.blockedDone, true, JSON.stringify(refused));+ assert.equal(refused.chatDeliveryUnconfirmed, true);+ assert.match(refused.error, /Do not send the message again/);+ assert.equal(unconfirmed.guard.workflowTerminalEvidence, null);+ assert.deepEqual(await unconfirmed.agent.executeTool(unconfirmed.tab, 'done', { summary: 'Not confirmed.', outcome: 'partial' }),+ { done: true, summary: 'Not confirmed.', outcome: 'partial' });+ assert.deepEqual(await unconfirmed.agent.executeTool(unconfirmed.tab, 'done', { summary: 'Sent.', outcome: 'success' }),+ { done: true, summary: 'Sent.', outcome: 'success' });+ // The plain-final path shares the check.+ const plain = fixture({ delivered: false });+ assert.equal((await plain.agent._leanChatReceiptBlock(plain.tab))?.chatDeliveryUnconfirmed, true);+ assert.equal(await plain.agent._leanChatReceiptBlock(plain.tab), null);+ });++ test(`${build}: the live-message probe returns the page's visible status texts`, async () => {+ const agent = new Agent({ getActive: () => STRONG });+ const run = code => {+ const element = (text, width = 10) => ({ innerText: text, getBoundingClientRect: () => ({ width, height: 10 }) });+ const nodes = [element(' Message sent '), element('Hidden', 0), element('')];+ // The page's value arrives serialized, as CDP and executeScript return it.+ return JSON.parse(JSON.stringify(vm.runInNewContext(code, { document: { querySelectorAll: () => nodes },+ window: { getComputedStyle: () => ({ visibility: 'visible', display: 'block', opacity: '1' }) } })));+ };+ if (build === 'chrome') {+ const { cdpClient } = await import('../src/chrome/src/cdp/cdp-client.js');+ const original = cdpClient.evaluate;+ cdpClient.evaluate = async (_tab, expression) => ({ result: { value: run(expression) } });+ try { assert.deepEqual(await agent._liveRegionMessages(1), ['Message sent']); } finally { cdpClient.evaluate = original; }+ } else {+ const original = globalThis.browser.tabs.executeScript;+ globalThis.browser.tabs.executeScript = async (_tab, { code }) => [run(code)];+ try { assert.deepEqual(await agent._liveRegionMessages(1), ['Message sent']); } finally { globalThis.browser.tabs.executeScript = original; }+ }+ });+test(`${build}: in a lean run an unverified CodeMirror write is a plain success with no note or lock`, async () => {const fileText = 'line one\n\nline three\n';const editorMeta = { tag: 'div', contentEditable: true, codeMirror: true, ariaLabel: 'Editing README.md file contents' };