Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Let a correction during the hand-off check pick the mode, and never acknowledge one away
Claudef7728cec28f1
diff --git a/docs/architecture.md b/docs/architecture.mdindex 08e78a114..f51dd789a 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -75,7 +75,7 @@ Model tiering is separate from mode: `compact | mid | full` controls how many no1. Run state decides alone when it can: a continuation keeps its run's mode; a turn for an in-browser (WebGPU) model, which runs as a tiny Ask, a selected-text scope, the standalone window, a Cloud run and a message that carries an `<untrusted_page_content>` block (a selection shortcut with page context) stay in Ask; and a video attachment, which can only be uploaded, needs Act. Context-menu and selection prompts are sent as Ask, as in Manual.2. Otherwise the decision model answers the fixed-choice `run_mode` question (`semantic-decisions.js`) about the user's own words, the previous turn's mode and the earlier task. It never sees page text, title or address. The question starts as soon as the turn's cost state exists and is awaited after `_enrichUserMessageWithCurrentPage`, so it overlaps page capture; after capture the turn waits at most `AUTO_MODE_ROUTE_GRACE_MS` (250 ms). Only a sure `act` or `dev` answer (the choices are `authorizing`) selects an action mode; Compact maps Dev to Act. The routed mode replaces the provisional Ask setup: run mode, system prompt, the turn's trusted runtime line and dialog handling (Chrome) or the BiDi run (Firefox).-3. Without a sure answer, or with no decision model, the turn starts in Ask at once. Its tools add `request_action_mode`, and its system prompt adds `AUTO_MODE_ASK_NOTE`. The tool is never dispatched. The loop accepts the call only when its `evidence` quotes the user's own words exactly (the request, or a correction sent during the turn) and, because the calling model has read the page, when the same `run_mode` question, about the request with its corrections and asked again without any page content (decision model, then the active model with a quoted reason; a sure routing answer is reused from the semantic-decision cache) also answers act or dev; a correction sent while that check runs is asked about again. Corrections sent during the read-only pass, applied or still pending, go with the hand-off: they stay journaled under the same request, and the action run queues them before its first model or tool boundary, so it never acts on the uncorrected request. An accepted hand-off restores the conversation and per-tab turn state captured before the turn gathered its context (approved plans, site-adapter and API notes, the follow-up allowance, the durable-turn marker) and returns; the background's `chat` handler then restarts the same request with `modeEscalation` in the requested mode, under the same request ID, so Stop still cancels it. The action run's planner gate reads the request again before any action. A refused call gets a tool result with the reason, and a second refusal withdraws the tool. Chrome drains Dev diagnostics only when a routed turn after Dev ends without handing itself to Dev.+3. Without a sure answer, or with no decision model, the turn starts in Ask at once. Its tools add `request_action_mode`, and its system prompt adds `AUTO_MODE_ASK_NOTE`. The tool is never dispatched. The loop accepts the call only when its `evidence` quotes the user's own words exactly (the request, or a correction sent during the turn) and, because the calling model has read the page, when the same `run_mode` question, about the request with its corrections and asked again without any page content (decision model, then the active model with a quoted reason; a sure routing answer is reused from the semantic-decision cache) also answers act or dev. A correction sent while that check runs supersedes the call like any pending correction: the next step applies it and the model asks again, so a correction that needs Dev gets a Dev hand-off. Corrections sent during the read-only pass, applied or still pending, go with the hand-off: they stay journaled under the same request, and the action run queues them before its first model or tool boundary, so it never acts on the uncorrected request. An accepted hand-off restores the conversation and per-tab turn state captured before the turn gathered its context (approved plans, site-adapter and API notes, the follow-up allowance, the durable-turn marker) and returns; the background's `chat` handler then restarts the same request with `modeEscalation` in the requested mode, under the same request ID, so Stop still cancels it. The action run's planner gate reads the request again before any action. A refused call gets a tool result with the reason, and a second refusal withdraws the tool. Chrome drains Dev diagnostics only when a routed turn after Dev ends without handing itself to Dev.The agent reports the routed mode with a `run_mode` update (`{ mode, source }`), which the run journal stores as the snapshot's mode and the panel stores on the assistant message for Retry, Continue and recovery. Traces record a `mode_route` note with the source, the decision latency and the time the turn waited. In Automatic, slash commands and the Ctrl+Shift+A/X/D shortcuts choose a mode for one message only; the shortcuts write the slash command in front of the draft.diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 0be72d87b..253a7faf4 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -26305,7 +26305,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dthis._finalizeSelectionGroundingScope(tabId, messages, submittedUserMessage);}await this._persistSubmittedTurn(tabId, runOptions?.detachedRequestId);- if (enriched.webbrainTurnKind === 'acknowledge' && runOptions.steeringRevision == null) {+ // A pending correction (one sent during the turn, or carried by a mode+ // hand-off) is never acknowledged away: the gate goes on and revalidates it.+ if (enriched.webbrainTurnKind === 'acknowledge' && runOptions.steeringRevision == null && !this._hasPendingSteering(tabId)) {this.plannerFollowUpSkipTabs.delete(tabId);const message = 'Acknowledged.';messages.push({ role: 'assistant', content: message });@@ -27124,8 +27126,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// during the turn. Because the calling model has read the page, the// page-free `run_mode` question about the corrected request must also// answer act or dev (a sure decision-model "answer" from routing is reused- // from the cache). Returns { accepted: { mode, evidence }, corrections }- // or { refusal: text }; the corrections go with the hand-off.+ // from the cache). Returns { accepted: { mode, evidence }, corrections },+ // { refusal: text }, or { superseded: true } when the user corrected the+ // request while the check ran: the call no longer covers the request, and+ // the model asks again (for Act or Dev) once it has read the correction.async _autoModeEscalation(tabId, toolCalls, enriched, provider, question, messages, snapshot = null) {const call = (toolCalls || []).find(item => item?.function?.name === REQUEST_ACTION_MODE_TOOL_NAME);const request = typeof enriched?.webbrainUserAuthoredText === 'string'@@ -27138,26 +27142,19 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (!requested) {return { refusal: 'request_action_mode was refused: `evidence` must quote the user\'s own words that ask for the action (from their request or a correction they sent), exactly as written. Answer the request read-only if you can.' };}- // A correction sent while the check runs is asked about again, so the- // answer always covers everything the user said.- const sameCorrections = (a, b) => a.length === b.length && a.every((item, index) => item.id === b[index].id);- for (let check = 0; check < 3; check += 1) {- const base = question || { request, page: {} };- const asked = corrections.length ? { ...base, request: corrected(corrections) } : base;- // A sure "answer" from routing settles it; an "unclear" one asks again.- const verdict = await this._semanticDecision(tabId, 'run_mode', asked, { provider, retryChoices: ['unclear'] });- const latest = this._autoModeTurnCorrections(tabId, messages, snapshot);- if (!sameCorrections(latest, corrections)) {- corrections = latest;- continue;- }- const mode = confirmedEscalationMode(requested, verdict.choice, { tier });- if (!mode) {- return { refusal: 'request_action_mode was refused: read on its own, the user\'s request does not ask for browser actions. Answer it read-only.' };- }- return { accepted: { mode, evidence: requested.evidence }, corrections };+ const base = question || { request, page: {} };+ const asked = corrections.length ? { ...base, request: corrected(corrections) } : base;+ // A sure "answer" from routing settles it; an "unclear" one asks again.+ const verdict = await this._semanticDecision(tabId, 'run_mode', asked, { provider, retryChoices: ['unclear'] });+ const latest = this._autoModeTurnCorrections(tabId, messages, snapshot);+ if (latest.length !== corrections.length || latest.some((item, index) => item.id !== corrections[index].id)) {+ return { superseded: true };+ }+ const mode = confirmedEscalationMode(requested, verdict.choice, { tier });+ if (!mode) {+ return { refusal: 'request_action_mode was refused: read on its own, the user\'s request does not ask for browser actions. Answer it read-only.' };}- return { refusal: 'request_action_mode was refused because the user kept correcting the request during the check. Answer the latest request read-only; the user can send it again to have it done.' };+ return { accepted: { mode, evidence: requested.evidence }, corrections };}// The conversation and per-tab turn state before a routed turn gathers its@@ -55285,6 +55282,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (mode === 'ask' && this._autoModeEscalationTabs.has(tabId)&& result.toolCalls?.some(call => call?.function?.name === REQUEST_ACTION_MODE_TOOL_NAME)) {const handoff = await this._autoModeEscalation(tabId, result.toolCalls, enriched, provider, autoModeQuestion, messages, autoModeTurnStart);+ if (handoff.superseded) {+ // Like any response a pending correction supersedes: the next step+ // applies the correction and the model decides again, so a+ // correction that needs Dev gets a Dev hand-off.+ onUpdate('text', { content: '', replace: true });+ continue;+ }const escalation = handoff.accepted;if (escalation) {// The caller restarts this turn in the requested mode, from thediff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 05881d45a..5592a280f 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -23352,7 +23352,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dthis._finalizeSelectionGroundingScope(tabId, messages, submittedUserMessage);}await this._persistSubmittedTurn(tabId, runOptions?.detachedRequestId);- if (enriched.webbrainTurnKind === 'acknowledge' && runOptions.steeringRevision == null) {+ // A pending correction (one sent during the turn, or carried by a mode+ // hand-off) is never acknowledged away: the gate goes on and revalidates it.+ if (enriched.webbrainTurnKind === 'acknowledge' && runOptions.steeringRevision == null && !this._hasPendingSteering(tabId)) {this.plannerFollowUpSkipTabs.delete(tabId);const message = 'Acknowledged.';messages.push({ role: 'assistant', content: message });@@ -24157,8 +24159,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// during the turn. Because the calling model has read the page, the// page-free `run_mode` question about the corrected request must also// answer act or dev (a sure decision-model "answer" from routing is reused- // from the cache). Returns { accepted: { mode, evidence }, corrections }- // or { refusal: text }; the corrections go with the hand-off.+ // from the cache). Returns { accepted: { mode, evidence }, corrections },+ // { refusal: text }, or { superseded: true } when the user corrected the+ // request while the check ran: the call no longer covers the request, and+ // the model asks again (for Act or Dev) once it has read the correction.async _autoModeEscalation(tabId, toolCalls, enriched, provider, question, messages, snapshot = null) {const call = (toolCalls || []).find(item => item?.function?.name === REQUEST_ACTION_MODE_TOOL_NAME);const request = typeof enriched?.webbrainUserAuthoredText === 'string'@@ -24171,26 +24175,19 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (!requested) {return { refusal: 'request_action_mode was refused: `evidence` must quote the user\'s own words that ask for the action (from their request or a correction they sent), exactly as written. Answer the request read-only if you can.' };}- // A correction sent while the check runs is asked about again, so the- // answer always covers everything the user said.- const sameCorrections = (a, b) => a.length === b.length && a.every((item, index) => item.id === b[index].id);- for (let check = 0; check < 3; check += 1) {- const base = question || { request, page: {} };- const asked = corrections.length ? { ...base, request: corrected(corrections) } : base;- // A sure "answer" from routing settles it; an "unclear" one asks again.- const verdict = await this._semanticDecision(tabId, 'run_mode', asked, { provider, retryChoices: ['unclear'] });- const latest = this._autoModeTurnCorrections(tabId, messages, snapshot);- if (!sameCorrections(latest, corrections)) {- corrections = latest;- continue;- }- const mode = confirmedEscalationMode(requested, verdict.choice, { tier });- if (!mode) {- return { refusal: 'request_action_mode was refused: read on its own, the user\'s request does not ask for browser actions. Answer it read-only.' };- }- return { accepted: { mode, evidence: requested.evidence }, corrections };+ const base = question || { request, page: {} };+ const asked = corrections.length ? { ...base, request: corrected(corrections) } : base;+ // A sure "answer" from routing settles it; an "unclear" one asks again.+ const verdict = await this._semanticDecision(tabId, 'run_mode', asked, { provider, retryChoices: ['unclear'] });+ const latest = this._autoModeTurnCorrections(tabId, messages, snapshot);+ if (latest.length !== corrections.length || latest.some((item, index) => item.id !== corrections[index].id)) {+ return { superseded: true };+ }+ const mode = confirmedEscalationMode(requested, verdict.choice, { tier });+ if (!mode) {+ return { refusal: 'request_action_mode was refused: read on its own, the user\'s request does not ask for browser actions. Answer it read-only.' };}- return { refusal: 'request_action_mode was refused because the user kept correcting the request during the check. Answer the latest request read-only; the user can send it again to have it done.' };+ return { accepted: { mode, evidence: requested.evidence }, corrections };}// The conversation and per-tab turn state before a routed turn gathers its@@ -46691,6 +46688,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (mode === 'ask' && this._autoModeEscalationTabs.has(tabId)&& result.toolCalls?.some(call => call?.function?.name === REQUEST_ACTION_MODE_TOOL_NAME)) {const handoff = await this._autoModeEscalation(tabId, result.toolCalls, enriched, provider, autoModeQuestion, messages, autoModeTurnStart);+ if (handoff.superseded) {+ // Like any response a pending correction supersedes: the next step+ // applies the correction and the model decides again, so a+ // correction that needs Dev gets a Dev hand-off.+ onUpdate('text', { content: '', replace: true });+ continue;+ }const escalation = handoff.accepted;if (escalation) {// The caller restarts this turn in the requested mode, from thediff --git a/test/auto-mode.mjs b/test/auto-mode.mjsindex 58a7dd28f..8176ac894 100644--- a/test/auto-mode.mjs+++ b/test/auto-mode.mjs@@ -351,6 +351,13 @@ for (const build of BUILDS) {webbrainSteering: { requestId: `auto-${f.tab}`, messageId: 's1', revision: 1 } });return handoff('act', 'downlaod this and then send it to Defne on whatsapp!');},+ messages => {+ // The correction that arrived during the check superseded the first+ // call; the model read it before asking again.+ assert.ok(messages.some(message => message.role === 'user' && String(message.content).startsWith('Use the full-size image.')));+ assert.equal(messages.some(message => message.role === 'tool'), false, 'the superseded call is not a refusal');+ return handoff('act', 'downlaod this and then send it to Defne on whatsapp!');+ },],});assert.equal(await f.run('downlaod this'), '');@@ -358,7 +365,7 @@ for (const build of BUILDS) {'downlaod this','downlaod this\nand then send it to Defne on whatsapp!','downlaod this\nand then send it to Defne on whatsapp!\nUse the full-size image.',- ], 'the page-free check reads every correction, asking again when one arrives during it');+ ], 'the page-free check reads every correction');assert.equal(f.updates.some(update => update.type === 'steering_queued'), false, 'nothing is handed back to the composer');const conversation = f.agent.conversations.get(f.tab);assert.equal(conversation.some(message => message.role === 'user'), false, 'the turn and its applied correction leave together');@@ -366,6 +373,7 @@ for (const build of BUILDS) {// The background restarts the same request in Act: both corrections are// applied, in order, before its first model call.+ const restartFrom = f.updates.length;fixtureReplies(f, [messages => {const users = messages.filter(message => message.role === 'user').map(message => String(message.content));const at = text => users.findIndex(content => content.includes(text));@@ -376,22 +384,20 @@ for (const build of BUILDS) {return { content: 'Stopping.' };}]);assert.match(await f.run('downlaod this', 'act', [], { modeEscalation: { mode: 'act' } }), /Stopped/);- assert.deepEqual(f.updates.filter(update => update.type === 'steering_applied').map(update => update.data.id), ['s1', 's2']);+ assert.deepEqual(f.updates.slice(restartFrom).filter(update => update.type === 'steering_applied').map(update => update.data.id), ['s1', 's2']);assert.equal(f.agent._modeEscalationSteering.has(f.tab), false, 'carried once');});test(`${build}: corrections are carried only into the hand-off's own restart`, async () => {const f = fixture(Agent, {- decide: (_q, options) => {- if (options.mainModel === false) return { choice: null, source: 'unavailable' };- const pending = f.agent._steeringRuns.get(f.tab).messages;- if (!pending.length && f.questions.length === 2) pending.push({ id: 's3', text: 'Only the 10 am slot.' });- return { choice: 'act', source: 'main_model' };- },+ decide: (_q, options) => (options.mainModel === false ? { choice: null, source: 'unavailable' } : { choice: 'act', source: 'main_model' }),replies: [handoff('act', 'book the 9 am slot'), done('Answered.')],});+ // Sent after the hand-off was accepted, before the turn returned.+ f.agent._persistNow = async () => { f.agent._steeringRuns.get(f.tab)?.messages.push({ id: 's3', text: 'Only the 10 am slot.' }); };assert.equal(await f.run('Please book the 9 am slot for me.'), '');assert.deepEqual(f.agent._modeEscalationSteering.get(f.tab), [{ id: 's3', text: 'Only the 10 am slot.' }]);+ f.agent._persistNow = async () => null;// A new message, not the restart: the carried correction is dropped.f.agent.takeModeEscalation(f.tab);f.agent._semanticDecision = async () => ({ choice: 'answer', source: 'decision_model' });@@ -400,6 +406,46 @@ for (const build of BUILDS) {assert.equal(f.updates.some(update => update.type === 'steering_applied'), false);});+ test(`${build}: a correction that needs Dev while the check runs gets a Dev hand-off`, async () => {+ const f = fixture(Agent, {+ decide: (question, options) => {+ if (options.mainModel === false) return { choice: null, source: 'unavailable' };+ if (f.questions.length === 2) f.agent._steeringRuns.get(f.tab).messages.push({ id: 's6', text: 'instead inspect the console errors' });+ return /console/.test(question.request) ? { choice: 'dev', source: 'main_model' } : { choice: 'act', source: 'main_model' };+ },+ replies: [+ handoff('act', 'Fix the login button'),+ messages => {+ assert.ok(messages.some(message => message.role === 'user' && String(message.content).startsWith('instead inspect the console errors')));+ return handoff('dev', 'inspect the console errors');+ },+ ],+ });+ assert.equal(await f.run('Fix the login button.'), '');+ assert.deepEqual(f.agent.takeModeEscalation(f.tab), { mode: 'dev', evidence: 'inspect the console errors' },+ 'the Act call made before the correction is not combined with a Dev answer');+ });++ test(`${build}: a carried correction is not acknowledged away by a short confirmation`, async () => {+ const f = fixture(Agent);+ f.agent._maybeRunPlannerGate = Agent.prototype._maybeRunPlannerGate;+ f.agent._beginSteeringRun(f.tab, () => {}, { detachedRequestId: `auto-${f.tab}` });+ f.agent._steeringRuns.get(f.tab).messages.push({ id: 's7', text: 'but use the 10 am slot' });+ let revalidated = 0;+ f.agent._runPlannerIntentGate = async () => ({ proceed: true, requestKind: 'execute', requiresStateChange: true });+ f.agent._revalidatePendingSteering = async () => { revalidated += 1; return { changed: true, gate: { proceed: true, requestKind: 'execute', requiresStateChange: true } }; };+ const messages = [{ role: 'system', content: 'x' }];+ const gate = await f.agent._maybeRunPlannerGate(f.tab, messages,+ { role: 'user', content: 'yes', webbrainUserAuthoredText: 'yes', webbrainTurnKind: 'acknowledge' }, () => {}, 'act', null, null, null, {});+ assert.notEqual(gate.reason, 'acknowledged');+ assert.equal(revalidated, 1, 'the pending correction is revalidated');+ assert.equal(messages.some(message => message.content === 'Acknowledged.'), false);+ f.agent._steeringRuns.delete(f.tab);+ const plain = await f.agent._maybeRunPlannerGate(f.tab, [{ role: 'system', content: 'x' }],+ { role: 'user', content: 'thanks', webbrainUserAuthoredText: 'thanks', webbrainTurnKind: 'acknowledge' }, () => {}, 'act', null, null, null, {});+ assert.equal(plain.reason, 'acknowledged', 'without a correction an acknowledgment still ends the turn');+ });+test(`${build}: a correction that never asks for an action keeps the turn in Ask`, async () => {const f = fixture(Agent, {decide: (question, options) => {