Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Route a hand-off restart again when a correction arrives after its check
Claudec09152af29d4
diff --git a/docs/architecture.md b/docs/architecture.mdindex f51dd789a..2a2514a30 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -75,7 +75,7 @@ Model tiering is separate from mode: `compact | mid | full` controls how many no1. Run state decides alone when it can: a continuation keeps its run's mode; a turn for an in-browser (WebGPU) model, which runs as a tiny Ask, a selected-text scope, the standalone window, a Cloud run and a message that carries an `<untrusted_page_content>` block (a selection shortcut with page context) stay in Ask; and a video attachment, which can only be uploaded, needs Act. Context-menu and selection prompts are sent as Ask, as in Manual.2. Otherwise the decision model answers the fixed-choice `run_mode` question (`semantic-decisions.js`) about the user's own words, the previous turn's mode and the earlier task. It never sees page text, title or address. The question starts as soon as the turn's cost state exists and is awaited after `_enrichUserMessageWithCurrentPage`, so it overlaps page capture; after capture the turn waits at most `AUTO_MODE_ROUTE_GRACE_MS` (250 ms). Only a sure `act` or `dev` answer (the choices are `authorizing`) selects an action mode; Compact maps Dev to Act. The routed mode replaces the provisional Ask setup: run mode, system prompt, the turn's trusted runtime line and dialog handling (Chrome) or the BiDi run (Firefox).-3. Without a sure answer, or with no decision model, the turn starts in Ask at once. Its tools add `request_action_mode`, and its system prompt adds `AUTO_MODE_ASK_NOTE`. The tool is never dispatched. The loop accepts the call only when its `evidence` quotes the user's own words exactly (the request, or a correction sent during the turn) and, because the calling model has read the page, when the same `run_mode` question, about the request with its corrections and asked again without any page content (decision model, then the active model with a quoted reason; a sure routing answer is reused from the semantic-decision cache) also answers act or dev. A correction sent while that check runs supersedes the call like any pending correction: the next step applies it and the model asks again, so a correction that needs Dev gets a Dev hand-off. Corrections sent during the read-only pass, applied or still pending, go with the hand-off: they stay journaled under the same request, and the action run queues them before its first model or tool boundary, so it never acts on the uncorrected request. An accepted hand-off restores the conversation and per-tab turn state captured before the turn gathered its context (approved plans, site-adapter and API notes, the follow-up allowance, the durable-turn marker) and returns; the background's `chat` handler then restarts the same request with `modeEscalation` in the requested mode, under the same request ID, so Stop still cancels it. The action run's planner gate reads the request again before any action. A refused call gets a tool result with the reason, and a second refusal withdraws the tool. Chrome drains Dev diagnostics only when a routed turn after Dev ends without handing itself to Dev.+3. Without a sure answer, or with no decision model, the turn starts in Ask at once. Its tools add `request_action_mode`, and its system prompt adds `AUTO_MODE_ASK_NOTE`. The tool is never dispatched. The loop accepts the call only when its `evidence` quotes the user's own words exactly (the request, or a correction sent during the turn) and, because the calling model has read the page, when the same `run_mode` question, about the request with its corrections and asked again without any page content (decision model, then the active model with a quoted reason; a sure routing answer is reused from the semantic-decision cache) also answers act or dev. A correction sent while that check runs supersedes the call like any pending correction: the next step applies it and the model asks again, so a correction that needs Dev gets a Dev hand-off. Corrections sent during the read-only pass, applied or still pending, go with the hand-off: they stay journaled under the same request, and the action run queues them before its first model or tool boundary, so it never acts on the uncorrected request. A correction that arrives after the check (while the hand-off is saved) marks the escalation `reroute`: the background then restarts the request in Automatic again, and that route reads the request with its carried corrections; the last of at most three restarts runs in the mode it asked for. An accepted hand-off restores the conversation and per-tab turn state captured before the turn gathered its context (approved plans, site-adapter and API notes, the follow-up allowance, the durable-turn marker) and returns; the background's `chat` handler then restarts the same request with `modeEscalation` in the requested mode, under the same request ID, so Stop still cancels it. The action run's planner gate reads the request again before any action. A refused call gets a tool result with the reason, and a second refusal withdraws the tool. Chrome drains Dev diagnostics only when a routed turn after Dev ends without handing itself to Dev.The agent reports the routed mode with a `run_mode` update (`{ mode, source }`), which the run journal stores as the snapshot's mode and the panel stores on the assistant message for Retry, Continue and recovery. Traces record a `mode_route` note with the source, the decision latency and the time the turn waited. In Automatic, slash commands and the Ctrl+Shift+A/X/D shortcuts choose a mode for one message only; the shortcuts write the slash command in front of the draft.diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 253a7faf4..728998f73 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -7573,6 +7573,10 @@ export class Agent extends LoopDetector {const carriedIds = new Set(carried.map(item => item.id));if (carried.length) {this._modeEscalationSteering.set(tabId, carried);+ // One sent after the hand-off was checked may need another mode: the+ // restart routes the corrected request again instead of trusting it.+ const checked = run.handoffChecked || new Set();+ if (carried.some(item => !checked.has(item.id))) this._pendingModeEscalations.get(tabId).reroute = true;void this._steeringJournal.forget(tabId, [...carriedIds]);for (const item of carried) void this._steeringJournal.accept(tabId, run.requestId, item);} else if (run?.messages.length) {@@ -7593,9 +7597,9 @@ export class Agent extends LoopDetector {this._modeEscalationSteering.delete(tabId);const run = this._steeringRuns.get(tabId);const fresh = run ? carried.filter(item => !run.acceptedIds.has(item.id)) : [];- if (!fresh.length) return;for (const item of fresh) run.acceptedIds.add(item.id);- run.messages.unshift(...fresh);+ if (fresh.length) run.messages.unshift(...fresh);+ return carried;}async _completeInitialResponseOnlyTurn(tabId, messages, onUpdate, mode, provider, costState, runId, runOptions, gate, enriched, priorMessages) {@@ -27031,11 +27035,12 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// The `run_mode` question of a routed turn, read before the turn joins the// conversation: the user's words, the previous turn's mode and the earlier// task, never page content.- _autoModeQuestion(messages, userMessage, previousMode = null) {+ _autoModeQuestion(messages, userMessage, previousMode = null, corrections = []) {const binding = this._activeTaskBinding(messages || []);const task = binding.index >= 0 ? String(binding.requestText || binding.text || '') : '';return {- request: this._messageText(userMessage),+ // A rerouted hand-off restart also reads the corrections it carries.+ request: [this._messageText(userMessage), ...(Array.isArray(corrections) ? corrections : [])].join('\n'),page: {previous_mode: isRunMode(previousMode) ? previousMode : 'none',// An earlier selection prompt with page context is not the user's own words.@@ -53846,8 +53851,15 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dawait this._hydrate(tabId);this._pendingModeEscalations.delete(tabId);if (mode === AUTO_MODE) {- this._modeEscalationSteering.delete(tabId);+ // A hand-off restart routed again keeps its carried corrections, and+ // its route reads them with the request.+ const reroute = runOptions?.modeEscalation?.reroute === true;+ if (!reroute) this._modeEscalationSteering.delete(tabId);({ mode, runOptions } = this._beginAutoModeRun(tabId, userMessage, attachments, runOptions, onUpdate));+ const carried = reroute ? this._takeModeEscalationSteering(tabId) : [];+ if (carried.length && runOptions?.autoModeRouting) {+ runOptions = { ...runOptions, autoModeRouting: { ...runOptions.autoModeRouting, corrections: carried.map(item => item.text) } };+ }} else if (runOptions?.modeEscalation && isRunMode(mode)) {runOptions = { ...runOptions, modeRoute: { requested: AUTO_MODE, mode, source: 'escalation' } };onUpdate('run_mode', { mode, source: 'escalation' });@@ -54261,7 +54273,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// An automatically routed turn asks the decision model while its page// context is gathered below (see auto-mode.js).const autoModeQuestion = runOptions?.autoModeRouting- ? this._autoModeQuestion(messages, userMessage, runOptions.autoModeRouting.previousMode)+ ? this._autoModeQuestion(messages, userMessage, runOptions.autoModeRouting.previousMode, runOptions.autoModeRouting.corrections): null;const autoModeRoute = autoModeQuestion ? this._routeAutoRunMode(tabId, autoModeQuestion) : null;@@ -55299,6 +55311,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (steeringRun) {const pendingIds = new Set(steeringRun.messages.map(item => String(item.id)));steeringRun.handoffApplied = (handoff.corrections || []).filter(item => !pendingIds.has(item.id));+ steeringRun.handoffChecked = new Set((handoff.corrections || []).map(item => item.id));}this._restoreAutoModeTurn(tabId, messages, autoModeTurnStart);this._pendingModeEscalations.set(tabId, escalation);diff --git a/src/chrome/src/background.js b/src/chrome/src/background.jsindex 2e232391c..4cd754300 100644--- a/src/chrome/src/background.js+++ b/src/chrome/src/background.js@@ -4050,13 +4050,17 @@ async function handleMessage(msg, sender) {} else {result = await agent.processMessage(tabId, msg.text, publishUpdate, mode, msg.attachments, runOptions);// An automatically routed Ask turn may hand itself to Act or Dev; it- // restarts here under the same request, so Stop still cancels it.- const escalation = mode === 'auto' ? agent.takeModeEscalation(tabId) : null;- if (escalation) {- result = await agent.processMessage(tabId, msg.text, publishUpdate, escalation.mode, msg.attachments, {+ // restarts here under the same request, so Stop still cancels it. A+ // correction that arrived after the hand-off was checked routes the+ // restart again; the last restart runs in the mode it asked for.+ let escalation = mode === 'auto' ? agent.takeModeEscalation(tabId) : null;+ for (let restarts = 0; escalation; restarts += 1) {+ const reroute = escalation.reroute === true && restarts < 2;+ result = await agent.processMessage(tabId, msg.text, publishUpdate, reroute ? 'auto' : escalation.mode, msg.attachments, {...runOptions,- modeEscalation: { mode: escalation.mode },+ modeEscalation: { mode: escalation.mode, ...(reroute ? { reroute: true } : {}) },});+ escalation = reroute ? agent.takeModeEscalation(tabId) : null;}}diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 5592a280f..0eaada322 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -7114,6 +7114,10 @@ export class Agent extends LoopDetector {const carriedIds = new Set(carried.map(item => item.id));if (carried.length) {this._modeEscalationSteering.set(tabId, carried);+ // One sent after the hand-off was checked may need another mode: the+ // restart routes the corrected request again instead of trusting it.+ const checked = run.handoffChecked || new Set();+ if (carried.some(item => !checked.has(item.id))) this._pendingModeEscalations.get(tabId).reroute = true;void this._steeringJournal.forget(tabId, [...carriedIds]);for (const item of carried) void this._steeringJournal.accept(tabId, run.requestId, item);} else if (run?.messages.length) {@@ -7134,9 +7138,9 @@ export class Agent extends LoopDetector {this._modeEscalationSteering.delete(tabId);const run = this._steeringRuns.get(tabId);const fresh = run ? carried.filter(item => !run.acceptedIds.has(item.id)) : [];- if (!fresh.length) return;for (const item of fresh) run.acceptedIds.add(item.id);- run.messages.unshift(...fresh);+ if (fresh.length) run.messages.unshift(...fresh);+ return carried;}async _completeInitialResponseOnlyTurn(tabId, messages, onUpdate, mode, provider, costState, runId, runOptions, gate, enriched, priorMessages) {@@ -24076,11 +24080,12 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// The `run_mode` question of a routed turn, read before the turn joins the// conversation: the user's words, the previous turn's mode and the earlier// task, never page content.- _autoModeQuestion(messages, userMessage, previousMode = null) {+ _autoModeQuestion(messages, userMessage, previousMode = null, corrections = []) {const binding = this._activeTaskBinding(messages || []);const task = binding.index >= 0 ? String(binding.requestText || binding.text || '') : '';return {- request: this._messageText(userMessage),+ // A rerouted hand-off restart also reads the corrections it carries.+ request: [this._messageText(userMessage), ...(Array.isArray(corrections) ? corrections : [])].join('\n'),page: {previous_mode: isRunMode(previousMode) ? previousMode : 'none',// An earlier selection prompt with page context is not the user's own words.@@ -45422,8 +45427,15 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dawait this._hydrate(tabId);this._pendingModeEscalations.delete(tabId);if (mode === AUTO_MODE) {- this._modeEscalationSteering.delete(tabId);+ // A hand-off restart routed again keeps its carried corrections, and+ // its route reads them with the request.+ const reroute = runOptions?.modeEscalation?.reroute === true;+ if (!reroute) this._modeEscalationSteering.delete(tabId);({ mode, runOptions } = this._beginAutoModeRun(tabId, userMessage, attachments, runOptions, onUpdate));+ const carried = reroute ? this._takeModeEscalationSteering(tabId) : [];+ if (carried.length && runOptions?.autoModeRouting) {+ runOptions = { ...runOptions, autoModeRouting: { ...runOptions.autoModeRouting, corrections: carried.map(item => item.text) } };+ }} else if (runOptions?.modeEscalation && isRunMode(mode)) {runOptions = { ...runOptions, modeRoute: { requested: AUTO_MODE, mode, source: 'escalation' } };onUpdate('run_mode', { mode, source: 'escalation' });@@ -45780,7 +45792,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// An automatically routed turn asks the decision model while its page// context is gathered below (see auto-mode.js).const autoModeQuestion = runOptions?.autoModeRouting- ? this._autoModeQuestion(messages, userMessage, runOptions.autoModeRouting.previousMode)+ ? this._autoModeQuestion(messages, userMessage, runOptions.autoModeRouting.previousMode, runOptions.autoModeRouting.corrections): null;const autoModeRoute = autoModeQuestion ? this._routeAutoRunMode(tabId, autoModeQuestion) : null;@@ -46705,6 +46717,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dif (steeringRun) {const pendingIds = new Set(steeringRun.messages.map(item => String(item.id)));steeringRun.handoffApplied = (handoff.corrections || []).filter(item => !pendingIds.has(item.id));+ steeringRun.handoffChecked = new Set((handoff.corrections || []).map(item => item.id));}this._restoreAutoModeTurn(tabId, messages, autoModeTurnStart);this._pendingModeEscalations.set(tabId, escalation);diff --git a/src/firefox/src/background.js b/src/firefox/src/background.jsindex 71266ca3d..a7d86a7ef 100644--- a/src/firefox/src/background.js+++ b/src/firefox/src/background.js@@ -3499,13 +3499,17 @@ async function handleMessage(msg, sender) {} else {result = await agent.processMessage(tabId, msg.text, publishUpdate, mode, msg.attachments, runOptions);// An automatically routed Ask turn may hand itself to Act or Dev; it- // restarts here under the same request, so Stop still cancels it.- const escalation = mode === 'auto' ? agent.takeModeEscalation(tabId) : null;- if (escalation) {- result = await agent.processMessage(tabId, msg.text, publishUpdate, escalation.mode, msg.attachments, {+ // restarts here under the same request, so Stop still cancels it. A+ // correction that arrived after the hand-off was checked routes the+ // restart again; the last restart runs in the mode it asked for.+ let escalation = mode === 'auto' ? agent.takeModeEscalation(tabId) : null;+ for (let restarts = 0; escalation; restarts += 1) {+ const reroute = escalation.reroute === true && restarts < 2;+ result = await agent.processMessage(tabId, msg.text, publishUpdate, reroute ? 'auto' : escalation.mode, msg.attachments, {...runOptions,- modeEscalation: { mode: escalation.mode },+ modeEscalation: { mode: escalation.mode, ...(reroute ? { reroute: true } : {}) },});+ escalation = reroute ? agent.takeModeEscalation(tabId) : null;}}diff --git a/test/auto-mode.mjs b/test/auto-mode.mjsindex 8176ac894..5f3ba2123 100644--- a/test/auto-mode.mjs+++ b/test/auto-mode.mjs@@ -426,6 +426,51 @@ for (const build of BUILDS) {'the Act call made before the correction is not combined with a Dev answer');});+ test(`${build}: a correction sent after the hand-off was checked routes the restart again`, async () => {+ const f = fixture(Agent, {+ decide: (question, options) => {+ if (options.mainModel === false) {+ return /console/.test(question.request) ? { choice: 'dev', source: 'decision_model' } : { choice: null, source: 'unavailable' };+ }+ return { choice: 'act', source: 'main_model' };+ },+ replies: [handoff('act', 'Fix the login button')],+ });+ // Sent while the accepted hand-off is being saved.+ f.agent._persistNow = async () => { f.agent._steeringRuns.get(f.tab)?.messages.push({ id: 's8', text: 'instead inspect the console errors' }); };+ assert.equal(await f.run('Fix the login button.'), '');+ f.agent._persistNow = async () => null;+ const escalation = f.agent.takeModeEscalation(f.tab);+ assert.deepEqual(escalation, { mode: 'act', evidence: 'Fix the login button', reroute: true }, 'the Act answer does not cover the late correction');++ // The background routes the restart again: the route reads the+ // correction with the request and picks Dev.+ const from = f.updates.length;+ fixtureReplies(f, [messages => {+ assert.ok(messages.some(message => message.role === 'user' && /runtime_mode=dev; mutation_tools_enabled=true/.test(String(message.content))), 'the restart runs in Dev');+ assert.ok(messages.some(message => message.role === 'user' && String(message.content).startsWith('instead inspect the console errors')));+ f.agent.abort(f.tab);+ return { content: 'Stopping.' };+ }]);+ assert.match(await f.run('Fix the login button.', 'auto', [], { modeEscalation: { mode: 'act', reroute: true } }), /Stopped/);+ assert.equal(f.questions.at(-1).question.request, 'Fix the login button.\ninstead inspect the console errors');+ assert.deepEqual(runModes(f.updates.slice(from)), [{ mode: 'dev', source: 'decision_model' }]);+ assert.equal(f.agent._modeEscalationSteering.has(f.tab), false);+ });++ test(`${build}: corrections checked with the hand-off do not route the restart again`, async () => {+ const f = fixture(Agent, {+ decide: (_q, options) => (options.mainModel === false ? { choice: null, source: 'unavailable' } : { choice: 'act', source: 'main_model' }),+ replies: [() => {+ f.agent.conversations.get(f.tab).push({ role: 'user', content: 'use the 10 am slot',+ webbrainSteering: { requestId: `auto-${f.tab}`, messageId: 's9', revision: 1 } });+ return handoff('act', 'book the 9 am slot');+ }],+ });+ assert.equal(await f.run('Please book the 9 am slot for me.'), '');+ assert.deepEqual(f.agent.takeModeEscalation(f.tab), { mode: 'act', evidence: 'book the 9 am slot' });+ });+test(`${build}: a carried correction is not acknowledged away by a short confirmation`, async () => {const f = fixture(Agent);f.agent._maybeRunPlannerGate = Agent.prototype._maybeRunPlannerGate;@@ -723,7 +768,8 @@ for (const build of BUILDS) {test(`${build}: the background restarts a hand-off under the same request and reports the routed mode`, () => {const background = source(build, 'background.js');const chat = background.slice(background.indexOf(" case 'chat': {"), background.indexOf(" case 'chat_stream': {"));- assert.match(chat, /const escalation = mode === 'auto' \? agent\.takeModeEscalation\(tabId\) : null;\s*if \(escalation\) \{\s*result = await agent\.processMessage\(tabId, msg\.text, publishUpdate, escalation\.mode, msg\.attachments, \{\s*\.\.\.runOptions,\s*modeEscalation: \{ mode: escalation\.mode \},/);+ assert.match(chat, /let escalation = mode === 'auto' \? agent\.takeModeEscalation\(tabId\) : null;\s*for \(let restarts = 0; escalation; restarts \+= 1\) \{\s*const reroute = escalation\.reroute === true && restarts < 2;\s*result = await agent\.processMessage\(tabId, msg\.text, publishUpdate, reroute \? 'auto' : escalation\.mode, msg\.attachments, \{\s*\.\.\.runOptions,\s*modeEscalation: \{ mode: escalation\.mode, \.\.\.\(reroute \? \{ reroute: true \} : \{\}\) \},\s*\}\);\s*escalation = reroute \? agent\.takeModeEscalation\(tabId\) : null;\s*\}/,+ 'a late correction routes the restart again, and the last restart always runs in the mode it asked for');assert.match(chat, /if \(type === 'run_mode' && \['ask', 'act', 'dev'\]\.includes\(data\?\.mode\)\) effectiveMode = data\.mode;/);assert.match(chat, /mode: effectiveMode,/);assert.match(chat, /effectiveMode === 'ask' \? askCompletionSucceededForBadge/);