Open-source AI browser agent for Chrome and Firefox (monorepo) π§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Automatic Ask turns hand actions off instead of sending the user to Act
Claudef6c2f81ff541
diff --git a/docs/architecture.md b/docs/architecture.mdindex 1bfa58094..c4696f30e 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -77,7 +77,7 @@ Model tiering is separate from mode: `compact | mid | full` controls how many no2. Otherwise the decision model answers the fixed-choice `run_mode` question (`semantic-decisions.js`) about the user's own words, the previous turn's mode and the earlier task. It never sees page text, title or address. The question starts as soon as the turn's cost state exists and is awaited after `_enrichUserMessageWithCurrentPage`, so it overlaps page capture; after capture the turn waits at most `AUTO_MODE_ROUTE_GRACE_MS` (250 ms). Only a sure `act` or `dev` answer (the choices are `authorizing`) selects an action mode; Compact maps Dev to Act. The routed mode replaces the provisional Ask setup: run mode, system prompt, the turn's trusted runtime line and dialog handling (Chrome) or the BiDi run (Firefox).3. Without a sure answer, or with no decision model, the turn starts in Ask at once. So does a turn whose user sent a correction while the route ran or while the action run's browser setup started (`route_corrected`): the route did not read it, so an action answer is not trusted. That check follows the setup's last await, so nothing arrives between it and the committed mode. Its tools add `request_action_mode`, and its system prompt adds `AUTO_MODE_ASK_NOTE`. The tool is never dispatched. The loop accepts the call only when its `evidence` quotes the user's own words exactly (the request, or a correction sent during the turn) and, because the calling model has read the page, when the same `run_mode` question, about the request with its corrections and asked again without any page content (decision model, then the active model with a quoted reason; a sure routing answer is reused from the semantic-decision cache) also answers act or dev. A correction sent while that check runs supersedes the call like any pending correction: the next step applies it and the model asks again, so a correction that needs Dev gets a Dev hand-off. Corrections sent during the read-only pass, applied or still pending, go with the hand-off: they stay journaled under the same request, and the action run queues them before its first model or tool boundary, so it never acts on the uncorrected request. A correction that arrives after the check (while the hand-off is saved) marks the escalation `reroute`: the background then restarts the request in Automatic again, and that route reads the request with its carried corrections; a third such restart runs read-only (`reroute_limit`, without the hand-off tool): it applies the corrections and answers, and never runs in a mode chosen before the latest correction. An accepted hand-off restores the conversation and per-tab turn state captured before the turn gathered its context (approved plans, site-adapter and API notes, the follow-up allowance, the durable-turn marker) and returns; the background's `chat` handler then restarts the same request with `modeEscalation` in the requested mode, under the same request ID, so Stop still cancels it. The action run's planner gate reads the request again before any action. A refused call gets a tool result with the reason, and a second refusal withdraws the tool. Chrome drains Dev diagnostics only when a routed turn after Dev ends without handing itself to Dev.-The agent reports the routed mode with a `run_mode` update (`{ mode, source }`), which the run journal stores as the snapshot's mode and the panel stores on the assistant message for Retry, Continue and recovery. Traces record a `mode_route` note with the source, the decision latency and the time the turn waited. Slash commands and the Ctrl+Shift+A/X/D shortcuts choose a mode for one message only, also while a mode is kept; the shortcuts write the slash command in front of the draft. Context-menu and selection prompts stay Ask either way.+The agent reports the routed mode with a `run_mode` update (`{ mode, source }`), which the run journal stores as the snapshot's mode and the panel stores on the assistant message for Retry, Continue and recovery. Traces record a `mode_route` note with the source, the decision latency and the time the turn waited. Slash commands and the Ctrl+Shift+A/X/D shortcuts choose a mode for one message only, also while a mode is kept; the shortcuts write the slash command in front of the draft. Context-menu and selection prompts stay Ask either way. The Ask system prompt, shared by chosen and automatic Ask turns, defers to `AUTO_MODE_ASK_NOTE` wherever it would tell the user to switch to Act mode, and the note says it replaces those instructions. An automatically routed Ask answer gets no one-click "Switch to Act & retry" offer (`ask_mode_handoff`): the turn could hand itself off, and the user's next message, a "yes" to an offer included, is routed again. A chosen Ask turn keeps the offer.The user types a message, the panel sends a detached `{action: 'chat_start', text, mode, tabId, requestId}` request, then reconnects to the background-owned run journal for `agent_update` events. The acknowledged start becomes the existing `chat` handler and `agent.processMessage()` lifecycle; closing or reloading the panel does not transfer ownership or start the run again. The panel renders tool calls, results, plan-review cards, clarification prompts, and the final answer incrementally.diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex c406807fa..b6ef52324 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -27255,7 +27255,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}async _maybeEmitAskModeHandoff(tabId, mode, userMessage, finalResponse, onUpdate, runOptions = {}) {+ // An automatically routed turn could hand itself to Act, and the user's+ // next message is routed again ("yes" to an offer included), so it is+ // not offered a one-click switch to a mode the user never picked.if (mode !== 'ask'+ || runOptions?.autoModeRouting|| runOptions?.cloudRun|| this._isStandaloneChatRun(runOptions)|| typeof finalResponse !== 'string'diff --git a/src/chrome/src/agent/auto-mode.js b/src/chrome/src/agent/auto-mode.jsindex 528203be4..cd4375ddb 100644--- a/src/chrome/src/agent/auto-mode.js+++ b/src/chrome/src/agent/auto-mode.js@@ -135,7 +135,7 @@ export function requestActionModeTool() {// Appended to the Ask system prompt of an automatically routed turn, after// the cache-stable base prompt.export const AUTO_MODE_ASK_NOTE = `AUTOMATIC MODE:-- WebBrain chose this read-only turn from the request; the user did not pick a mode. Never ask the user to switch to Act or Dev mode.+- WebBrain chose this read-only turn from the request; the user did not pick a mode. This section replaces every instruction above that tells the user to switch to Act or Dev mode: never ask the user to switch modes.- When the latest request needs browser actions or Dev page work that this turn cannot do, call \`request_action_mode\` alone, before answering, quoting the user's words that ask for it. WebBrain restarts the request with the right tools.- When reading, explaining, summarizing, analyzing or drafting text answers the request, answer it here.`;diff --git a/src/chrome/src/agent/tools.js b/src/chrome/src/agent/tools.jsindex e5e3e948e..d6609ae5b 100644--- a/src/chrome/src/agent/tools.js+++ b/src/chrome/src/agent/tools.js@@ -2151,7 +2151,7 @@ OPERATING ENVIRONMENT β read this carefully:- You operate inside the user's authenticated browser session. Every site they are logged into (GitHub, Gmail, banking, internal tools, etc.) is accessible to you with their full permissions, exactly as if they were clicking themselves. There is no separate "AI account" β you ARE the user, from the website's point of view.- You do NOT need API tokens, OAuth flows, or "permission to act on the user's behalf". The browser session already has all that.- Never refuse a task by saying things like "I don't have permission", "I can't access your account", "you'll need to do this manually", or "as an AI I can't authenticate". Those refusals are wrong in this environment. If a logged-in human could do it through the browser UI, you can do it.-- The only legitimate reasons to decline are: (a) the action is genuinely harmful or destructive and the user hasn't confirmed, (b) the required UI element doesn't exist or can't be located after honest attempts, or (c) the user is in Ask mode and the task requires Act mode.+- The only legitimate reasons to decline are: (a) the action is genuinely harmful or destructive and the user hasn't confirmed, (b) the required UI element doesn't exist or can't be located after honest attempts, or (c) the task needs browser actions that this read-only Ask turn cannot do. When an AUTOMATIC MODE section follows, hand the request off as it says instead of declining.- You CANNOT schedule, sleep, set timers, or "check back later". Each user turn is a single live session β there is no cron, no background polling, no way for you to resume on your own. If a task needs to wait for an external event (a build to finish, an email to arrive, a deploy to complete, prices to change), call \`done\` with what you have and tell the user to re-invoke you when ready. NEVER tell the user you'll "check back in a few minutes", "come back later", or "wait and try again" β those are lies about a capability you don't have.UNTRUSTED PAGE CONTENT:@@ -2219,7 +2219,7 @@ READING THE CURRENT TAB vs. FETCHING URLS β read this:Guidelines:1. Read the page first (a11y tree by default) to understand the context, then answer the user's question.2. Be conversational and helpful β answer in natural language, not raw data dumps.-3. If the user asks you to do something that requires clicking or typing, let them know they need to switch to Act mode.+3. If the user asks you to do something that requires clicking or typing: when an AUTOMATIC MODE section follows, call \`request_action_mode\` as it says; otherwise let them know they need to switch to Act mode.4. Summarize, analyze, and explain β that's your strength in this mode.LISTINGS & PAGINATION β read this:diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 88e9fa925..8d3ca4599 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -24290,7 +24290,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}async _maybeEmitAskModeHandoff(tabId, mode, userMessage, finalResponse, onUpdate, runOptions = {}) {+ // An automatically routed turn could hand itself to Act, and the user's+ // next message is routed again ("yes" to an offer included), so it is+ // not offered a one-click switch to a mode the user never picked.if (mode !== 'ask'+ || runOptions?.autoModeRouting|| runOptions?.cloudRun|| this._isStandaloneChatRun(runOptions)|| typeof finalResponse !== 'string'diff --git a/src/firefox/src/agent/auto-mode.js b/src/firefox/src/agent/auto-mode.jsindex 528203be4..cd4375ddb 100644--- a/src/firefox/src/agent/auto-mode.js+++ b/src/firefox/src/agent/auto-mode.js@@ -135,7 +135,7 @@ export function requestActionModeTool() {// Appended to the Ask system prompt of an automatically routed turn, after// the cache-stable base prompt.export const AUTO_MODE_ASK_NOTE = `AUTOMATIC MODE:-- WebBrain chose this read-only turn from the request; the user did not pick a mode. Never ask the user to switch to Act or Dev mode.+- WebBrain chose this read-only turn from the request; the user did not pick a mode. This section replaces every instruction above that tells the user to switch to Act or Dev mode: never ask the user to switch modes.- When the latest request needs browser actions or Dev page work that this turn cannot do, call \`request_action_mode\` alone, before answering, quoting the user's words that ask for it. WebBrain restarts the request with the right tools.- When reading, explaining, summarizing, analyzing or drafting text answers the request, answer it here.`;diff --git a/src/firefox/src/agent/tools.js b/src/firefox/src/agent/tools.jsindex c61e526cf..1962836af 100644--- a/src/firefox/src/agent/tools.js+++ b/src/firefox/src/agent/tools.js@@ -2057,7 +2057,7 @@ OPERATING ENVIRONMENT β read this carefully:- You operate inside the user's authenticated browser session. Every site they are logged into (GitHub, Gmail, banking, internal tools, etc.) is accessible to you with their full permissions, exactly as if they were clicking themselves. There is no separate "AI account" β you ARE the user, from the website's point of view.- You do NOT need API tokens, OAuth flows, or "permission to act on the user's behalf". The browser session already has all that.- Never refuse a task by saying things like "I don't have permission", "I can't access your account", "you'll need to do this manually", or "as an AI I can't authenticate". Those refusals are wrong in this environment. If a logged-in human could do it through the browser UI, you can do it.-- The only legitimate reasons to decline are: (a) the action is genuinely harmful or destructive and the user hasn't confirmed, (b) the required UI element doesn't exist or can't be located after honest attempts, or (c) the user is in Ask mode and the task requires Act mode.+- The only legitimate reasons to decline are: (a) the action is genuinely harmful or destructive and the user hasn't confirmed, (b) the required UI element doesn't exist or can't be located after honest attempts, or (c) the task needs browser actions that this read-only Ask turn cannot do. When an AUTOMATIC MODE section follows, hand the request off as it says instead of declining.- You CANNOT schedule, sleep, set timers, or "check back later". Each user turn is a single live session β there is no cron, no background polling, no way for you to resume on your own. If a task needs to wait for an external event (a build to finish, an email to arrive, a deploy to complete, prices to change), call \`done\` with what you have and tell the user to re-invoke you when ready. NEVER tell the user you'll "check back in a few minutes", "come back later", or "wait and try again" β those are lies about a capability you don't have.UNTRUSTED PAGE CONTENT:@@ -2105,7 +2105,7 @@ READING THE CURRENT TAB vs. FETCHING URLS β read this:Guidelines:1. Read the page first to understand the context, then answer the user's question.2. Be conversational and helpful β answer in natural language, not raw data dumps.-3. If the user asks you to do something that requires clicking or typing, let them know they need to switch to Act mode.+3. If the user asks you to do something that requires clicking or typing: when an AUTOMATIC MODE section follows, call \`request_action_mode\` as it says; otherwise let them know they need to switch to Act mode.4. Summarize, analyze, and explain β that's your strength in this mode.LISTINGS & PAGINATION β read this:diff --git a/test/auto-mode.mjs b/test/auto-mode.mjsindex b1ca2cd5e..ef49b0cf4 100644--- a/test/auto-mode.mjs+++ b/test/auto-mode.mjs@@ -684,6 +684,47 @@ for (const build of BUILDS) {assert.equal(f.agent.takeModeEscalation(f.tab), null);});+ test(`${build}: the read-only prompt hands an action off instead of sending the user to Act mode`, async () => {+ // The base Ask prompt is shared by chosen and automatic Ask turns; where+ // it would send the user to Act mode, it defers to the automatic note.+ const { SYSTEM_PROMPT_ASK } = await import(`../src/${build}/src/agent/tools.js`);+ assert.match(SYSTEM_PROMPT_ASK, /requires clicking or typing: when an AUTOMATIC MODE section follows, call `request_action_mode` as it says; otherwise let them know they need to switch to Act mode\./);+ assert.match(SYSTEM_PROMPT_ASK, /\(c\) the task needs browser actions that this read-only Ask turn cannot do\. When an AUTOMATIC MODE section follows, hand the request off as it says instead of declining\./);+ const { AUTO_MODE_ASK_NOTE } = await import(`../src/${build}/src/agent/auto-mode.js`);+ assert.match(AUTO_MODE_ASK_NOTE, /This section replaces every instruction above that tells the user to switch to Act or Dev mode: never ask the user to switch modes\./);+ const f = fixture(Agent, { replies: [done('Answer.')] });+ await f.run('What is on this page?');+ const system = f.calls[0].messages[0].content;+ assert.ok(system.startsWith(SYSTEM_PROMPT_ASK), 'the cache-stable base prompt comes first');+ assert.ok(system.indexOf(AUTO_MODE_ASK_NOTE) > SYSTEM_PROMPT_ASK.length, 'the note follows the instructions it replaces');+ });++ test(`${build}: an automatic read-only answer is not offered a one-click switch to Act`, async () => {+ // "Would you like me to open it?" after a routed answer: the user's "yes"+ // is routed and hands the turn off, so the Ask-to-Act button is noise.+ const offer = 'There is a "Learn more" link. Would you like me to open it for you?';+ const settle = () => new Promise(resolve => setImmediate(resolve));+ const classifierCalls = f => {+ const calls = [];+ delete f.agent._maybeEmitAskModeHandoff;+ f.agent._jevClassify = async (...args) => { calls.push(args[0]); return 'act'; };+ return calls;+ };+ const routed = fixture(Agent, { decide: () => ({ choice: 'answer', evidence: '', source: 'decision_model' }), replies: [done(offer)] });+ const routedCalls = classifierCalls(routed);+ assert.equal(await routed.run('Is there anything here I could read more about? Offer to open it for me.'), offer);+ await settle();+ assert.deepEqual(routedCalls, [], 'no Act suggestion is classified for a routed turn');+ assert.equal(routed.updates.some(update => update.type === 'ask_mode_handoff'), false);+ // A chosen Ask turn still gets the offer.+ const chosen = fixture(Agent, { replies: [done(offer)] });+ const chosenCalls = classifierCalls(chosen);+ assert.equal(await chosen.run('Is there anything here I could read more about? Offer to open it for me.', 'ask'), offer);+ await settle();+ assert.equal(chosenCalls.length, 1);+ assert.deepEqual(chosen.updates.filter(update => update.type === 'ask_mode_handoff').map(update => update.data), [{ value: 'act' }]);+ });+test(`${build}: a slow decision model leaves the turn read-only after the grace period`, async () => {// The decision never answers; the turn goes on without it.const f = fixture(Agent, { decide: () => new Promise(() => {}), replies: [done('Read-only answer.')] });diff --git a/test/run.js b/test/run.jsindex 76b38745d..6bf0f4280 100644--- a/test/run.js+++ b/test/run.js@@ -12103,6 +12103,8 @@ test('Ask mode handoff classification is strict, silent, and mode guarded', asyn{ cloudRun: true },{ standaloneChat: true },{ sourceGrounding: selectionGrounding },+ // An automatically routed turn hands itself off; its follow-up is routed again.+ { autoModeRouting: { previousMode: null } },]) {const skipped = createAgent();await skipped.agent._maybeEmitAskModeHandoff(53002, 'ask', 'Click the button', finalResponse, () => { }, runOptions);