Open-source AI browser agent for Chrome and Firefox (monorepo) đź§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Let an answer take back the send
Claude9691d80f928b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex cbe2cd91d..a21628333 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -19,7 +19,7 @@ This changelog was generated from the repository Git history and release tags. V- Providers → Decision accepts a local decision server on another machine on your network, such as `http://192.168.1.51:8091`. **Load available models** and **Test** used to stop with "Decision endpoints require HTTPS outside localhost." for any address but this computer's. Private addresses (10.x, 172.16–31.x, 192.168.x, Tailscale's 100.64–127.x, link-local, IPv6 unique-local) and `.local`, `.home.arpa` and `.internal` names now work over plain HTTP; public addresses still need HTTPS. In Chrome, decision requests to such a server go through the same offscreen proxy as chat models on your network. Vision and speech endpoints already accepted these addresses.- Act can send a photo or video in an Instagram direct message. Instagram's message box keeps its file input hidden, outside any form, deep in a long conversation, where WebBrain's check that an action still targets what the model saw cannot reach it. Every attachment was refused, even with the exact selector WebBrain itself listed for that input, often right after you had confirmed the recipient on WebBrain's card; the model then sent a link to the picture, or said Instagram blocks attachments. With Strict checks off, an upload to the very input WebBrain listed on that page, named by the selector it gave, now goes through with a note to the model, and the recipient checks still apply. A selector WebBrain did not list is still refused, and so is an input the page re-created after the list under the same selector, until WebBrain lists the page's controls again. When clicking "Add Photo or Video" opens a file chooser, WebBrain now names the input with that same selector, and the Instagram site notes say to attach the file to that input instead of clicking the button, and that a link to a picture is not the picture.- Act can attach images and videos to a post on X again. While the post composer is open at x.com/compose/post, the timeline behind it keeps a second, identical file input, and WebBrain's list of the page's controls gave neither input a selector. After the model's first selector matched both, WebBrain refused every later upload until it saw a selector it could not get, so the run ended at its cost limit with nothing attached. The list now gives every file input a selector that matches only that input, scoped to the open dialog where that is what tells them apart, and marks the dialog's input `inDialog`. The X site notes name the dialog's input, and when no selector can single out an input, WebBrain tells the model to report the upload as blocked instead of retrying.-- Act no longer asks you the same question again when the answer is on the page. Asked to post the same thing on LinkedIn while the X post was open, the planner asked which text and files to use, and after “look at the screen — the same” it asked again: the planner works from your words, the page's address and title and a short summary of the chat, never the page itself or your attachments. That summary held only the last ten messages, tool calls and page observations included, each cut to 300 characters, so the earlier answer that named the post's text and your screenshot file had dropped out of it. It now holds the last ten turns of the conversation without the agent's step-by-step notes, gives the latest answer up to 1,200 characters, and names the files you attached earlier. With the Planner on (Try or Strict), when you have just answered a planner question and the planner wants to ask again, it first takes a second look, told that it cannot see the page while the agent can, and plans the task instead; the plan is then checked against your words. So an answer such as "every five minutes" to a question about a scheduled task, the name of the person to write to, "yes, send it" or opening the conversation it asked you to open is planned. It never takes who a message goes to, an account or an amount from the page, and never writes to anyone your request or answer did not name; a schedule follows what you said, including "just do it once". With the Planner off, nothing checks a plan against your words, so it takes no second look at your answer. A planner running Qwen 3, QwQ or DeepSeek R1, which is told not to think so that it answers faster, now thinks before it asks you a first question, and with the Planner on also before asking again, unless reasoning is turned off for that provider. Chrome and Firefox.+- Act no longer asks you the same question again when the answer is on the page. Asked to post the same thing on LinkedIn while the X post was open, the planner asked which text and files to use, and after “look at the screen — the same” it asked again: the planner works from your words, the page's address and title and a short summary of the chat, never the page itself or your attachments. That summary held only the last ten messages, tool calls and page observations included, each cut to 300 characters, so the earlier answer that named the post's text and your screenshot file had dropped out of it. It now holds the last ten turns of the conversation without the agent's step-by-step notes, gives the latest answer up to 1,200 characters, and names the files you attached earlier. With the Planner on (Try or Strict), when you have just answered a planner question and the planner wants to ask again, it first takes a second look, told that it cannot see the page while the agent can, and plans the task instead; the plan is then checked against your words. So an answer such as "every five minutes" to a question about a scheduled task, the name of the person to write to, "yes, send it", "don't send it, save a draft" or opening the conversation it asked you to open is planned. It never takes who a message goes to, an account or an amount from the page, and never writes to anyone your request or answer did not name; a schedule follows what you said, including "just do it once". With the Planner off, nothing checks a plan against your words, so it takes no second look at your answer. A planner running Qwen 3, QwQ or DeepSeek R1, which is told not to think so that it answers faster, now thinks before it asks you a first question, and with the Planner on also before asking again, unless reasoning is turned off for that provider. Chrome and Firefox.## [42.3.1] - 2026-10-20diff --git a/docs/architecture.md b/docs/architecture.mdindex fc1ad74b1..e946e2b02 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -936,7 +936,7 @@ The recent-history digest (`_buildPlannerHistoryDigest()`) takes the last ten auA `clarify` from the planner or the intent check gets one second look before the user sees it (`_plannerClarifyRetry()`). A model that was sent `/no_think` (Qwen 3, QwQ, DeepSeek R1) answers again with thinking on and up to 8,192 output tokens (`_plannerCanRethink()`: not when reasoning is off in the provider's settings or extra request body, and not for Qwen3 Instruct-2507 and Coder models, which have no thinking mode). That rethink is the same request and is judged like the first answer.-In the full planner gate, on the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs), the second look also re-asks: `PLANNER_REASK_GUIDANCE` is added to the system prompt. It says the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. A turn that answers a planner question always has history, so its plan is then checked for alignment with the user's words, answer included, and that check is what reads which person the answer chose, whether it approved sending ("yes, send it"), and a schedule it gave, changed or dropped ("every five minutes", "just do it once"). The intent check (Planner off) has no alignment check, so it takes no second look at an answer at all, neither a re-ask nor a rethink: a retry that read the answer could run a recipient, schedule or submission nothing there checks, so its first answer stands.+In the full planner gate, on the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs), the second look also re-asks: `PLANNER_REASK_GUIDANCE` is added to the system prompt. It says the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. A turn that answers a planner question always has history, so its plan is then checked for alignment with the user's words, answer included, and that check is what reads which person the answer chose, whether it approved or took back sending ("yes, send it", "don't send it, save a draft"), and a schedule it gave, changed or dropped ("every five minutes", "just do it once"). A re-ask plan, or its alignment replacement, that drops the submission the answered question carried is therefore not treated as an inconsistency (`clarification_submit_intent_dropped`); the check reads whether the answer took it back. The intent check (Planner off) has no alignment check, so it takes no second look at an answer at all, neither a re-ask nor a rethink: a retry that read the answer could run a recipient, schedule or submission nothing there checks, so its first answer stands.An execute plan from a re-ask may still not address anyone the user did not back (`_plannerSecondLookRejection()`, trace note `planner_second_look_rejected` with the reason `adds_recipient`): the alignment check cannot keep out a recipient taken from the page when the answer points at it ("use the current chat", "look at the screen"), and the re-ask guidance forbids taking one. A planner question keeps the people to address of the task it asks about (`normalizePlan()` keeps `draft_recipients` on a `clarify` plan for a message it does not send; nothing runs from it), and the stored question records them as `messageTarget`. A question asked because the open conversation could not be pinned records that conversation too, so the reply can be planned once the user has opened it; it is pinned again before anything is sent. The plan may address only the people the answered question addressed (in any order or casing) or people the user's own words in the task name, every one of them (the task binding is built from user turns only). The open conversation is never taken from the page, and the first answer on that turn, model output like the second look, addresses no one. Another question, an answer or a plan to show runs nothing and is not checked. A plan the alignment check has the planner replace is held to the same limit, and when it goes beyond it the user keeps the question (`planner_alignment_repair` in the trace note).diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 272e21bb8..aa508299a 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -27333,7 +27333,14 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}await this._tracePlannerAttemptResponse(runId, 0, provider, result, phase, 1, startedAt, chatOptions);const plan = parsePlanFromContent(result?.content, parseOptions);- if (!plan || this._plannerIntentConsistencyIssue(plan, followUpContext)) return null;+ if (!plan) return null;+ const consistencyIssue = this._plannerIntentConsistencyIssue(plan, followUpContext);+ // An answer may take back the submission its question carried ("don't+ // send it, save a draft"). On a re-ask the alignment check that follows+ // reads that against the answer, as a first answer's focused recheck+ // would (_plannerIntentConsistencyRepairMessages). Any other issue keeps+ // the question.+ if (consistencyIssue && !(reask && consistencyIssue.kind === 'clarification_submit_intent_dropped')) return null;const rejection = reask ? this._plannerSecondLookRejection(plan, followUpContext) : null;if (rejection) {this._tracePlannerSecondLookRejection(runId, phase, rejection);@@ -28589,6 +28596,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dplan = secondLook;}}+ // A plan from a re-ask may take back the submission the answered+ // question carried; the alignment check below reads that against the+ // answer (_plannerClarifyRetry).+ if (reaskFirstPlan) consistencyRepairKind = 'clarification_submit_intent_dropped';if (this._plannerIntentUnresolvedConsistencyIssue(plan, consistencyRepairKind, followUpContext)) {if (runOptions.steeringRevision != null) return this._plannerActContinuation(runOptions, onUpdate, runId, 'inconsistent_intent');return this._normalizePlanBeforeActMode(plannerMode) === 'strict'diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex d42d53589..d12b7aeab 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -24371,7 +24371,14 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}await this._tracePlannerAttemptResponse(runId, 0, provider, result, phase, 1, startedAt, chatOptions);const plan = parsePlanFromContent(result?.content, parseOptions);- if (!plan || this._plannerIntentConsistencyIssue(plan, followUpContext)) return null;+ if (!plan) return null;+ const consistencyIssue = this._plannerIntentConsistencyIssue(plan, followUpContext);+ // An answer may take back the submission its question carried ("don't+ // send it, save a draft"). On a re-ask the alignment check that follows+ // reads that against the answer, as a first answer's focused recheck+ // would (_plannerIntentConsistencyRepairMessages). Any other issue keeps+ // the question.+ if (consistencyIssue && !(reask && consistencyIssue.kind === 'clarification_submit_intent_dropped')) return null;const rejection = reask ? this._plannerSecondLookRejection(plan, followUpContext) : null;if (rejection) {this._tracePlannerSecondLookRejection(runId, phase, rejection);@@ -25612,6 +25619,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dplan = secondLook;}}+ // A plan from a re-ask may take back the submission the answered+ // question carried; the alignment check below reads that against the+ // answer (_plannerClarifyRetry).+ if (reaskFirstPlan) consistencyRepairKind = 'clarification_submit_intent_dropped';if (this._plannerIntentUnresolvedConsistencyIssue(plan, consistencyRepairKind, followUpContext)) {if (runOptions.steeringRevision != null) return this._plannerActContinuation(runOptions, onUpdate, runId, 'inconsistent_intent');return this._normalizePlanBeforeActMode(plannerMode) === 'strict'diff --git a/test/planner-clarify-loop.mjs b/test/planner-clarify-loop.mjsindex 6e1fccc4d..c9ca0f687 100644--- a/test/planner-clarify-loop.mjs+++ b/test/planner-clarify-loop.mjs@@ -556,8 +556,7 @@ for (const build of ['chrome', 'firefox']) {['unparseable', ['not JSON'], {}],['request error', [new Error('vLLM: 400 max_tokens is too large')], {}],['cost allowance', [costAgent._costAllowanceError('The cloud cost allowance for this session is used up.')], {}],- ['drops the submission the earlier question kept', [plannerPlan({ requires_submission: false })],- { plannerClarification: { requiresSubmission: true, pageUrl: TWEET_URL } }],+ ['an answer-only plan that lists tools', [plannerPlan({ request_kind: 'respond', read_scope: 'none', requires_state_change: false, requires_submission: false })], {}],];for (const [label, retryAnswers, extraContext] of cases) {const { agent, calls } = scriptedAgent({ provider: PLAIN_MODEL, answers: [clarifyPlan(), ...retryAnswers] });@@ -567,6 +566,21 @@ for (const build of ['chrome', 'firefox']) {}});+ test(`${build}: an answer that takes back the send is planned, and the alignment check reads it`, async () => {+ const draftOnly = plannerPlan({ requires_submission: false, summary: 'Write the same post on LinkedIn as a draft without publishing it.' });+ const { agent, calls, alignmentChecks } = scriptedAgent({ provider: PLAIN_MODEL, answers: [clarifyPlan(), draftOnly] });+ const result = await fullGate(agent, {+ message: user("don't post it, just save a draft"), digest: ANSWERED_DIGEST,+ followUpContext: { answersPlannerQuestion: true, plannerClarification: { requiresSubmission: true, pageUrl: TWEET_URL } },+ });+ assert.equal(result.proceed, true, JSON.stringify(result));+ assert.equal(result.requiresSubmission, false, 'the answer took the send back');+ assert.equal(calls.length, 2);+ assert.equal(alignmentChecks.length, 1, 'the alignment check read the plan against the answer');+ assert.equal(alignmentChecks[0].proposed_plan.requires_submission, false);+ assert.match(String(alignmentChecks[0].current_request), /just save a draft/);+ });+test(`${build}: a second look after a portable repair stays portable`, async () => {const { agent, calls } = scriptedAgent({answers: [new Error('400 Bad Request: response_format json_schema is not supported'), clarifyPlan(), plannerPlan()],