Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) đź§ 

Take no second look at an answer in the intent check

Claude2042c6f143c2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index dd9b5c301..a6445f9d5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,7 +19,7 @@ This changelog was generated from the repository Git history and release tags. V
- Providers → Decision accepts a local decision server on another machine on your network, such as `http://192.168.1.51:8091`. **Load available models** and **Test** used to stop with "Decision endpoints require HTTPS outside localhost." for any address but this computer's. Private addresses (10.x, 172.16–31.x, 192.168.x, Tailscale's 100.64–127.x, link-local, IPv6 unique-local) and `.local`, `.home.arpa` and `.internal` names now work over plain HTTP; public addresses still need HTTPS. In Chrome, decision requests to such a server go through the same offscreen proxy as chat models on your network. Vision and speech endpoints already accepted these addresses.
- Act can send a photo or video in an Instagram direct message. Instagram's message box keeps its file input hidden, outside any form, deep in a long conversation, where WebBrain's check that an action still targets what the model saw cannot reach it. Every attachment was refused, even with the exact selector WebBrain itself listed for that input, often right after you had confirmed the recipient on WebBrain's card; the model then sent a link to the picture, or said Instagram blocks attachments. With Strict checks off, an upload to the very input WebBrain listed on that page, named by the selector it gave, now goes through with a note to the model, and the recipient checks still apply. A selector WebBrain did not list is still refused, and so is an input the page re-created after the list under the same selector, until WebBrain lists the page's controls again. When clicking "Add Photo or Video" opens a file chooser, WebBrain now names the input with that same selector, and the Instagram site notes say to attach the file to that input instead of clicking the button, and that a link to a picture is not the picture.
- Act can attach images and videos to a post on X again. While the post composer is open at x.com/compose/post, the timeline behind it keeps a second, identical file input, and WebBrain's list of the page's controls gave neither input a selector. After the model's first selector matched both, WebBrain refused every later upload until it saw a selector it could not get, so the run ended at its cost limit with nothing attached. The list now gives every file input a selector that matches only that input, scoped to the open dialog where that is what tells them apart, and marks the dialog's input `inDialog`. The X site notes name the dialog's input, and when no selector can single out an input, WebBrain tells the model to report the upload as blocked instead of retrying.
-- Act no longer asks you the same question again when the answer is on the page. Asked to post the same thing on LinkedIn while the X post was open, the planner asked which text and files to use, and after “look at the screen — the same” it asked again: the planner works from your words, the page's address and title and a short summary of the chat, never the page itself or your attachments. That summary held only the last ten messages, tool calls and page observations included, each cut to 300 characters, so the earlier answer that named the post's text and your screenshot file had dropped out of it. It now holds the last ten turns of the conversation without the agent's step-by-step notes, gives the latest answer up to 1,200 characters, and names the files you attached earlier. With the Planner on (Try or Strict), when you have just answered a planner question and the planner wants to ask again, it first takes a second look, told that it cannot see the page while the agent can, and plans the task instead; the plan is then checked against your words. So an answer such as "every five minutes" to a question about a scheduled task, the name of the person to write to, "yes, send it" or opening the conversation it asked you to open is planned. It never takes who a message goes to, an account or an amount from the page, never adds a person or a schedule your request did not carry, and never runs a task you asked to repeat just once instead. With the Planner off, nothing checks a plan against your words, so it does not take that second look. A planner running Qwen 3, QwQ or DeepSeek R1, which is told not to think so that it answers faster, now thinks before any question it would ask you, unless reasoning is turned off for that provider. Chrome and Firefox.
+- Act no longer asks you the same question again when the answer is on the page. Asked to post the same thing on LinkedIn while the X post was open, the planner asked which text and files to use, and after “look at the screen — the same” it asked again: the planner works from your words, the page's address and title and a short summary of the chat, never the page itself or your attachments. That summary held only the last ten messages, tool calls and page observations included, each cut to 300 characters, so the earlier answer that named the post's text and your screenshot file had dropped out of it. It now holds the last ten turns of the conversation without the agent's step-by-step notes, gives the latest answer up to 1,200 characters, and names the files you attached earlier. With the Planner on (Try or Strict), when you have just answered a planner question and the planner wants to ask again, it first takes a second look, told that it cannot see the page while the agent can, and plans the task instead; the plan is then checked against your words. So an answer such as "every five minutes" to a question about a scheduled task, the name of the person to write to, "yes, send it" or opening the conversation it asked you to open is planned. It never takes who a message goes to, an account or an amount from the page, never adds a person or a schedule your request did not carry, and never runs a task you asked to repeat just once instead. With the Planner off, nothing checks a plan against your words, so it takes no second look at your answer. A planner running Qwen 3, QwQ or DeepSeek R1, which is told not to think so that it answers faster, now thinks before it asks you a first question, and with the Planner on also before asking again, unless reasoning is turned off for that provider. Chrome and Firefox.
## [42.3.1] - 2026-10-20
diff --git a/docs/architecture.md b/docs/architecture.md
index cf6651b57..033cced50 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -936,7 +936,7 @@ The recent-history digest (`_buildPlannerHistoryDigest()`) takes the last ten au
A `clarify` from the planner or the intent check gets one second look before the user sees it (`_plannerClarifyRetry()`). A model that was sent `/no_think` (Qwen 3, QwQ, DeepSeek R1) answers again with thinking on and up to 8,192 output tokens (`_plannerCanRethink()`: not when reasoning is off in the provider's settings or extra request body, and not for Qwen3 Instruct-2507 and Coder models, which have no thinking mode). That rethink is the same request and is judged like the first answer.
-In the full planner gate, on the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs), the second look also re-asks: `PLANNER_REASK_GUIDANCE` is added to the system prompt. It says the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. A turn that answers a planner question always has history, so its plan is then checked for alignment with the user's words, answer included, and that check is what reads which person the answer chose, whether it gave a schedule's timing ("every five minutes", not "I'm not sure yet") and whether it approved sending ("yes, send it"). The intent check (Planner off) has no alignment check, so it never re-asks.
+In the full planner gate, on the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs), the second look also re-asks: `PLANNER_REASK_GUIDANCE` is added to the system prompt. It says the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. A turn that answers a planner question always has history, so its plan is then checked for alignment with the user's words, answer included, and that check is what reads which person the answer chose, whether it gave a schedule's timing ("every five minutes", not "I'm not sure yet") and whether it approved sending ("yes, send it"). The intent check (Planner off) has no alignment check, so it takes no second look at an answer at all, neither a re-ask nor a rethink: a retry that read the answer could run a recipient, schedule or submission nothing there checks, so its first answer stands.
An execute plan from a re-ask is also measured against what the user backed, the question they answered as the app stored it and their own words in the task, so that it cannot take what the alignment check could miss (`_plannerSecondLookRejection()`, trace note `planner_second_look_rejected` with the reason). The first answer on that turn is model output like the second look: it can keep the question but never authorizes more. A planner question keeps the people to address and the schedule of the task it asks about (`normalizePlan()` keeps `scheduling` and, for a message it does not send, `draft_recipients` on a `clarify` plan; nothing runs from it), and the stored question records them as `messageTarget` and `schedulingTool`. A question asked because the open conversation could not be pinned records that conversation too, so the reply can be planned once the user has opened it; it is pinned again before anything is sent. The plan may not address a recipient, for a message or a draft, other than the people the answered question addressed (in any order or casing), unless the user's own words in the task name every person (`adds_recipient`; the task binding is built from user turns only, and the open conversation is never taken from the page); plan a schedule of another kind than the answered question carried (`adds_schedule`: a `schedule_resume` pause never becomes a recurring `schedule_task`); or drop a `schedule_task` that the answered question or the first answer carried (`drops_schedule`: the run would do the task once instead). Another question, an answer or a plan to show runs nothing and is not checked. A plan the alignment check has the planner replace is held to the same limits, and when it goes beyond them the user keeps the question (`planner_alignment_repair` in the trace note).
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index e8a363e24..5cc03043b 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -28252,12 +28252,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
if (this._hasPendingSteering(tabId) || this._steeringGateSuperseded(tabId, runOptions)) {
return { proceed: false, steeringSuperseded: true };
}
- // A rethink only. The intent check has no alignment check that could
- // read the user's answer, so it never re-asks (_plannerSecondLookRejection).
- const secondLook = await this._plannerClarifyRetry(tabId, provider, plannerMessages, {
- parseOptions: plannerParseOptions, intentOnly: true,
- portable: plannerPortable, costState, runId, runtimeMode, onUpdate, followUpContext,
- });
+ // A rethink only, and none on a turn that answers a planner question:
+ // the intent check has no alignment check that could read the user's
+ // answer, so it never re-asks, and a retry that read the answer could
+ // run a recipient, schedule or submission nothing checked
+ // (_plannerSecondLookRejection). There the first answer stands.
+ const secondLook = followUpContext.answersPlannerQuestion === true ? null
+ : await this._plannerClarifyRetry(tabId, provider, plannerMessages, {
+ parseOptions: plannerParseOptions, intentOnly: true,
+ portable: plannerPortable, costState, runId, runtimeMode, onUpdate, followUpContext,
+ });
if (this._checkAbort(tabId)) return { proceed: false, message: '[Stopped by user]', reason: 'cancelled' };
if (this._hasPendingSteering(tabId) || this._steeringGateSuperseded(tabId, runOptions)) {
return { proceed: false, steeringSuperseded: true };
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index a5264d558..b1f624af6 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -25279,12 +25279,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
if (this._hasPendingSteering(tabId) || this._steeringGateSuperseded(tabId, runOptions)) {
return { proceed: false, steeringSuperseded: true };
}
- // A rethink only. The intent check has no alignment check that could
- // read the user's answer, so it never re-asks (_plannerSecondLookRejection).
- const secondLook = await this._plannerClarifyRetry(tabId, provider, plannerMessages, {
- parseOptions: plannerParseOptions, intentOnly: true,
- portable: plannerPortable, costState, runId, runtimeMode, onUpdate, followUpContext,
- });
+ // A rethink only, and none on a turn that answers a planner question:
+ // the intent check has no alignment check that could read the user's
+ // answer, so it never re-asks, and a retry that read the answer could
+ // run a recipient, schedule or submission nothing checked
+ // (_plannerSecondLookRejection). There the first answer stands.
+ const secondLook = followUpContext.answersPlannerQuestion === true ? null
+ : await this._plannerClarifyRetry(tabId, provider, plannerMessages, {
+ parseOptions: plannerParseOptions, intentOnly: true,
+ portable: plannerPortable, costState, runId, runtimeMode, onUpdate, followUpContext,
+ });
if (this._checkAbort(tabId)) return { proceed: false, message: '[Stopped by user]', reason: 'cancelled' };
if (this._hasPendingSteering(tabId) || this._steeringGateSuperseded(tabId, runOptions)) {
return { proceed: false, steeringSuperseded: true };
diff --git a/test/planner-clarify-loop.mjs b/test/planner-clarify-loop.mjs
index d5cd17934..95f405bb7 100644
--- a/test/planner-clarify-loop.mjs
+++ b/test/planner-clarify-loop.mjs
@@ -12,7 +12,7 @@
// that must come from the user. That plan may not take a recipient or a
// schedule the user did not back, and the alignment check then reads it
// against the user's words. The intent check, which has no alignment
-// check, never re-asks.
+// check, takes no second look at an answer.
import assert from 'node:assert/strict';
import { test } from 'node:test';
@@ -323,17 +323,18 @@ for (const build of ['chrome', 'firefox']) {
assert.deepEqual(second.options, first.options, 'a model that may think gets the usual options');
});
- test(`${build}: the intent check never re-asks, and a model told not to think only rethinks there`, async () => {
- const followUpContext = { answersPlannerQuestion: true };
- const plain = scriptedAgent({ provider: PLAIN_MODEL, answers: [intentPlan(clarifyPlan()), intentPlan(plannerPlan())] });
- const kept = await intentGate(plain.agent, { followUpContext });
- assert.deepEqual([kept.proceed, kept.requestKind, kept.message], [false, 'clarify', QUESTION], JSON.stringify(kept));
- assert.equal(plain.calls.length, 1, 'no alignment check could read the answer, so no second look');
+ test(`${build}: the intent check takes no second look at an answer, and only rethinks a first question`, async () => {
+ for (const provider of [PLAIN_MODEL, VLLM_QWEN]) {
+ const { agent, calls } = scriptedAgent({ provider, answers: [intentPlan(clarifyPlan()), intentPlan(plannerPlan())] });
+ const kept = await intentGate(agent, { followUpContext: { answersPlannerQuestion: true } });
+ assert.deepEqual([kept.proceed, kept.requestKind, kept.message], [false, 'clarify', QUESTION], `${provider.model}: ${JSON.stringify(kept)}`);
+ assert.equal(calls.length, 1, `${provider.model}: nothing there could read the answer, so the first answer stands`);
+ }
const qwen = scriptedAgent({ answers: [intentPlan(clarifyPlan()), intentPlan(plannerPlan())] });
- const rethought = await intentGate(qwen.agent, { followUpContext });
+ const rethought = await intentGate(qwen.agent, { message: user('Post the same thing on LinkedIn too') });
assert.equal(rethought.requestKind, 'execute', JSON.stringify(rethought));
- assert.equal(qwen.calls.length, 2);
- assert.doesNotMatch(qwen.calls[1].messages[0].content, REASK, 'the same request with thinking on, judged like a first answer');
+ assert.equal(qwen.calls.length, 2, 'a first question is rethought by a model told not to think');
+ assert.doesNotMatch(qwen.calls[1].messages[0].content, REASK, 'the same request with thinking on');
assert.ok(!contentText(qwen.calls[1].messages[1].content).startsWith('/no_think'));
});
@@ -606,11 +607,11 @@ for (const build of ['chrome', 'firefox']) {
});
test(`${build}: the intent recheck gets no re-ask guidance, and a pending correction no second look`, async () => {
- for (const [provider, expectedCalls] of [[PLAIN_MODEL, 1], [VLLM_QWEN, 2]]) {
+ for (const [provider, answered, expectedCalls] of [[PLAIN_MODEL, false, 1], [VLLM_QWEN, false, 2], [VLLM_QWEN, true, 1]]) {
const recheck = scriptedAgent({ provider, answers: [intentPlan(clarifyPlan())] });
- const result = await intentGate(recheck.agent, { runOptions: { plannerIntentRecheckOnly: true }, followUpContext: { answersPlannerQuestion: true } });
+ const result = await intentGate(recheck.agent, { runOptions: { plannerIntentRecheckOnly: true }, followUpContext: { answersPlannerQuestion: answered } });
assert.deepEqual([result.proceed, result.requestKind], [false, 'clarify'], JSON.stringify(result));
- assert.equal(recheck.calls.length, expectedCalls, `${provider.model}: only a no-think model rethinks`);
+ assert.equal(recheck.calls.length, expectedCalls, `${provider.model}, answer turn ${answered}: only a no-think model rethinks, and never an answer`);
assert.ok(recheck.calls.every(call => !REASK.test(call.messages[0].content)), `${provider.model}: no re-ask guidance`);
}