Open-source AI browser agent for Chrome and Firefox (monorepo) đź§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Keep the conversation a pinning question was about
Claudeb1e6dce497f4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6c3592aff..da95a179e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -19,7 +19,7 @@ This changelog was generated from the repository Git history and release tags. V- Providers → Decision accepts a local decision server on another machine on your network, such as `http://192.168.1.51:8091`. **Load available models** and **Test** used to stop with "Decision endpoints require HTTPS outside localhost." for any address but this computer's. Private addresses (10.x, 172.16–31.x, 192.168.x, Tailscale's 100.64–127.x, link-local, IPv6 unique-local) and `.local`, `.home.arpa` and `.internal` names now work over plain HTTP; public addresses still need HTTPS. In Chrome, decision requests to such a server go through the same offscreen proxy as chat models on your network. Vision and speech endpoints already accepted these addresses.- Act can send a photo or video in an Instagram direct message. Instagram's message box keeps its file input hidden, outside any form, deep in a long conversation, where WebBrain's check that an action still targets what the model saw cannot reach it. Every attachment was refused, even with the exact selector WebBrain itself listed for that input, often right after you had confirmed the recipient on WebBrain's card; the model then sent a link to the picture, or said Instagram blocks attachments. With Strict checks off, an upload to the very input WebBrain listed on that page, named by the selector it gave, now goes through with a note to the model, and the recipient checks still apply. A selector WebBrain did not list is still refused, and so is an input the page re-created after the list under the same selector, until WebBrain lists the page's controls again. When clicking "Add Photo or Video" opens a file chooser, WebBrain now names the input with that same selector, and the Instagram site notes say to attach the file to that input instead of clicking the button, and that a link to a picture is not the picture.- Act can attach images and videos to a post on X again. While the post composer is open at x.com/compose/post, the timeline behind it keeps a second, identical file input, and WebBrain's list of the page's controls gave neither input a selector. After the model's first selector matched both, WebBrain refused every later upload until it saw a selector it could not get, so the run ended at its cost limit with nothing attached. The list now gives every file input a selector that matches only that input, scoped to the open dialog where that is what tells them apart, and marks the dialog's input `inDialog`. The X site notes name the dialog's input, and when no selector can single out an input, WebBrain tells the model to report the upload as blocked instead of retrying.-- Act no longer asks you the same question again when the answer is on the page. Asked to post the same thing on LinkedIn while the X post was open, the planner asked which text and files to use, and after “look at the screen — the same” it asked again: the planner works from your words, the page's address and title and a short summary of the chat, never the page itself or your attachments. That summary held only the last ten messages, tool calls and page observations included, each cut to 300 characters, so the earlier answer that named the post's text and your screenshot file had dropped out of it. It now holds the last ten turns of the conversation without the agent's step-by-step notes, gives the latest answer up to 1,200 characters, and names the files you attached earlier. When you have just answered a planner question and the planner wants to ask again, it first takes a second look, told that it cannot see the page while the agent can, and plans the task instead. That second look cannot add anything you did not ask for: posting, sending or a schedule your request did not already carry stays a question for you. When a scheduled task should run, who a message goes to, an account or an amount stay questions too until you give them, and it never takes them from the page: an answer such as "every five minutes" to a question about a scheduled task, or the name of the person to write to, is planned, and a task you asked to repeat is never run just once instead. A planner running Qwen 3, QwQ or DeepSeek R1, which is told not to think so that it answers faster, now thinks during that second look and before any question it would ask you, unless reasoning is turned off for that provider. Chrome and Firefox.+- Act no longer asks you the same question again when the answer is on the page. Asked to post the same thing on LinkedIn while the X post was open, the planner asked which text and files to use, and after “look at the screen — the same” it asked again: the planner works from your words, the page's address and title and a short summary of the chat, never the page itself or your attachments. That summary held only the last ten messages, tool calls and page observations included, each cut to 300 characters, so the earlier answer that named the post's text and your screenshot file had dropped out of it. It now holds the last ten turns of the conversation without the agent's step-by-step notes, gives the latest answer up to 1,200 characters, and names the files you attached earlier. When you have just answered a planner question and the planner wants to ask again, it first takes a second look, told that it cannot see the page while the agent can, and plans the task instead. That second look cannot add anything you did not ask for: posting, sending or a schedule your request did not already carry stays a question for you. When a scheduled task should run, who a message goes to, an account or an amount stay questions too until you give them, and it never takes them from the page. An answer such as "every five minutes" to a question about a scheduled task, the name of the person to write to, or opening the conversation it asked you to open is planned, and a task you asked to repeat is never run just once instead. A planner running Qwen 3, QwQ or DeepSeek R1, which is told not to think so that it answers faster, now thinks during that second look and before any question it would ask you, unless reasoning is turned off for that provider. Chrome and Firefox.## [42.3.1] - 2026-10-20diff --git a/docs/architecture.md b/docs/architecture.mdindex 84a319696..9a20eaf8a 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -934,7 +934,7 @@ Planner calls are traced with `phase: "planner"` when trace recording is enabledThe recent-history digest (`_buildPlannerHistoryDigest()`) takes the last ten authored turns, after app-owned notes, pinned state, tool results and the narration sent with tool calls are filtered out, and spends a 2,000-character budget newest first: an assistant turn keeps up to 1,200 characters and a user turn up to 600, and older turns that no longer fit are left out whole. A user turn names its attachments (`[attached: …]`, at most four names, sanitized like the executor's attachment notice), because the planner never receives attachment bytes.-A `clarify` from the planner or the intent check gets one second look before the user sees it (`_plannerClarifyRetry()`). A model that was sent `/no_think` (Qwen 3, QwQ, DeepSeek R1) answers again with thinking on and up to 8,192 output tokens (`_plannerCanRethink()`: not when reasoning is off in the provider's settings or extra request body, and not for Qwen3 Instruct-2507 and Coder models, which have no thinking mode). On the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs or the intent recheck), `PLANNER_REASK_GUIDANCE` is added to the system prompt: the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. Such a second look is measured against what the task already carried, in its first answer on that turn and in the question the user answered (`_plannerSecondLookRejection()`, trace note `planner_second_look_rejected` with the reason). A planner question keeps the submission, the people to address and the schedule of the task it asks about (`normalizePlan()` keeps `scheduling` and, for a message it does not send, `draft_recipients` on a `clarify` plan; nothing runs from it), and the stored question records the target as `messageTarget` and the schedule's tool as `schedulingTool`. The second look may not add a submission neither of them carried (`adds_submission`); a recipient, for a message or a draft, that neither of them named, unless the user's own words in the task (the task binding, built from user turns only) name every person (`adds_recipient`; the open conversation is never taken from the page); or a schedule of a kind neither of them carried (`adds_schedule`), so an answer that only gives a scheduled task its timing ("every five minutes") is planned, while a `schedule_resume` pause never becomes a recurring `schedule_task`. An execute plan may not drop the `schedule_task` the task carried (`drops_schedule`): the intent check has no alignment check that would notice the run doing the task once instead. The trace phase is `planner_rethink` or `planner_reask` (`intent_rethink`, `intent_reask` for the intent check). The second answer is parsed and consistency-checked like the first and then takes the first answer's place in the rest of the gate, which in the planner gate includes the alignment check, recipient pinning and review. A failed, unparseable, inconsistent or rejected second answer, including one stopped by the cost allowance, keeps the first question, and a pending correction gets no second look.+A `clarify` from the planner or the intent check gets one second look before the user sees it (`_plannerClarifyRetry()`). A model that was sent `/no_think` (Qwen 3, QwQ, DeepSeek R1) answers again with thinking on and up to 8,192 output tokens (`_plannerCanRethink()`: not when reasoning is off in the provider's settings or extra request body, and not for Qwen3 Instruct-2507 and Coder models, which have no thinking mode). On the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs or the intent recheck), `PLANNER_REASK_GUIDANCE` is added to the system prompt: the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. Such a second look is measured against what the task already carried, in its first answer on that turn and in the question the user answered (`_plannerSecondLookRejection()`, trace note `planner_second_look_rejected` with the reason). A planner question keeps the submission, the people to address and the schedule of the task it asks about (`normalizePlan()` keeps `scheduling` and, for a message it does not send, `draft_recipients` on a `clarify` plan; nothing runs from it), and the stored question records the target as `messageTarget` and the schedule's tool as `schedulingTool`. A question asked because the open conversation could not be pinned records that conversation too, so the reply can be planned once the user has opened it; it is pinned again before anything is sent. The second look may not add a submission neither of them carried (`adds_submission`); a recipient, for a message or a draft, that neither of them named, unless the user's own words in the task (the task binding, built from user turns only) name every person (`adds_recipient`; the open conversation is never taken from the page); or a schedule of a kind neither of them carried (`adds_schedule`), so an answer that only gives a scheduled task its timing ("every five minutes") is planned, while a `schedule_resume` pause never becomes a recurring `schedule_task`. An execute plan may not drop the `schedule_task` the task carried (`drops_schedule`): the intent check has no alignment check that would notice the run doing the task once instead. The trace phase is `planner_rethink` or `planner_reask` (`intent_rethink`, `intent_reask` for the intent check). The second answer is parsed and consistency-checked like the first and then takes the first answer's place in the rest of the gate, which in the planner gate includes the alignment check, recipient pinning and review. A failed, unparseable, inconsistent or rejected second answer, including one stopped by the cost allowance, keeps the first question, and a pending correction gets no second look.LLM-request trace events include privacy-safe prompt provenance: the controlledprompt variant, system-prompt and aggregate message character counts, messagediff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 7da8e69bc..437608b21 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -28297,6 +28297,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout drequiresStateChange: false,requiresSubmission: true,schedulingTool: plan.scheduling?.tool || null,+ // The open conversation could not be pinned yet. Once the user has+ // opened it, a second look may address it again, and it is pinned+ // again before anything is sent (_plannerSecondLookRejection).+ messageTarget: plan.messaging,};}plan.messaging = messagingPin.target;@@ -28318,6 +28322,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout drequiresStateChange: false,requiresSubmission: false,schedulingTool: plan.scheduling?.tool || null,+ // The same for the conversation a draft is written into.+ messageTarget: plan.draft_recipients,};}plan.draft_recipients = draftPin.target;@@ -28807,6 +28813,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout drequiresStateChange: false,requiresSubmission: true,schedulingTool: plan.scheduling?.tool || null,+ // The open conversation could not be pinned yet. Once the user has+ // opened it, a second look may address it again, and it is pinned+ // again before anything is sent (_plannerSecondLookRejection).+ messageTarget: plan.messaging,};}plan.messaging = messagingPin.target;@@ -28828,6 +28838,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout drequiresStateChange: false,requiresSubmission: false,schedulingTool: plan.scheduling?.tool || null,+ // The same for the conversation a draft is written into.+ messageTarget: plan.draft_recipients,};}plan.draft_recipients = draftPin.target;diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 743b79a81..4dd6a9012 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -25324,6 +25324,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout drequiresStateChange: false,requiresSubmission: true,schedulingTool: plan.scheduling?.tool || null,+ // The open conversation could not be pinned yet. Once the user has+ // opened it, a second look may address it again, and it is pinned+ // again before anything is sent (_plannerSecondLookRejection).+ messageTarget: plan.messaging,};}plan.messaging = messagingPin.target;@@ -25345,6 +25349,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout drequiresStateChange: false,requiresSubmission: false,schedulingTool: plan.scheduling?.tool || null,+ // The same for the conversation a draft is written into.+ messageTarget: plan.draft_recipients,};}plan.draft_recipients = draftPin.target;@@ -25830,6 +25836,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout drequiresStateChange: false,requiresSubmission: true,schedulingTool: plan.scheduling?.tool || null,+ // The open conversation could not be pinned yet. Once the user has+ // opened it, a second look may address it again, and it is pinned+ // again before anything is sent (_plannerSecondLookRejection).+ messageTarget: plan.messaging,};}plan.messaging = messagingPin.target;@@ -25851,6 +25861,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout drequiresStateChange: false,requiresSubmission: false,schedulingTool: plan.scheduling?.tool || null,+ // The same for the conversation a draft is written into.+ messageTarget: plan.draft_recipients,};}plan.draft_recipients = draftPin.target;diff --git a/test/planner-clarify-loop.mjs b/test/planner-clarify-loop.mjsindex 5cbd74a5e..780f3dee7 100644--- a/test/planner-clarify-loop.mjs+++ b/test/planner-clarify-loop.mjs@@ -608,17 +608,49 @@ for (const build of ['chrome', 'firefox']) {}});- test(`${build}: a recipient question about a scheduled task records the schedule`, async () => {- const scheduledMessage = plannerPlan({ messaging: { target_kind: 'active_conversation', recipients: [] }, scheduling: EVERY_FIVE });+ test(`${build}: a question about a conversation that could not be pinned records it and the schedule`, async () => {+ const cases = [+ ['a scheduled message', plannerPlan({ messaging: OPEN_CHAT, scheduling: EVERY_FIVE }), true, 'schedule_task'],+ ['a message', plannerPlan({ messaging: OPEN_CHAT }), true, null],+ ['a draft', plannerPlan({ requires_submission: false, messaging: OPEN_CHAT }), false, null],+ ];for (const gate of ['planner', 'intent']) {- for (const [label, plan, scheduled] of [['scheduled', scheduledMessage, true], ['unscheduled', plannerPlan({ messaging: scheduledMessage.messaging }), false]]) {+ for (const [label, plan, sends, schedulingTool] of cases) {const { agent } = scriptedAgent({ provider: PLAIN_MODEL, answers: [gate === 'planner' ? plan : intentPlan(plan)] });- agent._pinActiveConversationMessagingTarget = async () => ({ ok: false, error: 'Open the conversation first.' });+ agent._pinActiveConversationMessagingTarget = async (_tabId, target) => (target+ ? { ok: false, error: 'Open the conversation first.' } : { ok: true, target: null });const result = gate === 'planner' ? await fullGate(agent) : await intentGate(agent);- assert.deepEqual([result.reason, result.plannerClarification, result.schedulingTool],- ['active_recipient_unverified', true, scheduled ? 'schedule_task' : null],- `${gate} ${label}: ${JSON.stringify(result)}`);+ assert.deepEqual([result.reason, result.plannerClarification, result.requiresSubmission, result.schedulingTool, result.messageTarget],+ ['active_recipient_unverified', true, sends, schedulingTool, OPEN_CHAT], `${gate} ${label}: ${JSON.stringify(result)}`);}}});++ test(`${build}: once the user opens the conversation a question asked for, the reply is planned and pinned again`, async () => {+ const tabId = 41;+ const reply = plannerPlan({ messaging: OPEN_CHAT, summary: 'Reply in the open conversation that I will be late.' });+ const pinned = { target_kind: 'named', recipients: [{ identity: 'Bob', role: 'to' }] };+ const { agent, calls } = scriptedAgent({ provider: PLAIN_MODEL, answers: [reply, clarifyPlan({ summary: 'Which conversation?' }), reply] });+ let open = false;+ const pins = [];+ agent._pinActiveConversationMessagingTarget = async (_tabId, target) => {+ if (target) pins.push(open);+ if (!target) return { ok: true, target: null };+ return open ? { ok: true, target: pinned } : { ok: false, error: 'Open the conversation first.' };+ };+ const messages = [{ role: 'system', content: 'sys' }];+ agent.conversations.set(tabId, messages);+ agent._persistSubmittedTurn = async () => {};+ agent._persist = () => {};+ const asked = await agent._maybeRunPlannerGate(tabId, messages, user('Reply in this chat that I will be late'), () => {}, 'act', null, null, TAB, {});+ assert.deepEqual([asked.proceed, asked.message], [false, 'Open the conversation first.'], JSON.stringify(asked));+ assert.deepEqual(messages.at(-1).webbrainPlannerClarification?.messageTarget, OPEN_CHAT, 'the question records the conversation');+ open = true;+ const gate = await agent._maybeRunPlannerGate(tabId, messages, user('It is open now'), () => {}, 'act', null, null, TAB, {});+ assert.equal(calls.length, 3);+ assert.match(calls[2].messages[0].content, REASK, 'the answer got a second look');+ assert.equal(gate.proceed, true, JSON.stringify(gate));+ assert.deepEqual(gate.messaging, pinned, 'the conversation is pinned again before anything is sent');+ assert.deepEqual(pins, [false, true]);+ });}