Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) đź§ 

Let the alignment check read a send the answer approved, and check its replacements

Claude5b9caf0d0f28
diff --git a/CHANGELOG.md b/CHANGELOG.md
index da95a179e..8017d99b3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,7 +19,7 @@ This changelog was generated from the repository Git history and release tags. V
- Providers → Decision accepts a local decision server on another machine on your network, such as `http://192.168.1.51:8091`. **Load available models** and **Test** used to stop with "Decision endpoints require HTTPS outside localhost." for any address but this computer's. Private addresses (10.x, 172.16–31.x, 192.168.x, Tailscale's 100.64–127.x, link-local, IPv6 unique-local) and `.local`, `.home.arpa` and `.internal` names now work over plain HTTP; public addresses still need HTTPS. In Chrome, decision requests to such a server go through the same offscreen proxy as chat models on your network. Vision and speech endpoints already accepted these addresses.
- Act can send a photo or video in an Instagram direct message. Instagram's message box keeps its file input hidden, outside any form, deep in a long conversation, where WebBrain's check that an action still targets what the model saw cannot reach it. Every attachment was refused, even with the exact selector WebBrain itself listed for that input, often right after you had confirmed the recipient on WebBrain's card; the model then sent a link to the picture, or said Instagram blocks attachments. With Strict checks off, an upload to the very input WebBrain listed on that page, named by the selector it gave, now goes through with a note to the model, and the recipient checks still apply. A selector WebBrain did not list is still refused, and so is an input the page re-created after the list under the same selector, until WebBrain lists the page's controls again. When clicking "Add Photo or Video" opens a file chooser, WebBrain now names the input with that same selector, and the Instagram site notes say to attach the file to that input instead of clicking the button, and that a link to a picture is not the picture.
- Act can attach images and videos to a post on X again. While the post composer is open at x.com/compose/post, the timeline behind it keeps a second, identical file input, and WebBrain's list of the page's controls gave neither input a selector. After the model's first selector matched both, WebBrain refused every later upload until it saw a selector it could not get, so the run ended at its cost limit with nothing attached. The list now gives every file input a selector that matches only that input, scoped to the open dialog where that is what tells them apart, and marks the dialog's input `inDialog`. The X site notes name the dialog's input, and when no selector can single out an input, WebBrain tells the model to report the upload as blocked instead of retrying.
-- Act no longer asks you the same question again when the answer is on the page. Asked to post the same thing on LinkedIn while the X post was open, the planner asked which text and files to use, and after “look at the screen — the same” it asked again: the planner works from your words, the page's address and title and a short summary of the chat, never the page itself or your attachments. That summary held only the last ten messages, tool calls and page observations included, each cut to 300 characters, so the earlier answer that named the post's text and your screenshot file had dropped out of it. It now holds the last ten turns of the conversation without the agent's step-by-step notes, gives the latest answer up to 1,200 characters, and names the files you attached earlier. When you have just answered a planner question and the planner wants to ask again, it first takes a second look, told that it cannot see the page while the agent can, and plans the task instead. That second look cannot add anything you did not ask for: posting, sending or a schedule your request did not already carry stays a question for you. When a scheduled task should run, who a message goes to, an account or an amount stay questions too until you give them, and it never takes them from the page. An answer such as "every five minutes" to a question about a scheduled task, the name of the person to write to, or opening the conversation it asked you to open is planned, and a task you asked to repeat is never run just once instead. A planner running Qwen 3, QwQ or DeepSeek R1, which is told not to think so that it answers faster, now thinks during that second look and before any question it would ask you, unless reasoning is turned off for that provider. Chrome and Firefox.
+- Act no longer asks you the same question again when the answer is on the page. Asked to post the same thing on LinkedIn while the X post was open, the planner asked which text and files to use, and after “look at the screen — the same” it asked again: the planner works from your words, the page's address and title and a short summary of the chat, never the page itself or your attachments. That summary held only the last ten messages, tool calls and page observations included, each cut to 300 characters, so the earlier answer that named the post's text and your screenshot file had dropped out of it. It now holds the last ten turns of the conversation without the agent's step-by-step notes, gives the latest answer up to 1,200 characters, and names the files you attached earlier. When you have just answered a planner question and the planner wants to ask again, it first takes a second look, told that it cannot see the page while the agent can, and plans the task instead. That second look cannot add anything you did not ask for: posting, sending or a schedule your request did not already carry stays a question for you. When a scheduled task should run, who a message goes to, an account or an amount stay questions too until you give them, and it never takes them from the page. An answer such as "every five minutes" to a question about a scheduled task, the name of the person to write to, opening the conversation it asked you to open or, unless the Planner is off, "yes, send it" is planned, and a task you asked to repeat is never run just once instead. A planner running Qwen 3, QwQ or DeepSeek R1, which is told not to think so that it answers faster, now thinks during that second look and before any question it would ask you, unless reasoning is turned off for that provider. Chrome and Firefox.
## [42.3.1] - 2026-10-20
diff --git a/docs/architecture.md b/docs/architecture.md
index 6ca0d2418..e6f528612 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -934,7 +934,7 @@ Planner calls are traced with `phase: "planner"` when trace recording is enabled
The recent-history digest (`_buildPlannerHistoryDigest()`) takes the last ten authored turns, after app-owned notes, pinned state, tool results and the narration sent with tool calls are filtered out, and spends a 2,000-character budget newest first: an assistant turn keeps up to 1,200 characters and a user turn up to 600, and older turns that no longer fit are left out whole. A user turn names its attachments (`[attached: …]`, at most four names, sanitized like the executor's attachment notice), because the planner never receives attachment bytes.
-A `clarify` from the planner or the intent check gets one second look before the user sees it (`_plannerClarifyRetry()`). A model that was sent `/no_think` (Qwen 3, QwQ, DeepSeek R1) answers again with thinking on and up to 8,192 output tokens (`_plannerCanRethink()`: not when reasoning is off in the provider's settings or extra request body, and not for Qwen3 Instruct-2507 and Coder models, which have no thinking mode). On the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs or the intent recheck), `PLANNER_REASK_GUIDANCE` is added to the system prompt: the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. An execute plan from such a second look may run only what the user backed: the question they answered, as the app stored it, and their own words in the task (`_plannerSecondLookRejection()`, trace note `planner_second_look_rejected` with the reason). The first answer on that turn is model output like the second look, so it can keep the question but never authorizes more. A planner question keeps the submission, the people to address and the schedule of the task it asks about (`normalizePlan()` keeps `scheduling` and, for a message it does not send, `draft_recipients` on a `clarify` plan; nothing runs from it), and the stored question records them as `requiresSubmission`, `messageTarget` and `schedulingTool`. A question asked because the open conversation could not be pinned records that conversation too, so the reply can be planned once the user has opened it; it is pinned again before anything is sent. The plan may not add a submission the answered question did not carry (`adds_submission`); a recipient, for a message or a draft, other than the people the answered question addressed (in any order or casing), unless the user's own words in the task name every person (`adds_recipient`; the task binding is built from user turns only, and the open conversation is never taken from the page); or a schedule of another kind than the answered question carried (`adds_schedule`), so an answer that only gives a scheduled task its timing ("every five minutes") is planned, while a `schedule_resume` pause never becomes a recurring `schedule_task`. It may not drop a `schedule_task` that the answered question or the first answer carried (`drops_schedule`): the intent check has no alignment check that would notice the run doing the task once instead. Another question, an answer or a plan to show runs nothing and is not checked. The trace phase is `planner_rethink` or `planner_reask` (`intent_rethink`, `intent_reask` for the intent check). The second answer is parsed and consistency-checked like the first and then takes the first answer's place in the rest of the gate, which in the planner gate includes the alignment check, recipient pinning and review. A failed, unparseable, inconsistent or rejected second answer, including one stopped by the cost allowance, keeps the first question, and a pending correction gets no second look.
+A `clarify` from the planner or the intent check gets one second look before the user sees it (`_plannerClarifyRetry()`). A model that was sent `/no_think` (Qwen 3, QwQ, DeepSeek R1) answers again with thinking on and up to 8,192 output tokens (`_plannerCanRethink()`: not when reasoning is off in the provider's settings or extra request body, and not for Qwen3 Instruct-2507 and Coder models, which have no thinking mode). On the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs or the intent recheck), `PLANNER_REASK_GUIDANCE` is added to the system prompt: the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. An execute plan from such a second look may run only what the user backed: the question they answered, as the app stored it, and their own words in the task (`_plannerSecondLookRejection()`, trace note `planner_second_look_rejected` with the reason). The first answer on that turn is model output like the second look, so it can keep the question but never authorizes more. A planner question keeps the submission, the people to address and the schedule of the task it asks about (`normalizePlan()` keeps `scheduling` and, for a message it does not send, `draft_recipients` on a `clarify` plan; nothing runs from it), and the stored question records them as `requiresSubmission`, `messageTarget` and `schedulingTool`. A question asked because the open conversation could not be pinned records that conversation too, so the reply can be planned once the user has opened it; it is pinned again before anything is sent. The plan may not add a submission the answered question did not carry (`adds_submission`), except in the full planner gate, where the alignment check then reads the user's words: an answer such as "yes, send it" may itself be the approval, and the intent check, which has no such check, keeps the question; a recipient, for a message or a draft, other than the people the answered question addressed (in any order or casing), unless the user's own words in the task name every person (`adds_recipient`; the task binding is built from user turns only, and the open conversation is never taken from the page); or a schedule of another kind than the answered question carried (`adds_schedule`), so an answer that only gives a scheduled task its timing ("every five minutes") is planned, while a `schedule_resume` pause never becomes a recurring `schedule_task`. It may not drop a `schedule_task` that the answered question or the first answer carried (`drops_schedule`): the intent check has no alignment check that would notice the run doing the task once instead. Another question, an answer or a plan to show runs nothing and is not checked. A plan the alignment check has the planner replace is held to the same limits, and when it goes beyond them the user keeps the question (`planner_alignment_repair` in the trace note). The trace phase is `planner_rethink` or `planner_reask` (`intent_rethink`, `intent_reask` for the intent check). The second answer is parsed and consistency-checked like the first and then takes the first answer's place in the rest of the gate, which in the planner gate includes the alignment check, recipient pinning and review. A failed, unparseable, inconsistent or rejected second answer, including one stopped by the cost allowance, keeps the first question, and a pending correction gets no second look.
LLM-request trace events include privacy-safe prompt provenance: the controlled
prompt variant, system-prompt and aggregate message character counts, message
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 0912135d0..1b8d8d634 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -27251,7 +27251,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// keep the question but never authorizes more. Returns why an execute plan
// is rejected, or null. Another question, an answer or a plan to show runs
// nothing and passes. A rejected plan leaves the question with the user.
- // - adds_submission: a submission the answered question did not carry.
+ // - adds_submission: a submission the answered question did not carry,
+ // unless the plan is then checked for alignment with the user's words
+ // (alignmentChecks, the full planner gate): the answer may itself be the
+ // approval ("yes, send it"), and only that independent check can read it.
+ // The intent check has none, so there the user keeps the question.
// - adds_recipient: a recipient, for a message or a draft (normalizePlan
// keeps a draft's addressees in draft_recipients), other than the people
// the answered question addressed, unless the user's own words name every
@@ -27262,9 +27266,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// - drops_schedule: no schedule_task where the answered question or the
// first answer carried one. The intent check has no alignment check that
// would notice the run doing the task once instead.
- _plannerSecondLookRejection(firstPlan, plan, followUpContext = {}) {
+ _plannerSecondLookRejection(firstPlan, plan, followUpContext = {}, { alignmentChecks = false } = {}) {
if (plan?.request_kind !== 'execute') return null;
- if (plan.requires_submission === true && followUpContext?.answeredQuestionRequiresSubmission !== true) return 'adds_submission';
+ if (plan.requires_submission === true && followUpContext?.answeredQuestionRequiresSubmission !== true
+ && !alignmentChecks) return 'adds_submission';
for (const target of [plan.messaging, plan.draft_recipients]) {
if (!target || this._sameMessageTarget(followUpContext?.answeredQuestionMessageTarget, target)) continue;
if (target.target_kind === 'named'
@@ -27300,7 +27305,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// consistency repair.
async _plannerClarifyRetry(tabId, provider, plannerMessages, {
parseOptions = {}, intentOnly = false, reask = false, portable = false, costState = null, runId = null,
- runtimeMode = 'act', onUpdate = null, followUpContext = {}, firstPlan = null,
+ runtimeMode = 'act', onUpdate = null, followUpContext = {}, firstPlan = null, alignmentChecks = false,
} = {}) {
const rethink = this._plannerCanRethink(provider);
if (!rethink && !reask) return null;
@@ -27331,19 +27336,21 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
await this._tracePlannerAttemptResponse(runId, 0, provider, result, phase, 1, startedAt, chatOptions);
const plan = parsePlanFromContent(result?.content, parseOptions);
if (!plan || this._plannerIntentConsistencyIssue(plan, followUpContext)) return null;
- const rejection = reask ? this._plannerSecondLookRejection(firstPlan, plan, followUpContext) : null;
+ const rejection = reask ? this._plannerSecondLookRejection(firstPlan, plan, followUpContext, { alignmentChecks }) : null;
if (rejection) {
- if (runId) {
- try {
- Promise.resolve(trace.recordNote(runId, 0, 'planner_second_look_rejected', { phase, reason: rejection }))
- .catch(() => {});
- } catch {}
- }
+ this._tracePlannerSecondLookRejection(runId, phase, rejection);
return null;
}
return plan;
}
+ _tracePlannerSecondLookRejection(runId, phase, reason) {
+ if (!runId) return;
+ try {
+ Promise.resolve(trace.recordNote(runId, 0, 'planner_second_look_rejected', { phase, reason })).catch(() => {});
+ } catch {}
+ }
+
_plannerIntentRecheckFallback() {
return {
proceed: true,
@@ -28557,19 +28564,29 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
: this._plannerActContinuation(runOptions, onUpdate, runId, 'invalid_output');
}
hasValidPlannerResponse = true;
+ // History is present on every turn that answers a planner question, so
+ // a plan for it is checked for alignment with the user's words below.
+ const checksAlignment = !!(historyDigest || followUpContext.priorUserTask || followUpContext.scratchpadFacts);
+ // The question a re-ask second look answered for the user. A plan that
+ // later replaces that answer is held to the same limits, or the user
+ // keeps this question (alignment repair below).
+ let reaskFirstPlan = null;
if (plan.request_kind === 'clarify') {
if (this._hasPendingSteering(tabId) || this._steeringGateSuperseded(tabId, runOptions)) {
return { proceed: false, steeringSuperseded: true };
}
const secondLook = await this._plannerClarifyRetry(tabId, provider, plannerMessages, {
parseOptions: plannerParseOptions, reask: followUpContext.answersPlannerQuestion === true, firstPlan: plan,
- portable: plannerPortable, costState, runId, runtimeMode, onUpdate, followUpContext,
+ alignmentChecks: checksAlignment, portable: plannerPortable, costState, runId, runtimeMode, onUpdate, followUpContext,
});
if (this._checkAbort(tabId)) return { proceed: false, message: '[Stopped by user]' };
if (this._hasPendingSteering(tabId) || this._steeringGateSuperseded(tabId, runOptions)) {
return { proceed: false, steeringSuperseded: true };
}
- if (secondLook) plan = secondLook;
+ if (secondLook) {
+ if (followUpContext.answersPlannerQuestion === true) reaskFirstPlan = plan;
+ plan = secondLook;
+ }
}
if (this._plannerIntentUnresolvedConsistencyIssue(plan, consistencyRepairKind, followUpContext)) {
if (runOptions.steeringRevision != null) return this._plannerActContinuation(runOptions, onUpdate, runId, 'inconsistent_intent');
@@ -28644,7 +28661,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// authorize a plan for an earlier task. Check the exact current candidate
// before pinning recipients, execution guards, or review approval.
let alignmentReplanned = false;
- if (historyDigest || followUpContext.priorUserTask || followUpContext.scratchpadFacts) {
+ if (checksAlignment) {
let aligned = false;
let alignmentFailure = null;
let alignmentDenialReason = '';
@@ -28766,6 +28783,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
alignmentFailure = 'invalid_replacement_plan';
break;
}
+ // A replacement for a second look's plan is held to the same
+ // limits, and is checked for alignment next. Otherwise the user
+ // keeps the question that plan answered.
+ const replacementRejection = reaskFirstPlan
+ ? this._plannerSecondLookRejection(reaskFirstPlan, plan, followUpContext, { alignmentChecks: true })
+ : null;
+ if (replacementRejection) {
+ this._tracePlannerSecondLookRejection(runId, 'planner_alignment_repair', replacementRejection);
+ return await routeNonExecutionPlan(reaskFirstPlan);
+ }
alignmentReplanned = true;
// A validated correction can remove execution entirely. Preserve
// its normal terminal route without pinning or approving a plan.
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 364424ae0..a2bf752cb 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -24289,7 +24289,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// keep the question but never authorizes more. Returns why an execute plan
// is rejected, or null. Another question, an answer or a plan to show runs
// nothing and passes. A rejected plan leaves the question with the user.
- // - adds_submission: a submission the answered question did not carry.
+ // - adds_submission: a submission the answered question did not carry,
+ // unless the plan is then checked for alignment with the user's words
+ // (alignmentChecks, the full planner gate): the answer may itself be the
+ // approval ("yes, send it"), and only that independent check can read it.
+ // The intent check has none, so there the user keeps the question.
// - adds_recipient: a recipient, for a message or a draft (normalizePlan
// keeps a draft's addressees in draft_recipients), other than the people
// the answered question addressed, unless the user's own words name every
@@ -24300,9 +24304,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// - drops_schedule: no schedule_task where the answered question or the
// first answer carried one. The intent check has no alignment check that
// would notice the run doing the task once instead.
- _plannerSecondLookRejection(firstPlan, plan, followUpContext = {}) {
+ _plannerSecondLookRejection(firstPlan, plan, followUpContext = {}, { alignmentChecks = false } = {}) {
if (plan?.request_kind !== 'execute') return null;
- if (plan.requires_submission === true && followUpContext?.answeredQuestionRequiresSubmission !== true) return 'adds_submission';
+ if (plan.requires_submission === true && followUpContext?.answeredQuestionRequiresSubmission !== true
+ && !alignmentChecks) return 'adds_submission';
for (const target of [plan.messaging, plan.draft_recipients]) {
if (!target || this._sameMessageTarget(followUpContext?.answeredQuestionMessageTarget, target)) continue;
if (target.target_kind === 'named'
@@ -24338,7 +24343,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// consistency repair.
async _plannerClarifyRetry(tabId, provider, plannerMessages, {
parseOptions = {}, intentOnly = false, reask = false, portable = false, costState = null, runId = null,
- runtimeMode = 'act', onUpdate = null, followUpContext = {}, firstPlan = null,
+ runtimeMode = 'act', onUpdate = null, followUpContext = {}, firstPlan = null, alignmentChecks = false,
} = {}) {
const rethink = this._plannerCanRethink(provider);
if (!rethink && !reask) return null;
@@ -24369,19 +24374,21 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
await this._tracePlannerAttemptResponse(runId, 0, provider, result, phase, 1, startedAt, chatOptions);
const plan = parsePlanFromContent(result?.content, parseOptions);
if (!plan || this._plannerIntentConsistencyIssue(plan, followUpContext)) return null;
- const rejection = reask ? this._plannerSecondLookRejection(firstPlan, plan, followUpContext) : null;
+ const rejection = reask ? this._plannerSecondLookRejection(firstPlan, plan, followUpContext, { alignmentChecks }) : null;
if (rejection) {
- if (runId) {
- try {
- Promise.resolve(trace.recordNote(runId, 0, 'planner_second_look_rejected', { phase, reason: rejection }))
- .catch(() => {});
- } catch {}
- }
+ this._tracePlannerSecondLookRejection(runId, phase, rejection);
return null;
}
return plan;
}
+ _tracePlannerSecondLookRejection(runId, phase, reason) {
+ if (!runId) return;
+ try {
+ Promise.resolve(trace.recordNote(runId, 0, 'planner_second_look_rejected', { phase, reason })).catch(() => {});
+ } catch {}
+ }
+
_plannerIntentRecheckFallback() {
return {
proceed: true,
@@ -25580,19 +25587,29 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
: this._plannerActContinuation(runOptions, onUpdate, runId, 'invalid_output');
}
hasValidPlannerResponse = true;
+ // History is present on every turn that answers a planner question, so
+ // a plan for it is checked for alignment with the user's words below.
+ const checksAlignment = !!(historyDigest || followUpContext.priorUserTask || followUpContext.scratchpadFacts);
+ // The question a re-ask second look answered for the user. A plan that
+ // later replaces that answer is held to the same limits, or the user
+ // keeps this question (alignment repair below).
+ let reaskFirstPlan = null;
if (plan.request_kind === 'clarify') {
if (this._hasPendingSteering(tabId) || this._steeringGateSuperseded(tabId, runOptions)) {
return { proceed: false, steeringSuperseded: true };
}
const secondLook = await this._plannerClarifyRetry(tabId, provider, plannerMessages, {
parseOptions: plannerParseOptions, reask: followUpContext.answersPlannerQuestion === true, firstPlan: plan,
- portable: plannerPortable, costState, runId, runtimeMode, onUpdate, followUpContext,
+ alignmentChecks: checksAlignment, portable: plannerPortable, costState, runId, runtimeMode, onUpdate, followUpContext,
});
if (this._checkAbort(tabId)) return { proceed: false, message: '[Stopped by user]' };
if (this._hasPendingSteering(tabId) || this._steeringGateSuperseded(tabId, runOptions)) {
return { proceed: false, steeringSuperseded: true };
}
- if (secondLook) plan = secondLook;
+ if (secondLook) {
+ if (followUpContext.answersPlannerQuestion === true) reaskFirstPlan = plan;
+ plan = secondLook;
+ }
}
if (this._plannerIntentUnresolvedConsistencyIssue(plan, consistencyRepairKind, followUpContext)) {
if (runOptions.steeringRevision != null) return this._plannerActContinuation(runOptions, onUpdate, runId, 'inconsistent_intent');
@@ -25667,7 +25684,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// authorize a plan for an earlier task. Check the exact current candidate
// before pinning recipients, execution guards, or review approval.
let alignmentReplanned = false;
- if (historyDigest || followUpContext.priorUserTask || followUpContext.scratchpadFacts) {
+ if (checksAlignment) {
let aligned = false;
let alignmentFailure = null;
let alignmentDenialReason = '';
@@ -25789,6 +25806,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
alignmentFailure = 'invalid_replacement_plan';
break;
}
+ // A replacement for a second look's plan is held to the same
+ // limits, and is checked for alignment next. Otherwise the user
+ // keeps the question that plan answered.
+ const replacementRejection = reaskFirstPlan
+ ? this._plannerSecondLookRejection(reaskFirstPlan, plan, followUpContext, { alignmentChecks: true })
+ : null;
+ if (replacementRejection) {
+ this._tracePlannerSecondLookRejection(runId, 'planner_alignment_repair', replacementRejection);
+ return await routeNonExecutionPlan(reaskFirstPlan);
+ }
alignmentReplanned = true;
// A validated correction can remove execution entirely. Preserve
// its normal terminal route without pinning or approving a plan.
diff --git a/test/planner-clarify-loop.mjs b/test/planner-clarify-loop.mjs
index d7c908b4b..3a12dcef5 100644
--- a/test/planner-clarify-loop.mjs
+++ b/test/planner-clarify-loop.mjs
@@ -108,8 +108,9 @@ for (const build of ['chrome', 'firefox']) {
// A planner that answers from a script and records what it was sent. An
// Error in the script is thrown instead of answered. The plan alignment
- // check, which runs when there is conversation history, approves.
- function scriptedAgent({ provider = VLLM_QWEN, answers, planning = 'try', review = 'never' }) {
+ // check, which runs when there is conversation history, approves unless
+ // `alignment` lists its verdicts in order.
+ function scriptedAgent({ provider = VLLM_QWEN, answers, planning = 'try', review = 'never', alignment = [] }) {
const agent = newAgent(provider);
agent.setPlanBeforeActMode(planning);
agent.setPlanReviewSettings({ mode: review });
@@ -119,7 +120,9 @@ for (const build of ['chrome', 'firefox']) {
if (meta?.generationName === 'planner_alignment') {
const request = JSON.parse(contentText(messages.at(-1).content));
alignmentChecks.push(request);
- return { content: JSON.stringify({ task_binding: request.task_binding, aligned: true, reason: 'The plan does what the user asked.' }) };
+ const aligned = alignment.length ? alignment.shift() : true;
+ return { content: JSON.stringify({ task_binding: request.task_binding, aligned,
+ reason: aligned ? 'The plan does what the user asked.' : 'The plan does something the user did not ask for.' }) };
}
calls.push({ name: meta?.generationName, messages, options });
const next = answers.length > 1 ? answers.shift() : answers[0];
@@ -462,6 +465,10 @@ for (const build of ['chrome', 'firefox']) {
assert.equal(reason(plannerPlan(), { ...ASKED, answeredQuestionSchedulingTool: 'schedule_task' }), 'drops_schedule');
assert.equal(reason(plannerPlan(), ASKED, clarifyPlan({ scheduling: OPEN_SCHEDULE })), 'drops_schedule', 'the first answer may keep the question');
assert.equal(reason(plannerPlan({ scheduling: EVERY_FIVE }), { ...ASKED, answeredQuestionSchedulingTool: 'schedule_task' }), null);
+ assert.equal(agent._plannerSecondLookRejection(parse(clarifyPlan()), parse(plannerPlan()), {}, { alignmentChecks: true }), null,
+ 'the alignment check reads a send the answer approved');
+ assert.equal(agent._plannerSecondLookRejection(parse(clarifyPlan()), parse(plannerPlan({ messaging: ALI })), {}, { alignmentChecks: true }),
+ 'adds_recipient', 'but it never stands in for a recipient');
for (const request_kind of ['clarify', 'respond', 'plan_only']) {
const second = request_kind === 'clarify'
? clarifyPlan({ summary: SECOND_QUESTION, messaging: ALI, scheduling: EVERY_FIVE })
@@ -495,6 +502,44 @@ for (const build of ['chrome', 'firefox']) {
assert.equal(agent._userTextNamesRecipients('', people('Ali')), false);
});
+ test(`${build}: a send the answer approves is planned where the alignment check reads it, and stays a question in the intent check`, async () => {
+ const SEND = 'Should I send it now, or keep it as a draft?';
+ const followUpContext = { answersPlannerQuestion: true };
+ const planner = scriptedAgent({ provider: PLAIN_MODEL, answers: [clarifyPlan({ requires_submission: false, summary: SEND }), plannerPlan()] });
+ const planned = await fullGate(planner.agent, { message: user('yes, send it'), digest: `User: Reply to Ali\nAssistant: ${SEND}`, followUpContext });
+ assert.equal(planned.proceed, true, JSON.stringify(planned));
+ assert.equal(planned.requiresSubmission, true);
+ assert.equal(planner.alignmentChecks.length, 1, 'the alignment check read the approval');
+ assert.equal(planner.alignmentChecks[0].proposed_plan.requires_submission, true);
+ const intent = scriptedAgent({ provider: PLAIN_MODEL, answers: [intentPlan(clarifyPlan({ requires_submission: false, summary: SEND })), intentPlan(plannerPlan())] });
+ const kept = await intentGate(intent.agent, { message: user('yes, send it'), followUpContext });
+ assert.deepEqual([kept.proceed, kept.requestKind, kept.message], [false, 'clarify', SEND], JSON.stringify(kept));
+ });
+
+ test(`${build}: a plan the alignment check has replaced stays within what the user backed`, async () => {
+ const digest = `User: Post the same thing on LinkedIn too\nAssistant: ${QUESTION}`;
+ const followUpContext = { answersPlannerQuestion: true, ...ASKED };
+ for (const [label, replacement, accepted] of [
+ ['a replacement that adds a recipient', plannerPlan({ messaging: ALI }), false],
+ ['a replacement that adds a recurring schedule', plannerPlan({ scheduling: EVERY_FIVE }), false],
+ ['a replacement within what the user backed', plannerPlan({ summary: 'Publish the same post on LinkedIn and check that it is live.' }), true],
+ ]) {
+ const { agent, calls, alignmentChecks } = scriptedAgent({
+ provider: PLAIN_MODEL, answers: [clarifyPlan(), plannerPlan(), replacement], alignment: [false, true],
+ });
+ const result = await fullGate(agent, { digest, followUpContext });
+ assert.equal(calls.length, 3, `${label}: the first answer, the second look and the replacement`);
+ if (accepted) {
+ assert.equal(result.proceed, true, `${label}: ${JSON.stringify(result)}`);
+ assert.equal(alignmentChecks.length, 2, `${label}: the replacement is checked too`);
+ assert.match(result.approvedScratchpadText, /check that it is live/);
+ } else {
+ assert.deepEqual([result.proceed, result.requestKind, result.message], [false, 'clarify', QUESTION], `${label}: ${JSON.stringify(result)}`);
+ assert.equal(alignmentChecks.length, 1, label);
+ }
+ }
+ });
+
test(`${build}: only a model with a thinking mode it may use rethinks`, () => {
const agent = newAgent();
const withConfig = (config, model = VLLM_QWEN.model) => ({ ...VLLM_QWEN, model, config: { ...VLLM_QWEN.config, model, ...config } });