Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Let a second look run only what the user backed

Claudeb86021528f08
diff --git a/docs/architecture.md b/docs/architecture.md
index 9a20eaf8a..6ca0d2418 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -934,7 +934,7 @@ Planner calls are traced with `phase: "planner"` when trace recording is enabled
The recent-history digest (`_buildPlannerHistoryDigest()`) takes the last ten authored turns, after app-owned notes, pinned state, tool results and the narration sent with tool calls are filtered out, and spends a 2,000-character budget newest first: an assistant turn keeps up to 1,200 characters and a user turn up to 600, and older turns that no longer fit are left out whole. A user turn names its attachments (`[attached: …]`, at most four names, sanitized like the executor's attachment notice), because the planner never receives attachment bytes.
-A `clarify` from the planner or the intent check gets one second look before the user sees it (`_plannerClarifyRetry()`). A model that was sent `/no_think` (Qwen 3, QwQ, DeepSeek R1) answers again with thinking on and up to 8,192 output tokens (`_plannerCanRethink()`: not when reasoning is off in the provider's settings or extra request body, and not for Qwen3 Instruct-2507 and Coder models, which have no thinking mode). On the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs or the intent recheck), `PLANNER_REASK_GUIDANCE` is added to the system prompt: the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. Such a second look is measured against what the task already carried, in its first answer on that turn and in the question the user answered (`_plannerSecondLookRejection()`, trace note `planner_second_look_rejected` with the reason). A planner question keeps the submission, the people to address and the schedule of the task it asks about (`normalizePlan()` keeps `scheduling` and, for a message it does not send, `draft_recipients` on a `clarify` plan; nothing runs from it), and the stored question records the target as `messageTarget` and the schedule's tool as `schedulingTool`. A question asked because the open conversation could not be pinned records that conversation too, so the reply can be planned once the user has opened it; it is pinned again before anything is sent. The second look may not add a submission neither of them carried (`adds_submission`); a recipient, for a message or a draft, that neither of them named, unless the user's own words in the task (the task binding, built from user turns only) name every person (`adds_recipient`; the open conversation is never taken from the page); or a schedule of a kind neither of them carried (`adds_schedule`), so an answer that only gives a scheduled task its timing ("every five minutes") is planned, while a `schedule_resume` pause never becomes a recurring `schedule_task`. An execute plan may not drop the `schedule_task` the task carried (`drops_schedule`): the intent check has no alignment check that would notice the run doing the task once instead. The trace phase is `planner_rethink` or `planner_reask` (`intent_rethink`, `intent_reask` for the intent check). The second answer is parsed and consistency-checked like the first and then takes the first answer's place in the rest of the gate, which in the planner gate includes the alignment check, recipient pinning and review. A failed, unparseable, inconsistent or rejected second answer, including one stopped by the cost allowance, keeps the first question, and a pending correction gets no second look.
+A `clarify` from the planner or the intent check gets one second look before the user sees it (`_plannerClarifyRetry()`). A model that was sent `/no_think` (Qwen 3, QwQ, DeepSeek R1) answers again with thinking on and up to 8,192 output tokens (`_plannerCanRethink()`: not when reasoning is off in the provider's settings or extra request body, and not for Qwen3 Instruct-2507 and Coder models, which have no thinking mode). On the turn that answers the planner's previous question (`_turnAnswersPlannerQuestion()`, which reads an answer as the task binding does; not in scheduled runs or the intent recheck), `PLANNER_REASK_GUIDANCE` is added to the system prompt: the planner cannot see the page and the executor can, so it may ask again only about schedule timing or a calendar recurrence, a recipient, an account, an amount, a credential, whether to publish, send or submit at all, or conflicting answers, none of which it may take from page content. An execute plan from such a second look may run only what the user backed: the question they answered, as the app stored it, and their own words in the task (`_plannerSecondLookRejection()`, trace note `planner_second_look_rejected` with the reason). The first answer on that turn is model output like the second look, so it can keep the question but never authorizes more. A planner question keeps the submission, the people to address and the schedule of the task it asks about (`normalizePlan()` keeps `scheduling` and, for a message it does not send, `draft_recipients` on a `clarify` plan; nothing runs from it), and the stored question records them as `requiresSubmission`, `messageTarget` and `schedulingTool`. A question asked because the open conversation could not be pinned records that conversation too, so the reply can be planned once the user has opened it; it is pinned again before anything is sent. The plan may not add a submission the answered question did not carry (`adds_submission`); a recipient, for a message or a draft, other than the people the answered question addressed (in any order or casing), unless the user's own words in the task name every person (`adds_recipient`; the task binding is built from user turns only, and the open conversation is never taken from the page); or a schedule of another kind than the answered question carried (`adds_schedule`), so an answer that only gives a scheduled task its timing ("every five minutes") is planned, while a `schedule_resume` pause never becomes a recurring `schedule_task`. It may not drop a `schedule_task` that the answered question or the first answer carried (`drops_schedule`): the intent check has no alignment check that would notice the run doing the task once instead. Another question, an answer or a plan to show runs nothing and is not checked. The trace phase is `planner_rethink` or `planner_reask` (`intent_rethink`, `intent_reask` for the intent check). The second answer is parsed and consistency-checked like the first and then takes the first answer's place in the rest of the gate, which in the planner gate includes the alignment check, recipient pinning and review. A failed, unparseable, inconsistent or rejected second answer, including one stopped by the cost allowance, keeps the first question, and a pending correction gets no second look.
LLM-request trace events include privacy-safe prompt provenance: the controlled
prompt variant, system-prompt and aggregate message character counts, message
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 437608b21..0912135d0 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -27168,19 +27168,23 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// For the planner's second look (_plannerClarifyRetry): whether the user
// turn at `index` answers the planner's previous question, and what the
- // task carried (_plannerSecondLookRejection): the schedule tool and message
- // target of the question it answers, and the user's own words in the task.
- // A correction or a scheduled run answers no question.
+ // user backed (_plannerSecondLookRejection): the submission, message target
+ // and schedule tool that question stored, and the user's own words in the
+ // task. A correction or a scheduled run answers no question.
_plannerQuestionAnswerContext(messages, index, runOptions = {}) {
const answersPlannerQuestion = runOptions?.steeringRevision == null
&& runOptions?.scheduledRun !== true && runOptions?.scheduledResume !== true
&& this._turnAnswersPlannerQuestion(messages, index);
if (!answersPlannerQuestion) {
- return { answersPlannerQuestion: false, answeredQuestionSchedulingTool: null, answeredQuestionMessageTarget: null, taskUserText: '' };
+ return {
+ answersPlannerQuestion: false, answeredQuestionRequiresSubmission: false,
+ answeredQuestionSchedulingTool: null, answeredQuestionMessageTarget: null, taskUserText: '',
+ };
}
const question = messages[this._plannerClarificationIndexBefore(messages, index)]?.webbrainPlannerClarification || {};
return {
answersPlannerQuestion: true,
+ answeredQuestionRequiresSubmission: question.requiresSubmission === true,
answeredQuestionSchedulingTool: question.schedulingTool || null,
answeredQuestionMessageTarget: normalizeMessageTarget(question.messageTarget),
// The task binding is built from user turns only: the request and each
@@ -27210,6 +27214,19 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
});
}
+ // Whether two message targets address the same people: the same kind and
+ // the same roles and identities, in any order and casing.
+ _sameMessageTarget(a, b) {
+ const key = value => {
+ const target = normalizeMessageTarget(value);
+ return target && JSON.stringify([target.target_kind, target.recipients
+ .map(recipient => `${recipient.role}:${normalizeRecipientIdentity(recipient.identity)}`)
+ .sort()]);
+ };
+ const left = key(a);
+ return !!left && left === key(b);
+ }
+
// Whether a second look may think: only a model planning tells not to
// think (/no_think). Reasoning turned off in the provider's settings or in
// its extra request body stays off (mergeProviderRequestBody merges both
@@ -27226,41 +27243,38 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
.reasoningEffort !== 'off';
}
- // What a second look told not to re-ask may plan, measured against what
- // the user's task already carried: the first answer on this turn (a
+ // What a second look told not to re-ask may run, measured against what the
+ // user backed: the question they just answered, as the app stored it (a
// question keeps the submission, the people to address and the schedule of
- // the task it asks about) and the question the user just answered. Returns
- // why the plan is rejected, or null. A rejected plan leaves the question
- // with the user.
- // - adds_submission: a submission neither of them carried.
+ // the task it asks about), and their own words in the task. The first
+ // answer on this turn is model output like the second look itself: it may
+ // keep the question but never authorizes more. Returns why an execute plan
+ // is rejected, or null. Another question, an answer or a plan to show runs
+ // nothing and passes. A rejected plan leaves the question with the user.
+ // - adds_submission: a submission the answered question did not carry.
// - adds_recipient: a recipient, for a message or a draft (normalizePlan
- // keeps a draft's addressees in draft_recipients), that neither of them
- // named, unless the user's own words in the task name every person. The
- // open conversation is never taken from the page on a second look.
- // - adds_schedule: a schedule of a kind neither of them carried. The user's
- // answer is often the timing of a schedule the task carried ("every five
- // minutes"), but a schedule_resume pause never becomes a recurring
- // schedule_task.
- // - drops_schedule: an execute plan without the schedule_task the task
- // carried. The intent check has no alignment check that would notice the
- // run doing the task once instead.
+ // keeps a draft's addressees in draft_recipients), other than the people
+ // the answered question addressed, unless the user's own words name every
+ // person. The open conversation is never taken from the page.
+ // - adds_schedule: a schedule of another kind than the answered question
+ // carried. The user's answer is often its timing ("every five minutes"),
+ // but a schedule_resume pause never becomes a recurring schedule_task.
+ // - drops_schedule: no schedule_task where the answered question or the
+ // first answer carried one. The intent check has no alignment check that
+ // would notice the run doing the task once instead.
_plannerSecondLookRejection(firstPlan, plan, followUpContext = {}) {
- const submissionCarried = firstPlan?.requires_submission === true
- || followUpContext?.plannerClarification?.requiresSubmission === true;
- if (plan?.requires_submission === true && !submissionCarried) return 'adds_submission';
- const carriedTargets = [firstPlan?.messaging, firstPlan?.draft_recipients, followUpContext?.answeredQuestionMessageTarget]
- .filter(Boolean)
- .map(target => JSON.stringify(target));
- for (const target of [plan?.messaging, plan?.draft_recipients]) {
- if (!target || carriedTargets.includes(JSON.stringify(target))) continue;
+ if (plan?.request_kind !== 'execute') return null;
+ if (plan.requires_submission === true && followUpContext?.answeredQuestionRequiresSubmission !== true) return 'adds_submission';
+ for (const target of [plan.messaging, plan.draft_recipients]) {
+ if (!target || this._sameMessageTarget(followUpContext?.answeredQuestionMessageTarget, target)) continue;
if (target.target_kind === 'named'
&& this._userTextNamesRecipients(followUpContext?.taskUserText, target.recipients)) continue;
return 'adds_recipient';
}
- const carriedSchedules = [firstPlan?.scheduling?.tool, followUpContext?.answeredQuestionSchedulingTool];
- if (plan?.scheduling && !carriedSchedules.includes(plan.scheduling.tool)) return 'adds_schedule';
- if (plan?.request_kind === 'execute' && carriedSchedules.includes('schedule_task')
- && plan.scheduling?.tool !== 'schedule_task') return 'drops_schedule';
+ const carriedSchedule = followUpContext?.answeredQuestionSchedulingTool || null;
+ if (plan.scheduling && plan.scheduling.tool !== carriedSchedule) return 'adds_schedule';
+ if (plan.scheduling?.tool !== 'schedule_task'
+ && [carriedSchedule, firstPlan?.scheduling?.tool].includes('schedule_task')) return 'drops_schedule';
return null;
}
@@ -27276,9 +27290,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// - No re-asking: when the user has just answered the planner's previous
// question, PLANNER_REASK_GUIDANCE reminds the planner that the executor
// can see what it cannot, and leaves a question only for values that have
- // to come from the user. Its plan may not add a submission, a recipient
- // or a schedule the task did not carry, or drop a recurring schedule it
- // did (_plannerSecondLookRejection).
+ // to come from the user. An execute plan from it may not add a
+ // submission, a recipient or a schedule the user did not back, or drop a
+ // recurring schedule the task carried (_plannerSecondLookRejection).
// The answer is parsed and checked like the first one, and an execute plan
// then goes through alignment, recipient pinning and review like any other.
// Returns null to keep the first answer: nothing to retry, or the retry
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 4dd6a9012..364424ae0 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -24206,19 +24206,23 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// For the planner's second look (_plannerClarifyRetry): whether the user
// turn at `index` answers the planner's previous question, and what the
- // task carried (_plannerSecondLookRejection): the schedule tool and message
- // target of the question it answers, and the user's own words in the task.
- // A correction or a scheduled run answers no question.
+ // user backed (_plannerSecondLookRejection): the submission, message target
+ // and schedule tool that question stored, and the user's own words in the
+ // task. A correction or a scheduled run answers no question.
_plannerQuestionAnswerContext(messages, index, runOptions = {}) {
const answersPlannerQuestion = runOptions?.steeringRevision == null
&& runOptions?.scheduledRun !== true && runOptions?.scheduledResume !== true
&& this._turnAnswersPlannerQuestion(messages, index);
if (!answersPlannerQuestion) {
- return { answersPlannerQuestion: false, answeredQuestionSchedulingTool: null, answeredQuestionMessageTarget: null, taskUserText: '' };
+ return {
+ answersPlannerQuestion: false, answeredQuestionRequiresSubmission: false,
+ answeredQuestionSchedulingTool: null, answeredQuestionMessageTarget: null, taskUserText: '',
+ };
}
const question = messages[this._plannerClarificationIndexBefore(messages, index)]?.webbrainPlannerClarification || {};
return {
answersPlannerQuestion: true,
+ answeredQuestionRequiresSubmission: question.requiresSubmission === true,
answeredQuestionSchedulingTool: question.schedulingTool || null,
answeredQuestionMessageTarget: normalizeMessageTarget(question.messageTarget),
// The task binding is built from user turns only: the request and each
@@ -24248,6 +24252,19 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
});
}
+ // Whether two message targets address the same people: the same kind and
+ // the same roles and identities, in any order and casing.
+ _sameMessageTarget(a, b) {
+ const key = value => {
+ const target = normalizeMessageTarget(value);
+ return target && JSON.stringify([target.target_kind, target.recipients
+ .map(recipient => `${recipient.role}:${normalizeRecipientIdentity(recipient.identity)}`)
+ .sort()]);
+ };
+ const left = key(a);
+ return !!left && left === key(b);
+ }
+
// Whether a second look may think: only a model planning tells not to
// think (/no_think). Reasoning turned off in the provider's settings or in
// its extra request body stays off (mergeProviderRequestBody merges both
@@ -24264,41 +24281,38 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
.reasoningEffort !== 'off';
}
- // What a second look told not to re-ask may plan, measured against what
- // the user's task already carried: the first answer on this turn (a
+ // What a second look told not to re-ask may run, measured against what the
+ // user backed: the question they just answered, as the app stored it (a
// question keeps the submission, the people to address and the schedule of
- // the task it asks about) and the question the user just answered. Returns
- // why the plan is rejected, or null. A rejected plan leaves the question
- // with the user.
- // - adds_submission: a submission neither of them carried.
+ // the task it asks about), and their own words in the task. The first
+ // answer on this turn is model output like the second look itself: it may
+ // keep the question but never authorizes more. Returns why an execute plan
+ // is rejected, or null. Another question, an answer or a plan to show runs
+ // nothing and passes. A rejected plan leaves the question with the user.
+ // - adds_submission: a submission the answered question did not carry.
// - adds_recipient: a recipient, for a message or a draft (normalizePlan
- // keeps a draft's addressees in draft_recipients), that neither of them
- // named, unless the user's own words in the task name every person. The
- // open conversation is never taken from the page on a second look.
- // - adds_schedule: a schedule of a kind neither of them carried. The user's
- // answer is often the timing of a schedule the task carried ("every five
- // minutes"), but a schedule_resume pause never becomes a recurring
- // schedule_task.
- // - drops_schedule: an execute plan without the schedule_task the task
- // carried. The intent check has no alignment check that would notice the
- // run doing the task once instead.
+ // keeps a draft's addressees in draft_recipients), other than the people
+ // the answered question addressed, unless the user's own words name every
+ // person. The open conversation is never taken from the page.
+ // - adds_schedule: a schedule of another kind than the answered question
+ // carried. The user's answer is often its timing ("every five minutes"),
+ // but a schedule_resume pause never becomes a recurring schedule_task.
+ // - drops_schedule: no schedule_task where the answered question or the
+ // first answer carried one. The intent check has no alignment check that
+ // would notice the run doing the task once instead.
_plannerSecondLookRejection(firstPlan, plan, followUpContext = {}) {
- const submissionCarried = firstPlan?.requires_submission === true
- || followUpContext?.plannerClarification?.requiresSubmission === true;
- if (plan?.requires_submission === true && !submissionCarried) return 'adds_submission';
- const carriedTargets = [firstPlan?.messaging, firstPlan?.draft_recipients, followUpContext?.answeredQuestionMessageTarget]
- .filter(Boolean)
- .map(target => JSON.stringify(target));
- for (const target of [plan?.messaging, plan?.draft_recipients]) {
- if (!target || carriedTargets.includes(JSON.stringify(target))) continue;
+ if (plan?.request_kind !== 'execute') return null;
+ if (plan.requires_submission === true && followUpContext?.answeredQuestionRequiresSubmission !== true) return 'adds_submission';
+ for (const target of [plan.messaging, plan.draft_recipients]) {
+ if (!target || this._sameMessageTarget(followUpContext?.answeredQuestionMessageTarget, target)) continue;
if (target.target_kind === 'named'
&& this._userTextNamesRecipients(followUpContext?.taskUserText, target.recipients)) continue;
return 'adds_recipient';
}
- const carriedSchedules = [firstPlan?.scheduling?.tool, followUpContext?.answeredQuestionSchedulingTool];
- if (plan?.scheduling && !carriedSchedules.includes(plan.scheduling.tool)) return 'adds_schedule';
- if (plan?.request_kind === 'execute' && carriedSchedules.includes('schedule_task')
- && plan.scheduling?.tool !== 'schedule_task') return 'drops_schedule';
+ const carriedSchedule = followUpContext?.answeredQuestionSchedulingTool || null;
+ if (plan.scheduling && plan.scheduling.tool !== carriedSchedule) return 'adds_schedule';
+ if (plan.scheduling?.tool !== 'schedule_task'
+ && [carriedSchedule, firstPlan?.scheduling?.tool].includes('schedule_task')) return 'drops_schedule';
return null;
}
@@ -24314,9 +24328,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// - No re-asking: when the user has just answered the planner's previous
// question, PLANNER_REASK_GUIDANCE reminds the planner that the executor
// can see what it cannot, and leaves a question only for values that have
- // to come from the user. Its plan may not add a submission, a recipient
- // or a schedule the task did not carry, or drop a recurring schedule it
- // did (_plannerSecondLookRejection).
+ // to come from the user. An execute plan from it may not add a
+ // submission, a recipient or a schedule the user did not back, or drop a
+ // recurring schedule the task carried (_plannerSecondLookRejection).
// The answer is parsed and checked like the first one, and an execute plan
// then goes through alignment, recipient pinning and review like any other.
// Returns null to keep the first answer: nothing to retry, or the retry
diff --git a/test/planner-clarify-loop.mjs b/test/planner-clarify-loop.mjs
index 780f3dee7..d7c908b4b 100644
--- a/test/planner-clarify-loop.mjs
+++ b/test/planner-clarify-loop.mjs
@@ -94,6 +94,9 @@ const HOW_OFTEN = 'How often should I check this page for a new release?';
const PAUSE = { tool: 'schedule_resume', hint: 'Wait until the deploy finishes, then continue.' };
const ALI = { target_kind: 'named', recipients: [{ identity: 'Ali Veli', role: 'to' }] };
const OPEN_CHAT = { target_kind: 'active_conversation', recipients: [] };
+const OPEN_CHAT_CC = { target_kind: 'active_conversation', recipients: [{ identity: 'Ali Veli', role: 'cc' }, { identity: 'Ayse Yilmaz', role: 'cc' }] };
+// The question the user answered carried a submission, as the app stored it.
+const ASKED = { answeredQuestionRequiresSubmission: true };
// The user's words in a task binding: the request and each answer.
const userWords = (request, ...answers) => `User task clarified by the latest answer: ${JSON.stringify({ request, answers })}`;
@@ -258,12 +261,16 @@ for (const build of ['chrome', 'firefox']) {
const invalid = asked({ schedulingTool: 'schedule_everything', messageTarget: { target_kind: 'everyone', recipients: [] } })[2].webbrainPlannerClarification;
assert.deepEqual([invalid.schedulingTool, invalid.messageTarget], [null, null], 'only a known tool and a valid target');
const context = agent._plannerQuestionAnswerContext(scheduled, 3);
- assert.deepEqual([context.answersPlannerQuestion, context.answeredQuestionSchedulingTool, context.answeredQuestionMessageTarget],
- [true, 'schedule_task', ALI]);
+ assert.deepEqual([context.answersPlannerQuestion, context.answeredQuestionRequiresSubmission, context.answeredQuestionSchedulingTool,
+ context.answeredQuestionMessageTarget], [true, false, 'schedule_task', ALI]);
+ assert.equal(agent._plannerQuestionAnswerContext(asked({ requiresSubmission: true }), 3).answeredQuestionRequiresSubmission, true);
assert.match(context.taskUserText, /Check this page for a new release on a schedule[\s\S]*every five minutes/, 'the user\'s words: request and answer');
assert.doesNotMatch(context.taskUserText, /How often/, 'never the question');
assert.deepEqual(agent._plannerQuestionAnswerContext(asked({}), 3).answeredQuestionSchedulingTool, null);
- const none = { answersPlannerQuestion: false, answeredQuestionSchedulingTool: null, answeredQuestionMessageTarget: null, taskUserText: '' };
+ const none = {
+ answersPlannerQuestion: false, answeredQuestionRequiresSubmission: false,
+ answeredQuestionSchedulingTool: null, answeredQuestionMessageTarget: null, taskUserText: '',
+ };
for (const runOptions of [{ steeringRevision: 1 }, { scheduledRun: true }, { scheduledResume: true }]) {
assert.deepEqual(agent._plannerQuestionAnswerContext(scheduled, 3, runOptions), none, JSON.stringify(runOptions));
}
@@ -297,7 +304,7 @@ for (const build of ['chrome', 'firefox']) {
for (const gate of ['planner', 'intent']) {
const answers = gate === 'planner' ? [clarifyPlan(), plannerPlan()] : [intentPlan(clarifyPlan()), intentPlan(plannerPlan())];
const { agent, calls } = scriptedAgent({ provider: PLAIN_MODEL, answers });
- const followUpContext = { answersPlannerQuestion: true };
+ const followUpContext = { answersPlannerQuestion: true, ...ASKED };
const result = gate === 'planner' ? await fullGate(agent, { followUpContext }) : await intentGate(agent, { followUpContext });
assert.equal(result.proceed, true, `${gate}: ${JSON.stringify(result)}`);
assert.equal(result.requestKind, 'execute', gate);
@@ -347,7 +354,8 @@ for (const build of ['chrome', 'firefox']) {
test(`${build}: a no-think model thinks during its second look, and a new plan runs`, async () => {
for (const answered of [false, true]) {
const { agent, calls } = scriptedAgent({ answers: [clarifyPlan(), plannerPlan()] });
- const result = await fullGate(agent, { message: user('Post the same thing on LinkedIn too'), followUpContext: { answersPlannerQuestion: answered } });
+ const result = await fullGate(agent, { message: user('Post the same thing on LinkedIn too'),
+ followUpContext: { answersPlannerQuestion: answered, answeredQuestionRequiresSubmission: answered } });
assert.equal(result.proceed, true, JSON.stringify(result));
assert.equal(result.requestKind, 'execute');
assert.equal(calls.length, 2);
@@ -368,7 +376,7 @@ for (const build of ['chrome', 'firefox']) {
await fullGate(first.agent);
assert.equal(first.calls.length, 1, 'no second look that could not think anyway');
const answered = scriptedAgent({ provider, answers: [clarifyPlan(), plannerPlan()] });
- const result = await fullGate(answered.agent, { followUpContext: { answersPlannerQuestion: true } });
+ const result = await fullGate(answered.agent, { followUpContext: { answersPlannerQuestion: true, ...ASKED } });
assert.equal(result.requestKind, 'execute');
const second = answered.calls[1];
assert.match(second.messages[0].content, REASK);
@@ -376,46 +384,50 @@ for (const build of ['chrome', 'firefox']) {
assert.equal(second.options.extraBody?.chat_template_kwargs?.enable_thinking, false);
});
- test(`${build}: a second look told not to re-ask may not add a submission, a recipient or a schedule the task did not carry`, async () => {
+ test(`${build}: a second look told not to re-ask runs only what the user backed`, async () => {
const named = ALI;
+ const task = { tool: 'schedule_task' };
+ // [label, first answer on this turn, second look, what the user backed, accepted]
const cases = [
['a submission nobody asked for', clarifyPlan({ requires_submission: false }), plannerPlan(), {}, false],
- ['a submission the answered question carried', clarifyPlan({ requires_submission: false }), plannerPlan(),
- { plannerClarification: { requiresSubmission: true, pageUrl: TWEET_URL } }, true],
- ['a submission the first answer carried', clarifyPlan(), plannerPlan(), {}, true],
- ['a recipient the first answer did not name', clarifyPlan(), plannerPlan({ messaging: named }), {}, false],
- ['the recipient the first answer named', clarifyPlan({ messaging: named }), plannerPlan({ messaging: named }), {}, true],
+ ['a submission only the first answer on this turn carried', clarifyPlan(), plannerPlan(), {}, false],
+ ['a submission the answered question carried', clarifyPlan({ requires_submission: false }), plannerPlan(), ASKED, true],
+ ['a recipient only the first answer on this turn named', clarifyPlan({ messaging: named }), plannerPlan({ messaging: named }), ASKED, false],
+ ['the recipient the answered question named', clarifyPlan(), plannerPlan({ messaging: named }), { ...ASKED, answeredQuestionMessageTarget: named }, true],
+ ['the people the answered question named, in another order and casing', clarifyPlan(),
+ plannerPlan({ messaging: { target_kind: 'active_conversation', recipients: [{ identity: 'AYSE YILMAZ', role: 'cc' }, { identity: 'ali veli', role: 'cc' }] } }),
+ { ...ASKED, answeredQuestionMessageTarget: OPEN_CHAT_CC }, true],
+ ['someone else in the conversation the answered question addressed', clarifyPlan(),
+ plannerPlan({ messaging: { target_kind: 'active_conversation', recipients: [{ identity: 'Ali Veli', role: 'cc' }, { identity: 'Bob', role: 'cc' }] } }),
+ { ...ASKED, answeredQuestionMessageTarget: OPEN_CHAT_CC }, false],
+ ['the open conversation where the answered question named a person', clarifyPlan(), plannerPlan({ messaging: OPEN_CHAT }),
+ { ...ASKED, answeredQuestionMessageTarget: named }, false],
['a draft into the open conversation', clarifyPlan({ requires_submission: false }),
- plannerPlan({ requires_submission: false, messaging: { target_kind: 'active_conversation', recipients: [] } }), {}, false],
- ['a draft for a recipient the first answer did not name', clarifyPlan({ requires_submission: false }),
+ plannerPlan({ requires_submission: false, messaging: OPEN_CHAT }), {}, false],
+ ['a draft for a recipient only the first answer on this turn named', clarifyPlan({ requires_submission: false, messaging: named }),
plannerPlan({ requires_submission: false, messaging: named }), {}, false],
- ['a draft for the recipient the first answer named', clarifyPlan({ messaging: named }),
- plannerPlan({ requires_submission: false, messaging: named }), {}, true],
- ['a draft for the recipient of the draft the first answer asked about', clarifyPlan({ requires_submission: false, messaging: named }),
- plannerPlan({ requires_submission: false, messaging: named }), {}, true],
- ['a recipient the answered question named', clarifyPlan(), plannerPlan({ messaging: named }),
- { answeredQuestionMessageTarget: named }, true],
+ ['a draft for the recipient the answered question named', clarifyPlan({ requires_submission: false }),
+ plannerPlan({ requires_submission: false, messaging: named }), { answeredQuestionMessageTarget: named }, true],
['a recipient the user named in the answer', clarifyPlan(), plannerPlan({ messaging: named }),
- { taskUserText: userWords('Send the report to my accountant', 'It is Ali Veli') }, true],
+ { ...ASKED, taskUserText: userWords('Send the report to my accountant', 'It is Ali Veli') }, true],
['a recipient only the page could have named', clarifyPlan(), plannerPlan({ messaging: named }),
- { taskUserText: userWords('Send the report to my accountant', 'the one on the screen') }, false],
+ { ...ASKED, taskUserText: userWords('Send the report to my accountant', 'the one on the screen') }, false],
['the open conversation, whatever the user wrote', clarifyPlan({ requires_submission: false }),
plannerPlan({ requires_submission: false, messaging: OPEN_CHAT }),
{ taskUserText: userWords('Draft a reply', 'use the current chat') }, false],
['the open conversation the answered question addressed', clarifyPlan({ requires_submission: false }),
plannerPlan({ requires_submission: false, messaging: OPEN_CHAT }), { answeredQuestionMessageTarget: OPEN_CHAT }, true],
- ['a schedule the task did not carry', clarifyPlan(), plannerPlan({ scheduling: { tool: 'schedule_task', hint: 'Every 43200 minutes.' } }), {}, false],
- ['a schedule the first answer carried', clarifyPlan({ scheduling: OPEN_SCHEDULE }), plannerPlan({ scheduling: EVERY_FIVE }), {}, true],
+ ['a schedule the task did not carry', clarifyPlan(), plannerPlan({ scheduling: { ...task, hint: 'Every 43200 minutes.' } }), ASKED, false],
+ ['a schedule only the first answer on this turn carried', clarifyPlan({ scheduling: OPEN_SCHEDULE }), plannerPlan({ scheduling: EVERY_FIVE }), ASKED, false],
['a schedule the answered question carried', clarifyPlan(), plannerPlan({ scheduling: EVERY_FIVE }),
- { answeredQuestionSchedulingTool: 'schedule_task' }, true],
- ['a recurring schedule where the first answer carried only a pause', clarifyPlan({ scheduling: PAUSE }),
- plannerPlan({ scheduling: EVERY_FIVE }), {}, false],
+ { ...ASKED, answeredQuestionSchedulingTool: 'schedule_task' }, true],
['a recurring schedule where the answered question carried only a pause', clarifyPlan(),
- plannerPlan({ scheduling: EVERY_FIVE }), { answeredQuestionSchedulingTool: 'schedule_resume' }, false],
- ['a run without the recurring schedule the first answer carried', clarifyPlan({ scheduling: OPEN_SCHEDULE }), plannerPlan(), {}, false],
+ plannerPlan({ scheduling: EVERY_FIVE }), { ...ASKED, answeredQuestionSchedulingTool: 'schedule_resume' }, false],
['a run without the recurring schedule the answered question carried', clarifyPlan(), plannerPlan(),
- { answeredQuestionSchedulingTool: 'schedule_task' }, false],
- ['a run without the pause the first answer carried', clarifyPlan({ scheduling: PAUSE }), plannerPlan(), {}, true],
+ { ...ASKED, answeredQuestionSchedulingTool: 'schedule_task' }, false],
+ ['a run without the recurring schedule the first answer on this turn carried', clarifyPlan({ scheduling: OPEN_SCHEDULE }), plannerPlan(), ASKED, false],
+ ['a run without the pause the answered question carried', clarifyPlan(), plannerPlan(),
+ { ...ASKED, answeredQuestionSchedulingTool: 'schedule_resume' }, true],
];
for (const [label, first, second, extraContext, accepted] of cases) {
for (const gate of ['planner', 'intent']) {
@@ -438,20 +450,37 @@ for (const build of ['chrome', 'firefox']) {
assert.equal(result.requiredSchedulingTool, 'schedule_task');
});
- test(`${build}: a rejected second look names why`, async () => {
+ test(`${build}: a rejected second look names why, and only a plan that runs is checked`, async () => {
const { parsePlanFromContent } = await import(`../src/${build}/src/agent/planner.js`);
const parse = plan => parsePlanFromContent(JSON.stringify(plan), { requireIntent: true, locale: 'en' });
const agent = newAgent();
- const reason = (first, second, context = {}) => agent._plannerSecondLookRejection(parse(first), parse(second), context);
- assert.equal(reason(clarifyPlan({ requires_submission: false }), plannerPlan()), 'adds_submission');
- assert.equal(reason(clarifyPlan(), plannerPlan({ messaging: ALI })), 'adds_recipient');
- assert.equal(reason(clarifyPlan({ requires_submission: false }), plannerPlan({ requires_submission: false, messaging: ALI })), 'adds_recipient',
- 'a draft\'s recipient too');
- assert.equal(reason(clarifyPlan(), plannerPlan({ scheduling: EVERY_FIVE })), 'adds_schedule');
- assert.equal(reason(clarifyPlan({ scheduling: OPEN_SCHEDULE }), plannerPlan()), 'drops_schedule');
- assert.equal(reason(clarifyPlan({ scheduling: OPEN_SCHEDULE }), clarifyPlan({ summary: SECOND_QUESTION })), null,
- 'another question runs nothing, so it drops nothing');
- assert.equal(reason(clarifyPlan({ scheduling: OPEN_SCHEDULE }), plannerPlan({ scheduling: EVERY_FIVE })), null);
+ const reason = (second, context = {}, first = clarifyPlan()) => agent._plannerSecondLookRejection(parse(first), parse(second), context);
+ assert.equal(reason(plannerPlan()), 'adds_submission', 'a first answer on this turn authorizes nothing');
+ assert.equal(reason(plannerPlan({ messaging: ALI }), ASKED, clarifyPlan({ messaging: ALI })), 'adds_recipient');
+ assert.equal(reason(plannerPlan({ requires_submission: false, messaging: ALI })), 'adds_recipient', 'a draft\'s recipient too');
+ assert.equal(reason(plannerPlan({ scheduling: EVERY_FIVE }), ASKED, clarifyPlan({ scheduling: OPEN_SCHEDULE })), 'adds_schedule');
+ assert.equal(reason(plannerPlan(), { ...ASKED, answeredQuestionSchedulingTool: 'schedule_task' }), 'drops_schedule');
+ assert.equal(reason(plannerPlan(), ASKED, clarifyPlan({ scheduling: OPEN_SCHEDULE })), 'drops_schedule', 'the first answer may keep the question');
+ assert.equal(reason(plannerPlan({ scheduling: EVERY_FIVE }), { ...ASKED, answeredQuestionSchedulingTool: 'schedule_task' }), null);
+ for (const request_kind of ['clarify', 'respond', 'plan_only']) {
+ const second = request_kind === 'clarify'
+ ? clarifyPlan({ summary: SECOND_QUESTION, messaging: ALI, scheduling: EVERY_FIVE })
+ : plannerPlan({ request_kind, read_scope: 'none', requires_state_change: false, requires_submission: false });
+ assert.equal(reason(second), null, `${request_kind} runs nothing`);
+ }
+ });
+
+ test(`${build}: the same people in another order or casing are the same target`, () => {
+ const agent = newAgent();
+ const reordered = { target_kind: 'active_conversation', recipients: [{ identity: 'AYSE YILMAZ', role: 'cc' }, { identity: 'ali veli', role: 'cc' }] };
+ assert.equal(agent._sameMessageTarget(OPEN_CHAT_CC, reordered), true);
+ assert.equal(agent._sameMessageTarget(OPEN_CHAT, { target_kind: 'active_conversation', recipients: [] }), true);
+ assert.equal(agent._sameMessageTarget(OPEN_CHAT_CC, { ...reordered, target_kind: 'named' }), false, 'another kind');
+ assert.equal(agent._sameMessageTarget(OPEN_CHAT_CC, { ...reordered, recipients: [reordered.recipients[0], { identity: 'ali veli', role: 'to' }] }), false,
+ 'another role');
+ assert.equal(agent._sameMessageTarget(OPEN_CHAT_CC, { ...reordered, recipients: [reordered.recipients[0]] }), false, 'fewer people');
+ assert.equal(agent._sameMessageTarget(null, OPEN_CHAT), false);
+ assert.equal(agent._sameMessageTarget(null, null), false);
});
test(`${build}: only whole names in the user's own words name a recipient`, () => {
@@ -500,7 +529,7 @@ for (const build of ['chrome', 'firefox']) {
const { agent, calls } = scriptedAgent({
answers: [new Error('400 Bad Request: response_format json_schema is not supported'), clarifyPlan(), plannerPlan()],
});
- const result = await fullGate(agent, { followUpContext: { answersPlannerQuestion: true } });
+ const result = await fullGate(agent, { followUpContext: { answersPlannerQuestion: true, ...ASKED } });
assert.equal(result.requestKind, 'execute', JSON.stringify(result));
assert.equal(calls.length, 3);
assert.equal(calls[1].options.extraBody, undefined, 'fixture: the repair went without request controls');