Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Keep the planner for turns with attachments or a continued run

Emre Sokullud543a83b5cb6
diff --git a/docs/architecture.md b/docs/architecture.md
index ae3594d76..d02f1005d 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -352,7 +352,7 @@ Raw page-source access through `read_page_source` is available only in Dev mode.
Manual action-mode runs (Act or Dev) call the active provider once before the tool loop with `planner.js`'s structured JSON prompt. Off uses the compact intent schema; Try and Strict use the full plan schema. Unset storage defaults to Try, while explicit Off remains Off. The planner sees the user task, sanitized URL/title, and a short recent-history digest; page context is wrapped as untrusted data and image blocks are dropped.
-In Try, a turn that automatic mode routed to Act or Dev with a sure `plan_need: direct` answer also uses the compact intent schema (`_routedDirectTask`), and the trace records a `planner_skipped` note with `reason: 'direct_task'`. `direct` is an authorizing choice, so it needs the 0.85 threshold, and its description leaves out risky work (payments, purchases, bookings, deletion, bulk changes, credentials, messages to several people) as well as work with several goals, many items, several sites or pages, conditions, schedules or facts to carry across many steps: those answer `plan`. Strict, plan review set to Always, a steering revision, and any action turn without a sure `direct` answer, such as one in a mode the user chose, keep the full planner. So does a request the decision model could not read whole (over 6,000 characters, or over the decision state's 12,000-byte budget): a `direct` answer about part of it is not used.
+In Try, a turn that automatic mode routed to Act or Dev with a sure `plan_need: direct` answer also uses the compact intent schema (`_routedDirectTask`), and the trace records a `planner_skipped` note with `reason: 'direct_task'`. `direct` is an authorizing choice, so it needs the 0.85 threshold, and its description leaves out risky work (payments, purchases, bookings, deletion, bulk changes, credentials, messages to several people) as well as work with several goals, many items, several sites or pages, conditions, schedules or facts to carry across many steps: those answer `plan`. Strict, plan review set to Always, a steering revision, and any action turn without a sure `direct` answer, such as one in a mode the user chose, keep the full planner. So does a turn whose task the decision model could not read whole. That covers a request over 6,000 characters, an earlier task cut at 600, a decision state over its 12,000-byte budget, and text that redaction changed (a credential line, a URL query). It also covers a turn with attached files and the continuation of a read-only run, whose task lies partly outside the question. A `direct` answer is not used for these turns.
Planner latency grows with the tokens the model writes, so both prompts ask for little. `summary`, `steps` and `risks` are English; `localized` is `null` for an English `wbLocale`, and for another locale it is required only when the user reads the plan: `clarify`, `plan_only`, and an `execute` plan shown for review, whose rule the full prompt states from the current review settings (`plannerReviewRule()`). An English `clarify` uses its summary as the question. When a plan the user reads arrives without its copy (`normalizePlan()` marks it `localized.missing`), `_localizedPlanForDisplay()` translates the display fields in one small request (trace phase `planner_localize`) before the review card or the plan-only answer is shown. Execution keeps the English plan. A translation that fails or leaves out a step or risk is not used, so the user sees the English text rather than a mix. A non-English `clarify` without its copy is still rejected and repaired like other invalid planner output. `memory: null` turns off the scratchpad and the progress ledger together, and `normalizePlan()` reads it as a declared `disabled` ledger, so no progress-classifier call follows. The compact intent check writes no steps for `execute` (its plan-only steps stay), and full-plan steps are one short sentence each.
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index d07574358..b2ebdf649 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -27360,7 +27360,18 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}
this._autoModeEscalationTabs.add(tabId);
this._autoModeRoutePendingTabs.add(tabId);
- return { mode: 'ask', runOptions: { ...runOptions, autoModeRouting: { previousMode, ...(continuation ? { continuedTask } : {}) } } };
+ const hasAttachments = Array.isArray(attachments) && attachments.length > 0;
+ return { mode: 'ask', runOptions: { ...runOptions, autoModeRouting: { previousMode, ...(continuation ? { continuedTask } : {}),
+ ...(hasAttachments ? { hasAttachments } : {}) } } };
+ }
+
+ // plan_need's answer as the planner gate may use it. "direct" counts only
+ // when the question held everything the task is about: attached files (a
+ // CSV of contacts to import) and a continued run's earlier work are not in
+ // it, so those turns keep the planner.
+ _routePlanSignal(choice, { hasAttachments = false, continuation = false } = {}) {
+ const plan = autoModePlanForChoice(choice);
+ return plan === 'direct' && (hasAttachments || continuation) ? null : plan;
}
// The user's own words of the task a continuation resumes: its request and
@@ -27466,7 +27477,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
routeMs: Number.isFinite(routed.latencyMs) ? routed.latencyMs : null,
waitedMs: Date.now() - waitStartedAt,
// plan_need for an action turn (_routedDirectTask).
- ...(mode !== 'ask' && autoModePlanForChoice(routed.plan) ? { plan: routed.plan } : {}),
+ ...(() => {
+ const plan = mode === 'ask' ? null : this._routePlanSignal(routed.plan, {
+ hasAttachments: runOptions?.autoModeRouting?.hasAttachments === true,
+ continuation: runOptions?.trustedContinuation === true,
+ });
+ return plan ? { plan } : {};
+ })(),
},
},
};
@@ -54910,7 +54927,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
corrections: carried.map(item => item.text), correctionIds: carried.map(item => item.id) } };
}
} else if (runOptions?.modeEscalation && isRunMode(mode)) {
- const plan = autoModePlanForChoice(runOptions.modeEscalation.plan);
+ const plan = this._routePlanSignal(runOptions.modeEscalation.plan, {
+ hasAttachments: Array.isArray(attachments) && attachments.length > 0,
+ continuation: runOptions?.trustedContinuation === true,
+ });
runOptions = { ...runOptions, modeRoute: { requested: AUTO_MODE, mode, source: 'escalation', ...(plan ? { plan } : {}) } };
onUpdate('run_mode', { mode, source: 'escalation' });
this._takeModeEscalationSteering(tabId);
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 8be7e3a1d..8592800df 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -24406,7 +24406,18 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}
this._autoModeEscalationTabs.add(tabId);
this._autoModeRoutePendingTabs.add(tabId);
- return { mode: 'ask', runOptions: { ...runOptions, autoModeRouting: { previousMode, ...(continuation ? { continuedTask } : {}) } } };
+ const hasAttachments = Array.isArray(attachments) && attachments.length > 0;
+ return { mode: 'ask', runOptions: { ...runOptions, autoModeRouting: { previousMode, ...(continuation ? { continuedTask } : {}),
+ ...(hasAttachments ? { hasAttachments } : {}) } } };
+ }
+
+ // plan_need's answer as the planner gate may use it. "direct" counts only
+ // when the question held everything the task is about: attached files (a
+ // CSV of contacts to import) and a continued run's earlier work are not in
+ // it, so those turns keep the planner.
+ _routePlanSignal(choice, { hasAttachments = false, continuation = false } = {}) {
+ const plan = autoModePlanForChoice(choice);
+ return plan === 'direct' && (hasAttachments || continuation) ? null : plan;
}
// The user's own words of the task a continuation resumes: its request and
@@ -24502,7 +24513,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
routeMs: Number.isFinite(routed.latencyMs) ? routed.latencyMs : null,
waitedMs: Date.now() - waitStartedAt,
// plan_need for an action turn (_routedDirectTask).
- ...(mode !== 'ask' && autoModePlanForChoice(routed.plan) ? { plan: routed.plan } : {}),
+ ...(() => {
+ const plan = mode === 'ask' ? null : this._routePlanSignal(routed.plan, {
+ hasAttachments: runOptions?.autoModeRouting?.hasAttachments === true,
+ continuation: runOptions?.trustedContinuation === true,
+ });
+ return plan ? { plan } : {};
+ })(),
},
},
};
@@ -46422,7 +46439,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
corrections: carried.map(item => item.text), correctionIds: carried.map(item => item.id) } };
}
} else if (runOptions?.modeEscalation && isRunMode(mode)) {
- const plan = autoModePlanForChoice(runOptions.modeEscalation.plan);
+ const plan = this._routePlanSignal(runOptions.modeEscalation.plan, {
+ hasAttachments: Array.isArray(attachments) && attachments.length > 0,
+ continuation: runOptions?.trustedContinuation === true,
+ });
runOptions = { ...runOptions, modeRoute: { requested: AUTO_MODE, mode, source: 'escalation', ...(plan ? { plan } : {}) } };
onUpdate('run_mode', { mode, source: 'escalation' });
this._takeModeEscalationSteering(tabId);
diff --git a/test/auto-mode.mjs b/test/auto-mode.mjs
index 870f7a112..92b2289fb 100644
--- a/test/auto-mode.mjs
+++ b/test/auto-mode.mjs
@@ -1061,6 +1061,27 @@ for (const build of BUILDS) {
await plain.run('Please book the 9 am slot for me.', 'act', [], { modeEscalation: { mode: 'act', plan: 'something else' } });
assert.deepEqual(plainSeen[0].route, { requested: 'auto', mode: 'act', source: 'escalation' });
});
+
+ // plan_need reads the user's words, not attached files or a continued
+ // run's earlier work, so its "direct" does not count for those turns.
+ test(`${build}: an attachment or a continued run keeps the planner whatever plan_need says`, async () => {
+ const contacts = { kind: 'text', name: 'contacts.csv', mimeType: 'text/csv', textContent: 'name,email\nAda,[email protected]', source: 'user_upload' };
+ const routed = fixture(Agent, { decide: () => ({ choice: 'act', evidence: '', source: 'decision_model', alongside: { plan_need: 'direct' } }) });
+ const routedSeen = planRoute(routed);
+ fixtureReplies(routed, [stopAfterCall(routed)]);
+ await routed.run('Import the attached contacts.', 'auto', [contacts]);
+ assert.equal(routedSeen[0].mode, 'act');
+ assert.equal('plan' in routedSeen[0].route, false, 'the attachment is not in the plan_need question');
+ const restart = fixture(Agent);
+ const restartSeen = planRoute(restart);
+ fixtureReplies(restart, [stopAfterCall(restart)]);
+ await restart.run('Import the attached contacts.', 'act', [contacts], { modeEscalation: { mode: 'act', plan: 'direct' } });
+ assert.equal('plan' in restartSeen[0].route, false, 'nor in a hand-off restart');
+ const agent = routed.agent;
+ assert.equal(agent._routePlanSignal('direct', { continuation: true }), null, 'a continued run');
+ assert.equal(agent._routePlanSignal('plan', { hasAttachments: true, continuation: true }), 'plan', 'a plan answer still counts');
+ assert.equal(agent._routePlanSignal('direct'), 'direct');
+ });
}
for (const build of BUILDS) {