Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Keep the Act hand-off when the page changes while the model decides

Claude9adbe596fcff
diff --git a/docs/architecture.md b/docs/architecture.md
index 1bfa58094..1fd624108 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -75,7 +75,7 @@ Model tiering is separate from mode: `compact | mid | full` controls how many no
1. Run state decides alone when it can: a continuation keeps its run's mode; a turn for an in-browser (WebGPU) model, which runs as a tiny Ask, a selected-text scope, the standalone window, a Cloud run and a message that carries an `<untrusted_page_content>` block (a selection shortcut with page context) stay in Ask; and a video attachment, which can only be uploaded, needs Act. Context-menu and selection prompts are sent as Ask, as in Manual.
2. Otherwise the decision model answers the fixed-choice `run_mode` question (`semantic-decisions.js`) about the user's own words, the previous turn's mode and the earlier task. It never sees page text, title or address. The question starts as soon as the turn's cost state exists and is awaited after `_enrichUserMessageWithCurrentPage`, so it overlaps page capture; after capture the turn waits at most `AUTO_MODE_ROUTE_GRACE_MS` (250 ms). Only a sure `act` or `dev` answer (the choices are `authorizing`) selects an action mode; Compact maps Dev to Act. The routed mode replaces the provisional Ask setup: run mode, system prompt, the turn's trusted runtime line and dialog handling (Chrome) or the BiDi run (Firefox).
-3. Without a sure answer, or with no decision model, the turn starts in Ask at once. So does a turn whose user sent a correction while the route ran or while the action run's browser setup started (`route_corrected`): the route did not read it, so an action answer is not trusted. That check follows the setup's last await, so nothing arrives between it and the committed mode. Its tools add `request_action_mode`, and its system prompt adds `AUTO_MODE_ASK_NOTE`. The tool is never dispatched. The loop accepts the call only when its `evidence` quotes the user's own words exactly (the request, or a correction sent during the turn) and, because the calling model has read the page, when the same `run_mode` question, about the request with its corrections and asked again without any page content (decision model, then the active model with a quoted reason; a sure routing answer is reused from the semantic-decision cache) also answers act or dev. A correction sent while that check runs supersedes the call like any pending correction: the next step applies it and the model asks again, so a correction that needs Dev gets a Dev hand-off. Corrections sent during the read-only pass, applied or still pending, go with the hand-off: they stay journaled under the same request, and the action run queues them before its first model or tool boundary, so it never acts on the uncorrected request. A correction that arrives after the check (while the hand-off is saved) marks the escalation `reroute`: the background then restarts the request in Automatic again, and that route reads the request with its carried corrections; a third such restart runs read-only (`reroute_limit`, without the hand-off tool): it applies the corrections and answers, and never runs in a mode chosen before the latest correction. An accepted hand-off restores the conversation and per-tab turn state captured before the turn gathered its context (approved plans, site-adapter and API notes, the follow-up allowance, the durable-turn marker) and returns; the background's `chat` handler then restarts the same request with `modeEscalation` in the requested mode, under the same request ID, so Stop still cancels it. The action run's planner gate reads the request again before any action. A refused call gets a tool result with the reason, and a second refusal withdraws the tool. Chrome drains Dev diagnostics only when a routed turn after Dev ends without handing itself to Dev.
+3. Without a sure answer, or with no decision model, the turn starts in Ask at once. So does a turn whose user sent a correction while the route ran or while the action run's browser setup started (`route_corrected`): the route did not read it, so an action answer is not trusted. That check follows the setup's last await, so nothing arrives between it and the committed mode. Its tools add `request_action_mode`, and its system prompt adds `AUTO_MODE_ASK_NOTE`. The tool is never dispatched. Page updates that arrive while the model decides (a timeline's counters and relative times, page-script focus) do not discard the call (`page-feedback-policy.js` treats it like the other page-independent tools): it never runs on the page, and the action run reads the page again before acting. The person's own input on the page, a navigation or a correction still makes the model decide again. The loop accepts the call only when its `evidence` quotes the user's own words exactly (the request, or a correction sent during the turn) and, because the calling model has read the page, when the same `run_mode` question, about the request with its corrections and asked again without any page content (decision model, then the active model with a quoted reason; a sure routing answer is reused from the semantic-decision cache) also answers act or dev. A correction sent while that check runs supersedes the call like any pending correction: the next step applies it and the model asks again, so a correction that needs Dev gets a Dev hand-off. Corrections sent during the read-only pass, applied or still pending, go with the hand-off: they stay journaled under the same request, and the action run queues them before its first model or tool boundary, so it never acts on the uncorrected request. A correction that arrives after the check (while the hand-off is saved) marks the escalation `reroute`: the background then restarts the request in Automatic again, and that route reads the request with its carried corrections; a third such restart runs read-only (`reroute_limit`, without the hand-off tool): it applies the corrections and answers, and never runs in a mode chosen before the latest correction. An accepted hand-off restores the conversation and per-tab turn state captured before the turn gathered its context (approved plans, site-adapter and API notes, the follow-up allowance, the durable-turn marker) and returns; the background's `chat` handler then restarts the same request with `modeEscalation` in the requested mode, under the same request ID, so Stop still cancels it. The action run's planner gate reads the request again before any action. A refused call gets a tool result with the reason, and a second refusal withdraws the tool. Chrome drains Dev diagnostics only when a routed turn after Dev ends without handing itself to Dev.
The agent reports the routed mode with a `run_mode` update (`{ mode, source }`), which the run journal stores as the snapshot's mode and the panel stores on the assistant message for Retry, Continue and recovery. Traces record a `mode_route` note with the source, the decision latency and the time the turn waited. Slash commands and the Ctrl+Shift+A/X/D shortcuts choose a mode for one message only, also while a mode is kept; the shortcuts write the slash command in front of the draft. Context-menu and selection prompts stay Ask either way.
diff --git a/src/chrome/src/agent/page-feedback-policy.js b/src/chrome/src/agent/page-feedback-policy.js
index 1129be7f4..8a6773a50 100644
--- a/src/chrome/src/agent/page-feedback-policy.js
+++ b/src/chrome/src/agent/page-feedback-policy.js
@@ -15,6 +15,10 @@ const DETACHED_TOOLS = new Set([
// The handler validates a one-use token and sends its exact approved prompt,
// independently of the source page's DOM.
'delegate_research',
+ // A routed Ask turn's hand-off is checked against the user's own words
+ // alone and never runs on the page; the action run it starts reads the page
+ // again before acting.
+ 'request_action_mode',
]);
const AX_ACTIONS = new Set(['click_ax', 'type_ax', 'set_field', 'set_checked']);
// The arithmetic in a request is found by its structure: the one span the
diff --git a/src/firefox/src/agent/page-feedback-policy.js b/src/firefox/src/agent/page-feedback-policy.js
index 1129be7f4..8a6773a50 100644
--- a/src/firefox/src/agent/page-feedback-policy.js
+++ b/src/firefox/src/agent/page-feedback-policy.js
@@ -15,6 +15,10 @@ const DETACHED_TOOLS = new Set([
// The handler validates a one-use token and sends its exact approved prompt,
// independently of the source page's DOM.
'delegate_research',
+ // A routed Ask turn's hand-off is checked against the user's own words
+ // alone and never runs on the page; the action run it starts reads the page
+ // again before acting.
+ 'request_action_mode',
]);
const AX_ACTIONS = new Set(['click_ax', 'type_ax', 'set_field', 'set_checked']);
// The arithmetic in a request is found by its structure: the one span the
diff --git a/test/auto-mode.mjs b/test/auto-mode.mjs
index b1ca2cd5e..a37ac5c11 100644
--- a/test/auto-mode.mjs
+++ b/test/auto-mode.mjs
@@ -303,6 +303,60 @@ for (const build of BUILDS) {
assert.equal(conversation.filter(message => message.role === 'user').length, 1, 'the request is recorded once');
});
+ // A live page (a social timeline's relative times and counters) changes
+ // while the model decides. The hand-off reads only the user's words, and
+ // the action run reads the page again, so those updates must not discard
+ // it: on X and Mastodon they did so on every call until the turn ended as
+ // page_unstable or the model gave up read-only.
+ const watchedPage = f => {
+ f.agent._pageFeedbackIdleMs = 0;
+ const from = { tab: { id: f.tab }, frameId: 0, documentId: 'main-document', url: pageUrl };
+ let monitor = null, seq = 0;
+ const enrich = f.agent._enrichUserMessageWithCurrentPage;
+ f.agent._enrichUserMessageWithCurrentPage = async (...args) => {
+ monitor = f.agent.pageMonitorState(from, 'main-token');
+ return enrich(...args);
+ };
+ return event => {
+ assert.equal(monitor?.active, true, 'the page is watched during the turn');
+ return f.agent.observePageFeedback(from, { runToken: monitor.runToken, documentToken: 'main-token',
+ seq: ++seq, revision: seq, kind: 'dom', source: 'page', target: 'time.relative', ...event }).accepted;
+ };
+ };
+ const decidesAct = (_q, options) => (options.mainModel === false
+ ? { choice: null, evidence: '', source: 'unavailable' }
+ : { choice: 'act', evidence: 'Post on Mastodon', source: 'main_model' });
+
+ test(`${build}: page updates while the model decides do not discard its hand-off`, async () => {
+ const request = 'Post on Mastodon: WebBrain QA test, please ignore';
+ const f = fixture(Agent, { decide: decidesAct });
+ const emit = watchedPage(f);
+ fixtureReplies(f, [() => {
+ // DOM churn and page-script focus, but nothing the person did.
+ for (const event of [{}, { target: 'span.counter' }, { kind: 'activity', target: 'body' }]) assert.equal(emit(event), true);
+ return handoff('act', request);
+ }]);
+ assert.equal(await f.run(request), '');
+ assert.equal(f.calls.length, 1, 'the first hand-off counts');
+ assert.deepEqual(f.agent.takeModeEscalation(f.tab), { mode: 'act', evidence: request });
+ assert.deepEqual([...new Set(f.executed)], ['get_accessibility_tree'], 'only the page-update read ran');
+ assert.equal(f.agent.conversations.get(f.tab).some(message => message.role === 'user'), false,
+ 'the update read leaves with the turn it restarts');
+ });
+
+ test(`${build}: the person's own input on the page still makes the model decide again`, async () => {
+ const request = 'Post on Mastodon: WebBrain QA test, please ignore';
+ const f = fixture(Agent, { decide: decidesAct });
+ const emit = watchedPage(f);
+ fixtureReplies(f, [() => {
+ assert.equal(emit({ kind: 'input', source: 'user', target: 'textarea#compose' }), true);
+ return handoff('act', request);
+ }, handoff('act', request)]);
+ assert.equal(await f.run(request), '');
+ assert.equal(f.calls.length, 2, 'the hand-off is decided again after the person typed');
+ assert.deepEqual(f.agent.takeModeEscalation(f.tab), { mode: 'act', evidence: request });
+ });
+
test(`${build}: a hand-off quoting anything but the user's words is refused`, async () => {
const f = fixture(Agent, {
replies: [
diff --git a/test/page-feedback-policy.mjs b/test/page-feedback-policy.mjs
index fee22bdde..2525e8ba4 100644
--- a/test/page-feedback-policy.mjs
+++ b/test/page-feedback-policy.mjs
@@ -265,7 +265,10 @@ for (const build of ['chrome', 'firefox']) {
for (const [name, args] of [['calculate', { expression: '12345678901234567890 * 7 + 0.125' }],
['read_pdf', {}], ['get_captcha_capabilities', {}], ['find_text', { text: 'literal match' }],
['generate_image', { prompt: 'An authorized illustration' }], ['schedule_task', { title: 'Authorized reminder', prompt: 'Inspect the draft' }],
- ['schedule_resume', { after_seconds: 60 }], ['resize_window', { width: 1280, height: 720 }]]) {
+ ['schedule_resume', { after_seconds: 60 }], ['resize_window', { width: 1280, height: 720 }],
+ // An automatic Ask turn's hand-off: checked against the user's words,
+ // never run on the page.
+ ['request_action_mode', { mode: 'act', evidence: 'Post on Mastodon: hello' }]]) {
test(`${build}: self-validating ${name} does not starve under unrelated DOM feedback`, () => {
assert.notEqual(pageFeedbackCallPolicy(name, args, state, page).kind, 'unsafe');
assert.equal(canRetainPageFeedbackCalls([call(name, args)], state, page, passive, { pending: true }), true);