Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Finish an Ask answer without waiting for the person to stop typing
Claude5089d0bff14c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 74bb1f6d3..02760a8ce 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,7 +7,7 @@ This changelog was generated from the repository Git history and release tags. V## [Unreleased]### Fixed-- Ask mode: an answer being written no longer disappears and starts over when you use the page meanwhile. In a WhatsApp trace, typing a draft and clicking in the chat while the answer was written threw the answer away twice and wrote it again each time, and the answer that finally stayed, written after you stopped, was the shortest. Scrolling, selecting, clicking and typing on the page, and the page's changes from them, now leave the answer as it is: it is about the page as you asked about it. Navigating the page to another address still starts the answer over.+- Ask mode: an answer being written no longer disappears and starts over when you use the page meanwhile. In a WhatsApp trace, typing a draft and clicking in the chat while the answer was written threw the answer away twice and wrote it again each time, and the answer that finally stayed, written after you stopped, was the shortest. Scrolling, selecting, clicking and typing on the page, and the page's changes from them, now leave the answer as it is: it is about the page as you asked about it, and the run ends without waiting for you to stop typing. Navigating the page to another address still starts the answer over.- Posting on another site after a correction: after a Mastodon post, "ok now on X" no longer leaves WebBrain going back and forth between the two sites, typing the X post again each time. It counted X as the same platform as Mastodon, because it reads account names only on X and Bluesky, and asked to find the Mastodon post on X itself, which can never happen. The site a post was made on now names its platform when no account name was read, and a post on another site goes ahead once a page has been read since the correction, provided the task made no earlier post on that site. What the task attempted is saved with the conversation, and just before each action that may post or submit something, so this still holds when the task continues after WebBrain restarted, and a post already sent on X under the correction is not sent again by a later run of the task. When the task did post on that site before, or WebBrain has no full record of its earlier attempts, it stops once and says so, instead of searching for a post it cannot find. A search, a settings save or consent click on X, a click that sent nothing or a form the page rejected is not such a post; a click that Stop interrupted after it was sent counts. On the other hosts and clients of X and Bluesky, such as X Pro, whose Post buttons WebBrain does not recognize, anything sent that may have been a post counts as one, so a post from X Pro is not repeated on x.com; this includes a Post button whose caption no model read in time, which looked like a harmless button, while Enter in the post editor there edits the draft and is not a post. If that record cannot be saved just before a possible post on X or Bluesky, the post is not sent and WebBrain says why, as it already does when a run's recovery checkpoint cannot be saved. x.com and twitter.com are one platform.- Chrome: a page's "Leave site?" prompt no longer waits for you when WebBrain navigates away during a task that started on a page the debugger cannot attach to, such as chrome://extensions, or whose debugger connection dropped. WebBrain arms its dialog handling again at the next debugger attach and around each of its own navigations, and answers Leave for its own navigation as it already did on other runs. If you cancel the debugger, prompts stay yours to answer. Firefox offers extensions no way to answer these prompts.- Chat sites such as WhatsApp: a request that points at the open chat instead of naming the person, such as "send some test messages to this one" or "message this user", now sends there instead of asking "which contact, and what text?". The planner never sees the page, and its rules counted only words like "reply here" as the open chat, so it asked the same question every time without reading the page. WebBrain still reads the open chat's header itself and binds every send to that one contact, and stops if no single chat is open. The planner is also told it cannot see the page, no longer reads a note about a screenshot it is never given, and leaves the wording of described messages, such as test messages or a thank-you, to the agent. One request can now cover the open chat and someone named besides it, as in "test messages here, then thank Rustem Bey": the open chat needs no confirmation, the reviewed plan shows the named person, and that person is confirmed once on WebBrain's card before the first send to them. Words that refer back to someone named earlier still mean that person, and on mail sites the people named besides "reply here" are handled as before. On WhatsApp, where a contact is confirmed through its contact drawer, one run still confirms only one chat's contact, so the second chat of such a request needs a request of its own; the agent is now told so, instead of being sent to re-read the page and retry.diff --git a/docs/architecture.md b/docs/architecture.mdindex 2e9cdc172..0b87dd30d 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -458,7 +458,8 @@ invariants therefore keep one production lifecycle.An Ask answer is about the page as the person asked about it. What they do onthe page while it is written (scrolling, selecting, clicking, typing, and thepage's changes from those), and any scroll or viewport resize, does not make-it stale: the answer already shown is kept and is not written again. A+it stale: the answer already shown is kept and is not written again, and+its decision does not wait for the person to stop typing or scrolling. Anavigation of the page to another address, a new document, a correction, andthe page's own changes are weighed as before, so a page-dependent answer stillrefreshes after automatic churn within its retry budget.diff --git a/src/chrome/src/agent/page-feedback.js b/src/chrome/src/agent/page-feedback.jsindex c8c3315cc..5b6eec08a 100644--- a/src/chrome/src/agent/page-feedback.js+++ b/src/chrome/src/agent/page-feedback.js@@ -40,7 +40,7 @@ function automaticAnswerFeedback(events) {const relevant = askRelevantFeedback(events);return !!relevant.length && relevant.every(event => ['page', 'unknown'].includes(event.source)&& (event.kind === 'dom' || (event.kind === 'navigation' && event.frameId > 0)- || (['activity', 'resize'].includes(event.kind) && event.source === 'page')));+ || (event.kind === 'activity' && event.source === 'page')));}// Busy renderers can stall browser messaging indefinitely; Stop must not wait.@@ -711,12 +711,19 @@ export const pageFeedbackMethods = {return !!this._pageFeedbackRuns?.get(tabId)?.events.size;},- async _waitForPageFeedbackIdle(tabId, { automaticAskReply = false } = {}) {+ async _waitForPageFeedbackIdle(tabId, { automaticAskReply = false, askReply = false } = {}) {const run = this._pageFeedbackRuns?.get(tabId);const signal = this._runAbortSignal(tabId);while (run && this._pageFeedbackRuns.get(tabId) === run) {this._throwIfAborted(signal);if (this._hasPendingSteering?.(tabId)) return;+ // Nothing the person does on the page can change a read-only Ask+ // answer, so its decision does not wait for them to stop typing or+ // scrolling; only a change that can still make it stale is waited for.+ if (askReply) {+ const events = [...run.events.values()];+ if (!askRelevantFeedback(events).length || automaticAnswerFeedback(events)) return;+ }// Automatic child commits can continue indefinitely. A read-only terminal// candidate still waits for actual user gestures, then uses its reply scope// or bounded recovery; browser actions keep the full activity wait.@@ -1306,7 +1313,9 @@ export const pageFeedbackMethods = {|| (responseToolCalls === null && pendingRun?.askAutomaticRetry))&& this._pageFeedbackAskOwnerCurrent(tabId, pendingRun, pendingRun?.modelState)&& automaticAnswerFeedback([...pendingRun.events.values()]);- await this._waitForPageFeedbackIdle(tabId, { automaticAskReply });+ const askReply = !!askAnswerCandidate(responseToolCalls, responseWithoutTools, responseContent).trim()+ && this._pageFeedbackAskOwnerCurrent(tabId, pendingRun, pendingRun?.modelState);+ await this._waitForPageFeedbackIdle(tabId, { automaticAskReply, askReply });const run = this._pageFeedbackRuns?.get(tabId);if (!run || this._checkAbort(tabId)) return false;const events = [...run.events.values()], batchRevision = run.revision;diff --git a/src/firefox/src/agent/page-feedback.js b/src/firefox/src/agent/page-feedback.jsindex c8c3315cc..5b6eec08a 100644--- a/src/firefox/src/agent/page-feedback.js+++ b/src/firefox/src/agent/page-feedback.js@@ -40,7 +40,7 @@ function automaticAnswerFeedback(events) {const relevant = askRelevantFeedback(events);return !!relevant.length && relevant.every(event => ['page', 'unknown'].includes(event.source)&& (event.kind === 'dom' || (event.kind === 'navigation' && event.frameId > 0)- || (['activity', 'resize'].includes(event.kind) && event.source === 'page')));+ || (event.kind === 'activity' && event.source === 'page')));}// Busy renderers can stall browser messaging indefinitely; Stop must not wait.@@ -711,12 +711,19 @@ export const pageFeedbackMethods = {return !!this._pageFeedbackRuns?.get(tabId)?.events.size;},- async _waitForPageFeedbackIdle(tabId, { automaticAskReply = false } = {}) {+ async _waitForPageFeedbackIdle(tabId, { automaticAskReply = false, askReply = false } = {}) {const run = this._pageFeedbackRuns?.get(tabId);const signal = this._runAbortSignal(tabId);while (run && this._pageFeedbackRuns.get(tabId) === run) {this._throwIfAborted(signal);if (this._hasPendingSteering?.(tabId)) return;+ // Nothing the person does on the page can change a read-only Ask+ // answer, so its decision does not wait for them to stop typing or+ // scrolling; only a change that can still make it stale is waited for.+ if (askReply) {+ const events = [...run.events.values()];+ if (!askRelevantFeedback(events).length || automaticAnswerFeedback(events)) return;+ }// Automatic child commits can continue indefinitely. A read-only terminal// candidate still waits for actual user gestures, then uses its reply scope// or bounded recovery; browser actions keep the full activity wait.@@ -1306,7 +1313,9 @@ export const pageFeedbackMethods = {|| (responseToolCalls === null && pendingRun?.askAutomaticRetry))&& this._pageFeedbackAskOwnerCurrent(tabId, pendingRun, pendingRun?.modelState)&& automaticAnswerFeedback([...pendingRun.events.values()]);- await this._waitForPageFeedbackIdle(tabId, { automaticAskReply });+ const askReply = !!askAnswerCandidate(responseToolCalls, responseWithoutTools, responseContent).trim()+ && this._pageFeedbackAskOwnerCurrent(tabId, pendingRun, pendingRun?.modelState);+ await this._waitForPageFeedbackIdle(tabId, { automaticAskReply, askReply });const run = this._pageFeedbackRuns?.get(tabId);if (!run || this._checkAbort(tabId)) return false;const events = [...run.events.values()], batchRevision = run.revision;diff --git a/test/feedback-ask-viewing.mjs b/test/feedback-ask-viewing.mjsindex 7d4b54a37..1acc60fd7 100644--- a/test/feedback-ask-viewing.mjs+++ b/test/feedback-ask-viewing.mjs@@ -112,6 +112,36 @@ for (const build of ['chrome', 'firefox']) {});}+ // The trace's person typed a long draft while the answer was written; the+ // answer is delivered without waiting for them to stop typing.+ for (const entry of ['chat', 'chat_stream']) test(`${build}: a streamed Ask answer is delivered while the person keeps typing (${entry})`, async () => {+ const tab = nextTab++, updates = [];+ let calls = 0, typing = null, answered = 0;+ const agent = makeAgent(Agent, (owner, streamed) => {+ assert.equal(streamed, true);+ calls++;+ // They type for three seconds, longer than the answer takes.+ const type = () => owner.__send({ kind: 'input', source: 'user', target: 'div#composer' });+ const until = Date.now() + 3000;+ return { chunks: [{ text: 'The page lists A and B; ', after: () => {+ type(); typing = setInterval(() => { if (Date.now() < until) type(); else clearInterval(typing); }, 150);+ } },+ { text: 'B is the cheapest at $5.', after: () => { answered = Date.now(); } }] };+ });+ agent._pageFeedbackIdleMs = 1000;+ try {+ const options = { detachedRequestId: 'ask-' + tab, interactiveChat: true, askStreamingEnabled: true };+ const update = (type, data) => updates.push({ type, data });+ const result = entry === 'chat' ? await agent.processMessage(tab, 'Which product is cheapest?', update, 'ask', [], options)+ : await agent.processMessageStream(tab, 'Which product is cheapest?', update, 'ask', options);+ const settled = Date.now() - answered;+ assert.equal(result, ANSWER);+ assert.equal(calls, 1);+ assert.equal(retracted(updates), false);+ assert.ok(settled < 900, `The run finished ${settled} ms after the answer, while the person was still typing`);+ } finally { clearInterval(typing); }+ });+for (const streaming of [false, true]) for (const [name, event] of [['the person scrolling', VIEWING['the person scrolling']],['the person clicking', VIEWING['the person clicking']], ...Object.entries(CHANGING)]) {const survives = name in VIEWING;diff --git a/test/quality/ask-person-activity-mv3.mjs b/test/quality/ask-person-activity-mv3.mjsindex 052dc8309..e66c79437 100644--- a/test/quality/ask-person-activity-mv3.mjs+++ b/test/quality/ask-person-activity-mv3.mjs@@ -9,7 +9,8 @@ import { chromium } from 'playwright';// the answer was being written they typed a draft into the chat's composer,// clicked and scrolled. The answer appeared and vanished twice before a third// one stayed. Through the installed MV3 extension, with real page events from-// real input: the answer that is being written stays, and is written once.+// real input: the answer that is being written stays, is written once, and+// the run ends without waiting for the person to stop typing.// The page and the model are owned local fixtures; nothing leaves the host.const out = path.resolve(process.env.WEBBRAIN_QUALITY_ARTIFACTS || 'ci/artifacts/quality-audit-2026-10-15/ask-person-activity-mv3');await fs.mkdir(out, { recursive: true });@@ -131,12 +132,19 @@ try {await page.mouse.wheel(0, -600);await page.click('#last');await delay(1500);+ // They keep typing for five seconds after the answer is released.+ await page.click('#composer');+ const typingUntil = Date.now() + 5000;+ const typing = (async () => { while (Date.now() < typingUntil) await page.keyboard.type(' ok', { delay: 60 }); })();+ const released = Date.now();releaseMain.resolve();await panel.waitForFunction(() => window.ownedRun.finished, null, { timeout: 45000 });+ const finishedAfter = Date.now() - released;+ await typing;const run = await panel.evaluate(() => window.ownedRun);const events = await panel.evaluate(id => ownedEvents.filter(event => event.requestId === id), requestId);const current = await message(panel, 'agent_run_state', { tabId, requestId });- const draft = await page.evaluate(() => document.querySelector('#composer').textContent);+ const draft = (await page.evaluate(() => document.querySelector('#composer').textContent)).replace(/( ok)+$/, '');const descriptor = await message(panel, 'export_traces', { tabId, full: true });const decisions = descriptor?.ok ? await panel.evaluate(async ({ sessionId, extensionId }) => {const recorder = await import(`chrome-extension://${extensionId}/src/trace/recorder.js`);@@ -147,7 +155,7 @@ try {const retracted = events.filter(event => event.type === 'text' && event.data?.replace === true && event.data?.content === '');const main = calls.filter(call => call.stage === 'main');const evidence = { run, calls, current: { running: current?.running, finalContent: current?.runUi?.finalContent, status: current?.runUi?.status },- retracted: retracted.length, decisions, draft, network, runtimeErrors };+ retracted: retracted.length, finishedAfter, decisions, draft, network, runtimeErrors };await fs.writeFile(path.join(out, 'evidence.json'), JSON.stringify(evidence, null, 2));assert.deepEqual(runtimeErrors, []);assert.equal(draft, 'bi de bana sorumluluk', 'The person really typed into the page');@@ -155,10 +163,11 @@ try {`The person's activity reached WebBrain as page feedback: ${JSON.stringify(decisions)}`);assert.equal(main.length, 1, `The answer was written once, not again after the person's activity: ${JSON.stringify(calls)}`);assert.equal(retracted.length, 0, 'The answer being shown was never taken back');+ assert.ok(finishedAfter < 3500, `The run ended ${finishedAfter} ms after the answer was released, while the person kept typing for 5000 ms`);assert.equal(run.error, undefined);assert.equal(run.result?.content, answer);assert.equal(current?.running, false);- console.log(JSON.stringify({ passed: true, installedMV3: true, mainCalls: main.length, retracted: retracted.length,+ console.log(JSON.stringify({ passed: true, installedMV3: true, mainCalls: main.length, retracted: retracted.length, finishedAfter,decisions: decisions.map(decision => `${decision.stage}:${decision.disposition}`), blockedNetwork: network.length }));} finally {releaseMain.resolve();