Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Meter an incomplete Responses call, and stop a Bonsai prefill when its card is cancelled
Claude56e683a86ca4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c0656fecd..632ef0ae6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,7 +7,7 @@ This changelog was generated from the repository Git history and release tags. V## [Unreleased]### Changed-- Selecting text on a page now shows a small bar, **Translate | Explain | Proofread | More…**, instead of the single `?` button. Translate, Explain and Proofread answer in a card next to the selection: Translate shows the detected source language, the original text and a target-language picker, and every card has an "Ask a follow-up" box that, once the answer is there, continues in the side panel with the card's answer as context. **More…** opens the same popup the `?` button opened (Summarize, Quiz me, Humanize, a question box and Hide this; Translate, Explain and Proofread are on the bar, so they are not repeated there). A card sends the selected text to your configured model provider, like any other selection action; selections over 4,000 characters go to the side panel instead of a card. A card answer that reaches the model's output limit says it was cut off and offers to continue in the side panel, which picks up from the unfinished text instead of starting over (a side-panel reply that is cut off again says so, including when an OpenAI Responses model reports the limit as an error, in which case the text it wrote is kept), and closing or redoing a card, or reloading or leaving the page, cancels the request it no longer needs. Chrome and Firefox.+- Selecting text on a page now shows a small bar, **Translate | Explain | Proofread | More…**, instead of the single `?` button. Translate, Explain and Proofread answer in a card next to the selection: Translate shows the detected source language, the original text and a target-language picker, and every card has an "Ask a follow-up" box that, once the answer is there, continues in the side panel with the card's answer as context. **More…** opens the same popup the `?` button opened (Summarize, Quiz me, Humanize, a question box and Hide this; Translate, Explain and Proofread are on the bar, so they are not repeated there). A card sends the selected text to your configured model provider, like any other selection action; selections over 4,000 characters go to the side panel instead of a card. A card answer that reaches the model's output limit says it was cut off and offers to continue in the side panel, which picks up from the unfinished text instead of starting over (a side-panel reply that is cut off again says so, including when an OpenAI Responses model reports the limit as an error, in which case the text it wrote is kept and the call still counts toward your cloud-cost allowance), and closing or redoing a card, or reloading or leaving the page, cancels the request it no longer needs. Chrome and Firefox.### Fixed- Chrome: sending a follow-up or a selection action to WebBrain while the side panel is already open no longer closes it. Each side panel now tells the background whether it is on screen, and the background skips re-opening a panel that is already showing and hands it the prompt directly A panel that was open while the background restarted tells the new worker about itself at once instead of after a delay, so a request in that moment does not close it. The same holds for actions chosen on a selection in WebBrain's PDF viewer, where a panel that was closed is opened once rather than twice.diff --git a/docs/architecture.md b/docs/architecture.mdindex 44a725a38..7300bdb63 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -538,7 +538,7 @@ that is closed, retried or retargeted sends `WB_SELECTION_SHORTCUT_INLINE_CANCELbackground aborts the paid request it no longer needs (a page that unloads mid-answer sends thesame cancel from `pagehide`, since it can no longer close the card itself; for the in-browser WebGPU models theprovider also sends `webgpu-chat-cancel`, and the offscreen worker interrupts the running-generation or drops a queued one, and the offscreen host remembers a cancel that arrives while the request is still being routed to a worker, so the generation never starts; those workers report `outputTruncated` when an answer used+generation or drops a queued one, and the offscreen host remembers a cancel that arrives while the request is still being routed to a worker, so the generation never starts, and the Bonsai 2 Vision worker re-checks for a cancel before each prefill slice so a long prompt stops at the next slice boundary; those workers report `outputTruncated` when an answer usedits token budget, since they have no finish reason); the card also claims its request idbefore it waits for the language bundle, so a double click starts only one call, and **More…** takes a newer id on the same wait, so an action clicked just before it never reaches the model. Selectionsover 4,000 characters skip the card and use the side-panel flow. The card draws the answer as DOMdiff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 6ba64e8dd..0135d6a10 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -8480,6 +8480,13 @@ export class Agent extends LoopDetector {const after = await this._recordCostUsage(provider, result?.usage, costState);if (after) result.costAllowanceMessage = after;return result;+ } catch (error) {+ // A call that ends in a provider error can still have been billed (OpenAI Responses+ // stopped at max_output_tokens): count what it reports before the error moves on.+ if (error?.billedUsage) {+ try { await this._recordCostUsage(provider, error.billedUsage, costState); } catch { /* the original error matters more */ }+ }+ throw error;} finally {linked.dispose();}diff --git a/src/chrome/src/offscreen/inference-worker.js b/src/chrome/src/offscreen/inference-worker.jsindex 93746b4f1..c2e522274 100644--- a/src/chrome/src/offscreen/inference-worker.js+++ b/src/chrome/src/offscreen/inference-worker.js@@ -373,11 +373,16 @@ function presentToPastName(name) {* null when the prompt fits in one slice. generate() then continues from the* cache: the Qwen VL input preparation computes the full rotary positions and* feeds only the remaining token.+ *+ * `checkCancelled` throws when the request was cancelled. It runs before the embedding+ * passes and before every slice, so a card that was closed stops at the next slice+ * boundary instead of occupying the serialized GPU worker for the whole prefill.*/-export async function prefillBonsai2InSlices(runtime, inputs, sliceTokens) {+export async function prefillBonsai2InSlices(runtime, inputs, sliceTokens, checkCancelled = () => {}) {const { model, library } = runtime;const length = inputs.input_ids.dims.at(-1);if (!sliceTokens || length - 1 <= sliceTokens) return null;+ checkCancelled();let embeds = await model.encode_text({ input_ids: inputs.input_ids });let attentionMask = inputs.attention_mask;if (inputs.pixel_values) {@@ -401,6 +406,7 @@ export async function prefillBonsai2InSlices(runtime, inputs, sliceTokens) {let cache = null;try {for (let start = 0; start < length - 1; start += sliceTokens) {+ checkCancelled();const end = Math.min(start + sliceTokens, length - 1);const outputs = await model.forward({inputs_embeds: embeds.slice(null, [start, end], null),@@ -1790,8 +1796,11 @@ async function runMultimodalText(payload) {runtime,inputs,bonsai2PrefillSliceTokens(payload?.options?.prefillSliceTokens),+ () => throwIfTextCancelled(requestKey),);}+ // A cancel that landed during the last slice should not start a generation.+ throwIfTextCancelled(requestKey);outputs = await runtime.model.generate({...inputs,...(prefilled ? { past_key_values: prefilled } : {}),diff --git a/src/chrome/src/providers/openai.js b/src/chrome/src/providers/openai.jsindex b37cf89a9..af34aed98 100644--- a/src/chrome/src/providers/openai.js+++ b/src/chrome/src/providers/openai.js@@ -892,6 +892,9 @@ export class OpenAICompatibleProvider extends BaseLLMProvider {// What the model wrote before it stopped, for callers that can show a partial answer.const partial = Array.isArray(response?.output) ? this._responseText(response.output) : '';if (partial) error.partialContent = partial;+ // The call was billed even though it ended in an error; callers that meter cost read this.+ const billedUsage = this._normalizeResponsesUsage(response?.usage);+ if (billedUsage) error.billedUsage = billedUsage;return error;}diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 20cc3b9ac..3dd92326d 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -8202,6 +8202,13 @@ export class Agent extends LoopDetector {const after = await this._recordCostUsage(provider, result?.usage, costState);if (after) result.costAllowanceMessage = after;return result;+ } catch (error) {+ // A call that ends in a provider error can still have been billed (OpenAI Responses+ // stopped at max_output_tokens): count what it reports before the error moves on.+ if (error?.billedUsage) {+ try { await this._recordCostUsage(provider, error.billedUsage, costState); } catch { /* the original error matters more */ }+ }+ throw error;} finally {linked.dispose();}diff --git a/src/firefox/src/providers/openai.js b/src/firefox/src/providers/openai.jsindex b1624df23..c0e4c680e 100644--- a/src/firefox/src/providers/openai.js+++ b/src/firefox/src/providers/openai.js@@ -909,6 +909,9 @@ export class OpenAICompatibleProvider extends BaseLLMProvider {// What the model wrote before it stopped, for callers that can show a partial answer.const partial = Array.isArray(response?.output) ? this._responseText(response.output) : '';if (partial) error.partialContent = partial;+ // The call was billed even though it ended in an error; callers that meter cost read this.+ const billedUsage = this._normalizeResponsesUsage(response?.usage);+ if (billedUsage) error.billedUsage = billedUsage;return error;}diff --git a/test/run.js b/test/run.jsindex 6be9d538e..38ae80edf 100644--- a/test/run.js+++ b/test/run.js@@ -53897,6 +53897,69 @@ test('a response-only reply cut off as a Responses error keeps its text and says}});+test('an OpenAI Responses call that ends at the output limit is still metered', async () => {+ for (const [label, AgentClass, Provider] of [['chrome', AgentCh, OpenAIProviderCh], ['firefox', AgentFx, OpenAIProviderFx]]) {+ const responses = new Provider({+ category: 'cloud', providerName: 'openai', baseUrl: 'https://api.openai.com/v1', model: 'gpt-5.6-terra',+ });+ const incomplete = (usage, text = 'Half of it') => ({+ status: 'incomplete',+ incomplete_details: { reason: 'max_output_tokens' },+ ...(usage ? { usage } : {}),+ output: [{ type: 'message', content: [{ type: 'output_text', text }] }],+ });+ const thrown = (data) => {+ try { responses._parseResponsesData(data); } catch (error) { return error; }+ return null;+ };+ const error = thrown(incomplete({ input_tokens: 120, output_tokens: 800, total_tokens: 920 }));+ assert.equal(error?.billedUsage?.prompt_tokens, 120, `${label}: the error keeps what the call was billed for`);+ assert.equal(error?.billedUsage?.completion_tokens, 800);+ assert.equal('billedUsage' in thrown(incomplete(null)), false, `${label}: a response with no usage has nothing to meter`);++ const setup = (data) => {+ const recorded = [];+ const provider = {+ supportsTools: false, supportsVision: false, promptTier: 'full', contextWindow: 128000,+ model: 'test-model', name: 'test-provider',+ chat: async () => responses._parseResponsesData(data),+ };+ const agent = new AgentClass({ getActive: () => provider, getVisionProvider: async () => null });+ agent._checkCostAllowance = async () => null;+ agent._recordCostUsage = async (_provider, usage) => { recorded.push(usage); return null; };+ return { agent, provider, recorded };+ };+ const messages = [{ role: 'system', content: 'rules' }, { role: 'user', content: 'Translate it.' }];++ // The wrapper every caller goes through meters the call before the error moves on.+ let { agent, provider, recorded } = setup(incomplete({ input_tokens: 120, output_tokens: 800, total_tokens: 920 }));+ await assert.rejects(agent._chatWithCostAllowance(provider, messages, {}, agent._newCostRunState()), /Responses incomplete/);+ assert.equal(recorded.length, 1, `${label}: the incomplete call is counted once`);+ assert.equal(recorded[0].completion_tokens, 800);++ // So does the side-panel path that recovers the partial text.+ ({ agent, provider, recorded } = setup(incomplete({ input_tokens: 120, output_tokens: 800, total_tokens: 920 })));+ const text = await agent._generateContextOnlyResponse(+ 1, messages, provider, agent._newCostRunState(), null, { phase: 'response_only', step: 1 },+ );+ assert.match(text, /^Half of it/, `${label}: the partial text is still shown`);+ assert.equal(recorded.length, 1, `${label}: and the recovered call is counted once`);++ ({ agent, provider, recorded } = setup(incomplete(null)));+ await assert.rejects(agent._chatWithCostAllowance(provider, messages, {}, agent._newCostRunState()), /Responses incomplete/);+ assert.equal(recorded.length, 0, `${label}: nothing reported, nothing recorded`);++ // A call that fails for another reason is not counted, and a completed one is counted once as before.+ ({ agent, provider, recorded } = setup(null));+ provider.chat = async () => { throw new Error('network down'); };+ await assert.rejects(agent._chatWithCostAllowance(provider, messages, {}, agent._newCostRunState()), /network down/);+ assert.equal(recorded.length, 0);+ provider.chat = async () => ({ content: 'Done', toolCalls: null, usage: { prompt_tokens: 3, completion_tokens: 2 } });+ await agent._chatWithCostAllowance(provider, messages, {}, agent._newCostRunState());+ assert.equal(recorded.length, 1);+ }+});+test('ordinary attachments leave selection grounding and remain usable', async () => {for (const [label, AgentClass, buildSelectionPrompt, sourceGrounding] of [['chrome', AgentCh, buildSelectionPromptCh, SELECTION_ONLY_SOURCE_GROUNDING_CH],@@ -69239,7 +69302,41 @@ test('Bonsai 2 Vision prefill runs long prompts in cache-carrying slices', async'image features are merged once over the whole prompt before slicing');assert.equal(calls.length, 3);+ // A cancelled request stops at the next slice boundary and releases the cache built so far.+ calls.length = 0;+ let released = 0;+ class TrackedCache extends DynamicCache {+ async dispose() { released += 1; return super.dispose?.(); }+ }+ const trackedRuntime = { model, library: { DynamicCache: TrackedCache } };+ const abort = () => Object.assign(new Error('The generation was cancelled.'), { name: 'AbortError' });+ let encodes = 0;+ const countingRuntime = {+ model: Object.assign(Object.create(model), {+ encode_text: async (args) => { encodes += 1; return model.encode_text(args); },+ }),+ library: { DynamicCache: TrackedCache },+ };+ await assert.rejects(+ worker.prefillBonsai2InSlices(countingRuntime, { input_ids: ids, attention_mask: mask }, 4, () => { throw abort(); }),+ /cancelled/,+ );+ assert.equal(calls.length, 0, 'a request cancelled before the prefill starts runs no slice');+ assert.equal(encodes, 0, 'and does not even embed the prompt');+ await assert.rejects(+ worker.prefillBonsai2InSlices(trackedRuntime, { input_ids: ids, attention_mask: mask }, 4, () => { if (calls.length >= 1) throw abort(); }),+ /cancelled/,+ );+ assert.equal(calls.length, 1, 'a request cancelled during the first slice stops before the second');+ assert.equal(released, 1, 'the cache built so far is released');+ assert.equal(+ (await worker.prefillBonsai2InSlices(trackedRuntime, { input_ids: ids, attention_mask: mask }, 4, () => {})).get_seq_length(),+ 9, 'a request that is not cancelled prefills as before',+ );+const workerSource = fs.readFileSync(path.join(ROOT, 'src/chrome/src/offscreen/inference-worker.js'), 'utf8');+ assert.match(workerSource, /bonsai2PrefillSliceTokens\(payload\?\.options\?\.prefillSliceTokens\),\s*\(\) => throwIfTextCancelled\(requestKey\),\s*\);\s*\}\s*\/\/[^\n]*\n\s*throwIfTextCancelled\(requestKey\);\s*outputs = await runtime\.model\.generate/,+ 'the worker must pass its cancellation check to the prefill and re-check before generating');assert.match(workerSource, /past_key_values: prefilled/);assert.match(workerSource, /if \(prefilled\) await prefilled\.dispose\(\)/,'the caller-supplied cache must be released after generation');