Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Hand long selections over inside the click, and flag a side-panel reply that is cut off again
Claude33d37af969b5
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex aa2873f2c..7342b8b26 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,7 +7,7 @@ This changelog was generated from the repository Git history and release tags. V## [Unreleased]### Changed-- Selecting text on a page now shows a small bar, **Translate | Explain | Proofread | More…**, instead of the single `?` button. Translate, Explain and Proofread answer in a card next to the selection: Translate shows the detected source language, the original text and a target-language picker, and every card has an "Ask a follow-up" box that, once the answer is there, continues in the side panel with the card's answer as context. **More…** opens the same popup the `?` button opened (Summarize, Quiz me, Humanize, a question box and Hide this; Translate, Explain and Proofread are on the bar, so they are not repeated there). A card sends the selected text to your configured model provider, like any other selection action; selections over 4,000 characters go to the side panel instead of a card. A card answer that reaches the model's output limit says it was cut off and offers to continue in the side panel, which picks up from the unfinished text instead of starting over, and closing or redoing a card cancels the request it no longer needs. Chrome and Firefox.+- Selecting text on a page now shows a small bar, **Translate | Explain | Proofread | More…**, instead of the single `?` button. Translate, Explain and Proofread answer in a card next to the selection: Translate shows the detected source language, the original text and a target-language picker, and every card has an "Ask a follow-up" box that, once the answer is there, continues in the side panel with the card's answer as context. **More…** opens the same popup the `?` button opened (Summarize, Quiz me, Humanize, a question box and Hide this; Translate, Explain and Proofread are on the bar, so they are not repeated there). A card sends the selected text to your configured model provider, like any other selection action; selections over 4,000 characters go to the side panel instead of a card. A card answer that reaches the model's output limit says it was cut off and offers to continue in the side panel, which picks up from the unfinished text instead of starting over (a side-panel reply that is cut off again says so), and closing or redoing a card cancels the request it no longer needs. Chrome and Firefox.### Fixed- Chrome: sending a follow-up or a selection action to WebBrain while the side panel is already open no longer closes it. Each side panel now tells the background whether it is on screen, and the background skips re-opening a panel that is already showing and hands it the prompt directly.diff --git a/docs/architecture.md b/docs/architecture.mdindex f15ad6f87..986e778f0 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -529,7 +529,10 @@ is flagged `truncated`, and the card says so and offers **Continue in side panelof passing a cut-off translation off as complete. Continue sends the unfinished text as`continueFrom`; the background wraps it inside the selection's untrusted block (keeping itsend) with a trusted request to resume from where it stops, because running the action again-would meet the same output limit. Each call carries a job id, and a card+would meet the same output limit. A side-panel reply that is cut off again says so (the+response-only path appends a notice when the provider reports a length finish or WebGPU+reports `outputTruncated`), so a long remainder never passes as complete and can be+continued by asking. Each call carries a job id, and a cardthat is closed, retried or retargeted sends `WB_SELECTION_SHORTCUT_INLINE_CANCEL` so thebackground aborts the paid request it no longer needs (for the in-browser WebGPU models theprovider also sends `webgpu-chat-cancel`, and the offscreen worker interrupts the runningdiff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 268b74e3d..f8d8bc967 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -162,7 +162,7 @@ import { generateImage, readMediaConfig } from './generative-media.js';import { GENERATIVE_MEDIA_SETUP_NOTE, isImageGenConfigured, mediaPermissionUrl, validateMediaConfig } from './media-config.js';import { extractFirstJsonObject } from './json-extract.js';import { repairAssistantDisplayText, repairClarifyText, sanitizeText as sanitizePlannerText } from './text-sanitize.js';-import { emptyOutputFailureMessage, modelOutputDiagnostics } from './model-output-diagnostics.js';+import { RESPONSE_CUT_OFF_NOTICE, emptyOutputFailureMessage, modelOutputDiagnostics, outputWasCutOff } from './model-output-diagnostics.js';import { stepLimitUserMessage, terminalGuardUserMessage } from './terminal-messages.js';import { buildCustomSkillsPrompt, buildSkillLoaderDefinition, buildSkillToolDefinitions, buildSkillToolRegistry, getEligibleCustomSkills, getEligibleSkillCatalog, normalizeCustomSkills } from './skills.js';import { OTP_EMAIL_PROVIDER_IDS, OTP_EMAIL_SKILL_ID, OTP_EMAIL_TOOL_NAME, otpEmailCandidates, otpEmailProviderForUrl, otpEmailUrlLooksLikeMessage, otpOpenMessageRootRef, otpRedactRefs, otpServiceDisplay, otpServiceKey, otpVerificationMessageExcerpt, selectOtpMailboxTab, selectUniqueOtpCandidateByPreview } from './otp-email-tool.js';@@ -26081,7 +26081,12 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dconst displayContent = phase === 'terminal_recovery'? terminalRecoveryFinalText(result?.content): String(result?.content || '').trim();- return repairAssistantDisplayText(displayContent);+ const displayText = repairAssistantDisplayText(displayContent);+ // A reply that ran into the output limit must not read as finished. A selection+ // card's "Continue" lands here too, and the remainder can be cut off again.+ return phase === 'response_only' && displayText && outputWasCutOff(result)+ ? `${displayText}${RESPONSE_CUT_OFF_NOTICE}`+ : displayText;}_consumeContextOnlyAbort(tabId, messages, onUpdate) {diff --git a/src/chrome/src/agent/model-output-diagnostics.js b/src/chrome/src/agent/model-output-diagnostics.jsindex 1ff1fa77a..bcfce7a1b 100644--- a/src/chrome/src/agent/model-output-diagnostics.js+++ b/src/chrome/src/agent/model-output-diagnostics.js@@ -123,6 +123,18 @@ export function modelOutputDiagnostics(result, { requestedMaxTokens = null, reco};}+// A finish reason that means the reply stopped because it ran out of output budget.+const CUT_OFF_FINISH_RE = /^(?:length|max_tokens|max_output_tokens|output_limit)$/i;++export const RESPONSE_CUT_OFF_NOTICE = '\n\n[This reply reached the model\'s output limit and was cut off. Ask WebBrain to continue to get the rest.]';++// True when the provider says the reply was stopped by its output limit. In-browser+// WebGPU models have no finish reason, so they report `outputTruncated` instead.+export function outputWasCutOff(result) {+ if (result?.outputTruncated === true) return true;+ return CUT_OFF_FINISH_RE.test(finishReasonFor(result));+}+export function emptyOutputFailureMessage(diagnostics = {}) {switch (diagnostics.emptyReason) {case 'output_limit':diff --git a/src/chrome/src/content/selection-shortcut.js b/src/chrome/src/content/selection-shortcut.jsindex 498cf6a88..a599593af 100644--- a/src/chrome/src/content/selection-shortcut.js+++ b/src/chrome/src/content/selection-shortcut.js@@ -931,17 +931,19 @@async function openCard(kind, opener) {if (!snapshot || submitting || !INLINE_ACTIONS.includes(kind)) return;+ if (snapshot.text.length > INLINE_MAX_CHARS) {+ // Too long for a card: the side panel is the better place to read it. This is+ // decided before anything is awaited, because the panel can only be opened+ // inside the click's user gesture, which the localization wait would outlive.+ submitSelection(kind, '', interfaceLanguage);+ return;+ }// Claim the card before anything is awaited: a second click while the// localization loads takes a newer id, so this call stops instead of opening// a second card and starting a second paid request.const requestId = ++cardRequestId;if (!localization) await refreshLocalization(interfaceLanguage);if (requestId !== cardRequestId || !snapshot || submitting) return;- if (snapshot.text.length > INLINE_MAX_CHARS) {- // Too long for a card: the side panel is the better place to read it.- submitSelection(kind, '', interfaceLanguage);- return;- }ensureSurface();applyLocalization();closePopup(false);diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex b048e117f..b4bb899e7 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -154,7 +154,7 @@ import { generateImage, readMediaConfig } from './generative-media.js';import { GENERATIVE_MEDIA_SETUP_NOTE, isImageGenConfigured, mediaPermissionUrl, validateMediaConfig } from './media-config.js';import { extractFirstJsonObject } from './json-extract.js';import { repairAssistantDisplayText, repairClarifyText, sanitizeText as sanitizePlannerText } from './text-sanitize.js';-import { emptyOutputFailureMessage, modelOutputDiagnostics } from './model-output-diagnostics.js';+import { RESPONSE_CUT_OFF_NOTICE, emptyOutputFailureMessage, modelOutputDiagnostics, outputWasCutOff } from './model-output-diagnostics.js';import { stepLimitUserMessage, terminalGuardUserMessage } from './terminal-messages.js';import { buildCustomSkillsPrompt, buildSkillLoaderDefinition, buildSkillToolDefinitions, buildSkillToolRegistry, getEligibleCustomSkills, getEligibleSkillCatalog, normalizeCustomSkills } from './skills.js';import { OTP_EMAIL_PROVIDER_IDS, OTP_EMAIL_SKILL_ID, OTP_EMAIL_TOOL_NAME, otpEmailCandidates, otpEmailProviderForUrl, otpEmailUrlLooksLikeMessage, otpOpenMessageRootRef, otpRedactRefs, otpServiceDisplay, otpServiceKey, otpVerificationMessageExcerpt, selectOtpMailboxTab, selectUniqueOtpCandidateByPreview } from './otp-email-tool.js';@@ -23236,7 +23236,12 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dconst displayContent = phase === 'terminal_recovery'? terminalRecoveryFinalText(result?.content): String(result?.content || '').trim();- return repairAssistantDisplayText(displayContent);+ const displayText = repairAssistantDisplayText(displayContent);+ // A reply that ran into the output limit must not read as finished. A selection+ // card's "Continue" lands here too, and the remainder can be cut off again.+ return phase === 'response_only' && displayText && outputWasCutOff(result)+ ? `${displayText}${RESPONSE_CUT_OFF_NOTICE}`+ : displayText;}_consumeContextOnlyAbort(tabId, messages, onUpdate) {diff --git a/src/firefox/src/agent/model-output-diagnostics.js b/src/firefox/src/agent/model-output-diagnostics.jsindex 1ff1fa77a..bcfce7a1b 100644--- a/src/firefox/src/agent/model-output-diagnostics.js+++ b/src/firefox/src/agent/model-output-diagnostics.js@@ -123,6 +123,18 @@ export function modelOutputDiagnostics(result, { requestedMaxTokens = null, reco};}+// A finish reason that means the reply stopped because it ran out of output budget.+const CUT_OFF_FINISH_RE = /^(?:length|max_tokens|max_output_tokens|output_limit)$/i;++export const RESPONSE_CUT_OFF_NOTICE = '\n\n[This reply reached the model\'s output limit and was cut off. Ask WebBrain to continue to get the rest.]';++// True when the provider says the reply was stopped by its output limit. In-browser+// WebGPU models have no finish reason, so they report `outputTruncated` instead.+export function outputWasCutOff(result) {+ if (result?.outputTruncated === true) return true;+ return CUT_OFF_FINISH_RE.test(finishReasonFor(result));+}+export function emptyOutputFailureMessage(diagnostics = {}) {switch (diagnostics.emptyReason) {case 'output_limit':diff --git a/src/firefox/src/content/selection-shortcut.js b/src/firefox/src/content/selection-shortcut.jsindex 498cf6a88..a599593af 100644--- a/src/firefox/src/content/selection-shortcut.js+++ b/src/firefox/src/content/selection-shortcut.js@@ -931,17 +931,19 @@async function openCard(kind, opener) {if (!snapshot || submitting || !INLINE_ACTIONS.includes(kind)) return;+ if (snapshot.text.length > INLINE_MAX_CHARS) {+ // Too long for a card: the side panel is the better place to read it. This is+ // decided before anything is awaited, because the panel can only be opened+ // inside the click's user gesture, which the localization wait would outlive.+ submitSelection(kind, '', interfaceLanguage);+ return;+ }// Claim the card before anything is awaited: a second click while the// localization loads takes a newer id, so this call stops instead of opening// a second card and starting a second paid request.const requestId = ++cardRequestId;if (!localization) await refreshLocalization(interfaceLanguage);if (requestId !== cardRequestId || !snapshot || submitting) return;- if (snapshot.text.length > INLINE_MAX_CHARS) {- // Too long for a card: the side panel is the better place to read it.- submitSelection(kind, '', interfaceLanguage);- return;- }ensureSurface();applyLocalization();closePopup(false);diff --git a/test/fixtures/run.mjs b/test/fixtures/run.mjsindex e71247182..84cfccaa7 100644--- a/test/fixtures/run.mjs+++ b/test/fixtures/run.mjs@@ -2500,6 +2500,33 @@ for (const [label, sourcePath, manualOpen] of [}});+ test(`${label}: a long selection is handed over at once, not after the language bundle loads`, async (page) => {+ // The side panel can only be opened inside the click's user gesture, so the hand-over+ // must not wait for localization.+ await setupSelectionShortcut(page, sourcePath, { requiresManualOpen: manualOpen, holdLocalization: true });+ await page.evaluate(() => {+ const article = document.createElement('article');+ article.id = 'long-selection';+ for (let index = 0; index < 400; index += 1) {+ const line = document.createElement('div');+ line.textContent = `Selected article line ${index + 1}`;+ article.appendChild(line);+ }+ document.body.appendChild(article);+ });+ const selected = await selectFixtureText(page, '#long-selection');+ await clickRectCenter(page, selected.inlineRects.translate);+ await page.waitForFunction(() => window.__selectionMessages.length === 1, null, { timeout: 2000 });+ const result = await page.evaluate(() => ({+ message: window.__selectionMessages[0],+ inline: window.__selectionInlineMessages.length,+ }));+ if (result.message.action !== 'translate' || result.inline !== 0) {+ throw new Error(`a long selection should reach the side panel flow while localization is still loading: ${JSON.stringify(result)}`);+ }+ await page.evaluate(() => window.__releaseLocalization?.());+ });+test(`${label}: the card opens below a selection near the top edge`, async (page) => {await setupSelectionShortcut(page, sourcePath, { requiresManualOpen: manualOpen });await page.evaluate(() => {diff --git a/test/run.js b/test/run.jsindex 5de381e65..099e687e5 100644--- a/test/run.js+++ b/test/run.js@@ -53695,6 +53695,65 @@ test('selection-only response-only phases carry the scope note', async () => {}});+test('a response-only reply that hit the output limit says so instead of reading as finished', async () => {+ const { outputWasCutOff: cutCh, RESPONSE_CUT_OFF_NOTICE: noticeCh } = await import(+ pathToFileURL(path.join(ROOT, 'src/chrome/src/agent/model-output-diagnostics.js')).href+ );+ const { outputWasCutOff: cutFx, RESPONSE_CUT_OFF_NOTICE: noticeFx } = await import(+ pathToFileURL(path.join(ROOT, 'src/firefox/src/agent/model-output-diagnostics.js')).href+ );+ assert.equal(noticeCh, noticeFx);+ for (const outputWasCutOff of [cutCh, cutFx]) {+ for (const result of [+ { finishReason: 'length' },+ { finishReason: 'max_tokens' },+ { stop_reason: 'max_tokens' },+ { finishReason: 'MAX_TOKENS' },+ { finishReason: 'output_limit' },+ { raw: { choices: [{ finish_reason: 'length' }] } },+ { outputTruncated: true },+ ]) {+ assert.equal(outputWasCutOff(result), true, JSON.stringify(result));+ }+ for (const result of [+ {}, null, undefined, { finishReason: 'stop' }, { finishReason: 'end_turn' },+ { outputTruncated: false }, { outputTruncated: 'true' }, { raw: { choices: [{ finish_reason: 'stop' }] } },+ { finishReason: 'tool_calls' },+ ]) {+ assert.equal(outputWasCutOff(result), false, JSON.stringify(result));+ }+ }++ for (const [label, AgentClass, notice] of [['chrome', AgentCh, noticeCh], ['firefox', AgentFx, noticeFx]]) {+ const run = async (reply, phase = 'response_only') => {+ const provider = {+ supportsTools: false, supportsVision: false, promptTier: 'full', contextWindow: 128000,+ model: 'test-model', name: 'test-provider',+ chat: async () => reply,+ };+ const agent = new AgentClass({ getActive: () => provider, getVisionProvider: async () => null });+ agent._checkCostAllowance = async () => null;+ agent._recordCostUsage = async () => null;+ return agent._generateContextOnlyResponse(+ 1, [{ role: 'system', content: 'rules' }, { role: 'user', content: 'Translate it.' }],+ provider, agent._newCostRunState(), null, { phase, step: 1 },+ );+ };+ assert.equal(await run({ content: 'Partial translation', toolCalls: null, outputTruncated: true }),+ `Partial translation${notice}`, `${label}: a worker-reported cut-off is flagged`);+ assert.equal(await run({ content: 'Partial translation', toolCalls: null, finishReason: 'length' }),+ `Partial translation${notice}`, `${label}: a length finish is flagged`);+ assert.equal(await run({ content: 'Whole translation', toolCalls: null, finishReason: 'stop' }),+ 'Whole translation', `${label}: a finished reply is left alone`);+ assert.equal(await run({ content: 'Whole translation', toolCalls: null, outputTruncated: false }),+ 'Whole translation', `${label}: a worker that finished is left alone`);+ assert.equal(await run({ content: '', toolCalls: null, finishReason: 'length' }),+ '', `${label}: an empty cut-off reply keeps its existing empty-output handling`);+ assert.ok(!(await run({ content: '{"final_text":"Recovered"}', toolCalls: null, finishReason: 'length' }, 'terminal_recovery')).includes('output limit'),+ `${label}: only the response-only phase gets the notice`);+ }+});+test('ordinary attachments leave selection grounding and remain usable', async () => {for (const [label, AgentClass, buildSelectionPrompt, sourceGrounding] of [['chrome', AgentCh, buildSelectionPromptCh, SELECTION_ONLY_SOURCE_GROUNDING_CH],