Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Follow a side panel the browser moves to another window

Claudef9f236b3fc95
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 632ef0ae6..58dd8c883 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,7 +10,7 @@ This changelog was generated from the repository Git history and release tags. V
- Selecting text on a page now shows a small bar, **Translate | Explain | Proofread | More…**, instead of the single `?` button. Translate, Explain and Proofread answer in a card next to the selection: Translate shows the detected source language, the original text and a target-language picker, and every card has an "Ask a follow-up" box that, once the answer is there, continues in the side panel with the card's answer as context. **More…** opens the same popup the `?` button opened (Summarize, Quiz me, Humanize, a question box and Hide this; Translate, Explain and Proofread are on the bar, so they are not repeated there). A card sends the selected text to your configured model provider, like any other selection action; selections over 4,000 characters go to the side panel instead of a card. A card answer that reaches the model's output limit says it was cut off and offers to continue in the side panel, which picks up from the unfinished text instead of starting over (a side-panel reply that is cut off again says so, including when an OpenAI Responses model reports the limit as an error, in which case the text it wrote is kept and the call still counts toward your cloud-cost allowance), and closing or redoing a card, or reloading or leaving the page, cancels the request it no longer needs. Chrome and Firefox.
### Fixed
-- Chrome: sending a follow-up or a selection action to WebBrain while the side panel is already open no longer closes it. Each side panel now tells the background whether it is on screen, and the background skips re-opening a panel that is already showing and hands it the prompt directly A panel that was open while the background restarted tells the new worker about itself at once instead of after a delay, so a request in that moment does not close it. The same holds for actions chosen on a selection in WebBrain's PDF viewer, where a panel that was closed is opened once rather than twice.
+- Chrome: sending a follow-up or a selection action to WebBrain while the side panel is already open no longer closes it. Each side panel now tells the background whether it is on screen, and the background skips re-opening a panel that is already showing and hands it the prompt directly A panel that was open while the background restarted tells the new worker about itself at once instead of after a delay, so a request in that moment does not close it. The same holds for actions chosen on a selection in WebBrain's PDF viewer, where a panel that was closed is opened once rather than twice. A side panel that the browser moves to another window (its tab dragged out) now reports its new window, so an action in that window no longer closes it and an action in the old window opens a panel again.
- Stopping a task while WebBrain was waiting for page navigation to settle now stops it immediately. The pending settle step previously kept waiting and could let the run continue after you chose Stop.
- Mastodon: replying to toots in a narrow window no longer posts the same reply several times and stops before the next toot. There, a toot's Reply button opens the composer at /publish, and after posting Mastodon goes back to the previous column by itself. WebBrain took both page changes for something you did, so it reported each click's outcome as unknown; the agent then concluded its reply had not been posted, wrote it again, and never reached the next toot. In Chrome, a page's own Back is now told apart from your browser's Back button: the page a click opens counts as that click's verified effect, and a click after which the page goes back by itself counts as delivered, to be checked on the page. The Mastodon notes now cover replies (start with the prefilled @mentions once, check the reply under the toot before any retry) and stay active on Favourites, Bookmarks and the other columns. Firefox still treats a page's own Back as yours until its Back button reporting is verified.
- Finishing a form you asked WebBrain not to submit no longer takes most of a minute. The completion check removed every line that mentioned a password, so "fill out this form with a random password … don't submit" reached its judge as "[credential omitted]", and so did the approved plan and the agent's summary. Seeing only an unsubmitted signup form, the judge answered "pending" again and again; in the trace it took four attempts and about 50 seconds, ending only when a slow screenshot check ran out of time. The active model's check, which already holds these texts in the conversation, now receives every line of them, with only addresses shortened and bearer tokens and entered passwords hidden, as in all judge evidence. A decision model on this device, or WebBrain Cloud's own while WebBrain Cloud is also your provider, receives that same copy; any other decision model keeps whole-line redaction, and when that leaves it without the task, it is no longer asked while the active model can judge. A password WebBrain typed into a password field during the run is now hidden in the text judges receive, such as the agent's summary or a field a "show password" control revealed, unless the typing tool could not tell what kind of field it was; other entered values stay visible, so a judge can still catch a mistyped one. The judges are told that a task asking not to submit is complete once the staged values match, that a hidden value neither proves nor disproves an entry, and, for each recorded entry, whether its field read back what was entered, whether it is a credential field and which field it targeted; a credential field's line in the page evidence keeps its role and reference. Staged field edits are judged from the accessibility tree first: a screenshot shows passwords only as dots, and the focused field's blinking caret made each one stale before its verdict returned.
diff --git a/docs/architecture.md b/docs/architecture.md
index 56085bf1e..caa505312 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -559,6 +559,9 @@ was dropped by a background restart reconnects at once (only a port that is refu
away backs off), so the restarted worker is not left unaware of a panel that is still showing.
A new panel reports as soon as it knows its window, ahead of its other start-up awaits, so an
action arriving while it is still opening does not open it a second time.
+A panel the browser carries to another window (the tab dragged out) keeps its document; the window scope
+notices on its next live `windows.getCurrent()` lookup and the reporter (`setWindowId()`) names the new
+window, so the background does not treat the old window as occupied and the new one as empty.
A selection made in the bundled PDF viewer, whose message carries only a tab id, brings the
window too (Chrome's `sender.tab` for that tab, else the window the viewer reported from
`windows.getCurrent()`), so the same check applies there; that path opens the panel once in the
diff --git a/src/chrome/src/sidepanel-presence.js b/src/chrome/src/sidepanel-presence.js
index 19bbdee48..96f4f5711 100644
--- a/src/chrome/src/sidepanel-presence.js
+++ b/src/chrome/src/sidepanel-presence.js
@@ -55,8 +55,12 @@ export function createSidePanelPresence(api) {
}
/**
- * Side panel side: keeps a port open and reports whether this page is visible.
- * The port is reopened when the background worker restarts, which drops it.
+ * Side panel side: keeps a port open and reports whether this page is visible, and in
+ * which window. The port is reopened when the background worker restarts, which drops
+ * it. A browser can keep this document alive while its tab is dragged to another
+ * window, so the caller says so through `setWindowId()` and the next report names the
+ * window the panel is really in; a panel whose window was unknown at start connects
+ * once it learns it.
*/
export function reportSidePanelPresence({
runtime,
@@ -65,19 +69,22 @@ export function reportSidePanelPresence({
schedule = (callback, delay) => setTimeout(callback, delay),
now = () => Date.now(),
}) {
- if (!Number.isInteger(windowId)) return { stop() {} };
+ let currentWindowId = Number.isInteger(windowId) ? windowId : null;
let port = null;
let stopped = false;
+ let reconnectPending = false;
let quickCloses = 0;
const report = () => {
+ if (currentWindowId == null) return;
try {
- port?.postMessage({ visible: doc.visibilityState === 'visible', windowId });
+ port?.postMessage({ visible: doc.visibilityState === 'visible', windowId: currentWindowId });
} catch { /* the port closed under us; its disconnect handler reconnects */ }
};
const connect = () => {
- if (stopped) return;
+ reconnectPending = false;
+ if (stopped || currentWindowId == null) return;
let next;
try {
next = runtime.connect({ name: SIDEPANEL_PRESENCE_PORT });
@@ -97,6 +104,7 @@ export function reportSidePanelPresence({
const delay = quickCloses === 0
? 0
: Math.min(RECONNECT_MAX_MS, RECONNECT_BASE_MS * 2 ** (quickCloses - 1));
+ reconnectPending = true;
schedule(connect, delay);
});
report();
@@ -105,6 +113,12 @@ export function reportSidePanelPresence({
doc.addEventListener('visibilitychange', report);
connect();
return {
+ setWindowId(nextWindowId) {
+ if (stopped || !Number.isInteger(nextWindowId) || nextWindowId === currentWindowId) return;
+ currentWindowId = nextWindowId;
+ if (port) report();
+ else if (!reconnectPending) connect();
+ },
stop() {
stopped = true;
doc.removeEventListener('visibilitychange', report);
diff --git a/src/chrome/src/ui/sidepanel-window-scope.js b/src/chrome/src/ui/sidepanel-window-scope.js
index 5d9a6ec9a..4d0b3d957 100644
--- a/src/chrome/src/ui/sidepanel-window-scope.js
+++ b/src/chrome/src/ui/sidepanel-window-scope.js
@@ -16,15 +16,25 @@ export function createSidePanelWindowScope({
getRenderedTabId,
switchToTab,
settleWindowTransfer = () => new Promise(resolve => setTimeout(resolve, 0)),
+ // Told when a live lookup finds the panel in a different window than the last one
+ // announced, so whoever else keys on the window (the presence reporter) can follow.
+ onWindowChanged = () => {},
}) {
let ownWindowId = initialWindowId;
+ let announcedWindowId = initialWindowId;
let tabTransfer = null;
let syncGeneration = 0;
async function refreshOwnWindowId() {
try {
const ownWindow = await browserApi.windows.getCurrent();
- if (ownWindow?.id != null) ownWindowId = ownWindow.id;
+ if (ownWindow?.id != null) {
+ ownWindowId = ownWindow.id;
+ if (ownWindowId !== announcedWindowId) {
+ announcedWindowId = ownWindowId;
+ try { onWindowChanged(ownWindowId); } catch { /* a listener must not break tab sync */ }
+ }
+ }
} catch {
// Keep the last confirmed ID. A transient windows API failure should
// not make an unrelated window eligible to control this panel.
diff --git a/src/chrome/src/ui/sidepanel.js b/src/chrome/src/ui/sidepanel.js
index fcea26243..846bbb0ca 100644
--- a/src/chrome/src/ui/sidepanel.js
+++ b/src/chrome/src/ui/sidepanel.js
@@ -5032,8 +5032,9 @@ async function init() {
// is reported as soon as the window is known, ahead of the awaits below, so an action
// that arrives while the panel is still starting up does not open it a second time.
const presenceReportedEarly = !isBtwWindow && initialWindowId != null;
+ let presenceReporter = null;
if (presenceReportedEarly) {
- reportSidePanelPresence({ runtime: chrome.runtime, doc: document, windowId: initialWindowId });
+ presenceReporter = reportSidePanelPresence({ runtime: chrome.runtime, doc: document, windowId: initialWindowId });
}
const [tab] = await chrome.tabs.query(initialWindowId != null
? { active: true, windowId: initialWindowId }
@@ -5069,11 +5070,13 @@ async function init() {
getCurrentTabId: () => currentTabId,
getRenderedTabId: () => renderedTabId,
switchToTab,
+ // The browser may carry this panel to another window; the background must follow.
+ onWindowChanged: (windowId) => presenceReporter?.setWindowId(windowId),
});
// The window could not be read up front; fall back to the active tab's.
if (!isBtwWindow && !presenceReportedEarly) {
- reportSidePanelPresence({ runtime: chrome.runtime, doc: document, windowId: tab?.windowId ?? null });
+ presenceReporter = reportSidePanelPresence({ runtime: chrome.runtime, doc: document, windowId: tab?.windowId ?? null });
}
if (windowScope) {
diff --git a/src/firefox/src/ui/sidepanel-window-scope.js b/src/firefox/src/ui/sidepanel-window-scope.js
index 5d9a6ec9a..4d0b3d957 100644
--- a/src/firefox/src/ui/sidepanel-window-scope.js
+++ b/src/firefox/src/ui/sidepanel-window-scope.js
@@ -16,15 +16,25 @@ export function createSidePanelWindowScope({
getRenderedTabId,
switchToTab,
settleWindowTransfer = () => new Promise(resolve => setTimeout(resolve, 0)),
+ // Told when a live lookup finds the panel in a different window than the last one
+ // announced, so whoever else keys on the window (the presence reporter) can follow.
+ onWindowChanged = () => {},
}) {
let ownWindowId = initialWindowId;
+ let announcedWindowId = initialWindowId;
let tabTransfer = null;
let syncGeneration = 0;
async function refreshOwnWindowId() {
try {
const ownWindow = await browserApi.windows.getCurrent();
- if (ownWindow?.id != null) ownWindowId = ownWindow.id;
+ if (ownWindow?.id != null) {
+ ownWindowId = ownWindow.id;
+ if (ownWindowId !== announcedWindowId) {
+ announcedWindowId = ownWindowId;
+ try { onWindowChanged(ownWindowId); } catch { /* a listener must not break tab sync */ }
+ }
+ }
} catch {
// Keep the last confirmed ID. A transient windows API failure should
// not make an unrelated window eligible to control this panel.
diff --git a/test/run.js b/test/run.js
index c64d3069a..a05c0d298 100644
--- a/test/run.js
+++ b/test/run.js
@@ -48753,11 +48753,13 @@ test('sidepanel follows its live window when the represented tab is detached', a
},
},
};
+ const windowChanges = [];
const scope = createSidePanelWindowScope({
browserApi,
initialWindowId: 10,
getCurrentTabId: () => currentTabId,
getRenderedTabId: () => renderedTabId,
+ onWindowChanged: windowId => windowChanges.push(windowId),
settleWindowTransfer: () => new Promise(resolve => {
releaseWindowTransfer = resolve;
}),
@@ -48781,6 +48783,7 @@ test('sidepanel follows its live window when the represented tab is detached', a
await attaching;
assert.deepEqual(switches, [101], `${label}: attach should resync to the active tab in the panel's new window`);
switches.length = 0;
+ assert.deepEqual(windowChanges, [20], `${label}: the move to another window should be announced once`);
activeTabs.set(10, { id: 103, windowId: 10 });
await scope.handleActivated({ tabId: 103, windowId: 10 });
@@ -48790,6 +48793,43 @@ test('sidepanel follows its live window when the represented tab is detached', a
await scope.handleActivated({ tabId: 201, windowId: 20 });
assert.deepEqual(switches, [201], `${label}: activity in the panel's live window should still switch tab-scoped chat`);
assert.equal(queries.every(query => query.currentWindow == null && query.windowId != null), true, `${label}: active-tab queries must use an explicit live window ID`);
+ assert.deepEqual(windowChanges, [20], `${label}: lookups that find the panel where it already was announce nothing`);
+
+ // A move the browser reverses (the panel stayed where it was) is not a change.
+ let stayedWindowId = 30;
+ const stayedChanges = [];
+ const stayed = createSidePanelWindowScope({
+ browserApi: {
+ windows: { getCurrent: async () => ({ id: stayedWindowId }) },
+ tabs: { query: async () => [{ id: 301, windowId: stayedWindowId }] },
+ },
+ initialWindowId: 30,
+ getCurrentTabId: () => 301,
+ getRenderedTabId: () => 301,
+ onWindowChanged: windowId => stayedChanges.push(windowId),
+ settleWindowTransfer: async () => {},
+ switchToTab: async () => {},
+ });
+ stayed.handleDetached(301, { oldWindowId: 30 });
+ await stayed.handleAttached(301, { newWindowId: 31 });
+ assert.deepEqual(stayedChanges, [], `${label}: a panel the browser kept in its window is not reported as moved`);
+ stayedWindowId = 31;
+ await stayed.syncActiveTab();
+ assert.deepEqual(stayedChanges, [31], `${label}: a later lookup that finds it elsewhere announces the move`);
+
+ // A listener that throws must not break tab sync.
+ const noisy = createSidePanelWindowScope({
+ browserApi: {
+ windows: { getCurrent: async () => ({ id: 41 }) },
+ tabs: { query: async () => [{ id: 401, windowId: 41 }] },
+ },
+ initialWindowId: 40,
+ getCurrentTabId: () => 400,
+ getRenderedTabId: () => 400,
+ onWindowChanged: () => { throw new Error('listener failed'); },
+ switchToTab: async () => {},
+ });
+ assert.equal((await noisy.syncActiveTab())?.id, 401, `${label}: a failing window listener must not stop the tab sync`);
}
});
@@ -52775,7 +52815,51 @@ test('side panel presence: the background learns which windows show a panel, and
// Without a window it reports nothing, which keeps the old always-open behaviour.
const anonymous = reportSidePanelPresence({ runtime, doc, windowId: null });
assert.equal(connections.length, 5);
+
+ // ...until the window is learned, when it connects and says where it stands.
+ posted.length = 0;
+ anonymous.setWindowId(4);
+ assert.equal(connections.length, 6, 'learning the window opens the port');
+ assert.deepEqual(posted, [{ visible: true, windowId: 4 }]);
anonymous.stop();
+
+ // A browser can carry the panel document to another window: the background must be
+ // told, or it keeps the old window occupied and the new one empty.
+ posted.length = 0;
+ const mover = reportSidePanelPresence({
+ runtime,
+ doc,
+ windowId: 5,
+ schedule: (callback, delay) => timers.push({ callback, delay }),
+ now: () => clock,
+ });
+ const moverConnections = connections.length;
+ assert.deepEqual(posted, [{ visible: true, windowId: 5 }]);
+ mover.setWindowId(9);
+ assert.deepEqual(posted.at(-1), { visible: true, windowId: 9 }, 'a move to another window is reported on the open port');
+ assert.equal(connections.length, moverConnections, 'a move does not need a new port');
+ posted.length = 0;
+ mover.setWindowId(9);
+ mover.setWindowId(null);
+ mover.setWindowId('11');
+ mover.setWindowId(undefined);
+ assert.deepEqual(posted, [], 'a repeated or unusable window id says nothing');
+ doc.visibilityState = 'hidden';
+ doc.listeners.forEach(callback => callback());
+ assert.deepEqual(posted.at(-1), { visible: false, windowId: 9 }, 'later reports keep naming the new window');
+ doc.visibilityState = 'visible';
+
+ // A move while the port is down is picked up by the reconnect that is already coming.
+ clock += 60_000;
+ connections.at(-1).drop();
+ mover.setWindowId(12);
+ assert.equal(connections.length, moverConnections, 'a pending reconnect is not raced by a second port');
+ timers.splice(0).forEach(timer => timer.callback());
+ assert.equal(connections.length, moverConnections + 1);
+ assert.deepEqual(posted.at(-1), { visible: true, windowId: 12 }, 'the reconnect names the window the panel moved to');
+ mover.stop();
+ mover.setWindowId(13);
+ assert.equal(connections.length, moverConnections + 1, 'a stopped reporter ignores a move');
});
test('background does not re-open a side panel that is already showing, and still opens one that is not', () => {
@@ -52838,10 +52922,12 @@ test('background does not re-open a side panel that is already showing, and stil
assert.match(background, /chrome\.runtime\.onConnect\.addListener\(\(port\) => \{ sidePanelPresence\.attach\(port\); \}\);/,
'the presence port must be accepted at the top level of the worker');
const panel = fs.readFileSync(path.join(ROOT, 'src/chrome/src/ui/sidepanel.js'), 'utf8');
- assert.match(panel, /const initialWindowId = initialWindow\?\.id \?\? null;[\s\S]*?const presenceReportedEarly = !isBtwWindow && initialWindowId != null;\s*if \(presenceReportedEarly\) \{\s*reportSidePanelPresence\(\{ runtime: chrome\.runtime, doc: document, windowId: initialWindowId \}\);\s*\}\s*const \[tab\] = await chrome\.tabs\.query/,
+ assert.match(panel, /const initialWindowId = initialWindow\?\.id \?\? null;[\s\S]*?const presenceReportedEarly = !isBtwWindow && initialWindowId != null;\s*let presenceReporter = null;\s*if \(presenceReportedEarly\) \{\s*presenceReporter = reportSidePanelPresence\(\{ runtime: chrome\.runtime, doc: document, windowId: initialWindowId \}\);\s*\}\s*const \[tab\] = await chrome\.tabs\.query/,
'the side panel must tell the background it is on screen as soon as its window is known, before its other awaits');
- assert.match(panel, /if \(!isBtwWindow && !presenceReportedEarly\) \{\s*reportSidePanelPresence\(\{ runtime: chrome\.runtime, doc: document, windowId: tab\?\.windowId \?\? null \}\);/,
+ assert.match(panel, /if \(!isBtwWindow && !presenceReportedEarly\) \{\s*presenceReporter = reportSidePanelPresence\(\{ runtime: chrome\.runtime, doc: document, windowId: tab\?\.windowId \?\? null \}\);/,
'and still report when the window could only be learned from the active tab');
+ assert.match(panel, /createSidePanelWindowScope\(\{[^}]*?onWindowChanged: \(windowId\) => presenceReporter\?\.setWindowId\(windowId\),/,
+ 'a panel the browser moves to another window must tell the background, or the old window stays occupied and the new one looks empty');
});
test('a PDF viewer selection leaves a side panel that is already showing in its window alone', async () => {