The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
fix(browser-extension): keep queued telemetry through collector outages (#863) (#871)
claude[bot]e93aaa8f243e
diff --git a/browser-extension/README.md b/browser-extension/README.mdindex 1ff7ff5b..ccfc8bdf 100644--- a/browser-extension/README.md+++ b/browser-extension/README.md@@ -180,8 +180,8 @@ How the Firefox build differs, all generated from `src/manifest.json` byNot yet verified on a real Firefox (no Firefox runs in CI): the event page's POST to`http://127.0.0.1:4318/v1/logs`, and the retry cadence. Chrome clamps `chrome.alarms` to about-30 seconds; Firefox's clamp has not been measured. The retry code is the same on both, and the-durable queue means delivery is eventual on either engine.+30 seconds; Firefox's clamp has not been measured. The retry code and the queue's limits are the+same on both engines.## Retained content@@ -305,11 +305,21 @@ dashboard), including interleaved sessions with no cross-contamination.stall between chunks could drop a turn. The durable delivery queue (in `chrome.storage`) protectseverything *after* a turn is assembled, not during. Persisting partial assembler state is afuture item.-- **Delivery retry cadence.** Each POST has a ten-second deadline, so a collector that accepts- a connection but never responds cannot hold the queue indefinitely. Timed-out batches remain- durable and follow the existing retry limit. Retries use `chrome.alarms`, which MV3 clamps- to a ~30s minimum, so the first retry lands at ~30s rather than the computed sub-second backoff. Fine for a- local-loopback collector that is almost always reachable; the queue guarantees eventual delivery.+- **Delivery retry cadence and queue limits.** Each POST has a ten-second deadline, so a collector+ that accepts a connection but never responds cannot hold the queue indefinitely; a timed-out+ batch is retried like any other failure. Retries back off from half a second, doubling to a+ one-minute cap, and run on `chrome.alarms`, which MV3 clamps to a ~30s minimum, so a recovered+ collector receives the backlog within about a minute. Batches are delivered oldest first, and+ while the collector is down only the oldest batch is retried; newer ones wait behind it. A+ failed send (connection refused, timeout, or a retryable 429, 502, 503 or 504) never drops a+ batch. A batch is dropped undelivered when the collector rejects it with any other status,+ since sending it again cannot succeed, when it is older than **24 hours**, or when the queue+ passes **2,000 batches** or **6 MiB** of envelopes, which drops the oldest batches first (the+ byte cap keeps the queue inside `chrome.storage.local`'s 10 MiB quota). Within those limits, a+ collector outage of any length (restarting during an install or upgrade, not yet started after+ boot, crash-looping) loses nothing. The queue is re-checked each+ time the service worker starts, because a browser restart or extension update can clear the+ retry alarm.- **Capture depends on `window.fetch`.** Confirmed correct for claude.ai. Sites using XHR or a pageservice worker for streaming would need additional interception (deferred).- **Single capture path.** No DOM-scraping fallback yet; if a site changes its SSE wire format, thediff --git a/browser-extension/src/background/delivery.ts b/browser-extension/src/background/delivery.tsindex e38e3612..a8b1ce81 100644--- a/browser-extension/src/background/delivery.ts+++ b/browser-extension/src/background/delivery.ts@@ -2,6 +2,14 @@// chrome.storage.local so nothing is lost if the SW is suspended, then POSTed// to the local beacon collector. Failed sends retry with exponential backoff// driven by chrome.alarms (which survives suspension, unlike setTimeout).+//+// Retention policy: a queued batch is kept until it is delivered, until it is+// older than MAX_QUEUE_AGE_MS, or until newer batches push it past the queue's+// size caps (MAX_QUEUE_ITEMS, MAX_QUEUE_BYTES), which drop the oldest first.+// A failed send never drops a batch on its own, so a collector that is down+// for minutes (restarting during an install or upgrade, not yet started after+// boot, crash-looping) loses nothing it comes back from within the age limit.+// A batch the collector rejects (a non-retryable status) is dropped at once.import { ext } from '../shared/browser.js';import type { ChatTurn, Settings } from '../shared/types.js';@@ -13,16 +21,34 @@ import { serial } from '../shared/serial.js';const QUEUE_KEY = 'delivery_queue';const ALARM = 'beacon_flush';-const MAX_ATTEMPTS = 6;const POST_TIMEOUT_MS = 10_000;+/** A batch older than this is dropped undelivered. */+export const MAX_QUEUE_AGE_MS = 24 * 60 * 60 * 1000;+/** Most batches the queue holds; the oldest are dropped past it. */+export const MAX_QUEUE_ITEMS = 2_000;+/**+ * Most envelope bytes (UTF-8 JSON) the queue holds; the oldest batches are+ * dropped past it. Chrome's storage.local quota is 10 MiB for the whole+ * extension without `unlimitedStorage`, so this leaves room for settings and for+ * the queue's own bookkeeping fields.+ */+export const MAX_QUEUE_BYTES = 6 * 1024 * 1024;+/** Longest wait between retries, and so the longest delay after the collector recovers. */+export const MAX_RETRY_DELAY_MS = 60_000;+interface QueueItem {id: string;endpoint: string;envelope: LogsEnvelope;+ /** Failed sends so far. Sets the backoff; never a reason to drop the batch. */attempts: number;/** Persisted deadline so new turns cannot consume retries before backoff. */nextAttemptAt?: number;+ /** When the batch was queued (epoch ms); the age limit counts from here. */+ queuedAt?: number;+ /** UTF-8 size of the serialized envelope, counted against MAX_QUEUE_BYTES. */+ bytes?: number;}async function readQueue(): Promise<QueueItem[]> {@@ -50,52 +76,97 @@ export async function enqueueTurn(endpoint: settings.endpoint,envelope,attempts: 0,+ queuedAt: Date.now(),};await withQueueLock(async () => {const queue = await readQueue();queue.push(item);- await writeQueue(queue);+ await writeQueue(pruneQueue(queue, Date.now()));});await flush();}-/** Attempt to POST everything queued; requeue failures with backoff. */+/**+ * POST every queued batch that is due, oldest first; requeue failures with backoff.+ *+ * Once a batch for an endpoint fails or is still waiting out its backoff, the+ * endpoint's later batches wait behind it in this pass. A recovered collector+ * therefore receives batches in the order they were captured, and a collector+ * that is down gets one probe per pass instead of one POST per queued batch.+ */export async function flush(): Promise<void> {await withQueueLock(async () => {- const queue = await readQueue();- if (queue.length === 0) return;+ const stored = await readQueue();+ if (stored.length === 0) return;+ const queue = pruneQueue(stored, Date.now());const remaining: QueueItem[] = [];- let anyFailed = false;+ const waiting = new Set<string>();for (const item of queue) {// Checked again at send time, not only when settings are saved: an item// queued by an older build, or a tampered queue, must not reach anything// but the local collector. Such an item is dropped, not retried.if (!isAllowedEndpoint(item.endpoint)) continue;- if (item.nextAttemptAt != null && item.nextAttemptAt > Date.now()) {+ if (waiting.has(item.endpoint) || (item.nextAttemptAt ?? 0) > Date.now()) {remaining.push(item);- anyFailed = true;+ waiting.add(item.endpoint);continue;}- const ok = await post(item);- if (!ok) {- item.attempts += 1;- if (item.attempts < MAX_ATTEMPTS) {- item.nextAttemptAt = Date.now() + retryDelay(item.attempts);- remaining.push(item);- anyFailed = true;- } // else: give up, drop it (avoid an unbounded queue)- }+ const outcome = await post(item);+ // A batch the collector rejects outright would fail the same way forever+ // and hold every later batch behind it, so it is dropped, not retried.+ if (outcome !== 'retry') continue;+ item.attempts += 1;+ item.nextAttemptAt = Date.now() + retryDelay(item.attempts);+ remaining.push(item);+ waiting.add(item.endpoint);}await writeQueue(remaining);- if (anyFailed) scheduleRetry(remaining);+ scheduleRetry(remaining);});}-async function post(item: QueueItem): Promise<boolean> {- if (!isAllowedEndpoint(item.endpoint)) return false;+/**+ * Drop batches past the age limit, then the oldest batches until the queue fits+ * its caps. The newest batch is always kept. A batch from an older build has no+ * `queuedAt`; it is stamped now, so it gets a full age window rather than none.+ */+function pruneQueue(queue: QueueItem[], now: number): QueueItem[] {+ const fresh = queue.filter((item) => {+ item.queuedAt ??= now;+ return now - item.queuedAt < MAX_QUEUE_AGE_MS;+ });+ let bytes = fresh.reduce((sum, item) => sum + itemBytes(item), 0);+ let drop = 0;+ while (+ fresh.length - drop > 1 &&+ (fresh.length - drop > MAX_QUEUE_ITEMS || bytes > MAX_QUEUE_BYTES)+ ) {+ bytes -= itemBytes(fresh[drop]);+ drop += 1;+ }+ return fresh.slice(drop);+}++const utf8 = new TextEncoder();++function itemBytes(item: QueueItem): number {+ item.bytes ??= utf8.encode(JSON.stringify(item.envelope)).length;+ return item.bytes;+}++/**+ * Statuses the OTLP/HTTP spec marks retryable. Any other non-2xx response+ * rejects the batch itself, so sending it again cannot succeed.+ */+const RETRYABLE_STATUSES = new Set([429, 502, 503, 504]);++type PostOutcome = 'sent' | 'retry' | 'rejected';++async function post(item: QueueItem): Promise<PostOutcome> {+ if (!isAllowedEndpoint(item.endpoint)) return 'rejected';try {const res = await fetch(item.endpoint, {method: 'POST',@@ -107,28 +178,37 @@ async function post(item: QueueItem): Promise<boolean> {redirect: 'error',body: JSON.stringify(item.envelope),});- return res.ok;+ if (res.ok) return 'sent';+ return RETRYABLE_STATUSES.has(res.status) ? 'retry' : 'rejected';} catch {- return false;+ // Connection refused, timed out or redirected: the collector may come back.+ return 'retry';}}+/** Backoff after the given number of failed sends: 0.5s, 1s, 2s, ... capped at a minute. */function retryDelay(attempts: number): number {- return Math.min(250 * 2 ** attempts, 60_000);+ return Math.min(250 * 2 ** attempts, MAX_RETRY_DELAY_MS);}function scheduleRetry(queue: QueueItem[]): void {- // Preserve each failed batch's deadline when unrelated new turns arrive.- // Chrome may clamp alarm cadence, but an enqueue must never accelerate a- // retry or repeatedly postpone the existing earliest scheduled deadline.- const when = Math.min(...queue.map((item) =>- item.nextAttemptAt ?? Date.now() + retryDelay(item.attempts)));- ext.alarms.create(ALARM, { when });+ // Wake at the earliest stored deadline. A batch waiting behind another has+ // none of its own, so an unrelated new turn neither accelerates nor postpones+ // the retry. Chrome clamps the alarm to about 30 seconds.+ let when = Infinity;+ for (const item of queue) {+ if (item.nextAttemptAt != null && item.nextAttemptAt < when) when = item.nextAttemptAt;+ }+ if (when !== Infinity) ext.alarms.create(ALARM, { when });}-/** Wire the alarm listener (called once from the SW entry). */+/** Wire the alarm listener and resume delivery (called once from the SW entry). */export function installFlushAlarm(): void {ext.alarms.onAlarm.addListener((alarm) => {if (alarm.name === ALARM) void flush();});+ // Alarms can be cleared by a browser restart and are cleared by an extension+ // update, so a queue left from before would otherwise wait for the next turn.+ // Flushing on worker start sends what is due and re-arms the retry for the rest.+ void flush();}diff --git a/browser-extension/test/unit/delivery-outage.test.ts b/browser-extension/test/unit/delivery-outage.test.tsnew file mode 100644index 00000000..6226195f--- /dev/null+++ b/browser-extension/test/unit/delivery-outage.test.ts@@ -0,0 +1,431 @@+// Collector outages against the durable delivery queue (#863). A fake+// extension environment stands in for the browser: storage.local persists+// across simulated service-worker restarts, alarms follow Chrome's ~30s clamp,+// and a fake loopback collector records every POST it sees. Time is driven by+// fake timers, so a ten-minute or day-long outage runs instantly.+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import type { ChatTurn, Settings } from '../../src/shared/types.js';+import { DEFAULT_SETTINGS } from '../../src/shared/types.js';++const MINUTE = 60_000;+const ALARM_CLAMP_MS = 30_000;+const ENDPOINT = DEFAULT_SETTINGS.endpoint;++type Alarm = { name: string };+type Listener = (alarm: Alarm) => void;+const env = vi.hoisted(() => ({+ storage: new Map<string, unknown>(),+ alarms: new Map<string, number>(),+ listeners: [] as Listener[],+}));++vi.mock('../../src/shared/browser.js', () => ({+ ext: {+ storage: {+ local: {+ // Clone both ways, as the real storage serializes values.+ get: async (key: string) =>+ env.storage.has(key) ? { [key]: structuredClone(env.storage.get(key)) } : {},+ set: async (items: Record<string, unknown>) => {+ for (const [k, v] of Object.entries(items)) env.storage.set(k, structuredClone(v));+ },+ },+ },+ alarms: {+ // Chrome delays an alarm due sooner than ~30s from now to ~30s from now.+ create: (name: string, info: { when: number }) => {+ env.alarms.set(name, Math.max(info.when, Date.now() + ALARM_CLAMP_MS));+ },+ onAlarm: { addListener: (fn: Listener) => env.listeners.push(fn) },+ },+ },+}));++type Delivery = typeof import('../../src/background/delivery.js');+type StoredItem = {+ id: string; endpoint: string; envelope: unknown; attempts: number;+ nextAttemptAt?: number; queuedAt?: number; bytes?: number;+};++/**+ * A fake loopback collector: records every POST and accepts only while `up`.+ * `answers` gives the HTTP status it returns for a prompt id while up.+ */+const collector = {+ up: true,+ attempted: [] as string[],+ received: [] as string[],+ byEndpoint: new Map<string, boolean>(),+ answers: new Map<string, number>(),+};++function promptId(body: unknown): string | undefined {+ if (Array.isArray(body)) {+ for (const v of body) {+ const found = promptId(v);+ if (found) return found;+ }+ } else if (body && typeof body === 'object') {+ const o = body as Record<string, unknown>;+ if (o.key === 'beacon.prompt.id') return (o.value as { stringValue: string }).stringValue;+ for (const v of Object.values(o)) {+ const found = promptId(v);+ if (found) return found;+ }+ }+ return undefined;+}++async function fakeFetch(url: string, init: { body: string }): Promise<{ ok: boolean; status: number }> {+ const id = promptId(JSON.parse(init.body)) ?? '?';+ collector.attempted.push(id);+ const up = collector.byEndpoint.get(url) ?? collector.up;+ if (!up) throw new TypeError('Failed to fetch'); // connection refused+ const status = collector.answers.get(id) ?? 200;+ if (status !== 200) return { ok: false, status };+ collector.received.push(id);+ return { ok: true, status };+}++let delivery: Delivery;++/** Start (or restart) the service worker: fresh module state, persisted storage. */+async function startWorker(): Promise<void> {+ vi.resetModules();+ env.listeners.length = 0;+ delivery = await import('../../src/background/delivery.js');+ delivery.installFlushAlarm();+ await settle();+}++/** Let fire-and-forget flushes finish. The fakes never wait on a macrotask. */+async function settle(): Promise<void> {+ for (let i = 0; i < 5; i++) await new Promise((r) => setImmediate(r));+}++/** Advance the clock by `ms`, firing every alarm that comes due on the way. */+async function runFor(ms: number): Promise<number> {+ const end = Date.now() + ms;+ let fired = 0;+ for (;;) {+ const due = [...env.alarms.entries()].filter(([, t]) => t <= end).sort((a, b) => a[1] - b[1])[0];+ if (!due) break;+ vi.setSystemTime(Math.max(due[1], Date.now()));+ env.alarms.delete(due[0]);+ for (const fn of env.listeners) fn({ name: due[0] });+ fired += 1;+ await settle();+ }+ vi.setSystemTime(end);+ return fired;+}++function turn(id: string, promptText = `prompt ${id}`): ChatTurn {+ return {+ site: 'claude_web',+ sessionId: 'conv-1',+ turnId: id,+ promptText,+ inputMessages: [{ role: 'user', text: `prompt ${id}` }],+ outputMessages: [{ role: 'assistant', text: `answer ${id}` }],+ responseText: `answer ${id}`,+ toolCalls: [],+ startedAt: Date.now(),+ completedAt: Date.now(),+ captureMode: 'sse',+ };+}++const settings: Settings = { ...DEFAULT_SETTINGS };++function queue(): StoredItem[] {+ return (env.storage.get('delivery_queue') as StoredItem[] | undefined) ?? [];+}++function stored(id: string, over: Partial<StoredItem> = {}): StoredItem {+ return { id, endpoint: ENDPOINT, envelope: { id }, attempts: 0, queuedAt: Date.now(), ...over };+}++beforeEach(async () => {+ vi.useFakeTimers({ toFake: ['Date'] });+ vi.setSystemTime(1_000_000);+ env.storage.clear();+ env.alarms.clear();+ collector.up = true;+ collector.attempted = [];+ collector.received = [];+ collector.byEndpoint.clear();+ collector.answers.clear();+ vi.stubGlobal('fetch', vi.fn(fakeFetch));+ await startWorker();+});++afterEach(() => {+ vi.useRealTimers();+ vi.unstubAllGlobals();+});++describe('collector outage', () => {+ it('delivers every turn captured during a ten-minute outage once it recovers', async () => {+ collector.up = false;+ const ids = Array.from({ length: 8 }, (_, i) => `t${i}`);+ for (const id of ids) {+ await delivery.enqueueTurn(turn(id), settings);+ await runFor(75_000); // 8 x 75s = 10 minutes down+ }+ expect(queue().map((i) => i.id)).toEqual(ids);+ expect(collector.received).toEqual([]);++ collector.up = true;+ await runFor(delivery.MAX_RETRY_DELAY_MS);++ expect(collector.received).toEqual(ids); // all of them, in capture order+ expect(queue()).toEqual([]);+ });++ it('keeps retrying through an outage of many hours, well past the old six attempts', async () => {+ collector.up = false;+ await delivery.enqueueTurn(turn('long'), settings);+ const fired = await runFor(6 * 60 * MINUTE);+ expect(fired).toBeGreaterThan(300);+ expect(queue()).toEqual([expect.objectContaining({ id: 'long' })]);+ expect(queue()[0].attempts).toBeGreaterThan(300);++ collector.up = true;+ await runFor(delivery.MAX_RETRY_DELAY_MS);+ expect(collector.received).toEqual(['long']);+ });++ it('probes a down collector with the oldest batch only, not every queued batch', async () => {+ collector.up = false;+ for (const id of ['a', 'b', 'c', 'd']) {+ await delivery.enqueueTurn(turn(id), settings);+ await runFor(MINUTE);+ }+ await runFor(10 * MINUTE);+ expect(new Set(collector.attempted)).toEqual(new Set(['a']));+ });++ it('sends nothing twice: each turn reaches the collector exactly once', async () => {+ collector.up = false;+ const ids = ['x1', 'x2', 'x3', 'x4', 'x5'];+ for (const id of ids) await delivery.enqueueTurn(turn(id), settings);+ await runFor(4 * MINUTE);+ collector.up = true;+ // The backlog is due: the alarm, a popup "flush now" and a new turn all+ // race the same queue.+ vi.setSystemTime(queue()[0].nextAttemptAt!);+ env.alarms.clear();+ await Promise.all([+ ...env.listeners.map(async (fn) => fn({ name: 'beacon_flush' })),+ delivery.flush(),+ delivery.flush(),+ delivery.enqueueTurn(turn('x6'), settings),+ ]);+ await settle();+ expect(collector.received).toEqual([...ids, 'x6']);+ await runFor(10 * MINUTE);+ expect(collector.received).toEqual([...ids, 'x6']);+ expect(queue()).toEqual([]);+ });++ it('holds new turns behind the waiting backlog so a recovery delivers in order', async () => {+ collector.up = false;+ await delivery.enqueueTurn(turn('old'), settings);+ await runFor(5 * MINUTE);+ collector.up = true;+ // The collector is back, but the backlog's retry is not due yet. A new turn+ // must not jump ahead of it.+ await delivery.enqueueTurn(turn('new'), settings);+ expect(collector.received).toEqual([]);+ await runFor(delivery.MAX_RETRY_DELAY_MS);+ expect(collector.received).toEqual(['old', 'new']);+ });++ it('keeps the undelivered tail in order when the collector drops mid-drain', async () => {+ collector.up = false;+ const ids = ['m1', 'm2', 'm3', 'm4'];+ for (const id of ids) await delivery.enqueueTurn(turn(id), settings);+ await runFor(2 * MINUTE);++ collector.up = true;+ let died = false;+ vi.stubGlobal('fetch', vi.fn(async (url: string, init: { body: string }) => {+ if (!died && collector.received.length === 2) {+ died = true; // the collector dies after accepting two batches+ collector.up = false;+ }+ return fakeFetch(url, init);+ }));+ await runFor(delivery.MAX_RETRY_DELAY_MS);+ expect(collector.received).toEqual(['m1', 'm2']);+ expect(queue().map((i) => i.id)).toEqual(['m3', 'm4']);++ collector.up = true;+ await runFor(delivery.MAX_RETRY_DELAY_MS);+ expect(collector.received).toEqual(ids);+ });++ it('does not let one unreachable endpoint hold back batches for another', async () => {+ const other = 'http://localhost:4318/v1/logs';+ collector.byEndpoint.set(ENDPOINT, false);+ await delivery.enqueueTurn(turn('down'), settings);+ await delivery.enqueueTurn(turn('up'), { ...settings, endpoint: other });+ expect(collector.received).toEqual(['up']);+ expect(queue().map((i) => i.id)).toEqual(['down']);+ });++ it('drops a batch the collector rejects instead of holding the queue behind it', async () => {+ collector.answers.set('bad', 400);+ await delivery.enqueueTurn(turn('bad'), settings);+ await delivery.enqueueTurn(turn('next'), settings);+ expect(collector.attempted).toEqual(['bad', 'next']);+ expect(collector.received).toEqual(['next']);+ expect(queue()).toEqual([]);+ });++ it('retries a batch the collector answers with a retryable status', async () => {+ collector.answers.set('busy', 503);+ await delivery.enqueueTurn(turn('busy'), settings);+ await delivery.enqueueTurn(turn('after'), settings);+ expect(queue().map((i) => i.id)).toEqual(['busy', 'after']);++ collector.answers.clear();+ await runFor(delivery.MAX_RETRY_DELAY_MS);+ expect(collector.received).toEqual(['busy', 'after']);+ });+});++describe('service-worker restarts', () => {+ it('survives the worker being suspended and restarted mid-outage', async () => {+ collector.up = false;+ await delivery.enqueueTurn(turn('s1'), settings);+ await runFor(2 * MINUTE);+ await delivery.enqueueTurn(turn('s2'), settings);+ await startWorker(); // suspended: in-memory state is gone, storage and alarms are not+ await runFor(3 * MINUTE);+ collector.up = true;+ await runFor(delivery.MAX_RETRY_DELAY_MS);+ expect(collector.received).toEqual(['s1', 's2']);+ });++ it('resumes delivery on worker start when a restart or update cleared the alarm', async () => {+ collector.up = false;+ await delivery.enqueueTurn(turn('r1'), settings);+ await delivery.enqueueTurn(turn('r2'), settings);+ await runFor(5 * MINUTE);+ env.alarms.clear(); // browser restart / extension update+ collector.up = true;+ await runFor(10 * MINUTE);+ expect(collector.received).toEqual([]); // nothing would ever retry...++ await startWorker(); // ...until the worker starts again, with no new turn+ await runFor(delivery.MAX_RETRY_DELAY_MS);+ expect(collector.received).toEqual(['r1', 'r2']);+ expect(queue()).toEqual([]);+ });+});++describe('queue bounds', () => {+ it('expires batches older than the age limit and keeps younger ones', async () => {+ const now = Date.now();+ env.storage.set('delivery_queue', [+ stored('expired', { queuedAt: now - delivery.MAX_QUEUE_AGE_MS }),+ stored('young', { queuedAt: now - delivery.MAX_QUEUE_AGE_MS + MINUTE }),+ ]);+ const sentAt: number[] = [];+ vi.stubGlobal('fetch', vi.fn(async () => {+ sentAt.push(Date.now());+ throw new TypeError('Failed to fetch');+ }));+ await delivery.flush();+ expect(queue().map((i) => i.id)).toEqual(['young']);+ expect(sentAt).toEqual([now]); // only the young batch was tried++ await runFor(2 * MINUTE);+ expect(queue()).toEqual([]); // past its limit now+ const expiresAt = now + MINUTE;+ expect(sentAt.length).toBeGreaterThan(1);+ expect(sentAt.every((t) => t < expiresAt)).toBe(true); // never sent once expired+ });++ it('drops a batch captured at the start of an outage that outlasts the age limit', async () => {+ collector.up = false;+ await delivery.enqueueTurn(turn('first'), settings);+ await runFor(delivery.MAX_QUEUE_AGE_MS - MINUTE);+ await delivery.enqueueTurn(turn('late'), settings);+ await runFor(2 * MINUTE);+ collector.up = true;+ await runFor(delivery.MAX_RETRY_DELAY_MS);+ expect(collector.received).toEqual(['late']);+ });++ it('gives a batch queued by an older build a full age window', async () => {+ const legacy = stored('legacy', { attempts: 3 });+ delete legacy.queuedAt;+ env.storage.set('delivery_queue', [legacy]);+ collector.up = false;+ await delivery.flush();+ expect(queue()).toEqual([expect.objectContaining({ id: 'legacy', queuedAt: Date.now() })]);+ });++ it('caps the batch count, dropping the oldest first and keeping order', async () => {+ const backlog = Array.from({ length: delivery.MAX_QUEUE_ITEMS }, (_, i) =>+ stored(`q${i}`, { nextAttemptAt: Date.now() + MINUTE }));+ env.storage.set('delivery_queue', backlog);+ collector.up = false;+ for (const id of ['n1', 'n2', 'n3']) await delivery.enqueueTurn(turn(id), settings);++ const ids = queue().map((i) => i.id);+ expect(ids).toHaveLength(delivery.MAX_QUEUE_ITEMS);+ expect(ids.slice(0, 2)).toEqual(['q3', 'q4']);+ expect(ids.slice(-3)).toEqual(['n1', 'n2', 'n3']);+ });++ it('caps the queued bytes, dropping the oldest first', async () => {+ // Each envelope serializes to {"pad":"x...x"}: the pad plus 10 bytes. Four+ // of them come to 4000 bytes under the cap.+ const each = delivery.MAX_QUEUE_BYTES / 4 - 1000;+ const big = (id: string) => stored(id, { envelope: { pad: 'x'.repeat(each - 10) } });+ env.storage.set('delivery_queue', ['b0', 'b1', 'b2', 'b3'].map(big));+ collector.up = false;+ await delivery.flush();+ expect(queue().map((i) => i.bytes)).toEqual([each, each, each, each]);++ // A turn over 4000 bytes tips the queue over: the oldest batch goes.+ await delivery.enqueueTurn(turn('newest', 'p'.repeat(10_000)), settings);+ const q = queue();+ expect(q.map((i) => i.id)).toEqual(['b1', 'b2', 'b3', 'newest']);+ expect(q.reduce((sum, i) => sum + i.bytes!, 0)).toBeLessThanOrEqual(delivery.MAX_QUEUE_BYTES);+ });++ it('keeps a lone batch over the byte cap until a newer one replaces it', async () => {+ env.storage.set('delivery_queue', [+ stored('huge', { envelope: { pad: 'x'.repeat(delivery.MAX_QUEUE_BYTES) } }),+ ]);+ collector.up = false;+ await delivery.flush();+ expect(queue().map((i) => i.id)).toEqual(['huge']);+ await delivery.enqueueTurn(turn('after'), settings);+ expect(queue().map((i) => i.id)).toEqual(['after']);+ });+});++describe('retry backoff', () => {+ it('stores the deadline of each failure and wakes at it, clamped by the alarm floor', async () => {+ collector.up = false;+ await delivery.enqueueTurn(turn('b'), settings);+ const seen: Array<{ attempts: number; wait: number }> = [];+ for (let i = 0; i < 9; i++) {+ const [item] = queue();+ seen.push({ attempts: item.attempts, wait: item.nextAttemptAt! - Date.now() });+ expect(env.alarms.get('beacon_flush')).toBe(+ Math.max(item.nextAttemptAt!, Date.now() + ALARM_CLAMP_MS));+ await runFor(env.alarms.get('beacon_flush')! - Date.now());+ }+ expect(seen.map((s) => s.attempts)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9]);+ expect(seen.map((s) => s.wait)).toEqual([500, 1_000, 2_000, 4_000, 8_000, 16_000,+ 32_000, 60_000, 60_000]);+ expect(collector.attempted).toEqual(Array(10).fill('b'));+ });+});diff --git a/browser-extension/test/unit/retry-backoff.test.ts b/browser-extension/test/unit/retry-backoff.test.tsindex 1b26a473..2502cf62 100644--- a/browser-extension/test/unit/retry-backoff.test.ts+++ b/browser-extension/test/unit/retry-backoff.test.ts@@ -17,7 +17,7 @@ beforeEach(() => {api.get.mockImplementation(async () => ({ delivery_queue: structuredClone(queue) }));api.set.mockImplementation(async ({ delivery_queue }: { delivery_queue: StoredItem[] }) => { queue = structuredClone(delivery_queue); });api.alarm.mockClear();- send = vi.fn().mockResolvedValue({ ok: false });+ send = vi.fn().mockResolvedValue({ ok: false, status: 503 });vi.stubGlobal('fetch', send);});afterEach(() => { vi.useRealTimers(); vi.unstubAllGlobals(); });@@ -48,11 +48,11 @@ it('still delivers a legacy queue item without a persisted deadline', async () =expect(queue).toEqual([]);});-it('keeps the existing maximum attempt limit for due failures', async () => {+it('keeps a due failure queued past the former six-attempt limit', async () => {vi.setSystemTime(11_000);queue[0].attempts = 5;const { flush } = await import('../../src/background/delivery.js');await flush();expect(send).toHaveBeenCalledTimes(1);- expect(queue).toEqual([]);+ expect(queue).toEqual([expect.objectContaining({ id: 'turn', attempts: 6, nextAttemptAt: 11_000 + 16_000 })]);});diff --git a/docs/runtimes/browser-extension.mdx b/docs/runtimes/browser-extension.mdxindex 9f393489..41e787f4 100644--- a/docs/runtimes/browser-extension.mdx+++ b/docs/runtimes/browser-extension.mdx@@ -44,7 +44,7 @@ beacon endpoint statusA content script running in the page's main world tees the streamed chat response, a per-site adapter parses it into a normalized turn, and the extension's service worker posts OTLP GenAI logs to the local collector. The `beaconjson` exporter converts those into endpoint events and writes them to the runtime JSONL log, alongside every other source.-The extension never writes files and never contacts a remote endpoint. Delivery is queued in local extension storage and retried with backoff, so a turn survives the service worker being suspended mid-stream.+The extension never writes files and never contacts a remote endpoint. Delivery is queued in local extension storage and retried with backoff, so a turn survives the service worker being suspended mid-stream and the collector being down. A queued turn is kept until it is delivered, for up to 24 hours; past 2,000 queued turns or 6 MiB, the oldest are dropped first. A turn the collector rejects outright, rather than failing to reach it, is dropped.Events collected this way carry `harness.collection_method` of `otlp`, and a harness name of `claude_web` or `chatgpt_web`, so browser activity stays distinguishable from CLI agents in the runtime log and the dashboard.