Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

fix(browser-extension): keep queued telemetry through collector outages (#863) (#871)

claude[bot]e93aaa8f243e
diff --git a/browser-extension/README.md b/browser-extension/README.md
index 1ff7ff5b..ccfc8bdf 100644
--- a/browser-extension/README.md
+++ b/browser-extension/README.md
@@ -180,8 +180,8 @@ How the Firefox build differs, all generated from `src/manifest.json` by
Not yet verified on a real Firefox (no Firefox runs in CI): the event page's POST to
`http://127.0.0.1:4318/v1/logs`, and the retry cadence. Chrome clamps `chrome.alarms` to about
-30 seconds; Firefox's clamp has not been measured. The retry code is the same on both, and the
-durable queue means delivery is eventual on either engine.
+30 seconds; Firefox's clamp has not been measured. The retry code and the queue's limits are the
+same on both engines.
## Retained content
@@ -305,11 +305,21 @@ dashboard), including interleaved sessions with no cross-contamination.
stall between chunks could drop a turn. The durable delivery queue (in `chrome.storage`) protects
everything *after* a turn is assembled, not during. Persisting partial assembler state is a
future item.
-- **Delivery retry cadence.** Each POST has a ten-second deadline, so a collector that accepts
- a connection but never responds cannot hold the queue indefinitely. Timed-out batches remain
- durable and follow the existing retry limit. Retries use `chrome.alarms`, which MV3 clamps
- to a ~30s minimum, so the first retry lands at ~30s rather than the computed sub-second backoff. Fine for a
- local-loopback collector that is almost always reachable; the queue guarantees eventual delivery.
+- **Delivery retry cadence and queue limits.** Each POST has a ten-second deadline, so a collector
+ that accepts a connection but never responds cannot hold the queue indefinitely; a timed-out
+ batch is retried like any other failure. Retries back off from half a second, doubling to a
+ one-minute cap, and run on `chrome.alarms`, which MV3 clamps to a ~30s minimum, so a recovered
+ collector receives the backlog within about a minute. Batches are delivered oldest first, and
+ while the collector is down only the oldest batch is retried; newer ones wait behind it. A
+ failed send (connection refused, timeout, or a retryable 429, 502, 503 or 504) never drops a
+ batch. A batch is dropped undelivered when the collector rejects it with any other status,
+ since sending it again cannot succeed, when it is older than **24 hours**, or when the queue
+ passes **2,000 batches** or **6 MiB** of envelopes, which drops the oldest batches first (the
+ byte cap keeps the queue inside `chrome.storage.local`'s 10 MiB quota). Within those limits, a
+ collector outage of any length (restarting during an install or upgrade, not yet started after
+ boot, crash-looping) loses nothing. The queue is re-checked each
+ time the service worker starts, because a browser restart or extension update can clear the
+ retry alarm.
- **Capture depends on `window.fetch`.** Confirmed correct for claude.ai. Sites using XHR or a page
service worker for streaming would need additional interception (deferred).
- **Single capture path.** No DOM-scraping fallback yet; if a site changes its SSE wire format, the
diff --git a/browser-extension/src/background/delivery.ts b/browser-extension/src/background/delivery.ts
index e38e3612..a8b1ce81 100644
--- a/browser-extension/src/background/delivery.ts
+++ b/browser-extension/src/background/delivery.ts
@@ -2,6 +2,14 @@
// chrome.storage.local so nothing is lost if the SW is suspended, then POSTed
// to the local beacon collector. Failed sends retry with exponential backoff
// driven by chrome.alarms (which survives suspension, unlike setTimeout).
+//
+// Retention policy: a queued batch is kept until it is delivered, until it is
+// older than MAX_QUEUE_AGE_MS, or until newer batches push it past the queue's
+// size caps (MAX_QUEUE_ITEMS, MAX_QUEUE_BYTES), which drop the oldest first.
+// A failed send never drops a batch on its own, so a collector that is down
+// for minutes (restarting during an install or upgrade, not yet started after
+// boot, crash-looping) loses nothing it comes back from within the age limit.
+// A batch the collector rejects (a non-retryable status) is dropped at once.
import { ext } from '../shared/browser.js';
import type { ChatTurn, Settings } from '../shared/types.js';
@@ -13,16 +21,34 @@ import { serial } from '../shared/serial.js';
const QUEUE_KEY = 'delivery_queue';
const ALARM = 'beacon_flush';
-const MAX_ATTEMPTS = 6;
const POST_TIMEOUT_MS = 10_000;
+/** A batch older than this is dropped undelivered. */
+export const MAX_QUEUE_AGE_MS = 24 * 60 * 60 * 1000;
+/** Most batches the queue holds; the oldest are dropped past it. */
+export const MAX_QUEUE_ITEMS = 2_000;
+/**
+ * Most envelope bytes (UTF-8 JSON) the queue holds; the oldest batches are
+ * dropped past it. Chrome's storage.local quota is 10 MiB for the whole
+ * extension without `unlimitedStorage`, so this leaves room for settings and for
+ * the queue's own bookkeeping fields.
+ */
+export const MAX_QUEUE_BYTES = 6 * 1024 * 1024;
+/** Longest wait between retries, and so the longest delay after the collector recovers. */
+export const MAX_RETRY_DELAY_MS = 60_000;
+
interface QueueItem {
id: string;
endpoint: string;
envelope: LogsEnvelope;
+ /** Failed sends so far. Sets the backoff; never a reason to drop the batch. */
attempts: number;
/** Persisted deadline so new turns cannot consume retries before backoff. */
nextAttemptAt?: number;
+ /** When the batch was queued (epoch ms); the age limit counts from here. */
+ queuedAt?: number;
+ /** UTF-8 size of the serialized envelope, counted against MAX_QUEUE_BYTES. */
+ bytes?: number;
}
async function readQueue(): Promise<QueueItem[]> {
@@ -50,52 +76,97 @@ export async function enqueueTurn(
endpoint: settings.endpoint,
envelope,
attempts: 0,
+ queuedAt: Date.now(),
};
await withQueueLock(async () => {
const queue = await readQueue();
queue.push(item);
- await writeQueue(queue);
+ await writeQueue(pruneQueue(queue, Date.now()));
});
await flush();
}
-/** Attempt to POST everything queued; requeue failures with backoff. */
+/**
+ * POST every queued batch that is due, oldest first; requeue failures with backoff.
+ *
+ * Once a batch for an endpoint fails or is still waiting out its backoff, the
+ * endpoint's later batches wait behind it in this pass. A recovered collector
+ * therefore receives batches in the order they were captured, and a collector
+ * that is down gets one probe per pass instead of one POST per queued batch.
+ */
export async function flush(): Promise<void> {
await withQueueLock(async () => {
- const queue = await readQueue();
- if (queue.length === 0) return;
+ const stored = await readQueue();
+ if (stored.length === 0) return;
+ const queue = pruneQueue(stored, Date.now());
const remaining: QueueItem[] = [];
- let anyFailed = false;
+ const waiting = new Set<string>();
for (const item of queue) {
// Checked again at send time, not only when settings are saved: an item
// queued by an older build, or a tampered queue, must not reach anything
// but the local collector. Such an item is dropped, not retried.
if (!isAllowedEndpoint(item.endpoint)) continue;
- if (item.nextAttemptAt != null && item.nextAttemptAt > Date.now()) {
+ if (waiting.has(item.endpoint) || (item.nextAttemptAt ?? 0) > Date.now()) {
remaining.push(item);
- anyFailed = true;
+ waiting.add(item.endpoint);
continue;
}
- const ok = await post(item);
- if (!ok) {
- item.attempts += 1;
- if (item.attempts < MAX_ATTEMPTS) {
- item.nextAttemptAt = Date.now() + retryDelay(item.attempts);
- remaining.push(item);
- anyFailed = true;
- } // else: give up, drop it (avoid an unbounded queue)
- }
+ const outcome = await post(item);
+ // A batch the collector rejects outright would fail the same way forever
+ // and hold every later batch behind it, so it is dropped, not retried.
+ if (outcome !== 'retry') continue;
+ item.attempts += 1;
+ item.nextAttemptAt = Date.now() + retryDelay(item.attempts);
+ remaining.push(item);
+ waiting.add(item.endpoint);
}
await writeQueue(remaining);
- if (anyFailed) scheduleRetry(remaining);
+ scheduleRetry(remaining);
});
}
-async function post(item: QueueItem): Promise<boolean> {
- if (!isAllowedEndpoint(item.endpoint)) return false;
+/**
+ * Drop batches past the age limit, then the oldest batches until the queue fits
+ * its caps. The newest batch is always kept. A batch from an older build has no
+ * `queuedAt`; it is stamped now, so it gets a full age window rather than none.
+ */
+function pruneQueue(queue: QueueItem[], now: number): QueueItem[] {
+ const fresh = queue.filter((item) => {
+ item.queuedAt ??= now;
+ return now - item.queuedAt < MAX_QUEUE_AGE_MS;
+ });
+ let bytes = fresh.reduce((sum, item) => sum + itemBytes(item), 0);
+ let drop = 0;
+ while (
+ fresh.length - drop > 1 &&
+ (fresh.length - drop > MAX_QUEUE_ITEMS || bytes > MAX_QUEUE_BYTES)
+ ) {
+ bytes -= itemBytes(fresh[drop]);
+ drop += 1;
+ }
+ return fresh.slice(drop);
+}
+
+const utf8 = new TextEncoder();
+
+function itemBytes(item: QueueItem): number {
+ item.bytes ??= utf8.encode(JSON.stringify(item.envelope)).length;
+ return item.bytes;
+}
+
+/**
+ * Statuses the OTLP/HTTP spec marks retryable. Any other non-2xx response
+ * rejects the batch itself, so sending it again cannot succeed.
+ */
+const RETRYABLE_STATUSES = new Set([429, 502, 503, 504]);
+
+type PostOutcome = 'sent' | 'retry' | 'rejected';
+
+async function post(item: QueueItem): Promise<PostOutcome> {
+ if (!isAllowedEndpoint(item.endpoint)) return 'rejected';
try {
const res = await fetch(item.endpoint, {
method: 'POST',
@@ -107,28 +178,37 @@ async function post(item: QueueItem): Promise<boolean> {
redirect: 'error',
body: JSON.stringify(item.envelope),
});
- return res.ok;
+ if (res.ok) return 'sent';
+ return RETRYABLE_STATUSES.has(res.status) ? 'retry' : 'rejected';
} catch {
- return false;
+ // Connection refused, timed out or redirected: the collector may come back.
+ return 'retry';
}
}
+/** Backoff after the given number of failed sends: 0.5s, 1s, 2s, ... capped at a minute. */
function retryDelay(attempts: number): number {
- return Math.min(250 * 2 ** attempts, 60_000);
+ return Math.min(250 * 2 ** attempts, MAX_RETRY_DELAY_MS);
}
function scheduleRetry(queue: QueueItem[]): void {
- // Preserve each failed batch's deadline when unrelated new turns arrive.
- // Chrome may clamp alarm cadence, but an enqueue must never accelerate a
- // retry or repeatedly postpone the existing earliest scheduled deadline.
- const when = Math.min(...queue.map((item) =>
- item.nextAttemptAt ?? Date.now() + retryDelay(item.attempts)));
- ext.alarms.create(ALARM, { when });
+ // Wake at the earliest stored deadline. A batch waiting behind another has
+ // none of its own, so an unrelated new turn neither accelerates nor postpones
+ // the retry. Chrome clamps the alarm to about 30 seconds.
+ let when = Infinity;
+ for (const item of queue) {
+ if (item.nextAttemptAt != null && item.nextAttemptAt < when) when = item.nextAttemptAt;
+ }
+ if (when !== Infinity) ext.alarms.create(ALARM, { when });
}
-/** Wire the alarm listener (called once from the SW entry). */
+/** Wire the alarm listener and resume delivery (called once from the SW entry). */
export function installFlushAlarm(): void {
ext.alarms.onAlarm.addListener((alarm) => {
if (alarm.name === ALARM) void flush();
});
+ // Alarms can be cleared by a browser restart and are cleared by an extension
+ // update, so a queue left from before would otherwise wait for the next turn.
+ // Flushing on worker start sends what is due and re-arms the retry for the rest.
+ void flush();
}
diff --git a/browser-extension/test/unit/delivery-outage.test.ts b/browser-extension/test/unit/delivery-outage.test.ts
new file mode 100644
index 00000000..6226195f
--- /dev/null
+++ b/browser-extension/test/unit/delivery-outage.test.ts
@@ -0,0 +1,431 @@
+// Collector outages against the durable delivery queue (#863). A fake
+// extension environment stands in for the browser: storage.local persists
+// across simulated service-worker restarts, alarms follow Chrome's ~30s clamp,
+// and a fake loopback collector records every POST it sees. Time is driven by
+// fake timers, so a ten-minute or day-long outage runs instantly.
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import type { ChatTurn, Settings } from '../../src/shared/types.js';
+import { DEFAULT_SETTINGS } from '../../src/shared/types.js';
+
+const MINUTE = 60_000;
+const ALARM_CLAMP_MS = 30_000;
+const ENDPOINT = DEFAULT_SETTINGS.endpoint;
+
+type Alarm = { name: string };
+type Listener = (alarm: Alarm) => void;
+const env = vi.hoisted(() => ({
+ storage: new Map<string, unknown>(),
+ alarms: new Map<string, number>(),
+ listeners: [] as Listener[],
+}));
+
+vi.mock('../../src/shared/browser.js', () => ({
+ ext: {
+ storage: {
+ local: {
+ // Clone both ways, as the real storage serializes values.
+ get: async (key: string) =>
+ env.storage.has(key) ? { [key]: structuredClone(env.storage.get(key)) } : {},
+ set: async (items: Record<string, unknown>) => {
+ for (const [k, v] of Object.entries(items)) env.storage.set(k, structuredClone(v));
+ },
+ },
+ },
+ alarms: {
+ // Chrome delays an alarm due sooner than ~30s from now to ~30s from now.
+ create: (name: string, info: { when: number }) => {
+ env.alarms.set(name, Math.max(info.when, Date.now() + ALARM_CLAMP_MS));
+ },
+ onAlarm: { addListener: (fn: Listener) => env.listeners.push(fn) },
+ },
+ },
+}));
+
+type Delivery = typeof import('../../src/background/delivery.js');
+type StoredItem = {
+ id: string; endpoint: string; envelope: unknown; attempts: number;
+ nextAttemptAt?: number; queuedAt?: number; bytes?: number;
+};
+
+/**
+ * A fake loopback collector: records every POST and accepts only while `up`.
+ * `answers` gives the HTTP status it returns for a prompt id while up.
+ */
+const collector = {
+ up: true,
+ attempted: [] as string[],
+ received: [] as string[],
+ byEndpoint: new Map<string, boolean>(),
+ answers: new Map<string, number>(),
+};
+
+function promptId(body: unknown): string | undefined {
+ if (Array.isArray(body)) {
+ for (const v of body) {
+ const found = promptId(v);
+ if (found) return found;
+ }
+ } else if (body && typeof body === 'object') {
+ const o = body as Record<string, unknown>;
+ if (o.key === 'beacon.prompt.id') return (o.value as { stringValue: string }).stringValue;
+ for (const v of Object.values(o)) {
+ const found = promptId(v);
+ if (found) return found;
+ }
+ }
+ return undefined;
+}
+
+async function fakeFetch(url: string, init: { body: string }): Promise<{ ok: boolean; status: number }> {
+ const id = promptId(JSON.parse(init.body)) ?? '?';
+ collector.attempted.push(id);
+ const up = collector.byEndpoint.get(url) ?? collector.up;
+ if (!up) throw new TypeError('Failed to fetch'); // connection refused
+ const status = collector.answers.get(id) ?? 200;
+ if (status !== 200) return { ok: false, status };
+ collector.received.push(id);
+ return { ok: true, status };
+}
+
+let delivery: Delivery;
+
+/** Start (or restart) the service worker: fresh module state, persisted storage. */
+async function startWorker(): Promise<void> {
+ vi.resetModules();
+ env.listeners.length = 0;
+ delivery = await import('../../src/background/delivery.js');
+ delivery.installFlushAlarm();
+ await settle();
+}
+
+/** Let fire-and-forget flushes finish. The fakes never wait on a macrotask. */
+async function settle(): Promise<void> {
+ for (let i = 0; i < 5; i++) await new Promise((r) => setImmediate(r));
+}
+
+/** Advance the clock by `ms`, firing every alarm that comes due on the way. */
+async function runFor(ms: number): Promise<number> {
+ const end = Date.now() + ms;
+ let fired = 0;
+ for (;;) {
+ const due = [...env.alarms.entries()].filter(([, t]) => t <= end).sort((a, b) => a[1] - b[1])[0];
+ if (!due) break;
+ vi.setSystemTime(Math.max(due[1], Date.now()));
+ env.alarms.delete(due[0]);
+ for (const fn of env.listeners) fn({ name: due[0] });
+ fired += 1;
+ await settle();
+ }
+ vi.setSystemTime(end);
+ return fired;
+}
+
+function turn(id: string, promptText = `prompt ${id}`): ChatTurn {
+ return {
+ site: 'claude_web',
+ sessionId: 'conv-1',
+ turnId: id,
+ promptText,
+ inputMessages: [{ role: 'user', text: `prompt ${id}` }],
+ outputMessages: [{ role: 'assistant', text: `answer ${id}` }],
+ responseText: `answer ${id}`,
+ toolCalls: [],
+ startedAt: Date.now(),
+ completedAt: Date.now(),
+ captureMode: 'sse',
+ };
+}
+
+const settings: Settings = { ...DEFAULT_SETTINGS };
+
+function queue(): StoredItem[] {
+ return (env.storage.get('delivery_queue') as StoredItem[] | undefined) ?? [];
+}
+
+function stored(id: string, over: Partial<StoredItem> = {}): StoredItem {
+ return { id, endpoint: ENDPOINT, envelope: { id }, attempts: 0, queuedAt: Date.now(), ...over };
+}
+
+beforeEach(async () => {
+ vi.useFakeTimers({ toFake: ['Date'] });
+ vi.setSystemTime(1_000_000);
+ env.storage.clear();
+ env.alarms.clear();
+ collector.up = true;
+ collector.attempted = [];
+ collector.received = [];
+ collector.byEndpoint.clear();
+ collector.answers.clear();
+ vi.stubGlobal('fetch', vi.fn(fakeFetch));
+ await startWorker();
+});
+
+afterEach(() => {
+ vi.useRealTimers();
+ vi.unstubAllGlobals();
+});
+
+describe('collector outage', () => {
+ it('delivers every turn captured during a ten-minute outage once it recovers', async () => {
+ collector.up = false;
+ const ids = Array.from({ length: 8 }, (_, i) => `t${i}`);
+ for (const id of ids) {
+ await delivery.enqueueTurn(turn(id), settings);
+ await runFor(75_000); // 8 x 75s = 10 minutes down
+ }
+ expect(queue().map((i) => i.id)).toEqual(ids);
+ expect(collector.received).toEqual([]);
+
+ collector.up = true;
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+
+ expect(collector.received).toEqual(ids); // all of them, in capture order
+ expect(queue()).toEqual([]);
+ });
+
+ it('keeps retrying through an outage of many hours, well past the old six attempts', async () => {
+ collector.up = false;
+ await delivery.enqueueTurn(turn('long'), settings);
+ const fired = await runFor(6 * 60 * MINUTE);
+ expect(fired).toBeGreaterThan(300);
+ expect(queue()).toEqual([expect.objectContaining({ id: 'long' })]);
+ expect(queue()[0].attempts).toBeGreaterThan(300);
+
+ collector.up = true;
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+ expect(collector.received).toEqual(['long']);
+ });
+
+ it('probes a down collector with the oldest batch only, not every queued batch', async () => {
+ collector.up = false;
+ for (const id of ['a', 'b', 'c', 'd']) {
+ await delivery.enqueueTurn(turn(id), settings);
+ await runFor(MINUTE);
+ }
+ await runFor(10 * MINUTE);
+ expect(new Set(collector.attempted)).toEqual(new Set(['a']));
+ });
+
+ it('sends nothing twice: each turn reaches the collector exactly once', async () => {
+ collector.up = false;
+ const ids = ['x1', 'x2', 'x3', 'x4', 'x5'];
+ for (const id of ids) await delivery.enqueueTurn(turn(id), settings);
+ await runFor(4 * MINUTE);
+ collector.up = true;
+ // The backlog is due: the alarm, a popup "flush now" and a new turn all
+ // race the same queue.
+ vi.setSystemTime(queue()[0].nextAttemptAt!);
+ env.alarms.clear();
+ await Promise.all([
+ ...env.listeners.map(async (fn) => fn({ name: 'beacon_flush' })),
+ delivery.flush(),
+ delivery.flush(),
+ delivery.enqueueTurn(turn('x6'), settings),
+ ]);
+ await settle();
+ expect(collector.received).toEqual([...ids, 'x6']);
+ await runFor(10 * MINUTE);
+ expect(collector.received).toEqual([...ids, 'x6']);
+ expect(queue()).toEqual([]);
+ });
+
+ it('holds new turns behind the waiting backlog so a recovery delivers in order', async () => {
+ collector.up = false;
+ await delivery.enqueueTurn(turn('old'), settings);
+ await runFor(5 * MINUTE);
+ collector.up = true;
+ // The collector is back, but the backlog's retry is not due yet. A new turn
+ // must not jump ahead of it.
+ await delivery.enqueueTurn(turn('new'), settings);
+ expect(collector.received).toEqual([]);
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+ expect(collector.received).toEqual(['old', 'new']);
+ });
+
+ it('keeps the undelivered tail in order when the collector drops mid-drain', async () => {
+ collector.up = false;
+ const ids = ['m1', 'm2', 'm3', 'm4'];
+ for (const id of ids) await delivery.enqueueTurn(turn(id), settings);
+ await runFor(2 * MINUTE);
+
+ collector.up = true;
+ let died = false;
+ vi.stubGlobal('fetch', vi.fn(async (url: string, init: { body: string }) => {
+ if (!died && collector.received.length === 2) {
+ died = true; // the collector dies after accepting two batches
+ collector.up = false;
+ }
+ return fakeFetch(url, init);
+ }));
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+ expect(collector.received).toEqual(['m1', 'm2']);
+ expect(queue().map((i) => i.id)).toEqual(['m3', 'm4']);
+
+ collector.up = true;
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+ expect(collector.received).toEqual(ids);
+ });
+
+ it('does not let one unreachable endpoint hold back batches for another', async () => {
+ const other = 'http://localhost:4318/v1/logs';
+ collector.byEndpoint.set(ENDPOINT, false);
+ await delivery.enqueueTurn(turn('down'), settings);
+ await delivery.enqueueTurn(turn('up'), { ...settings, endpoint: other });
+ expect(collector.received).toEqual(['up']);
+ expect(queue().map((i) => i.id)).toEqual(['down']);
+ });
+
+ it('drops a batch the collector rejects instead of holding the queue behind it', async () => {
+ collector.answers.set('bad', 400);
+ await delivery.enqueueTurn(turn('bad'), settings);
+ await delivery.enqueueTurn(turn('next'), settings);
+ expect(collector.attempted).toEqual(['bad', 'next']);
+ expect(collector.received).toEqual(['next']);
+ expect(queue()).toEqual([]);
+ });
+
+ it('retries a batch the collector answers with a retryable status', async () => {
+ collector.answers.set('busy', 503);
+ await delivery.enqueueTurn(turn('busy'), settings);
+ await delivery.enqueueTurn(turn('after'), settings);
+ expect(queue().map((i) => i.id)).toEqual(['busy', 'after']);
+
+ collector.answers.clear();
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+ expect(collector.received).toEqual(['busy', 'after']);
+ });
+});
+
+describe('service-worker restarts', () => {
+ it('survives the worker being suspended and restarted mid-outage', async () => {
+ collector.up = false;
+ await delivery.enqueueTurn(turn('s1'), settings);
+ await runFor(2 * MINUTE);
+ await delivery.enqueueTurn(turn('s2'), settings);
+ await startWorker(); // suspended: in-memory state is gone, storage and alarms are not
+ await runFor(3 * MINUTE);
+ collector.up = true;
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+ expect(collector.received).toEqual(['s1', 's2']);
+ });
+
+ it('resumes delivery on worker start when a restart or update cleared the alarm', async () => {
+ collector.up = false;
+ await delivery.enqueueTurn(turn('r1'), settings);
+ await delivery.enqueueTurn(turn('r2'), settings);
+ await runFor(5 * MINUTE);
+ env.alarms.clear(); // browser restart / extension update
+ collector.up = true;
+ await runFor(10 * MINUTE);
+ expect(collector.received).toEqual([]); // nothing would ever retry...
+
+ await startWorker(); // ...until the worker starts again, with no new turn
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+ expect(collector.received).toEqual(['r1', 'r2']);
+ expect(queue()).toEqual([]);
+ });
+});
+
+describe('queue bounds', () => {
+ it('expires batches older than the age limit and keeps younger ones', async () => {
+ const now = Date.now();
+ env.storage.set('delivery_queue', [
+ stored('expired', { queuedAt: now - delivery.MAX_QUEUE_AGE_MS }),
+ stored('young', { queuedAt: now - delivery.MAX_QUEUE_AGE_MS + MINUTE }),
+ ]);
+ const sentAt: number[] = [];
+ vi.stubGlobal('fetch', vi.fn(async () => {
+ sentAt.push(Date.now());
+ throw new TypeError('Failed to fetch');
+ }));
+ await delivery.flush();
+ expect(queue().map((i) => i.id)).toEqual(['young']);
+ expect(sentAt).toEqual([now]); // only the young batch was tried
+
+ await runFor(2 * MINUTE);
+ expect(queue()).toEqual([]); // past its limit now
+ const expiresAt = now + MINUTE;
+ expect(sentAt.length).toBeGreaterThan(1);
+ expect(sentAt.every((t) => t < expiresAt)).toBe(true); // never sent once expired
+ });
+
+ it('drops a batch captured at the start of an outage that outlasts the age limit', async () => {
+ collector.up = false;
+ await delivery.enqueueTurn(turn('first'), settings);
+ await runFor(delivery.MAX_QUEUE_AGE_MS - MINUTE);
+ await delivery.enqueueTurn(turn('late'), settings);
+ await runFor(2 * MINUTE);
+ collector.up = true;
+ await runFor(delivery.MAX_RETRY_DELAY_MS);
+ expect(collector.received).toEqual(['late']);
+ });
+
+ it('gives a batch queued by an older build a full age window', async () => {
+ const legacy = stored('legacy', { attempts: 3 });
+ delete legacy.queuedAt;
+ env.storage.set('delivery_queue', [legacy]);
+ collector.up = false;
+ await delivery.flush();
+ expect(queue()).toEqual([expect.objectContaining({ id: 'legacy', queuedAt: Date.now() })]);
+ });
+
+ it('caps the batch count, dropping the oldest first and keeping order', async () => {
+ const backlog = Array.from({ length: delivery.MAX_QUEUE_ITEMS }, (_, i) =>
+ stored(`q${i}`, { nextAttemptAt: Date.now() + MINUTE }));
+ env.storage.set('delivery_queue', backlog);
+ collector.up = false;
+ for (const id of ['n1', 'n2', 'n3']) await delivery.enqueueTurn(turn(id), settings);
+
+ const ids = queue().map((i) => i.id);
+ expect(ids).toHaveLength(delivery.MAX_QUEUE_ITEMS);
+ expect(ids.slice(0, 2)).toEqual(['q3', 'q4']);
+ expect(ids.slice(-3)).toEqual(['n1', 'n2', 'n3']);
+ });
+
+ it('caps the queued bytes, dropping the oldest first', async () => {
+ // Each envelope serializes to {"pad":"x...x"}: the pad plus 10 bytes. Four
+ // of them come to 4000 bytes under the cap.
+ const each = delivery.MAX_QUEUE_BYTES / 4 - 1000;
+ const big = (id: string) => stored(id, { envelope: { pad: 'x'.repeat(each - 10) } });
+ env.storage.set('delivery_queue', ['b0', 'b1', 'b2', 'b3'].map(big));
+ collector.up = false;
+ await delivery.flush();
+ expect(queue().map((i) => i.bytes)).toEqual([each, each, each, each]);
+
+ // A turn over 4000 bytes tips the queue over: the oldest batch goes.
+ await delivery.enqueueTurn(turn('newest', 'p'.repeat(10_000)), settings);
+ const q = queue();
+ expect(q.map((i) => i.id)).toEqual(['b1', 'b2', 'b3', 'newest']);
+ expect(q.reduce((sum, i) => sum + i.bytes!, 0)).toBeLessThanOrEqual(delivery.MAX_QUEUE_BYTES);
+ });
+
+ it('keeps a lone batch over the byte cap until a newer one replaces it', async () => {
+ env.storage.set('delivery_queue', [
+ stored('huge', { envelope: { pad: 'x'.repeat(delivery.MAX_QUEUE_BYTES) } }),
+ ]);
+ collector.up = false;
+ await delivery.flush();
+ expect(queue().map((i) => i.id)).toEqual(['huge']);
+ await delivery.enqueueTurn(turn('after'), settings);
+ expect(queue().map((i) => i.id)).toEqual(['after']);
+ });
+});
+
+describe('retry backoff', () => {
+ it('stores the deadline of each failure and wakes at it, clamped by the alarm floor', async () => {
+ collector.up = false;
+ await delivery.enqueueTurn(turn('b'), settings);
+ const seen: Array<{ attempts: number; wait: number }> = [];
+ for (let i = 0; i < 9; i++) {
+ const [item] = queue();
+ seen.push({ attempts: item.attempts, wait: item.nextAttemptAt! - Date.now() });
+ expect(env.alarms.get('beacon_flush')).toBe(
+ Math.max(item.nextAttemptAt!, Date.now() + ALARM_CLAMP_MS));
+ await runFor(env.alarms.get('beacon_flush')! - Date.now());
+ }
+ expect(seen.map((s) => s.attempts)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ expect(seen.map((s) => s.wait)).toEqual([500, 1_000, 2_000, 4_000, 8_000, 16_000,
+ 32_000, 60_000, 60_000]);
+ expect(collector.attempted).toEqual(Array(10).fill('b'));
+ });
+});
diff --git a/browser-extension/test/unit/retry-backoff.test.ts b/browser-extension/test/unit/retry-backoff.test.ts
index 1b26a473..2502cf62 100644
--- a/browser-extension/test/unit/retry-backoff.test.ts
+++ b/browser-extension/test/unit/retry-backoff.test.ts
@@ -17,7 +17,7 @@ beforeEach(() => {
api.get.mockImplementation(async () => ({ delivery_queue: structuredClone(queue) }));
api.set.mockImplementation(async ({ delivery_queue }: { delivery_queue: StoredItem[] }) => { queue = structuredClone(delivery_queue); });
api.alarm.mockClear();
- send = vi.fn().mockResolvedValue({ ok: false });
+ send = vi.fn().mockResolvedValue({ ok: false, status: 503 });
vi.stubGlobal('fetch', send);
});
afterEach(() => { vi.useRealTimers(); vi.unstubAllGlobals(); });
@@ -48,11 +48,11 @@ it('still delivers a legacy queue item without a persisted deadline', async () =
expect(queue).toEqual([]);
});
-it('keeps the existing maximum attempt limit for due failures', async () => {
+it('keeps a due failure queued past the former six-attempt limit', async () => {
vi.setSystemTime(11_000);
queue[0].attempts = 5;
const { flush } = await import('../../src/background/delivery.js');
await flush();
expect(send).toHaveBeenCalledTimes(1);
- expect(queue).toEqual([]);
+ expect(queue).toEqual([expect.objectContaining({ id: 'turn', attempts: 6, nextAttemptAt: 11_000 + 16_000 })]);
});
diff --git a/docs/runtimes/browser-extension.mdx b/docs/runtimes/browser-extension.mdx
index 9f393489..41e787f4 100644
--- a/docs/runtimes/browser-extension.mdx
+++ b/docs/runtimes/browser-extension.mdx
@@ -44,7 +44,7 @@ beacon endpoint status
A content script running in the page's main world tees the streamed chat response, a per-site adapter parses it into a normalized turn, and the extension's service worker posts OTLP GenAI logs to the local collector. The `beaconjson` exporter converts those into endpoint events and writes them to the runtime JSONL log, alongside every other source.
-The extension never writes files and never contacts a remote endpoint. Delivery is queued in local extension storage and retried with backoff, so a turn survives the service worker being suspended mid-stream.
+The extension never writes files and never contacts a remote endpoint. Delivery is queued in local extension storage and retried with backoff, so a turn survives the service worker being suspended mid-stream and the collector being down. A queued turn is kept until it is delivered, for up to 24 hours; past 2,000 queued turns or 6 MiB, the oldest are dropped first. A turn the collector rejects outright, rather than failing to reach it, is dropped.
Events collected this way carry `harness.collection_method` of `otlp`, and a harness name of `claude_web` or `chatgpt_web`, so browser activity stays distinguishable from CLI agents in the runtime log and the dashboard.