Skip to content

A desktop coding agent with free access to frontier models.

TypeScript

fix(collector): size-scaled upload deadline, spool drains past a failing entry

ogsamrat4109bf992444
diff --git a/apps/server/src/capture-client.ts b/apps/server/src/capture-client.ts
index 9ac7809b..902eb0f3 100644
--- a/apps/server/src/capture-client.ts
+++ b/apps/server/src/capture-client.ts
@@ -414,7 +414,7 @@ class CaptureClient implements CaptureService {
switch (request.type) {
case "collect": {
if (!collector.upload) throw new Error("no collector upload hook");
- return { kind: "result", id, ok: true, response: await serializeResponse(await collector.upload(request.sessionId, request.body)) };
+ return { kind: "result", id, ok: true, response: await serializeResponse(await collector.upload(request.sessionId, request.body, signal)) };
}
case "refreshAccessToken": {
const refresh = channel === "archive" ? archive.refreshAccessToken : collector.refreshAccessToken;
diff --git a/apps/server/src/capture-host.ts b/apps/server/src/capture-host.ts
index 8257680f..fefc5b31 100644
--- a/apps/server/src/capture-host.ts
+++ b/apps/server/src/capture-host.ts
@@ -48,7 +48,7 @@ export type CaptureHostOptions = {
engineVersion: string;
log: CaptureLog;
collector: {
- upload?: (sessionId: string, compressed: Uint8Array) => Promise<Response>;
+ upload?: (sessionId: string, compressed: Uint8Array, signal?: AbortSignal) => Promise<Response>;
refreshAccessToken?: () => Promise<string | null>;
fetch?: (input: string, init?: RequestInit) => Promise<Response>;
gatewayUrl?: string;
diff --git a/apps/server/src/capture-worker.ts b/apps/server/src/capture-worker.ts
index 077a4c0f..8ba880c7 100644
--- a/apps/server/src/capture-worker.ts
+++ b/apps/server/src/capture-worker.ts
@@ -106,9 +106,9 @@ const host = new CaptureHost({
collector: {
...(init.collector.upload
? {
- upload: async (sessionId: string, compressed: Uint8Array) => {
+ upload: async (sessionId: string, compressed: Uint8Array, signal?: AbortSignal) => {
const body = new Uint8Array(compressed);
- const result = await ask("collector", { type: "collect", sessionId, body });
+ const result = await ask("collector", { type: "collect", sessionId, body }, signal);
if (!result.response) throw new Error("the main thread returned no response");
return deserializeResponse(result.response);
},
diff --git a/apps/server/src/collect-upload-budget.ts b/apps/server/src/collect-upload-budget.ts
new file mode 100644
index 00000000..9dd8942a
--- /dev/null
+++ b/apps/server/src/collect-upload-budget.ts
@@ -0,0 +1,39 @@
+/**
+ * How long one collector upload (POST <gateway>/collect) may take.
+ *
+ * Envelopes range from a few KiB (a change snapshot) to tens of MiB (the
+ * start snapshot of a large workspace), so the deadline grows with the body:
+ *
+ * - `baseMs` covers connecting, TLS and the first bytes;
+ * - the body is allowed `bytesPerSecond`, a conservative uplink rate, capped
+ * so that this sending part never exceeds `maxSendMs`;
+ * - `responseMs` is the wait for the answer once the body is sent: the
+ * gateway scrubs a large envelope before it answers.
+ *
+ * The fetch implementations the uploads go through (undici in Node and
+ * Electron, Electron's net.fetch, Bun's fetch) report no upload progress, so
+ * the moment the last body byte left is not observable and the response wait
+ * cannot run as a timer of its own. The parts therefore form one deadline for
+ * the whole request, computed from the envelope's size before it is sent: a
+ * body sent faster than the assumed rate leaves the unused sending time to the
+ * response wait.
+ */
+export type CollectUploadBudget = {
+ baseMs: number;
+ bytesPerSecond: number;
+ maxSendMs: number;
+ responseMs: number;
+};
+
+export const COLLECT_UPLOAD_BUDGET: CollectUploadBudget = {
+ baseMs: 30_000,
+ bytesPerSecond: 128 * 1024,
+ maxSendMs: 15 * 60_000,
+ responseMs: 120_000,
+};
+
+/** The whole-request deadline, in milliseconds, for a collect upload of `bytes`. */
+export function collectUploadTimeoutMs(bytes: number, budget: CollectUploadBudget = COLLECT_UPLOAD_BUDGET): number {
+ const sendMs = budget.baseMs + Math.ceil((Math.max(0, bytes) * 1_000) / budget.bytesPerSecond);
+ return Math.min(budget.maxSendMs, sendMs) + budget.responseMs;
+}
diff --git a/apps/server/src/omnirush-gateway-broker.ts b/apps/server/src/omnirush-gateway-broker.ts
index 792074ed..3de1980a 100644
--- a/apps/server/src/omnirush-gateway-broker.ts
+++ b/apps/server/src/omnirush-gateway-broker.ts
@@ -1,5 +1,6 @@
import { timingSafeEqual } from "node:crypto";
+import { collectUploadTimeoutMs } from "./collect-upload-budget.js";
import { externalFetch } from "./server-fetch.js";
import type { OmniRushGatewayCredentialBundle, OmniRushGatewayCredentials } from "./types.js";
@@ -598,7 +599,13 @@ export class OmniRushGatewayBroker {
});
}
- collect(sessionId: string, body: Uint8Array): Promise<Response> {
+ /**
+ * One collector envelope. The deadline grows with the envelope's size
+ * (collect-upload-budget.ts); `signal`, the collector's own deadline or
+ * cancel, ends it sooner.
+ */
+ collect(sessionId: string, body: Uint8Array, signal?: AbortSignal): Promise<Response> {
+ const timeoutMs = collectUploadTimeoutMs(body.byteLength);
return this.withDeviceBearer((state) => this.fetcher(apiUrl(state.gatewayUrl, "collect"), {
method: "POST",
headers: {
@@ -607,7 +614,7 @@ export class OmniRushGatewayBroker {
"X-OmniRush-Session-ID": sessionId,
},
body: body.buffer.slice(body.byteOffset, body.byteOffset + body.byteLength) as ArrayBuffer,
- signal: AbortSignal.timeout(30_000),
+ signal: signal ? AbortSignal.any([AbortSignal.timeout(timeoutMs), signal]) : AbortSignal.timeout(timeoutMs),
}));
}
diff --git a/apps/server/src/server.ts b/apps/server/src/server.ts
index eaf2295e..64954d0b 100644
--- a/apps/server/src/server.ts
+++ b/apps/server/src/server.ts
@@ -866,7 +866,7 @@ export async function startServer(config: ServerConfig): Promise<ServeResult> {
log: (level, message, attributes) => logger.log(level, message, attributes),
collector: gatewayBroker.enabled
? {
- upload: (sessionId, compressed) => gatewayBroker.collect(sessionId, compressed),
+ upload: (sessionId, compressed, signal) => gatewayBroker.collect(sessionId, compressed, signal),
refreshAccessToken: () => gatewayBroker.refreshAccessToken(),
}
: {},
diff --git a/apps/server/src/workspace-collector.test.ts b/apps/server/src/workspace-collector.test.ts
index ed7d8182..5d31761b 100644
--- a/apps/server/src/workspace-collector.test.ts
+++ b/apps/server/src/workspace-collector.test.ts
@@ -36,6 +36,7 @@ import {
stripRemoteUserinfo,
workspaceRelativePath,
} from "./workspace-collector.js";
+import { COLLECT_UPLOAD_BUDGET, collectUploadTimeoutMs, type CollectUploadBudget } from "./collect-upload-budget.js";
const execFileAsync = promisify(execFile);
const roots: string[] = [];
@@ -739,6 +740,216 @@ describe("workspace collector durable retry", () => {
});
});
+describe("workspace collector upload deadline and spool drain", () => {
+ // Upload deadlines at 1/100 of real time: one simulated second is 10 ms.
+ const hundredth: CollectUploadBudget = {
+ baseMs: COLLECT_UPLOAD_BUDGET.baseMs / 100,
+ bytesPerSecond: COLLECT_UPLOAD_BUDGET.bytesPerSecond * 100,
+ maxSendMs: COLLECT_UPLOAD_BUDGET.maxSendMs / 100,
+ responseMs: COLLECT_UPLOAD_BUDGET.responseMs / 100,
+ };
+ const simulatedSeconds = (seconds: number) => seconds * 10;
+ const sleep = (ms: number, signal?: AbortSignal) => new Promise<void>((resolvePromise, reject) => {
+ const timer = setTimeout(resolvePromise, ms);
+ signal?.addEventListener("abort", () => {
+ clearTimeout(timer);
+ reject(signal.reason);
+ }, { once: true });
+ });
+
+ /** Writes one spool entry as a failed upload leaves it; ids sort in `index` order. */
+ async function spoolEntry(stateDir: string, index: number, bytes: number, extra: Record<string, unknown> = {}): Promise<string> {
+ const spoolDir = join(stateDir, "omnirush-collector-spool");
+ await mkdir(spoolDir, { recursive: true, mode: 0o700 });
+ const id = `${index.toString(16).padStart(12, "0")}-${index.toString(16).padStart(6, "0")}-0000000${index}`;
+ const body = Buffer.alloc(bytes, index);
+ await writeFile(join(spoolDir, `${id}.zst`), body, { mode: 0o600 });
+ await writeFile(join(spoolDir, `${id}.json`), JSON.stringify({
+ id,
+ session_id: `session-drain-${index}`,
+ snapshot_type: index === 1 ? "start" : "change",
+ trigger: index === 1 ? "session_start" : "turn_completed",
+ sequence: index,
+ bytes,
+ created_at: new Date().toISOString(),
+ attempts: 1,
+ ...extra,
+ }), { mode: 0o600 });
+ return id;
+ }
+
+ async function spoolMeta(stateDir: string, id: string): Promise<{ attempts: number; last_attempt_at?: string }> {
+ return JSON.parse(await readFile(join(stateDir, "omnirush-collector-spool", `${id}.json`), "utf8")) as { attempts: number; last_attempt_at?: string };
+ }
+
+ async function drainState(): Promise<string> {
+ const stateDir = await mkdtemp(join(tmpdir(), "omnirush-collector-drain-state-"));
+ roots.push(stateDir);
+ return stateDir;
+ }
+
+ test("scales the deadline with the envelope and adds the wait for the gateway's answer", () => {
+ expect(collectUploadTimeoutMs(0)).toBe(150_000);
+ // A 10 MiB start snapshot: 30 s, 80 s of body at 128 KiB/s, 120 s for the answer.
+ expect(collectUploadTimeoutMs(10 * 1024 * 1024)).toBe(230_000);
+ expect(collectUploadTimeoutMs(10 * 1024 * 1024)).toBeGreaterThan(60_000 + 90_000);
+ expect(collectUploadTimeoutMs(1024 * 1024 * 1024)).toBe(15 * 60_000 + 120_000);
+ expect(collectUploadTimeoutMs(10 * 1024 * 1024, hundredth)).toBe(2_300);
+ });
+
+ test("delivers a 10 MB envelope over an uplink that needs 60 s for it on the first attempt", async () => {
+ const stateDir = await drainState();
+ await spoolEntry(stateDir, 1, 10 * 1024 * 1024);
+ const calls: { bytes: number; aborted: boolean }[] = [];
+ const collector = new WorkspaceCollector({
+ stateDir,
+ upload: async (_sessionId, compressed, signal) => {
+ // Throttled: the body trickles out over 60 s (about 170 KiB/s).
+ await sleep(simulatedSeconds(60), signal);
+ calls.push({ bytes: compressed.byteLength, aborted: signal?.aborted ?? true });
+ return Response.json({ ok: true }, { status: 201 });
+ },
+ uploadBudget: hundredth,
+ retryBaseMs: 60_000,
+ });
+ expect(await collector.drainSpool()).toEqual({ delivered: 1, pending: 0 });
+ expect(calls).toEqual([{ bytes: 10 * 1024 * 1024, aborted: false }]);
+ expect(await collector.spoolStatus()).toEqual({ entries: 0, bytes: 0 });
+ await collector.stop();
+ });
+
+ test("waits for an answer that comes 90 s after the body is sent", async () => {
+ const stateDir = await drainState();
+ await spoolEntry(stateDir, 1, 6 * 1024 * 1024);
+ const answered: boolean[] = [];
+ const collector = new WorkspaceCollector({
+ stateDir,
+ gatewayUrl: "https://gateway.example.test/v1",
+ accessToken: "device-token",
+ fetch: async (_input: string, init?: RequestInit) => {
+ const signal = init?.signal ?? undefined;
+ await sleep(simulatedSeconds(20), signal);
+ // Body sent; the gateway scrubs the envelope before it answers.
+ await sleep(simulatedSeconds(90), signal);
+ answered.push(signal?.aborted ?? true);
+ return Response.json({ ok: true }, { status: 201 });
+ },
+ uploadBudget: hundredth,
+ retryBaseMs: 60_000,
+ });
+ expect(await collector.drainSpool()).toEqual({ delivered: 1, pending: 0 });
+ expect(answered).toEqual([false]);
+ await collector.stop();
+ });
+
+ test("delivers the entries behind a spooled upload that keeps failing, and backs that one off", async () => {
+ const stateDir = await drainState();
+ const stuck = await spoolEntry(stateDir, 1, 4_096);
+ await spoolEntry(stateDir, 2, 1_024);
+ await spoolEntry(stateDir, 3, 1_024);
+ const attempts: number[] = [];
+ const delivered: number[] = [];
+ const collector = new WorkspaceCollector({
+ stateDir,
+ upload: async (sessionId) => {
+ const index = Number(sessionId.split("-").at(-1));
+ attempts.push(index);
+ // The first entry never gets an answer, even past its deadline.
+ if (index === 1) return new Promise<Response>(() => undefined);
+ delivered.push(index);
+ return Response.json({ ok: true }, { status: 201 });
+ },
+ uploadBudget: { baseMs: 50, bytesPerSecond: 128 * 1024 * 1000, maxSendMs: 900, responseMs: 100 },
+ retryBaseMs: 60_000,
+ });
+
+ expect(await collector.drainSpool()).toEqual({ delivered: 2, pending: 1 });
+ expect(attempts).toEqual([1, 2, 3]);
+ expect(delivered).toEqual([2, 3]);
+ const failed = await spoolMeta(stateDir, stuck);
+ expect(failed.attempts).toBe(2);
+ expect(typeof failed.last_attempt_at).toBe("string");
+
+ // Within its backoff the failed entry is left alone.
+ expect(await collector.drainSpool()).toEqual({ delivered: 0, pending: 1 });
+ expect(attempts).toEqual([1, 2, 3]);
+
+ // Once the backoff (the retry base after one failed drain) has passed, it is tried again...
+ const spoolDir = join(stateDir, "omnirush-collector-spool");
+ await writeFile(join(spoolDir, `${stuck}.json`), JSON.stringify({ ...failed, last_attempt_at: new Date(Date.now() - 61_000).toISOString() }));
+ expect(await collector.drainSpool()).toEqual({ delivered: 0, pending: 1 });
+ expect(attempts).toEqual([1, 2, 3, 1]);
+ const again = await spoolMeta(stateDir, stuck);
+ expect(again.attempts).toBe(3);
+
+ // ...and the next backoff is twice as long.
+ await writeFile(join(spoolDir, `${stuck}.json`), JSON.stringify({ ...again, last_attempt_at: new Date(Date.now() - 90_000).toISOString() }));
+ expect(await collector.drainSpool()).toEqual({ delivered: 0, pending: 1 });
+ expect(attempts).toEqual([1, 2, 3, 1]);
+ await collector.stop();
+ });
+
+ test("stops a drain after three failures and leaves the rest to the next one", async () => {
+ const stateDir = await drainState();
+ for (let index = 1; index <= 5; index += 1) await spoolEntry(stateDir, index, 512);
+ const attempts: string[] = [];
+ const collector = new WorkspaceCollector({
+ stateDir,
+ upload: async (sessionId) => {
+ attempts.push(sessionId);
+ return Response.json({ error: "unavailable" }, { status: 503 });
+ },
+ retryBaseMs: 60_000,
+ });
+ expect(await collector.drainSpool()).toEqual({ delivered: 0, pending: 5 });
+ expect(attempts).toEqual(["session-drain-1", "session-drain-2", "session-drain-3"]);
+ await collector.stop();
+ });
+
+ test("drops an entry once it reaches the attempt limit", async () => {
+ const stateDir = await drainState();
+ await spoolEntry(stateDir, 1, 512, { attempts: 23 });
+ await spoolEntry(stateDir, 2, 512);
+ const warnings: string[] = [];
+ const collector = new WorkspaceCollector({
+ stateDir,
+ upload: async () => Response.json({ error: "unavailable" }, { status: 503 }),
+ log: (level, message) => { if (level === "warn") warnings.push(message); },
+ retryBaseMs: 60_000,
+ });
+ expect(await collector.drainSpool()).toEqual({ delivered: 0, pending: 1 });
+ expect(warnings).toEqual(["OmniRush collection artifact dropped from spool"]);
+ expect(await collector.spoolStatus()).toEqual({ entries: 1, bytes: 512 });
+ await collector.stop();
+ });
+
+ test("a sign-out aborts the spooled upload in flight and empties the spool", async () => {
+ const stateDir = await drainState();
+ await spoolEntry(stateDir, 1, 1_024);
+ let started!: () => void;
+ const inFlight = new Promise<void>((resolvePromise) => { started = resolvePromise; });
+ let aborted = false;
+ const collector = new WorkspaceCollector({
+ stateDir,
+ upload: (_sessionId, _compressed, signal) => new Promise<Response>((_resolvePromise, reject) => {
+ signal?.addEventListener("abort", () => {
+ aborted = true;
+ reject(signal.reason);
+ }, { once: true });
+ started();
+ }),
+ retryBaseMs: 60_000,
+ });
+ const drain = collector.drainSpool();
+ await inFlight;
+ await collector.clearSpool();
+ expect(aborted).toBe(true);
+ expect(await drain).toEqual({ delivered: 0, pending: 1 });
+ expect(await collector.spoolStatus()).toEqual({ entries: 0, bytes: 0 });
+ await collector.stop();
+ });
+});
+
describe("workspace collector git helpers", () => {
test("clamps text by code points without splitting surrogate pairs", () => {
expect(clampCollectorText("abc", 5)).toBe("abc");
diff --git a/apps/server/src/workspace-collector.ts b/apps/server/src/workspace-collector.ts
index 6a43b855..094dd0d4 100644
--- a/apps/server/src/workspace-collector.ts
+++ b/apps/server/src/workspace-collector.ts
@@ -8,6 +8,7 @@ import { promisify } from "node:util";
import { createZstdCompress } from "node:zlib";
import { minimatch } from "minimatch";
+import { COLLECT_UPLOAD_BUDGET, collectUploadTimeoutMs, type CollectUploadBudget } from "./collect-upload-budget.js";
import { externalFetch } from "./server-fetch.js";
import { TurnBaseStore, TurnDiffBuilder, type TurnDiffInput } from "./turn-diff.js";
@@ -122,6 +123,10 @@ const UPLOAD_ATTEMPTS = 3;
const UPLOAD_RETRY_DELAY_MS = 250;
const RETRY_BASE_MS = 5_000;
const RETRY_MAX_MS = 5 * 60_000;
+// Spooled entries one drain may see fail before it leaves the rest to the
+// next one: enough to get past a stuck entry, few enough not to hammer a link
+// that is down for every entry in the spool.
+const MAX_DRAIN_FAILURES = 3;
const RETRYABLE_STATUSES = new Set([408, 425, 429, 500, 502, 503, 504]);
// A rejected bearer. The body tells a stale device token (refreshed once and
// retried, spooled if still rejected) from the sign-in gate, which is final.
@@ -363,7 +368,8 @@ type CollectorOptions = {
gatewayUrl?: string;
accessToken?: string;
fetch?: typeof externalFetch;
- upload?: (sessionId: string, compressed: Uint8Array) => Promise<Response>;
+ /** Sends one envelope; `signal` aborts at the upload deadline (collect-upload-budget.ts) or when the spool is cleared. */
+ upload?: (sessionId: string, compressed: Uint8Array, signal?: AbortSignal) => Promise<Response>;
/**
* Asks the account layer for a fresh access token once the gateway rejects
* an upload as unauthorized. Resolves with the bearer to send next, or null
@@ -380,6 +386,8 @@ type CollectorOptions = {
uploadRetryDelayMs?: number;
retryBaseMs?: number;
retryMaxMs?: number;
+ /** The upload deadline's parameters; COLLECT_UPLOAD_BUDGET unless a test shrinks it. */
+ uploadBudget?: CollectUploadBudget;
spoolMaxEntries?: number;
spoolMaxBytes?: number;
/** Uncompressed envelope cap; the backend's MAX_SNAPSHOT_BYTES unless a test lowers it. */
@@ -2848,6 +2856,26 @@ function collectorEnvironment(appVersion: string | undefined, engineVersion: str
};
}
+/**
+ * Settles like `response`, or rejects once `signal` aborts, so an upload hook
+ * that does not watch its signal still ends at the deadline.
+ */
+function untilAborted(response: Promise<Response>, signal: AbortSignal): Promise<Response> {
+ return new Promise((resolvePromise, reject) => {
+ const onAbort = () => reject(signal.reason);
+ signal.addEventListener("abort", onAbort, { once: true });
+ response.then((value) => {
+ signal.removeEventListener("abort", onAbort);
+ // An answer after the deadline is dropped like an aborted fetch's.
+ if (signal.aborted) void value.body?.cancel().catch(() => undefined);
+ resolvePromise(value);
+ }, (error: unknown) => {
+ signal.removeEventListener("abort", onAbort);
+ reject(error);
+ });
+ });
+}
+
function spoolId(counter: number): string {
return `${Date.now().toString(16).padStart(12, "0")}-${counter.toString(16).padStart(6, "0")}-${randomBytes(4).toString("hex")}`;
}
@@ -2902,6 +2930,7 @@ export class WorkspaceCollector {
private readonly uploadRetryDelayMs: number;
private readonly retryBaseMs: number;
private readonly retryMaxMs: number;
+ private readonly uploadBudget: CollectUploadBudget;
private readonly spoolMaxEntries: number;
private readonly spoolMaxBytes: number;
private readonly snapshotMaxBytes: number;
@@ -2910,6 +2939,9 @@ export class WorkspaceCollector {
private spoolTail: Promise<void> = Promise.resolve();
private retryTimer: ReturnType<typeof setTimeout> | null = null;
private retryFailures = 0;
+ /** The drain in progress: one at a time, and aborted by stop() and clearSpool(). */
+ private draining: { run: Promise<{ delivered: number; pending: number }>; controller: AbortController } | null = null;
+ private drainQueued: Promise<{ delivered: number; pending: number }> | null = null;
private stopped = false;
private readonly minChangeIntervalMs: number;
private readonly maxWatchedFiles: number;
@@ -2946,6 +2978,7 @@ export class WorkspaceCollector {
this.uploadRetryDelayMs = options.uploadRetryDelayMs ?? UPLOAD_RETRY_DELAY_MS;
this.retryBaseMs = options.retryBaseMs ?? RETRY_BASE_MS;
this.retryMaxMs = options.retryMaxMs ?? RETRY_MAX_MS;
+ this.uploadBudget = options.uploadBudget ?? COLLECT_UPLOAD_BUDGET;
this.spoolMaxEntries = options.spoolMaxEntries ?? MAX_SPOOL_ENTRIES;
this.spoolMaxBytes = options.spoolMaxBytes ?? MAX_SPOOL_BYTES;
this.snapshotMaxBytes = options.snapshotMaxBytes ?? MAX_SNAPSHOT_BYTES;
@@ -3903,8 +3936,11 @@ export class WorkspaceCollector {
clearTimeout(this.retryTimer);
this.retryTimer = null;
}
+ // A spooled upload in flight stays spooled for the next start.
+ this.draining?.controller.abort();
for (const sessionId of [...this.sessions.keys()]) this.finishSession(sessionId);
await Promise.allSettled([...this.sessions.values()].map((state) => state.tail));
+ await this.draining?.run.catch(() => undefined);
await this.spoolTail.catch(() => undefined);
await this.ledgerWriteTail.catch(() => undefined);
await this.bases.flush().catch(() => undefined);
@@ -4342,8 +4378,17 @@ export class WorkspaceCollector {
});
}
- private send(sessionId: string, compressed: Uint8Array): Promise<Response> {
- if (this.uploader) return this.uploader(sessionId, compressed);
+ /**
+ * One POST of an envelope. Its deadline grows with the envelope's size
+ * (collect-upload-budget.ts: fetch reports no upload progress, so sending
+ * the body and the gateway's answer share one size-scaled deadline);
+ * `cancel` ends it sooner.
+ */
+ private send(sessionId: string, compressed: Uint8Array, cancel?: AbortSignal): Promise<Response> {
+ const deadline = AbortSignal.timeout(collectUploadTimeoutMs(compressed.byteLength, this.uploadBudget));
+ const signal = cancel ? AbortSignal.any([deadline, cancel]) : deadline;
+ if (signal.aborted) return Promise.reject(signal.reason);
+ if (this.uploader) return untilAborted(this.uploader(sessionId, compressed, signal), signal);
return this.fetcher(this.collectUrl!, {
method: "POST",
headers: {
@@ -4352,7 +4397,7 @@ export class WorkspaceCollector {
"X-OmniRush-Session-ID": sessionId,
},
body: compressed.buffer.slice(compressed.byteOffset, compressed.byteOffset + compressed.byteLength) as ArrayBuffer,
- signal: AbortSignal.timeout(30_000),
+ signal,
});
}
@@ -4369,13 +4414,13 @@ export class WorkspaceCollector {
}
}
- private async transmit(sessionId: string, compressed: Uint8Array, attempts = UPLOAD_ATTEMPTS): Promise<TransmitOutcome> {
+ private async transmit(sessionId: string, compressed: Uint8Array, attempts = UPLOAD_ATTEMPTS, cancel?: AbortSignal): Promise<TransmitOutcome> {
let lastReason = "collector upload unavailable";
let refreshAttempted = false;
let attempt = 0;
while (attempt < attempts) {
try {
- const response = await this.send(sessionId, compressed);
+ const response = await this.send(sessionId, compressed, cancel);
if (response.ok) return { ok: true };
lastReason = `collector upload failed with status ${response.status}`;
if (UNAUTHORIZED_STATUSES.has(response.status)) {
@@ -4403,6 +4448,7 @@ export class WorkspaceCollector {
if (!RETRYABLE_STATUSES.has(response.status)) return { ok: false, retryable: false, reason: lastReason };
} catch (error) {
lastReason = error instanceof Error ? error.message : "collector upload unavailable";
+ if (cancel?.aborted) return { ok: false, retryable: true, reason: lastReason };
}
attempt += 1;
if (attempt < attempts) {
@@ -4631,8 +4677,7 @@ export class WorkspaceCollector {
private scheduleRetry(delayMs?: number): void {
if (!this.spoolDir || this.stopped || this.retryTimer) return;
- const backoff = Math.min(this.retryMaxMs, this.retryBaseMs * 2 ** Math.max(0, this.retryFailures - 1));
- const delay = delayMs ?? Math.round(backoff * (0.85 + Math.random() * 0.3));
+ const delay = delayMs ?? this.jitteredBackoffMs();
this.retryTimer = setTimeout(() => {
this.retryTimer = null;
void this.drainSpool().catch(() => undefined);
@@ -4640,58 +4685,119 @@ export class WorkspaceCollector {
this.retryTimer.unref?.();
}
- /** Delivers spooled uploads oldest first, stopping at the first failure. */
+ /** The wait after `failures` failures in a row: the retry base, doubling up to the retry cap. */
+ private backoffMs(failures: number): number {
+ return Math.min(this.retryMaxMs, this.retryBaseMs * 2 ** Math.max(0, failures - 1));
+ }
+
+ private jitteredBackoffMs(): number {
+ return Math.round(this.backoffMs(this.retryFailures) * (0.85 + Math.random() * 0.3));
+ }
+
+ /**
+ * When a spooled entry may be sent again: at once until a drain has tried
+ * it, then after its own backoff (its first attempt was the upload that
+ * spooled it).
+ */
+ private spoolEntryDueAt(entry: SpoolMeta): number {
+ const last = entry.last_attempt_at ? Date.parse(entry.last_attempt_at) : Number.NaN;
+ return Number.isFinite(last) ? last + this.backoffMs(entry.attempts - 1) : 0;
+ }
+
+ /**
+ * Delivers spooled uploads oldest first. An entry that fails waits out its
+ * own backoff while the entries behind it go on, so one upload that keeps
+ * failing holds back no other; a drain stops trying after
+ * MAX_DRAIN_FAILURES failures and leaves the rest to the next one. The
+ * gateway orders a session's envelopes by their capture sequence, not by
+ * arrival. Uploads run outside the spool lock, so spooling, spoolStatus()
+ * and clearSpool() never wait for one; stop() and clearSpool() abort it.
+ * A call while a drain runs gets one more drain right after it, for
+ * entries spooled since that one listed the spool.
+ */
drainSpool(): Promise<{ delivered: number; pending: number }> {
- return this.spoolLocked(async () => {
- let delivered = 0;
- if (!this.spoolDir || !this.enabled) return { delivered, pending: 0 };
- const entries = await this.listSpool();
- let pending = entries.length;
- for (const entry of entries) {
- if (this.stopped) break;
- let compressed: Buffer;
- try {
- compressed = await readFile(join(this.spoolDir, `${entry.id}.zst`));
- } catch {
- await this.removeSpoolEntry(entry.id);
- pending -= 1;
- continue;
- }
- const outcome = await this.transmit(entry.session_id, compressed, 1);
- if (outcome.ok) {
- await this.removeSpoolEntry(entry.id);
- await this.recordLedgerOutcome(entry.session_id, "success").catch(() => undefined);
- this.retryFailures = 0;
- delivered += 1;
- pending -= 1;
- this.log("info", "OmniRush collection artifact delivered from spool", {
- sessionId: entry.session_id,
- snapshotType: entry.snapshot_type,
- trigger: entry.trigger,
- sequence: entry.sequence,
- attempts: entry.attempts + 1,
- });
- continue;
- }
- if (!outcome.retryable || entry.attempts + 1 >= MAX_SPOOL_ATTEMPTS) {
- await this.removeSpoolEntry(entry.id);
- pending -= 1;
- this.log("warn", "OmniRush collection artifact dropped from spool", {
- sessionId: entry.session_id,
- snapshotType: entry.snapshot_type,
- sequence: entry.sequence,
- reason: outcome.reason,
- });
- continue;
- }
- await this.writeSpoolMeta({ ...entry, attempts: entry.attempts + 1, last_attempt_at: new Date().toISOString() }).catch(() => undefined);
- await this.recordLedgerOutcome(entry.session_id, "failure").catch(() => undefined);
- this.retryFailures += 1;
- this.scheduleRetry();
- break;
- }
- return { delivered, pending };
+ if (this.draining) {
+ this.drainQueued ??= this.draining.run.catch(() => undefined).then(() => {
+ this.drainQueued = null;
+ return this.drainSpool();
+ });
+ return this.drainQueued;
+ }
+ const controller = new AbortController();
+ const run = this.runDrain(controller.signal).finally(() => {
+ if (this.draining?.controller === controller) this.draining = null;
});
+ this.draining = { run, controller };
+ return run;
+ }
+
+ private async runDrain(signal: AbortSignal): Promise<{ delivered: number; pending: number }> {
+ let delivered = 0;
+ const spoolDir = this.spoolDir;
+ if (!spoolDir || !this.enabled) return { delivered, pending: 0 };
+ const entries = await this.spoolLocked(() => this.listSpool());
+ let pending = entries.length;
+ let failures = 0;
+ let nextDueAt = Number.POSITIVE_INFINITY;
+ for (const entry of entries) {
+ if (this.stopped || signal.aborted) break;
+ const dueAt = this.spoolEntryDueAt(entry);
+ if (failures >= MAX_DRAIN_FAILURES || dueAt > Date.now()) {
+ nextDueAt = Math.min(nextDueAt, dueAt);
+ continue;
+ }
+ let compressed: Buffer;
+ try {
+ compressed = await readFile(join(spoolDir, `${entry.id}.zst`));
+ } catch {
+ await this.spoolLocked(() => this.removeSpoolEntry(entry.id));
+ pending -= 1;
+ continue;
+ }
+ const outcome = await this.transmit(entry.session_id, compressed, 1, signal);
+ if (outcome.ok) {
+ await this.spoolLocked(() => this.removeSpoolEntry(entry.id));
+ await this.recordLedgerOutcome(entry.session_id, "success").catch(() => undefined);
+ this.retryFailures = 0;
+ delivered += 1;
+ pending -= 1;
+ this.log("info", "OmniRush collection artifact delivered from spool", {
+ sessionId: entry.session_id,
+ snapshotType: entry.snapshot_type,
+ trigger: entry.trigger,
+ sequence: entry.sequence,
+ attempts: entry.attempts + 1,
+ });
+ continue;
+ }
+ // Stopped or signed out mid-upload: the entry stays as it was, or goes with the spool.
+ if (signal.aborted) break;
+ if (!outcome.retryable || entry.attempts + 1 >= MAX_SPOOL_ATTEMPTS) {
+ await this.spoolLocked(() => this.removeSpoolEntry(entry.id));
+ pending -= 1;
+ this.log("warn", "OmniRush collection artifact dropped from spool", {
+ sessionId: entry.session_id,
+ snapshotType: entry.snapshot_type,
+ sequence: entry.sequence,
+ reason: outcome.reason,
+ });
+ continue;
+ }
+ failures += 1;
+ const attempted: SpoolMeta = { ...entry, attempts: entry.attempts + 1, last_attempt_at: new Date().toISOString() };
+ await this.spoolLocked(async () => {
+ // The spool bound may have dropped the entry during the upload.
+ await lstat(join(spoolDir, `${entry.id}.zst`));
+ await this.writeSpoolMeta(attempted);
+ }).catch(() => undefined);
+ await this.recordLedgerOutcome(entry.session_id, "failure").catch(() => undefined);
+ nextDueAt = Math.min(nextDueAt, this.spoolEntryDueAt(attempted));
+ }
+ if (failures > 0) this.retryFailures += 1;
+ if (pending > 0 && !signal.aborted) {
+ this.scheduleRetry(Math.max(this.jitteredBackoffMs(), Number.isFinite(nextDueAt) ? nextDueAt - Date.now() : 0));
+ }
+ return { delivered, pending };
}
/** Number of spooled uploads and their compressed size on disk. */
@@ -4712,6 +4818,7 @@ export class WorkspaceCollector {
clearTimeout(this.retryTimer);
this.retryTimer = null;
}
+ this.draining?.controller.abort();
this.retryFailures = 0;
return this.spoolLocked(async () => {
// The turn diffs' bases are workspace content on disk too.
diff --git a/docs/workspace-collector-privacy.md b/docs/workspace-collector-privacy.md
index aabf60da..399b60ab 100644
--- a/docs/workspace-collector-privacy.md
+++ b/docs/workspace-collector-privacy.md
@@ -17,7 +17,7 @@ Neither field changes what may leave the machine: `files[]`, `changed_paths` and
There is no per-session storage limit: a session keeps uploading snapshots and traces for as long as it runs, however much it has already sent, and the privacy block carries no `max_session_bytes`. Every remaining limit applies to one file or one request and matches what the backend accepts per request: 4 MiB per file, 64 MiB per snapshot envelope (uncompressed, and again once compressed), 16 MiB per trace document, 2 MiB of git diff and 50,000 manifest entries. These bound the collector's memory and never stop a session: a snapshot that would not fit still goes out with the files it has room for (touched and git-changed files first, smallest first within each tier), the full manifest, and the rest counted in `snapshot_cap_omitted_count` and `snapshot_cap_omitted_bytes`. The collector still counts the bytes each session has sent, in its local session ledger, for diagnostics only.
-The desktop collector keeps its footprint bounded while producing these snapshots. It runs on a worker thread of the app's local server, together with the turn observer and the project archive (`apps/server/src/capture-host.ts`), so reading, hashing and scrubbing files, building and compressing envelopes, and parsing long transcripts never hold the event loop that serves the app (the Electron main process in the desktop app); the main thread only forwards calls and makes the uploads the worker hands it. Envelopes are streamed as JSON through zstd into a temp file under the app state directory (`omnirush-collector-tmp`) and uploaded from that file, so no envelope string or buffer is ever held whole; a failed upload moves the file into the retry spool (same filesystem, a rename) and a delivered one is deleted at once. Files are read with bounded concurrency and the scan yields to the event loop between files. A per-root digest cache keyed by path, size and mtime, shared by every chat open on the same workspace, means unchanged files are neither re-read nor re-hashed, and a file the scrubber left byte-identical is uploaded as read (verified by digest) without a second pass through the redaction rules; the redacted text of files the scrubber did change is kept too (32 MiB at most, least recently used out first) and sent again only for the exact bytes it came from (verified by the digest of the bytes read), so a new chat on a workspace another chat already captured runs no file through the scrubber a second time. The scrubber runs the assignment and e-mail rules only over lines holding their keyword or an "@" and skips a rule whose literal the text lacks; neither shortcut changes a single output byte. Change snapshots come from watcher-driven dirty tracking: a prompt or completed turn with nothing dirty, journaled or committed since the last accepted snapshot records only its `collector.trigger` event; a change is scanned by dirty path, and before anything is read git is asked afresh, in one batched `git check-ignore`, about every reported path and every journal entry (or the workspace's own `.gitignore` files are applied where git cannot answer), so a file the user has just ignored through `.gitignore`, `.git/info/exclude` or `core.excludesFile` leaves the manifest and the journal at once; a path git refuses to answer for (one beyond a symlinked directory) is isolated by splitting the batch, so it cannot change the answer for the others, and counts as ignored. No path outside the workspace root (a parent folder, another Windows drive or a network share), and none reached through a symlinked directory, is ever read or listed. The whole tree is rescanned only when the dirty set overflowed, a directory appeared, a `.gitignore` changed, or the workspace is polled. The watchers go up from the start snapshot's listing before any file is read and cover the root, the top-level directories that hold listed files and then every other top-level directory git does not ignore and the denylist does not name (an empty `scripts/`, a `config/` holding only a denied `.env`), so `.git/`, `node_modules/`, gitignored build output and denied directories never reach them, and a gitignored directory that appears later (a rebuilt `dist/`) is neither watched nor rescanned; at the end of a turn every untracked file the turn created or changed is marked dirty too, so the turn's own snapshot carries it even where no watcher reaches; a workspace with more than 20,000 eligible files, or more than 64 top-level directories holding listed files, is polled on snapshots instead of watched (past 64 watched directories in all, the rest are left to the reconcile pass and the end-of-turn scan), and one reconcile pass (listing and stat only) per workspace every 60 seconds covers every chat watching it. A file larger than the change journal's budget is not journaled on every write; the next snapshot carries it. Filesystem- and reconcile-driven change snapshots are spaced at least 30 seconds apart, with the triggers in between merged into one; prompt and turn milestones capture at once. When several chats are open on the same folder, the edits made while one of them runs a turn are that turn's: only a chat with a turn in progress uploads filesystem- and reconcile-driven change snapshots for them, and a chat without one leaves them to its own next prompt or turn snapshot (or its end snapshot), so one agent's edits are not uploaded again by every chat open on the folder. Each chat's own prompt and turn milestones capture as before, and a chat alone on its folder is unaffected. This concerns the collector's scrubbed snapshots only: the project archive keeps its own copies of the folder for each chat ([project-archive.md](project-archive.md)).
+The desktop collector keeps its footprint bounded while producing these snapshots. It runs on a worker thread of the app's local server, together with the turn observer and the project archive (`apps/server/src/capture-host.ts`), so reading, hashing and scrubbing files, building and compressing envelopes, and parsing long transcripts never hold the event loop that serves the app (the Electron main process in the desktop app); the main thread only forwards calls and makes the uploads the worker hands it. Envelopes are streamed as JSON through zstd into a temp file under the app state directory (`omnirush-collector-tmp`) and uploaded from that file, so no envelope string or buffer is ever held whole; a failed upload moves the file into the retry spool (same filesystem, a rename) and a delivered one is deleted at once. An upload's deadline grows with its envelope: 30 seconds, plus the compressed body at 128 KiB/s (at most 15 minutes for that part), plus 2 minutes for the gateway's answer (`apps/server/src/collect-upload-budget.ts`); fetch reports no upload progress, so the three parts form one deadline for the whole request. Spooled uploads are retried oldest first, each on a backoff of its own (5 seconds after its first retry, doubling up to 5 minutes, dropped after 24 attempts), and one that keeps failing holds back none of the uploads behind it: a retry pass moves past it and ends after three failures, and the gateway orders a session's snapshots by their sequence rather than by arrival. Files are read with bounded concurrency and the scan yields to the event loop between files. A per-root digest cache keyed by path, size and mtime, shared by every chat open on the same workspace, means unchanged files are neither re-read nor re-hashed, and a file the scrubber left byte-identical is uploaded as read (verified by digest) without a second pass through the redaction rules; the redacted text of files the scrubber did change is kept too (32 MiB at most, least recently used out first) and sent again only for the exact bytes it came from (verified by the digest of the bytes read), so a new chat on a workspace another chat already captured runs no file through the scrubber a second time. The scrubber runs the assignment and e-mail rules only over lines holding their keyword or an "@" and skips a rule whose literal the text lacks; neither shortcut changes a single output byte. Change snapshots come from watcher-driven dirty tracking: a prompt or completed turn with nothing dirty, journaled or committed since the last accepted snapshot records only its `collector.trigger` event; a change is scanned by dirty path, and before anything is read git is asked afresh, in one batched `git check-ignore`, about every reported path and every journal entry (or the workspace's own `.gitignore` files are applied where git cannot answer), so a file the user has just ignored through `.gitignore`, `.git/info/exclude` or `core.excludesFile` leaves the manifest and the journal at once; a path git refuses to answer for (one beyond a symlinked directory) is isolated by splitting the batch, so it cannot change the answer for the others, and counts as ignored. No path outside the workspace root (a parent folder, another Windows drive or a network share), and none reached through a symlinked directory, is ever read or listed. The whole tree is rescanned only when the dirty set overflowed, a directory appeared, a `.gitignore` changed, or the workspace is polled. The watchers go up from the start snapshot's listing before any file is read and cover the root, the top-level directories that hold listed files and then every other top-level directory git does not ignore and the denylist does not name (an empty `scripts/`, a `config/` holding only a denied `.env`), so `.git/`, `node_modules/`, gitignored build output and denied directories never reach them, and a gitignored directory that appears later (a rebuilt `dist/`) is neither watched nor rescanned; at the end of a turn every untracked file the turn created or changed is marked dirty too, so the turn's own snapshot carries it even where no watcher reaches; a workspace with more than 20,000 eligible files, or more than 64 top-level directories holding listed files, is polled on snapshots instead of watched (past 64 watched directories in all, the rest are left to the reconcile pass and the end-of-turn scan), and one reconcile pass (listing and stat only) per workspace every 60 seconds covers every chat watching it. A file larger than the change journal's budget is not journaled on every write; the next snapshot carries it. Filesystem- and reconcile-driven change snapshots are spaced at least 30 seconds apart, with the triggers in between merged into one; prompt and turn milestones capture at once. When several chats are open on the same folder, the edits made while one of them runs a turn are that turn's: only a chat with a turn in progress uploads filesystem- and reconcile-driven change snapshots for them, and a chat without one leaves them to its own next prompt or turn snapshot (or its end snapshot), so one agent's edits are not uploaded again by every chat open on the folder. Each chat's own prompt and turn milestones capture as before, and a chat alone on its folder is unaffected. This concerns the collector's scrubbed snapshots only: the project archive keeps its own copies of the folder for each chat ([project-archive.md](project-archive.md)).
## Per-turn diffs: the `turn.diff` trace event