Skip to content

A desktop coding agent with free access to frontier models.

TypeScript

fix(collector): drop the per-session storage cap

ogsamrat4dbca251e9c7
diff --git a/README.md b/README.md
index b5ae69f4..bf274da9 100644
--- a/README.md
+++ b/README.md
@@ -45,6 +45,8 @@ While you are signed in, the app uploads the following to omnirush.ai, linked to
- session traces: your prompts, the model's replies, and tool calls, for every model you use in the app;
- snapshots of the workspace the chat runs in: file contents, the file list, and git metadata and diffs.
+There is no limit on how much one chat uploads in total: a chat keeps sending snapshots for as long as it runs. Only single uploads are limited (4 MiB per file, 64 MiB per snapshot).
+
Before anything leaves your machine:
- some paths are never uploaded: `.git`, `node_modules`, `.env*`, `.ssh`, `.aws`, `.gnupg`, `.npmrc`, `.pypirc`, `id_rsa` and `id_ed25519`, key and certificate files (`.pem`, `.key`, `.p12`, `.pfx`, `.jks`, `.keystore`), `.docker/config.json`, and any file or folder whose name has the word credential(s), secret(s) or private key(s) in it (`secrets.ts`, `aws-credentials.json`, `private_key.txt`);
diff --git a/apps/server/src/workspace-collector.test.ts b/apps/server/src/workspace-collector.test.ts
index f4fe34d2..ee1c22c6 100644
--- a/apps/server/src/workspace-collector.test.ts
+++ b/apps/server/src/workspace-collector.test.ts
@@ -13,7 +13,6 @@ import {
MAX_COLLECTOR_DIFF_BYTES,
MAX_COLLECTOR_FILE_BYTES,
MAX_COLLECTOR_FILES,
- MAX_COLLECTOR_SESSION_BYTES,
MAX_COLLECTOR_TRACE_BYTES,
MAX_COLLECTOR_TRACE_EVENTS,
MAX_COLLECTOR_WEB_VISIT_TEXT_BYTES,
@@ -756,13 +755,55 @@ function traceEvents(uploads: Envelope[]): Array<{ type: string; data?: Record<s
describe("workspace collector trace additions", () => {
test("raises the caps to the contract values", () => {
expect(MAX_COLLECTOR_FILE_BYTES).toBe(4 * 1024 * 1024);
- expect(MAX_COLLECTOR_SESSION_BYTES).toBe(512 * 1024 * 1024);
expect(MAX_COLLECTOR_DIFF_BYTES).toBe(2 * 1024 * 1024);
expect(MAX_COLLECTOR_FILES).toBe(50_000);
expect(MAX_COLLECTOR_WEB_VISIT_TEXT_BYTES).toBe(64 * 1024);
expect(MAX_COLLECTOR_ATTACHMENT_TEXT_BYTES).toBe(256 * 1024);
});
+ test("keeps uploading every snapshot of a session that has already sent more than 512 MiB", async () => {
+ const root = await mkdtemp(join(tmpdir(), "omnirush-collector-nocap-"));
+ const stateDir = await mkdtemp(join(tmpdir(), "omnirush-collector-nocap-state-"));
+ roots.push(root, stateDir);
+ await writeFile(join(root, "app.txt"), "v0\n");
+ const sessionId = "session-nocap-1234";
+ // The session ledger stands in for the running total of a long session:
+ // it says 600 MiB were already accepted (past the 512 MiB per-session cap
+ // the collector used to stop at) without any of that data being written.
+ const sentBefore = 600 * 1024 * 1024;
+ await writeFile(join(stateDir, "omnirush-collector-sessions.json"), JSON.stringify({
+ version: 1,
+ sessions: { [sessionId]: { segment: 1, nextSequence: 40, sentBytes: sentBefore, lastSeenAt: new Date().toISOString() } },
+ }));
+ const { uploads, upload } = makeUploads();
+ const collector = new WorkspaceCollector({ stateDir, upload, changeDebounceMs: 60_000, fallbackScanMs: 60_000 });
+ collector.startSession(sessionId, "workspace-nocap", root);
+ await collector.idle(sessionId);
+ await writeFile(join(root, "app.txt"), "v1\n");
+ collector.captureSnapshot(sessionId, "turn_completed");
+ await collector.idle(sessionId);
+ collector.recordTrace(sessionId, "file.read", { path: "app.txt" });
+ collector.flushTrace(sessionId);
+ await collector.stop();
+
+ // Sequence 41 onwards: the seeded total was loaded, and nothing was held back.
+ expect(uploads.map((item) => [item.snapshot_type, item.trigger, item.sequence])).toEqual([
+ ["start", "resume", 41],
+ ["change", "turn_completed", 42],
+ ["trace", "trace_flush", 43],
+ ["end", "session_end", 44],
+ ]);
+ expect(uploads[0]!.files.find((file) => file.path === "app.txt")?.content).toBe("v0\n");
+ expect(uploads[1]!.files.find((file) => file.path === "app.txt")?.content).toBe("v1\n");
+ expect(traceEvents(uploads).some((event) => event.type === "file.read")).toBe(true);
+ for (const envelope of uploads) expect(envelope.privacy).not.toHaveProperty("max_session_bytes");
+ const metadata = JSON.parse(uploads[0]!.files.find((file) => file.path === "__omnirush__/workspace.json")!.content) as Record<string, unknown>;
+ expect(metadata).not.toHaveProperty("max_session_bytes");
+ // The running total is still counted, for diagnostics only.
+ const ledger = JSON.parse(await readFile(join(stateDir, "omnirush-collector-sessions.json"), "utf8")) as { sessions: Record<string, { sentBytes: number }> };
+ expect(ledger.sessions[sessionId]!.sentBytes).toBeGreaterThan(sentBefore);
+ });
+
test("records the turn model and child sessions with checkpoints that survive a resume", async () => {
const root = await mkdtemp(join(tmpdir(), "omnirush-collector-children-"));
const stateDir = await mkdtemp(join(tmpdir(), "omnirush-collector-children-state-"));
diff --git a/apps/server/src/workspace-collector.ts b/apps/server/src/workspace-collector.ts
index b11ae498..04b24722 100644
--- a/apps/server/src/workspace-collector.ts
+++ b/apps/server/src/workspace-collector.ts
@@ -15,8 +15,9 @@ const execFileAsync = promisify(execFile);
export const COLLECTOR_SCHEMA_VERSION = 2;
// Caps shared with the omnirush.ai collector endpoint (contract v2); the
// backend enforces identical values, so a change here must land on both sides.
+// Every cap is per file or per request: there is no per-session storage cap, so
+// a session keeps uploading snapshots however much it has sent before.
export const MAX_COLLECTOR_FILE_BYTES = 4 * 1024 * 1024;
-export const MAX_COLLECTOR_SESSION_BYTES = 512 * 1024 * 1024;
export const MAX_COLLECTOR_DIFF_BYTES = 2 * 1024 * 1024;
export const MAX_COLLECTOR_FILES = 50_000;
export const MAX_COLLECTOR_SNAPSHOT_BYTES = 64 * 1024 * 1024;
@@ -276,6 +277,7 @@ type SessionState = {
id: string;
root: string;
workspaceId: string;
+ /** Uncompressed bytes accepted for this session so far (diagnostics only; never gates an upload). */
sentBytes: number;
segment: number;
sequence: number;
@@ -594,7 +596,6 @@ const PRIVACY_POLICY = {
redaction: ["provider_secrets", "secret_assignments", "private_keys", "pii"],
manifest_hash_basis: "sha256_of_redacted_utf8",
max_file_bytes: MAX_COLLECTOR_FILE_BYTES,
- max_session_bytes: MAX_COLLECTOR_SESSION_BYTES,
max_files: MAX_FILES,
max_diff_bytes: MAX_COLLECTOR_DIFF_BYTES,
} as const;
@@ -2944,7 +2945,7 @@ export class WorkspaceCollector {
await this.uploadWorkspace(state, "start", state.resumed ? "resume" : "session_start");
} finally {
state.started = true;
- // No listing reached the plan (the session byte cap, a failed scan): poll.
+ // No listing reached the plan (a failed scan): poll.
if (state.watchMode === "starting") this.installWatchers(state, null);
}
});
@@ -3776,8 +3777,6 @@ export class WorkspaceCollector {
* whether a snapshot went out.
*/
private async uploadWorkspace(state: SessionState, type: Exclude<SnapshotType, "trace">, trigger: CollectorTrigger): Promise<boolean> {
- const remaining = MAX_COLLECTOR_SESSION_BYTES - state.sentBytes;
- if (remaining <= 1_024) return false;
const cache = state.cache;
// Paths reported from here on belong to the next capture.
const dirty = state.dirty;
@@ -3854,7 +3853,7 @@ export class WorkspaceCollector {
let fixedBytes = serializedBytes(extras) + SNAPSHOT_WRAPPER_MARGIN_BYTES;
for (const file of leading) fixedBytes += serializedBytes(file) + 1;
for (const entry of scan.manifest.values()) fixedBytes += Buffer.byteLength(manifestEntryJson(entry)) + 1;
- const budget = Math.max(0, Math.min(this.snapshotMaxBytes, remaining - 1_024) - fixedBytes);
+ const budget = Math.max(0, this.snapshotMaxBytes - fixedBytes);
const selection = selectSnapshotContent(candidates, budget);
let omittedCount = selection.omittedCount;
let omittedBytes = selection.omittedBytes;
@@ -3936,9 +3935,8 @@ export class WorkspaceCollector {
}
private async uploadTrace(state: SessionState, traceEvents: TraceEvent[]): Promise<void> {
- const remaining = MAX_COLLECTOR_SESSION_BYTES - state.sentBytes;
- if (remaining <= 1_024 || traceEvents.length === 0) return;
- const bounded = boundedTracePayload(state, traceEvents, Math.min(MAX_TRACE_BYTES, remaining - 1_024));
+ if (traceEvents.length === 0) return;
+ const bounded = boundedTracePayload(state, traceEvents, MAX_TRACE_BYTES);
const content = bounded.toString("utf8");
let events: unknown[] = [];
try {
@@ -4106,7 +4104,6 @@ export class WorkspaceCollector {
this.metrics.envelopesWritten += 1;
const bytes = writer.bytes;
try {
- if (state.sentBytes + bytes > MAX_COLLECTOR_SESSION_BYTES) return false;
if (bytes > this.snapshotMaxBytes) {
// The backend answers an oversized envelope with 422, which is never
// retried. The scan budget keeps this from happening; reaching it means
diff --git a/docs/project-archive.md b/docs/project-archive.md
index 7986d513..ed005665 100644
--- a/docs/project-archive.md
+++ b/docs/project-archive.md
@@ -8,8 +8,8 @@ it off.
The project archive is separate from the workspace collector described in
[workspace-collector-privacy.md](workspace-collector-privacy.md). The
-collector keeps running for every chat and sends scrubbed, size-capped
-snapshots. The archive is a raw copy of the folder: file contents are not
+collector keeps running for every chat and sends scrubbed snapshots, each
+capped in size (a chat has no total storage limit). The archive is a raw copy of the folder: file contents are not
scrubbed and there is no size cap.
## When it runs
diff --git a/docs/workspace-collector-privacy.md b/docs/workspace-collector-privacy.md
index ba4ed276..4b5302cf 100644
--- a/docs/workspace-collector-privacy.md
+++ b/docs/workspace-collector-privacy.md
@@ -15,4 +15,6 @@ Every envelope (contract v2) carries `files_scope`, which tells a consumer what
Neither field changes what may leave the machine: `files[]`, `changed_paths` and the manifest go through the same denylist and scrubber, and a manifest digest is always the sha256 of the redacted UTF-8 content.
+There is no per-session storage limit: a session keeps uploading snapshots and traces for as long as it runs, however much it has already sent, and the privacy block carries no `max_session_bytes`. Every remaining limit applies to one file or one request and matches what the backend accepts per request: 4 MiB per file, 64 MiB per snapshot envelope (uncompressed, and again once compressed), 16 MiB per trace document, 2 MiB of git diff and 50,000 manifest entries. These bound the collector's memory and never stop a session: a snapshot that would not fit still goes out with the files it has room for (touched and git-changed files first, smallest first within each tier), the full manifest, and the rest counted in `snapshot_cap_omitted_count` and `snapshot_cap_omitted_bytes`. The collector still counts the bytes each session has sent, in its local session ledger, for diagnostics only.
+
The desktop collector keeps its footprint bounded while producing these snapshots. Envelopes are streamed as JSON through zstd into a temp file under the app state directory (`omnirush-collector-tmp`) and uploaded from that file, so no envelope string or buffer is ever held whole; a failed upload moves the file into the retry spool (same filesystem, a rename) and a delivered one is deleted at once. Files are read with bounded concurrency and the scan yields to the event loop between files. A per-root digest cache keyed by path, size and mtime, shared by every chat open on the same workspace, means unchanged files are neither re-read nor re-hashed, and a file the scrubber left byte-identical is uploaded as read (verified by digest) without a second pass through the redaction rules. The scrubber runs the assignment and e-mail rules only over lines holding their keyword or an "@" and skips a rule whose literal the text lacks; neither shortcut changes a single output byte. Change snapshots come from watcher-driven dirty tracking: a prompt or completed turn with nothing dirty, journaled or committed since the last accepted snapshot records only its `collector.trigger` event; a change is scanned by dirty path, and before anything is read git is asked afresh, in one batched `git check-ignore`, about every reported path and every journal entry (or the workspace's own `.gitignore` files are applied where git cannot answer), so a file the user has just ignored through `.gitignore`, `.git/info/exclude` or `core.excludesFile` leaves the manifest and the journal at once; a path git refuses to answer for (one beyond a symlinked directory) is isolated by splitting the batch, so it cannot change the answer for the others, and counts as ignored. No path reached through a symlinked directory is ever read or listed. The whole tree is rescanned only when the dirty set overflowed, a directory appeared, a `.gitignore` changed, or the workspace is polled. The watchers go up from the start snapshot's listing before any file is read and cover the root, the top-level directories that hold listed files and then every other top-level directory git does not ignore and the denylist does not name (an empty `scripts/`, a `config/` holding only a denied `.env`), so `.git/`, `node_modules/`, gitignored build output and denied directories never reach them, and a gitignored directory that appears later (a rebuilt `dist/`) is neither watched nor rescanned; at the end of a turn every untracked file the turn created or changed is marked dirty too, so the turn's own snapshot carries it even where no watcher reaches; a workspace with more than 20,000 eligible files, or more than 64 top-level directories holding listed files, is polled on snapshots instead of watched (past 64 watched directories in all, the rest are left to the reconcile pass and the end-of-turn scan), and one reconcile pass (listing and stat only) per workspace every 60 seconds covers every chat watching it. A file larger than the change journal's budget is not journaled on every write; the next snapshot carries it. Filesystem- and reconcile-driven change snapshots are spaced at least 30 seconds apart, with the triggers in between merged into one; prompt and turn milestones capture at once.