Skip to content

A desktop coding agent with free access to frontier models.

TypeScript
18.8 KB
Raw Blame

Session data upload: privacy rails

The project archive, which uploads an encrypted, unscrubbed copy of git project folders (and, in any other folder, the files the agent opens or changes there, as they are, including binaries, while omnirush.ai has that setting on for the account; or the whole folder for accounts where omnirush.ai turns on archiving every folder; on by default, up to 100 GiB per upload and per chat; opt out by emailing [email protected]), is described in project-archive.md. OmniRush also saves the files the agent reads, runs or creates during a session (including temporary files and a few allowlisted config files, with secrets removed), so the session can be replayed; see "Files used" in project-archive.md for what is saved and what never is. The session uploader tells the archive which paths inside the chat folder a session touched (tool paths in the trace and file changes it sees); paths outside the folder are never passed on.

The desktop server's session uploader (apps/server/src/session-uploader.ts) snapshots consented workspace files, git metadata and session traces for omnirush.ai; the backend applies the same denylist and scrubber server-side, so a rule changed on one side must land on the other, and the desktop side may only ever be the stricter of the two. Before anything leaves the machine, credential-named paths are dropped and counted in the envelope's privacy.denied_file_count: any path component whose words (split on space, _, -, .) include key(s), secret(s), token(s), password(s), passwd, credential(s), id_rsa, id_ed25519, netrc, npmrc, pypirc, kubeconfig, wallet, seed or mnemonic, plus .aws, .ssh, .gnupg and .docker/config.json, on top of the older .env*, credentials*, secrets*, keys/, *.pem/*.key rules. A file with a source-code or docs extension (.ts, .py, .md, .json, .yml, ...) is kept instead and scrubbed: a secret-named assignment (password:, api_key =, "token":, export CLIENT_SECRET=, in INI/YAML/JSON/dotenv/shell/source spelling; the key is split into segments on _, -, ., :, camelCase and letter/digit boundaries, and qualifies only when a whole segment or adjacent pair is a listed family word, so author, keywords and tokenizer_class never do) loses its value under value rule v2: in SOURCE mode (files with a programming-language extension: js, cjs, mjs, ts, tsx, jsx, py, go, rs, java, kt, c, cc, cpp, h, hpp, rb, php, swift, cs, scala, sh, bash, zsh, ps1, lua, dart, vue, svelte, map, patch, diff) a value is redacted only when it is 8+ non-space characters, not a code expression, <placeholder> or filesystem path, not an unquoted identifier or member expression (config.token, tokenFile), and carries a digit or is 20+ mixed-case characters, while CONFIG mode (every other extension, none, the envelope's git diff and the trace) keeps the permissive rule of 8+ characters with a letter, skipping only filesystem paths and unquoted dotted identifiers, and a key whose last segment is a structural word (length, ttl, url, path, name, id, header, file, filename, dir, mode, method, role, owner, type, kind, enabled, estimate, hash, digest, at, config, client, prefix, suffix, format, scheme, provider, status, state, label, description, title, class, field, fields, list, names, version, timeout, limit, max, min, interval, retries, port) never names a secret. A Bearer value is redacted only when its token carries a digit or is 20+ mixed-case characters that are not a bare identifier, the AWS 40-character proximity pass runs only on documents of three or more lines (a single-line JSON document relies on its key, or on a sibling naming AWS), and provider token shapes (AWS access key ids and, near them, secret access keys; GitHub, Slack, Google, Stripe, OpenAI, Anthropic, JWTs, Discord, Bearer values, private key blocks, URL userinfo) become [REDACTED], and personal identifiers become [REDACTED_PII]. .json files and the trace document are scrubbed as JSON (every string, keys included) so a redaction can never break an escape sequence; git diffs stay text. As on the backend, a .json file first gets the AWS proximity pass over its raw text, an object or array whose keys or string values name aws/secret makes all its values AWS context (the container rail, which also covers the trace), every pair of a repeated key is kept and scrubbed (JSON.parse alone would drop the earlier value unscrubbed), and a document nested deeper than 512 levels gets the text scrub instead. The same denylist filters the manifest, touched paths, git status and diff sections.

The backend's second-sample refinements (46ee95e) are mirrored exactly: in SOURCE mode an unquoted value must also be token-shaped (only letters, digits and _ . / + = ~ -, so stagedClient?.plaintext, params[8] and [0-9a-f]{16} stay) and the key must start the text or follow whitespace, {, ,, (, a quote or backtick, + or # (so ?token= or pipeline-token:... inside a string literal stays), while CONFIG keeps its word-start guard. 53 more structural last segments never name a secret (in, out, expires, ms, bytes, encryption, algorithm, policy, source, handler, event, message, error, envelope, ref, alias, ...; auth_code and password_plaintext still do), a JSON key that looks like a path or filename (holds / or \, or ends in a 1-5 character extension) is not a secret key in the JSON rail though its value still gets the provider-token scrub, and URL userinfo is redacted only when it holds a password (scheme://user:pass@host becomes scheme://[REDACTED]@host; a bare user@host stays). The assignment rule reads whitespace exactly as the backend's Python \s does, and in SOURCE mode a desktop key may also start right after any non-ASCII character (a byte order mark, or the é of café_token, which the backend's Unicode \w spans instead), so the desktop never redacts less than the backend there. URL userinfo with an empty user (redis://:<password>@host, Redis's AUTH form) is redacted too; the backend at 46ee95e misses it, so there the desktop is the stricter side.

Snapshot scope, incremental captures and the files_scope field

Every envelope (contract v2) carries files_scope, which tells a consumer what files[] covers:

  • "full": files[] holds the content of every eligible file the snapshot cap allowed (the start snapshot of a session, whether new or resumed) or, for a trace envelope, its single trace document.
  • "changed": files[] holds only the files whose redacted sha256 differs from the previous snapshot the gateway accepted (delivered or spooled) for this session; a change or end snapshot. The envelope also carries changed_paths, the redacted paths of every such file, including any the snapshot cap kept out of files[], and the manifest still lists every eligible file with its sha256 and size, so a consumer can tell a deleted file (gone from the manifest) from an unchanged one (same digest as before) without content. A snapshot that failed to upload advances nothing: the same files are sent again with the next one.

Neither field changes what may leave the machine: files[], changed_paths and the manifest go through the same denylist and scrubber, and a manifest digest is always the sha256 of the redacted UTF-8 content.

There is no per-session storage limit: a session keeps uploading snapshots and traces for as long as it runs, however much it has already sent, and the privacy block carries no max_session_bytes. Every remaining limit applies to one file or one request and matches what the backend accepts per request: 4 MiB per file, 64 MiB per snapshot envelope (uncompressed, and again once compressed), 16 MiB per trace document, 2 MiB of git diff and 50,000 manifest entries. These bound the session uploader's memory and never stop a session: a snapshot that would not fit still goes out with the files it has room for (touched and git-changed files first, smallest first within each tier), the full manifest, and the rest counted in snapshot_cap_omitted_count and snapshot_cap_omitted_bytes. The session uploader still counts the bytes each session has sent, in its local session ledger, for diagnostics only.

The desktop session uploader keeps its footprint bounded while producing these snapshots. It runs on a worker thread of the app's local server, together with the turn observer and the project archive (apps/server/src/capture-host.ts), so reading, hashing and scrubbing files, building and compressing envelopes, and parsing long transcripts never hold the event loop that serves the app (the Electron main process in the desktop app); the main thread only forwards calls and makes the uploads the worker hands it. Envelopes are streamed as JSON through zstd into a temp file under the app state directory (omnirush-upload-tmp) and uploaded from that file, so no envelope string or buffer is ever held whole; a failed upload moves the file into the retry spool (same filesystem, a rename) and a delivered one is deleted at once. An upload's deadline grows with its envelope: 30 seconds, plus the compressed body at 128 KiB/s (at most 15 minutes for that part), plus 2 minutes for the gateway's answer (apps/server/src/session-upload-budget.ts); fetch reports no upload progress, so the three parts form one deadline for the whole request. An upload that runs out its deadline goes to the spool at once instead of being tried again on the spot, so the chat's later snapshots are not held up behind it; a quick failure (a dropped connection, a 5xx answer) is still tried twice more first. Spooled uploads are retried oldest first, each on a backoff of its own (5 seconds after its first retry, doubling up to 5 minutes, dropped after 24 attempts), and one that keeps failing holds back none of the uploads behind it: a retry pass moves past it and ends after three failures, and the gateway orders a session's snapshots by their sequence rather than by arrival. A sign-out cancels every upload in flight, live or from the spool, and an envelope it catches is deleted rather than spooled, so nothing captured for one account waits on disk for the next. Files are read with bounded concurrency and the scan yields to the event loop between files. A per-root digest cache keyed by path, size and mtime, shared by every chat open on the same workspace, means unchanged files are neither re-read nor re-hashed, and a file the scrubber left byte-identical is uploaded as read (verified by digest) without a second pass through the redaction rules; the redacted text of files the scrubber did change is kept too (32 MiB at most, least recently used out first) and sent again only for the exact bytes it came from (verified by the digest of the bytes read), so a new chat on a workspace another chat already captured runs no file through the scrubber a second time. The scrubber runs the assignment and e-mail rules only over lines holding their keyword or an "@" and skips a rule whose literal the text lacks; neither shortcut changes a single output byte. Change snapshots come from watcher-driven dirty tracking: a prompt or completed turn with nothing dirty, journaled or committed since the last accepted snapshot records only its collector.trigger event (the trace event names the backend receives kept their old collector. prefix when the desktop code was renamed); a change is scanned by dirty path, and before anything is read git is asked afresh, in one batched git check-ignore, about every reported path and every journal entry (or the workspace's own .gitignore files are applied where git cannot answer), so a file the user has just ignored through .gitignore, .git/info/exclude or core.excludesFile leaves the manifest and the journal at once; a path git refuses to answer for (one beyond a symlinked directory) is isolated by splitting the batch, so it cannot change the answer for the others, and counts as ignored. No path outside the workspace root (a parent folder, another Windows drive or a network share), and none reached through a symlinked directory, is ever read or listed. The whole tree is rescanned only when the dirty set overflowed, a directory appeared, a .gitignore changed, or the workspace is polled. The watchers go up from the start snapshot's listing before any file is read and cover the root, the top-level directories that hold listed files and then every other top-level directory git does not ignore and the denylist does not name (an empty scripts/, a config/ holding only a denied .env), so .git/, node_modules/, gitignored build output and denied directories never reach them, and a gitignored directory that appears later (a rebuilt dist/) is neither watched nor rescanned; at the end of a turn every untracked file the turn created or changed is marked dirty too, so the turn's own snapshot carries it even where no watcher reaches; a workspace with more than 20,000 eligible files, or more than 64 top-level directories holding listed files, is polled on snapshots instead of watched (past 64 watched directories in all, the rest are left to the reconcile pass and the end-of-turn scan), and one reconcile pass (listing and stat only) per workspace every 60 seconds covers every chat watching it. A file larger than the change journal's budget is not journaled on every write; the next snapshot carries it. Filesystem- and reconcile-driven change snapshots are spaced at least 30 seconds apart, with the triggers in between merged into one; prompt and turn milestones capture at once. When several chats are open on the same folder, the edits made while one of them runs a turn are that turn's: only a chat with a turn in progress uploads filesystem- and reconcile-driven change snapshots for them, and a chat without one leaves them to its own next prompt or turn snapshot (or its end snapshot), so one agent's edits are not uploaded again by every chat open on the folder. Each chat's own prompt and turn milestones capture as before, and a chat alone on its folder is unaffected. This concerns the session uploader's scrubbed snapshots only: the project archive keeps its own copies of the folder for each chat (project-archive.md).

Per-turn diffs: the turn.diff trace event

At the end of each turn the trace gets one turn.diff event (apps/server/src/turn-diff.ts), pairing the turn with what it changed in the workspace: every file whose redacted sha256 moved between the workspace as the turn's prompt found it and the last accepted manifest. The turn begins when its prompt is sent, not when the queued prompt snapshot runs: a file written after that moment (an edit landing while an earlier upload still holds the queue) is measured from what the session uploader last saw of it before the prompt, and one it only ever saw already written by the turn is reported as no_base without a "before" digest. Likewise, when the next prompt goes out before a turn's own snapshot has run, what the next turn wrote is left to it. The turn observer follows a turn until it settles, however long it runs, reading the engine's status every second at first and less often as the turn goes on (every 10 seconds once it has run ten minutes); it waits out an engine that times out, drops the connection or restarts, noting session.engine_unavailable once the engine has not answered for a minute and session.engine_recovered when it answers again, and stops following only after 24 hours (session.observer_timeout) or on an unexpected error. A turn it stops following still gets its snapshot and turn.diff, and the project archive is told the turn ended without completing (see project-archive.md); its messages, like those of a turn cut short by the app quitting, stay after the session's checkpoint and go out with the session's next settled turn, after an app restart too. One trace carries at most 8 MiB of messages: the older messages of a longer turn, or of such an unsent backlog, go ahead in turn.messages events (part of parts), each flushed as its own trace, up to 32 MiB in all, and any older ones are counted in session.messages_omitted. Each entry carries the workspace-relative redacted path, a status (added, modified, deleted, skipped for a binary or over-4-MiB file the turn wrote, sent without content, or no_base, for a changed or deleted file, when there is no text from before the turn to diff against: no longer held, or never seen), both redacted digests, the +/- line counts and a unified diff of the scrubbed text sent before the turn against the scrubbed text sent after it. A binary or oversized file the turn only read (a traced read, an attachment) is not an entry: only one written since the prompt counts. The diff work is bounded: one file's line matching stops after 5 million comparisons and writes the rest of its change as removed and added lines (still a patch that turns the old text into the new one, its counts then an upper bound), and a turn's diffs get 250 ms of work in all, after which the files still waiting for a diff are only counted in omitted_file_count. A file diff is cut at 256 KiB and the whole event, envelope included, at 2 MiB (JSON-encoded), both marked truncated; files past the event cap are only counted in omitted_file_count, while entries without diff text (skipped, no_base) still go in as long as they fit. The diff is built from text the session uploader already sent, so the denylist and scrubber apply unchanged, and the trace is scrubbed again as a whole like every other event. To have the "before" side, the session uploader keeps the scrubbed texts it sent (start snapshots seed it) in a content-addressed store under the app state directory (omnirush-upload-bases, owner-only files, 64 MiB at most, least recently used out first), so bases survive a restart; a sign-out deletes it with the spool. Version 1.0.10 and earlier neither read nor delete that directory: after a downgrade it stays on disk, up to 64 MiB of scrubbed workspace text, until it is removed by hand or a later version signs out. Before version 2.2.2 the session uploader's files under the app state directory were named omnirush-collector-tmp, omnirush-collector-sessions.json, omnirush-collector-spool and omnirush-collector-bases; 2.2.2 moves them to omnirush-upload-tmp, omnirush-upload-sessions.json, omnirush-upload-spool and omnirush-upload-bases on its first start (apps/server/src/session-upload-state.ts), before it reads any of them, so the session ledger, pending uploads and bases carry over and no session is uploaded again. A sign-out also deletes the spool and bases under the old names. Version 2.2.1 and earlier do not read the new names: after a downgrade the moved spool and bases stay on disk until a later version starts and signs out, and a chat resumed on the older version finds no session ledger, so it is uploaded as a new session (a full start snapshot and its whole transcript).