Skip to content

A desktop coding agent with free access to frontier models.

TypeScript

feat(capture): archive reproducible tasks like CLI 2.0.0

ogsamrat399f6d18a17e
diff --git a/apps/server/src/session-archive/README.md b/apps/server/src/session-archive/README.md
index ef91e5f2..f197748c 100644
--- a/apps/server/src/session-archive/README.md
+++ b/apps/server/src/session-archive/README.md
@@ -320,12 +320,13 @@ against a loopback sink).
- **The policy probe.** Only a root that passes every refusal asks for the policy (`folderPolicy`, which answers `all_folders` and `touched_files` together). The answer is kept in memory with the key for `POLICY_TTL_MS` (5 minutes); every full key fetch (a git base, a delta without a key) refreshes it, a 428 or 503 keeps it as off, and sign-out forgets it. Without a kept answer the archiver sends one probe (`ArchiveUploader.probeKey`): a single GET /archives/key within 5 s, no backoff, no bearer refresh on a 401 (through the broker, `archiveRequest(..., { refresh: false })`), shared by the sessions that start meanwhile. Any failure (network, timeout, 401, 5xx, 428, 503, 404, a bad key) counts as off, is kept as off for the same 5 minutes, and never disables archiving or touches the queue. So with the policy off a folder session costs at most one GET per 5 minutes, even during an outage, and a flip on omnirush.ai reaches a running app within 5 minutes with no release.
- **Key and consent.** A folder base reuses the key from the probe it just made. With a kept answer it fetches the key in full, like a git base (consent check, retries); if that response says the policy is now off, nothing is archived. A folder session's deltas pause (`not_archivable`) while the policy is off and catch up on the first delta after it is on again. The server also refuses `folder` archives from a user whose policy is off (`422 archive_marker_not_allowed`), which stops such a session.
- **Touched files.** GET /archives/key also carries `policy.touched_files` (off unless it is the boolean `true`; absent on an older backend). Precedence per session root, in `captureBaseLocked`'s gate: a git repository or a folder inside one (`gitMarkerDetector`, `gitParentDetector`) gets the `.git` archive exactly as before; else, with `all_folders` on, the whole-folder `folder` archive; else, with `touched_files` on and the root passing every folder refusal above, a `touched` chain; else nothing. The same probe, TTL and kept answer serve both flags.
- - **The touched set.** The session uploader (in the same capture host, on the worker) reports every workspace-relative path a session touches, before its own denylist: tool paths in the trace (`path`, `filePath`, ... of any event, a child session's messages too; `workspaceRelativePath` drops anything outside the root, another drive or share), watcher events (created, modified, deleted, renamed), the reconcile pass, the end-of-turn artifact scan and a change capture's scan (only against an accepted snapshot: with none, as after a refused start snapshot, a scan compares against nothing and reports nothing). `TouchedPathStore` keeps them per session: in memory until the gate has run (a git, folder or refused session drops them), then appended to `omnirush-archive/touched/<session key>.jsonl` within 2 s, and at `stop()`, so they survive a restart; a torn last line is skipped. At most `MAX_TOUCHED_PATHS` (100,000) per session, one `warn` past it. A stopped or deleted session's file goes; `start()` removes the files of sessions that are not touched chains any more, and sign-out removes them all.
+ - **The touched set.** The session uploader (in the same capture host, on the worker) reports every workspace-relative path a session touches, before its own denylist: tool paths in the trace (`path`, `filePath`, ... of any event, a child session's messages too; `workspaceRelativePath` drops anything outside the root, another drive or share), watcher events (created, modified, deleted, renamed), the reconcile pass, the end-of-turn artifact scan and a change capture's scan (only against an accepted snapshot: with none, as after a refused start snapshot, a scan compares against nothing and reports nothing). `TouchedPathStore` keeps them per session: in memory until the gate has run (a refused session drops them), then appended to `omnirush-archive/touched/<session key>.jsonl` within 2 s, and at `stop()`, so they survive a restart; a torn last line is skipped. At most `MAX_TOUCHED_PATHS` (100,000) per session, one `warn` past it. A stopped or deleted session's file goes; `start()` removes the files of sessions that are stopped, ended or unknown (every archived chain keeps its touched paths across a restart), and sign-out removes them all.
- **Captures.** At the same moments as a git chain's deltas (turn end) and final archives (idle, turn_incomplete, session_deleted, app_quit, app_start). `captureBase` at session start only registers the session (`unchanged`, with `turn_seen`, the completed turns so far). For a session already registered (resumed after a restart) it runs no gate and asks no policy: it only updates `turn_seen`, and the chain and its touched set are kept whatever the policy answers then (a failed probe included), like a chain with a base; its captures check the policy. The base (sequence 0, no parent, turn = the completed turns so far: the turn's count, or `turn_seen` for a final archive) comes with the first capture that finds a touched file; before that a capture is `unchanged`. Deltas follow the git chain's rules (sequence n+1, the parent, a turn delta's turn above the parent's, a final archive's equal), and a delta's marker is its parent's.
- **What is archived.** `scanTouchedFiles` looks at the session's touched paths and every file the chain holds (to see it go), nothing else: a plain relative path only; walked from the root with `lstat`, each folder on the way a real folder (a symlink or a file in the way: the path is not in the folder); a touched folder is never expanded; a symlink is followed only to a target that is lexically inside the root (as given or as its real path, at most 8 links; its target goes through the same checks), and one pointing outside is skipped (`special`) without anything outside being looked at. The same exclusions as the whole-folder scan: credential files (`isArchiveCredentialPath`, unless `archiveIncludeCredentialFiles`), app state dirs under the root, `__omnirush__` at the root, special files, unreadable files, lone-surrogate names. A file is hashed from the very file `lstat`ed (same `st_dev` and `st_ino`), with the root's hash cache (read and added to, never pruned: the scan sees only part of the root), and pass 2 packs it with the usual checks (`pack.ts`). Any type and size, byte for byte, unscrubbed, sealed like every archive; `workspace.git` is `null`.
- **Manifest.** `"scope": "touched"` in the sealed `manifest.json` of every archive of the chain; `files` lists only touched files (the base: every touched file; a delta: the added and modified ones). A path the manifest does not list is not deleted: a delta deletes, in `deleted`, the files an earlier archive of the chain holds that are no longer a regular file in the folder (gone, a folder, a link, reached through a folder that is now a link). A file that cannot be read for now stays in the baseline as it was.
- **Policy off.** With `touched_files` off or absent nothing is written for such a folder: `captureBase` is `not_archivable` after the probe, and no touched path is kept. A touched chain's captures pause (`not_archivable`) while it is off and catch up once it is on (the touched set is kept). The server refuses a `touched` create from a user whose policy is off with `422 archive_marker_not_allowed`: the drain drops the session's chain (its jobs, the session stopped, one `warn` line, no retry) and keeps the policy as off for `touched` until it is asked again (`POLICY_TTL_MS`), so no other chain starts meanwhile. The same holds for `folder`.
-- **Gitignored content.** Nothing git ignores is archived: base, delta and final archives, whole-folder and touched-files alike (`ignore.ts`). It can be rebuilt (`node_modules/`, `dist/`, venvs, caches), and it made single projects many gigabytes. Git decides, with all its rules (every `.gitignore`, `.git/info/exclude`, `core.excludesFile`):
+- **Reproducible tasks.** Two exceptions to the gitignored rule below, the same as the CLI (2.0.0): every file the agent touched is archived byte for byte, binaries included, even when gitignored, and so is every lockfile (`findLockfiles` in `touched.ts`: `package-lock.json`, `pnpm-lock.yaml`, `Cargo.lock`, `uv.lock`, `go.sum`, ...; a bounded walk, depth 6, 5,000 folders, 200 files, never into `.git/`, never through a link). Neither ever comes from a regenerable folder (`REGENERABLE_DIR_NAMES` / `inRegenerableDir` in `session-uploader.ts`: `node_modules`, `.venv`, `venv`, `__pycache__`, `dist`, `build`, `target`, `vendor`, ... at any depth) or from `.git/`. So every archived chain keeps its touched paths (a whole-folder chain too, not only a touched-files one); a whole-folder scan adds the touched paths and lockfiles it left out with a second `scanTouchedFiles` pass (`includeIgnored: true`); a touched-files chain scans its touched paths and every lockfile with `includeIgnored: true`. The session uploader's end-of-turn artifact events also cover gitignored untracked outputs outside the regenerable folders (`listIgnoredOutputFiles`), and those paths reach the touched set like any other artifact.
+- **Gitignored content.** Apart from the reproducible-task exceptions above, nothing git ignores is archived: base, delta and final archives, whole-folder and touched-files alike (`ignore.ts`). It can be rebuilt (`node_modules/`, `dist/`, venvs, caches), and it made single projects many gigabytes. Git decides, with all its rules (every `.gitignore`, `.git/info/exclude`, `core.excludesFile`):
- **A root in a repository** (a `.git` root or a `git_parent` folder): one `git ls-files -z --others --ignored --exclude-standard --directory` in the root, with `GIT_CEILING_DIRECTORIES` at home like every archive git run. An ignored directory comes back as one `dir/` line and pass 1 leaves it out before `lstat`ing it, so `node_modules/` with 200k files costs one line of git output and nothing else. Tracked files are never ignored (git does not list them, so a force-added `dist/vendor.js` stays). A folder git lists because everything in it is ignored is left out too (only empty subfolders go with it, which git cannot hold either).
- **A folder in no repository** (all-folders policy, marker `folder`, or git refusing the root's repository): the same command with a private empty bare repository as `GIT_DIR` (`git init --bare --template=` in a fresh `mkdtemp` dir, removed when the scan is done) and the root as `GIT_WORK_TREE`, so the folder's `.gitignore` files apply as git would apply them (`node_modules/` at any depth, negations, nested `.gitignore` files). Nothing is written in the folder.
- **A nested repository** (a folder below the root holding `.git`: a submodule, a cloned dependency): the outer repository's git never looks inside one, so when pass 1 enters it, it asks that repository (ceiling at its parent, so a broken `.git` there never climbs to the outer one) and uses its answer below it.
diff --git a/apps/server/src/session-archive/index.test.ts b/apps/server/src/session-archive/index.test.ts
index cb2cedc6..32198311 100644
--- a/apps/server/src/session-archive/index.test.ts
+++ b/apps/server/src/session-archive/index.test.ts
@@ -1026,10 +1026,10 @@ describe("SessionArchiver touched files", () => {
// The agent reads the PDF, writes a binary, and names a credential file and a link out of the folder.
subject.recordTouched(id, "brief.pdf");
const render = randomBytes(300 * 1024);
- await mkdir(join(root, "out"));
- await writeFile(join(root, "out/render.png"), render);
- subject.recordTouched(id, "out/render.png");
- subject.recordTouched(id, "out");
+ await mkdir(join(root, "renders"));
+ await writeFile(join(root, "renders/render.png"), render);
+ subject.recordTouched(id, "renders/render.png");
+ subject.recordTouched(id, "renders");
subject.recordTouched(id, ".env");
subject.recordTouched(id, "link-out");
@@ -1038,20 +1038,20 @@ describe("SessionArchiver touched files", () => {
const [base] = server.objects();
expect(base!.request).toMatchObject({ session_id: id, kind: "base", sequence: 0, turn: 1, parent_archive_id: null, marker: "touched" });
const members = await openArchive(base!.object!);
- expect(members.map((member) => member.name)).toEqual(["__omnirush__/manifest.json", "brief.pdf", "out/render.png"]);
+ expect(members.map((member) => member.name)).toEqual(["__omnirush__/manifest.json", "brief.pdf", "renders/render.png"]);
expect(members[1]!.content.equals(brief)).toBe(true);
expect(members[2]!.content.equals(render)).toBe(true);
const manifest = manifestOf(members);
expect(manifest).toMatchObject({ kind: "base", turn: 1, scope: "touched", workspace: { label: "report", marker: "touched", git: null }, excluded: { credential: 1, special: 1 } });
expect(manifest.files).toEqual([
{ path: "brief.pdf", type: "file", mode: expect.any(Number), size: brief.length, sha256: sha256(brief) },
- { path: "out/render.png", type: "file", mode: expect.any(Number), size: render.length, sha256: sha256(render) },
+ { path: "renders/render.png", type: "file", mode: expect.any(Number), size: render.length, sha256: sha256(render) },
]);
expect("deleted" in manifest).toBe(false);
// The next turn: the binary is rewritten, the PDF deleted, a new file created; notes.txt is still untouched.
const rerender = randomBytes(1024);
- await writeFile(join(root, "out/render.png"), rerender);
+ await writeFile(join(root, "renders/render.png"), rerender);
await rm(join(root, "brief.pdf"));
await writeFile(join(root, "summary.md"), "# summary\n");
subject.recordTouched(id, "summary.md");
@@ -1061,7 +1061,7 @@ describe("SessionArchiver touched files", () => {
const delta = server.objects()[1]!;
expect(delta.request).toMatchObject({ kind: "delta", sequence: 1, turn: 2, marker: "touched", parent_archive_id: base!.request.archive_id });
const deltaMembers = await openArchive(delta.object!);
- expect(deltaMembers.slice(1).map((member) => member.name)).toEqual(["out/render.png", "summary.md"]);
+ expect(deltaMembers.slice(1).map((member) => member.name)).toEqual(["renders/render.png", "summary.md"]);
expect(deltaMembers[1]!.content.equals(rerender)).toBe(true);
expect(manifestOf(deltaMembers)).toMatchObject({ kind: "delta", trigger: "turn", scope: "touched", deleted: ["brief.pdf"] });
@@ -1074,7 +1074,7 @@ describe("SessionArchiver touched files", () => {
}
});
- test("a git folder is archived as today with touched_files on, and keeps no touched paths", async () => {
+ test("a git folder is archived as today with touched_files on, and keeps its touched paths (reproducible tasks)", async () => {
const { server, state, make } = await touchedSetup();
const { root } = await project();
const subject = make();
@@ -1085,7 +1085,7 @@ describe("SessionArchiver touched files", () => {
expect(await names(server.objects()[0]!)).toEqual(expect.arrayContaining([".git/HEAD", "assets/scene.blend", "src/app.ts"]));
expect(manifestOf(await openArchive(server.objects()[0]!.object!))).not.toHaveProperty("scope");
await subject.stop();
- expect(await readdir(join(state, ARCHIVE_STATE_DIRECTORY, "touched"))).toEqual([]);
+ expect(await readdir(join(state, ARCHIVE_STATE_DIRECTORY, "touched"))).toHaveLength(1);
});
test("all_folders wins over touched_files: the whole folder", async () => {
diff --git a/apps/server/src/session-archive/index.ts b/apps/server/src/session-archive/index.ts
index 863dc0de..600850a3 100644
--- a/apps/server/src/session-archive/index.ts
+++ b/apps/server/src/session-archive/index.ts
@@ -33,6 +33,7 @@ import {
ArchiveHashCache,
archiveLabel,
baselineChunks,
+ compareArchivePaths,
computeArchiveDelta,
isArchiveDeltaEmpty,
manifestSource,
@@ -50,7 +51,8 @@ import {
import { writeSealedArchive } from "./pack.js";
import { POLICY_OFF, type ArchivePolicy } from "./policy.js";
import { SEAL_CONTENT } from "./seal.js";
-import { scanTouchedFiles, touchedChange, TouchedPathStore } from "./touched.js";
+import { findLockfiles, scanTouchedFiles, touchedChange, TouchedPathStore } from "./touched.js";
+import { inRegenerableDir } from "../session-uploader.js";
import {
ARCHIVE_MARKER_NOT_ALLOWED,
ArchiveUploader,
@@ -379,7 +381,9 @@ export class SessionArchiver {
modeOf: async (sessionId) => {
const state = await this.loadSession(sessionId);
if (!state) return "unknown";
- return state.marker === TOUCHED_MARKER && !state.stopped && !state.ended ? "tracked" : "ignored";
+ // Every archived chain keeps its touched paths, a whole-folder one for
+ // the ignored files the agent touched (archived with the lockfiles).
+ return !state.stopped && !state.ended ? "tracked" : "ignored";
},
log: this.log,
...(options.touchedFlushMs !== undefined ? { flushMs: options.touchedFlushMs } : {}),
@@ -730,9 +734,11 @@ export class SessionArchiver {
const probe: { fetched: KeyResult | null } = { fetched: null };
const gate = await isArchivableProject(root, [...this.detectors, this.folderDetector(generation, probe)], { appDirs: this.appDirs });
const touched = gate.marker === TOUCHED_MARKER;
- // Whatever else the gate said, the session keeps no touched paths.
- if (!touched) await this.touched.forget(sessionId);
- if (!gate.archivable || !gate.marker) return { status: "skipped", reason: "not_archivable" };
+ // A session that is not archived keeps no touched paths; any other keeps them.
+ if (!gate.archivable || !gate.marker) {
+ await this.touched.forget(sessionId);
+ return { status: "skipped", reason: "not_archivable" };
+ }
// The key and the consent check (7.2): a probe made for this base already holds them; else the full fetch.
const fetched = probe.fetched ?? await this.uploader.fetchKey();
if (!probe.fetched) this.rememberPolicy(fetched, generation, false);
@@ -742,7 +748,7 @@ export class SessionArchiver {
}
// The policy was turned off since the answer this folder was let in on.
if (fetched.status === "ok" && !markerAllowed(gate.marker, fetched.policy)) {
- if (touched) await this.touched.forget(sessionId);
+ await this.touched.forget(sessionId);
return { status: "skipped", reason: "not_archivable" };
}
const state: SessionState = {
@@ -759,7 +765,7 @@ export class SessionArchiver {
...(touched ? { turn_seen: turn } : {}),
last_activity_at: this.now().toISOString(),
};
- if (touched) this.touched.track(sessionId);
+ this.touched.track(sessionId);
if (fetched.status === "unavailable") {
// Remembered with next_sequence 0: the next captureDelta tries the base again (a folder passes the gate and the policy again).
if (generation === this.generation) await this.saveSession(state);
@@ -954,10 +960,12 @@ export class SessionArchiver {
await this.markStopped(sessionId, "baseline_missing");
return { status: "skipped", reason: "stopped" };
}
- // Every path the session touched, and every file the chain holds (to see it go).
- const paths = await this.touched.snapshot(sessionId);
+ // Every path the session touched and every lockfile, outside the
+ // regenerable folders, and every file the chain holds (to see it go).
+ // Gitignored files the agent touched are archived too.
+ const paths = new Set([...(await this.touched.snapshot(sessionId)), ...(await findLockfiles(state.root, signal))].filter((path) => !inRegenerableDir(path)));
for (const entry of baseline) paths.add(entry.path);
- const scan = await scanTouchedFiles(state.root, paths, { excludedDirs: this.appDirs, includeCredentialFiles: this.includeCredentials, hashCache: cache, ...(signal ? { signal } : {}) });
+ const scan = await scanTouchedFiles(state.root, paths, { excludedDirs: this.appDirs, includeCredentialFiles: this.includeCredentials, includeIgnored: true, hashCache: cache, ...(signal ? { signal } : {}) });
const change = touchedChange(baseline, scan);
if (change.files.length === 0 && change.deleted.length === 0) {
if (cache.changed) await this.saveHashCache(rootKey, cache);
@@ -975,6 +983,15 @@ export class SessionArchiver {
const withGit = state.marker !== FOLDER_MARKER;
const gitReading = kind === "base" && withGit ? readArchiveGit(state.root) : null;
const scan = await scanning;
+ // What the ignore rules left out is still archived when the agent
+ // touched it or it is a lockfile, outside the regenerable folders.
+ const held = new Set(scan.entries.map((entry) => entry.path));
+ const keepers = [...(await this.touched.snapshot(sessionId)), ...(await findLockfiles(state.root, signal))]
+ .filter((path) => !held.has(path) && !inRegenerableDir(path));
+ if (keepers.length > 0) {
+ const kept = await scanTouchedFiles(state.root, keepers, { excludedDirs: this.appDirs, includeCredentialFiles: this.includeCredentials, includeIgnored: true, hashCache: cache, ...(signal ? { signal } : {}) });
+ if (kept.entries.length > 0) scan.entries = [...scan.entries, ...kept.entries].sort((left, right) => compareArchivePaths(left.path, right.path));
+ }
files = scan.entries;
excluded = scan.excluded;
ignored = scan.ignored;
@@ -1605,10 +1622,10 @@ export class SessionArchiver {
for (const name of await readdir(this.dirs.hashCache)) {
if (name.endsWith(".tmp")) await rm(join(this.dirs.hashCache, name), { force: true });
}
- // Touched paths only of touched-files sessions still archiving.
+ // Touched paths only of sessions still archiving (every archived chain keeps them).
for (const name of await readdir(this.dirs.touched)) {
const session = name.endsWith(".jsonl") ? sessions.get(name.slice(0, -".jsonl".length)) : undefined;
- if (!session || session.marker !== TOUCHED_MARKER || session.stopped || session.ended) await rm(join(this.dirs.touched, name), { force: true });
+ if (!session || session.stopped || session.ended) await rm(join(this.dirs.touched, name), { force: true });
}
}
}
diff --git a/apps/server/src/session-archive/reproducible.test.ts b/apps/server/src/session-archive/reproducible.test.ts
new file mode 100644
index 00000000..74d05c24
--- /dev/null
+++ b/apps/server/src/session-archive/reproducible.test.ts
@@ -0,0 +1,153 @@
+import { afterEach, describe, expect, test } from "bun:test";
+import { execFile } from "node:child_process";
+import { createHash } from "node:crypto";
+import { mkdir, readdir, writeFile } from "node:fs/promises";
+import { dirname, join } from "node:path";
+import { promisify } from "node:util";
+
+import { inRegenerableDir } from "../session-uploader.js";
+import { FakeArchiveServer } from "./fake-archive-server.js";
+import { ARCHIVE_STATE_DIRECTORY, SessionArchiver, type SessionArchiverOptions } from "./index.js";
+import { cleanupTempDirs, manifestOf, openArchive, tempDir } from "./test-helpers.js";
+import { findLockfiles } from "./touched.js";
+
+// What a session must carry to be rebuilt as a reproducible task (the CLI's
+// test/capture-reproducible.test.js): the files the agent touched, binaries
+// and gitignored ones included, and every lockfile, but never the regenerable
+// folders (node_modules, virtualenvs, build output) that are reinstalled from
+// those lockfiles.
+
+const execFileAsync = promisify(execFile);
+
+afterEach(cleanupTempDirs);
+
+const sha256 = (bytes: Uint8Array | string) => createHash("sha256").update(bytes).digest("hex");
+const PNG = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0x00, 0x00, 0x00, 0x0d, 0x49, 0x48, 0x44, 0x52]);
+
+async function git(root: string, ...args: string[]): Promise<void> {
+ await execFileAsync("git", ["-C", root, "-c", "user.name=t", "-c", "[email protected]", "-c", "commit.gpgsign=false", ...args]);
+}
+
+/** The CLI test's tree: an ignored output folder, an ignored Cargo.lock, a nested lockfile and node_modules. */
+async function tree(root: string): Promise<void> {
+ const write = async (relative: string, content: string | Buffer) => {
+ await mkdir(dirname(join(root, relative)), { recursive: true });
+ await writeFile(join(root, relative), content);
+ };
+ await write(".gitignore", "outputs/\nnode_modules/\nCargo.lock\n");
+ await write("src/main.rs", "fn main() {}\n");
+ await write("outputs/plot.png", PNG);
+ await write("outputs/untouched.txt", "an ignored file the agent never touched\n");
+ await write("Cargo.lock", "# ignored lockfile\n");
+ await write("web/pnpm-lock.yaml", "lockfileVersion: 9\n");
+ await write("node_modules/dep/package-lock.json", "{}\n");
+ await write("node_modules/dep/index.js", "module.exports = 1;\n");
+}
+
+function archiver(server: FakeArchiveServer, stateDir: string, options: Partial<SessionArchiverOptions> = {}) {
+ return new SessionArchiver({
+ gatewayUrl: server.gatewayUrl,
+ accessToken: server.token,
+ fetch: server.respond,
+ stateDir,
+ retry: { baseMs: 1, maxMs: 2, attempts: 2 },
+ touchedFlushMs: 5,
+ ...options,
+ });
+}
+
+describe("reproducible-task capture", () => {
+ test("regenerable folders are recognised by any path component, .git content too", () => {
+ for (const regenerable of ["node_modules/a/b.js", "a/.venv/lib/x.py", "pkg/target/debug/app", "x/.git/HEAD", "dist/out.js"]) {
+ expect(inRegenerableDir(regenerable)).toBe(true);
+ }
+ for (const kept of ["outputs/plot.png", "docs/build.md", "node_modules", "src/target.rs"]) {
+ expect(inRegenerableDir(kept)).toBe(false);
+ }
+ });
+
+ test("lockfiles are found at any depth outside .git and the regenerable folders", async () => {
+ const root = await tempDir("repro");
+ await tree(root);
+ await mkdir(join(root, ".git"));
+ await writeFile(join(root, ".git", "yarn.lock"), "not a project lockfile\n");
+ expect(await findLockfiles(root)).toEqual(["Cargo.lock", "web/pnpm-lock.yaml"]);
+ });
+
+ test("a whole-folder archive keeps the ignored files the agent touched and every lockfile, never the regenerable folders", async () => {
+ const root = await tempDir("repro");
+ await tree(root);
+ await git(root, "init", "-q", "-b", "main");
+ await git(root, "add", ".gitignore", "src/main.rs", "web/pnpm-lock.yaml");
+ await git(root, "commit", "-q", "-m", "init");
+ const server = new FakeArchiveServer();
+ const state = await tempDir("state");
+ const subject = archiver(server, state);
+ const id = "ses_repro_whole_folder";
+
+ expect(await subject.captureBase(id, root)).toMatchObject({ status: "queued", kind: "base" });
+ expect((await subject.drain()).uploaded).toBe(1);
+ const base = await openArchive(server.objects()[0]!.object!);
+ const baseNames = base.map((member) => member.name);
+ // The gitignored lockfile is archived in full; the ignored folders are not.
+ expect(base.find((member) => member.name === "Cargo.lock")?.content.toString()).toBe("# ignored lockfile\n");
+ expect(baseNames).toEqual(expect.arrayContaining(["src/main.rs", "web/pnpm-lock.yaml", ".gitignore"]));
+ expect(baseNames.some((name) => name.startsWith("outputs/") || name.startsWith("node_modules/"))).toBe(false);
+ // Entries stay in archive order after the merge.
+ const files = (manifestOf(base).files as Array<{ path: string }>).map((file) => file.path);
+ expect(files).toEqual([...files].sort((left, right) => Buffer.compare(Buffer.from(left), Buffer.from(right))));
+
+ // The agent writes a gitignored binary and touches a file in node_modules.
+ const plot = Buffer.concat([PNG, Buffer.from([0, 1, 2, 255])]);
+ await writeFile(join(root, "outputs/plot.png"), plot);
+ subject.recordTouched(id, "outputs/plot.png");
+ subject.recordTouched(id, "node_modules/dep/index.js");
+ expect(await subject.captureDelta(id, root, 1)).toMatchObject({ status: "queued", kind: "delta" });
+ expect((await subject.drain()).uploaded).toBe(1);
+ const delta = await openArchive(server.objects()[1]!.object!);
+ expect(delta.map((member) => member.name)).toEqual(["__omnirush__/manifest.json", "outputs/plot.png"]);
+ expect(sha256(delta[1]!.content)).toBe(sha256(plot));
+ expect(manifestOf(delta).deleted ?? []).toEqual([]);
+
+ // The next turn changes nothing the agent touched: the touched file stays in the chain.
+ await writeFile(join(root, "outputs/untouched.txt"), "edited, still never touched\n");
+ expect(await subject.captureDelta(id, root, 2)).toEqual({ status: "skipped", reason: "unchanged" });
+ for (const object of server.objects()) {
+ const names = (await openArchive(object.object!)).map((member) => member.name);
+ expect(names).not.toContain("outputs/untouched.txt");
+ expect(names.some((name) => name.startsWith("node_modules/"))).toBe(false);
+ }
+ await subject.stop();
+ // The whole-folder chain kept its touched paths, and keeps them across a restart.
+ expect(await readdir(join(state, ARCHIVE_STATE_DIRECTORY, "touched"))).toHaveLength(1);
+ const restarted = archiver(server, state);
+ await restarted.start();
+ expect(await readdir(join(state, ARCHIVE_STATE_DIRECTORY, "touched"))).toHaveLength(1);
+ // The restarted app still holds the touched binary: no delta deletes it.
+ expect(await restarted.captureDelta(id, root, 3)).toEqual({ status: "skipped", reason: "unchanged" });
+ await restarted.stop();
+ });
+
+ test("a touched-files chain archives the gitignored files the agent touched and every lockfile, never node_modules", async () => {
+ const home = await tempDir("home");
+ const root = join(home, "report");
+ await mkdir(root);
+ await tree(root);
+ const server = new FakeArchiveServer();
+ server.policy = { touched_files: true };
+ const subject = archiver(server, await tempDir("state"), { folderGate: { homeDir: home } });
+ const id = "ses_repro_touched";
+
+ expect(await subject.captureBase(id, root)).toEqual({ status: "skipped", reason: "unchanged" });
+ subject.recordTouched(id, "outputs/plot.png");
+ subject.recordTouched(id, "node_modules/dep/index.js");
+ subject.recordTouched(id, "node_modules/dep/package-lock.json");
+ expect(await subject.captureDelta(id, root, 1)).toMatchObject({ status: "queued", kind: "base", sequence: 0 });
+ expect((await subject.drain()).uploaded).toBe(1);
+ const members = await openArchive(server.objects()[0]!.object!);
+ expect(members.map((member) => member.name)).toEqual(["__omnirush__/manifest.json", "Cargo.lock", "outputs/plot.png", "web/pnpm-lock.yaml"]);
+ expect(members[2]!.content.equals(PNG)).toBe(true);
+ expect(manifestOf(members)).toMatchObject({ scope: "touched", workspace: { marker: "touched" } });
+ await subject.stop();
+ });
+});
diff --git a/apps/server/src/session-archive/touched.test.ts b/apps/server/src/session-archive/touched.test.ts
index 84ee6fcf..b8c134b0 100644
--- a/apps/server/src/session-archive/touched.test.ts
+++ b/apps/server/src/session-archive/touched.test.ts
@@ -121,6 +121,18 @@ describe("scanTouchedFiles and gitignored files", () => {
expect(change.files.map((entry) => entry.path)).toEqual([".gitignore"]);
expect(change.next.map((entry) => entry.path)).toEqual([".gitignore", "app/main.py", "clone/src.c"]);
});
+
+ test("includeIgnored keeps the touched files git ignores (the reproducible-task rule), byte for byte", async () => {
+ const root = await tempDir("touched-include-ignored");
+ await writeFile(join(root, ".gitignore"), "outputs/\nCargo.lock\n");
+ await mkdir(join(root, "outputs"));
+ await writeFile(join(root, "outputs/plot.png"), Buffer.from([0x89, 0x50, 0x4e, 0x47, 0, 255]));
+ await writeFile(join(root, "Cargo.lock"), "# lock\n");
+ const scan = await scanTouchedFiles(root, ["outputs/plot.png", "Cargo.lock"], { includeIgnored: true });
+ expect(scan.entries.map((entry) => entry.path)).toEqual(["Cargo.lock", "outputs/plot.png"]);
+ expect(scan.ignored).toBe(0);
+ expect(scan.gone.size).toBe(0);
+ });
});
describe("touchedChange", () => {
@@ -183,6 +195,19 @@ describe("TouchedPathStore", () => {
expect(await readdir(dir)).toEqual([]);
});
+ test("a snapshot taken before any path is written leaves the file readable: the header comes with the first path", async () => {
+ const dir = await tempDir("touched-header");
+ const { store: first } = store(dir, "tracked");
+ expect([...(await first.snapshot("ses_touched_03"))]).toEqual([]);
+ expect(await readdir(dir)).toEqual([]);
+ first.note("ses_touched_03", "outputs/plot.png");
+ await first.flush();
+ const [file] = await readdir(dir);
+ expect((await readFile(join(dir, file!), "utf8")).split("\n")).toEqual(['{"v":1,"session_id":"ses_touched_03"}', '"outputs/plot.png"', ""]);
+ const { store: second } = store(dir, "tracked");
+ expect([...(await second.snapshot("ses_touched_03"))]).toEqual(["outputs/plot.png"]);
+ });
+
test("a session the record says is something else keeps nothing; clear() drops what is in memory", async () => {
const dir = await tempDir("touched-ignored");
const { store: ignored } = store(dir, "ignored");
diff --git a/apps/server/src/session-archive/touched.ts b/apps/server/src/session-archive/touched.ts
index 9096a344..dc7e0d0a 100644
--- a/apps/server/src/session-archive/touched.ts
+++ b/apps/server/src/session-archive/touched.ts
@@ -7,12 +7,13 @@
* disk, and scanTouchedFiles turns them into archive entries at capture
* time: regular files inside the root only, never through a symlinked
* folder, a touched folder never expanded, with the whole-folder scan's
- * exclusions, and never a file git ignores (`ignore.ts`). Nothing outside the root is ever opened or stat'ed: a path is
+ * exclusions, and never a file git ignores (`ignore.ts`) unless the caller
+ * keeps ignored files (includeIgnored: the archiver always does). Nothing outside the root is ever opened or stat'ed: a path is
* checked lexically first, then walked from the root one lstat at a time.
*/
import { createHash } from "node:crypto";
import type { BigIntStats } from "node:fs";
-import { appendFile, lstat, open, readFile, readlink, realpath, rm, writeFile } from "node:fs/promises";
+import { appendFile, lstat, open, readdir, readFile, readlink, realpath, rm, writeFile } from "node:fs/promises";
import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
import {
@@ -38,6 +39,7 @@ import {
} from "./manifest.js";
import { stateKey } from "./files.js";
import { touchedIgnoredPaths } from "./ignore.js";
+import { REGENERABLE_DIR_NAMES } from "../session-uploader.js";
import type { ArchiveLog } from "./upload.js";
/** A session keeps at most this many touched paths; later ones are not archived (one log line). */
@@ -60,6 +62,8 @@ export type TouchedScanOptions = {
hashCache?: ArchiveHashCache;
/** Stops the scan between paths: it then rejects with the signal's reason. */
signal?: AbortSignal;
+ /** Gitignored files are kept (the reproducible-task rule: touched files and lockfiles are archived even when ignored). */
+ includeIgnored?: boolean;
/** Tests: every absolute path the scan lstats, reads a link of or opens, before it does. */
onAccess?: (path: string) => void;
};
@@ -111,6 +115,49 @@ function linkTargetInside(path: string, target: string, roots: readonly string[]
return null;
}
+/** Lockfiles and dependency pins, archived in full wherever they sit (outside the regenerable folders). */
+const LOCKFILE_NAMES = new Set([
+ "package-lock.json", "npm-shrinkwrap.json", "yarn.lock", "pnpm-lock.yaml", "bun.lock", "bun.lockb", "deno.lock",
+ "Cargo.lock", "poetry.lock", "Pipfile.lock", "uv.lock", "pdm.lock", "pixi.lock", "conda-lock.yml",
+ "Gemfile.lock", "composer.lock", "go.sum", "go.work.sum", "mix.lock", "pubspec.lock", "Package.resolved",
+ "packages.lock.json", "gradle.lockfile", "flake.lock", "Podfile.lock", "Cartfile.resolved",
+]);
+const LOCKFILE_MAX_DEPTH = 6;
+const LOCKFILE_MAX_DIRS = 5_000;
+const LOCKFILE_MAX_FOUND = 200;
+
+/**
+ * The lockfiles under `root`, workspace-relative and portable: a bounded walk
+ * (depth, folders visited, results) that never follows a symlink and skips
+ * `.git/` and the regenerable folders. scanTouchedFiles checks each one again
+ * before anything is archived. Never throws.
+ */
+export async function findLockfiles(root: string, signal?: AbortSignal): Promise<string[]> {
+ const found: string[] = [];
+ let visited = 0;
+ const walk = async (rel: string, depth: number): Promise<void> => {
+ if (depth > LOCKFILE_MAX_DEPTH || visited >= LOCKFILE_MAX_DIRS || found.length >= LOCKFILE_MAX_FOUND || signal?.aborted) return;
+ visited += 1;
+ let entries;
+ try {
+ entries = await readdir(rel ? join(root, ...rel.split("/")) : root, { withFileTypes: true });
+ } catch {
+ return;
+ }
+ entries.sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0));
+ for (const entry of entries) {
+ const path = rel ? `${rel}/${entry.name}` : entry.name;
+ if (entry.isFile() && LOCKFILE_NAMES.has(entry.name)) {
+ if (found.length < LOCKFILE_MAX_FOUND) found.push(path);
+ } else if (entry.isDirectory() && entry.name !== ".git" && !REGENERABLE_DIR_NAMES.has(entry.name)) {
+ await walk(path, depth + 1);
+ }
+ }
+ };
+ await walk("", 0);
+ return found;
+}
+
/**
* Pass 1 of a touched-files capture: each path (and each symlink target
* inside the root it leads to) walked from the root with lstat, never
@@ -235,8 +282,8 @@ export async function scanTouchedFiles(root: string, paths: Iterable<string>, op
await forEachBounded([...paths], STAT_CONCURRENCY, (path) => visit(path, 0));
- // Git's answer for just the files found, so the folder is never walked.
- const ignoredPaths = await touchedIgnoredPaths(base, entries.map((entry) => entry.path), signal);
+ // Git's answer for just the files found, so the folder is never walked (none asked when ignored files are kept).
+ const ignoredPaths = options.includeIgnored === true ? new Set<string>() : await touchedIgnoredPaths(base, entries.map((entry) => entry.path), signal);
signal?.throwIfAborted();
if (ignoredPaths.size > 0) {
let kept = 0;
@@ -534,7 +581,12 @@ export class TouchedPathStore {
session.torn = loaded.torn;
}
const known = session.known;
- if (session.pending.size === 0) return;
+ if (session.pending.size === 0) {
+ // Nothing to write to a file that has no header yet: it is loaded (and started) again by the next write,
+ // else that write would append to a file without its header, unreadable after a restart.
+ if (header) session.known = null;
+ return;
+ }
const batch = [...session.pending];
session.pending = new Set();
session.writing = batch;
diff --git a/apps/server/src/session-uploader.test.ts b/apps/server/src/session-uploader.test.ts
index 63516033..35e6b156 100644
--- a/apps/server/src/session-uploader.test.ts
+++ b/apps/server/src/session-uploader.test.ts
@@ -1564,11 +1564,11 @@ describe("session uploader trace additions", () => {
expect(photo?.data).toEqual({ name: "photo.png", mime: "image/png", bytes: 12, sha256: sha256("png"), text: null, text_truncated: false });
});
- test("emits artifact events for untracked outputs a turn creates or modifies", async () => {
+ test("emits artifact events for untracked outputs a turn creates or modifies, gitignored ones outside the regenerable folders too", async () => {
const root = await mkdtemp(join(tmpdir(), "omnirush-upload-artifacts-"));
roots.push(root);
await git(root, "init", "-q");
- await writeFile(join(root, ".gitignore"), "ignored/\n");
+ await writeFile(join(root, ".gitignore"), "ignored/\nnode_modules/\n");
await writeFile(join(root, "tracked.txt"), "tracked\n");
await writeFile(join(root, "stale.txt"), "already there\n");
await git(root, "add", ".gitignore", "tracked.txt");
@@ -1587,13 +1587,16 @@ describe("session uploader trace additions", () => {
await writeFile(join(root, "notes.md"), "generated notes\n");
await mkdir(join(root, "ignored"));
await writeFile(join(root, "ignored", "cache.txt"), "ignored output\n");
+ await mkdir(join(root, "node_modules", "dep"), { recursive: true });
+ await writeFile(join(root, "node_modules", "dep", "index.js"), "module.exports = 1;\n");
sessionUploader.captureSnapshot(sessionId, "turn_completed");
await sessionUploader.idle(sessionId);
sessionUploader.flushTrace(sessionId);
await sessionUploader.stop();
const artifacts = traceEvents(uploads).filter((event) => event.type === "artifact").map((event) => event.data);
- expect(artifacts.map((artifact) => artifact?.path).sort()).toEqual(["notes.md", "out/report.bin"]);
+ expect(artifacts.map((artifact) => artifact?.path).sort()).toEqual(["ignored/cache.txt", "notes.md", "out/report.bin"]);
+ expect(artifacts.find((artifact) => artifact?.path === "ignored/cache.txt")).toEqual({ path: "ignored/cache.txt", sha256: sha256("ignored output\n"), bytes: 15 });
expect(artifacts.find((artifact) => artifact?.path === "out/report.bin")).toEqual({
path: "out/report.bin",
sha256: createHash("sha256").update(Buffer.from([0, 1, 2, 3, 255])).digest("hex"),
diff --git a/apps/server/src/session-uploader.ts b/apps/server/src/session-uploader.ts
index 60010d3f..88c97b74 100644
--- a/apps/server/src/session-uploader.ts
+++ b/apps/server/src/session-uploader.ts
@@ -2326,6 +2326,24 @@ async function gitPriorityPaths(root: string): Promise<Set<string>> {
return new Set(parseGitStatus(status.stdout.toString("utf8"), prefix).entries.map((entry) => entry.path));
}
+/**
+ * Folders whose content is reinstalled or rebuilt from the lockfiles (the
+ * reproducible-task rule's "not needed"): a touched or locked path in one is
+ * never archived for being ignored, and an ignored output in one is never an
+ * artifact.
+ */
+export const REGENERABLE_DIR_NAMES = new Set([
+ "node_modules", ".venv", "venv", "__pycache__", ".tox", ".nox", ".mypy_cache", ".pytest_cache", ".ruff_cache",
+ "dist", "build", "target", "out", ".next", ".nuxt", ".svelte-kit", ".turbo", ".parcel-cache", ".cache",
+ ".gradle", ".dart_tool", "Pods", "DerivedData", "vendor", "bower_components", "site-packages",
+]);
+
+/** Whether a workspace-relative portable path lies in a regenerable folder (or is `.git/` content). */
+export function inRegenerableDir(path: string): boolean {
+ const parts = path.split("/");
+ return parts.slice(0, -1).some((part) => part === ".git" || REGENERABLE_DIR_NAMES.has(part));
+}
+
/**
* Files inside the workspace that git does not track but does not ignore
* either: the outputs an agent leaves next to the source. A workspace without
@@ -2344,9 +2362,48 @@ async function listUntrackedFiles(root: string): Promise<string[]> {
}
}
+const MAX_IGNORED_OUTPUT_FILES = 5_000;
+
+/**
+ * The gitignored untracked files outside the regenerable folders (a build's
+ * `outputs/plot.png`, a downloaded `data.pdf`): `git ls-files --ignored
+ * --directory` names a wholly ignored folder once, and only the folders that
+ * are not regenerable are listed further. Empty outside a git work tree
+ * (walkFallback already lists every file there) or when git fails.
+ */
+async function listIgnoredOutputFiles(root: string): Promise<string[]> {
+ const listed = async (args: string[]): Promise<string[]> => {
+ const { stdout } = await execFileAsync("git", ["-C", root, "ls-files", "--others", "--ignored", "--exclude-standard", "-z", ...args], {
+ encoding: "buffer",
+ maxBuffer: 16 * 1024 * 1024,
+ timeout: 15_000,
+ });
+ return Buffer.from(stdout).toString("utf8").split("\0").filter(Boolean);
+ };
+ try {
+ const paths: string[] = [];
+ for (const path of await listed(["--directory"])) {
+ if (paths.length >= MAX_IGNORED_OUTPUT_FILES) break;
+ if (!path.endsWith("/")) {
+ if (!inRegenerableDir(path)) paths.push(path);
+ continue;
+ }
+ const dir = path.slice(0, -1);
+ if (inRegenerableDir(`${dir}/x`)) continue;
+ for (const inner of await listed(["--", dir])) {
+ if (paths.length >= MAX_IGNORED_OUTPUT_FILES) break;
+ if (!inRegenerableDir(inner)) paths.push(inner);
+ }
+ }
+ return filterListing(paths).paths;
+ } catch {
+ return [];
+ }
+}
+
async function artifactStats(root: string): Promise<Map<string, ArtifactStat>> {
const stats = new Map<string, ArtifactStat>();
- for (const path of await listUntrackedFiles(root)) {
+ for (const path of [...(await listUntrackedFiles(root)), ...(await listIgnoredOutputFiles(root))]) {
try {
if (workspaceRelativePath(root, path) === null) continue;
const absolute = resolve(root, path);
@@ -4320,8 +4377,8 @@ export class SessionUploader {
}
/**
- * Emits one "artifact" event per untracked-but-not-ignored file the turn
- * created or modified. Tracked files are already covered by the change
+ * Emits one "artifact" event per untracked file (gitignored ones outside the
+ * regenerable folders too) the turn created or modified. Tracked files are already covered by the change
* snapshot and touched_paths; this adds the outputs git would not list.
*/
private async captureArtifacts(state: SessionState): Promise<void> {