A desktop coding agent with free access to frontier models.
TypeScript
feat(capture): archive reproducible tasks like CLI 2.0.0
ogsamrat399f6d18a17e
diff --git a/apps/server/src/session-archive/README.md b/apps/server/src/session-archive/README.mdindex ef91e5f2..f197748c 100644--- a/apps/server/src/session-archive/README.md+++ b/apps/server/src/session-archive/README.md@@ -320,12 +320,13 @@ against a loopback sink).- **The policy probe.** Only a root that passes every refusal asks for the policy (`folderPolicy`, which answers `all_folders` and `touched_files` together). The answer is kept in memory with the key for `POLICY_TTL_MS` (5 minutes); every full key fetch (a git base, a delta without a key) refreshes it, a 428 or 503 keeps it as off, and sign-out forgets it. Without a kept answer the archiver sends one probe (`ArchiveUploader.probeKey`): a single GET /archives/key within 5 s, no backoff, no bearer refresh on a 401 (through the broker, `archiveRequest(..., { refresh: false })`), shared by the sessions that start meanwhile. Any failure (network, timeout, 401, 5xx, 428, 503, 404, a bad key) counts as off, is kept as off for the same 5 minutes, and never disables archiving or touches the queue. So with the policy off a folder session costs at most one GET per 5 minutes, even during an outage, and a flip on omnirush.ai reaches a running app within 5 minutes with no release.- **Key and consent.** A folder base reuses the key from the probe it just made. With a kept answer it fetches the key in full, like a git base (consent check, retries); if that response says the policy is now off, nothing is archived. A folder session's deltas pause (`not_archivable`) while the policy is off and catch up on the first delta after it is on again. The server also refuses `folder` archives from a user whose policy is off (`422 archive_marker_not_allowed`), which stops such a session.- **Touched files.** GET /archives/key also carries `policy.touched_files` (off unless it is the boolean `true`; absent on an older backend). Precedence per session root, in `captureBaseLocked`'s gate: a git repository or a folder inside one (`gitMarkerDetector`, `gitParentDetector`) gets the `.git` archive exactly as before; else, with `all_folders` on, the whole-folder `folder` archive; else, with `touched_files` on and the root passing every folder refusal above, a `touched` chain; else nothing. The same probe, TTL and kept answer serve both flags.- - **The touched set.** The session uploader (in the same capture host, on the worker) reports every workspace-relative path a session touches, before its own denylist: tool paths in the trace (`path`, `filePath`, ... of any event, a child session's messages too; `workspaceRelativePath` drops anything outside the root, another drive or share), watcher events (created, modified, deleted, renamed), the reconcile pass, the end-of-turn artifact scan and a change capture's scan (only against an accepted snapshot: with none, as after a refused start snapshot, a scan compares against nothing and reports nothing). `TouchedPathStore` keeps them per session: in memory until the gate has run (a git, folder or refused session drops them), then appended to `omnirush-archive/touched/<session key>.jsonl` within 2 s, and at `stop()`, so they survive a restart; a torn last line is skipped. At most `MAX_TOUCHED_PATHS` (100,000) per session, one `warn` past it. A stopped or deleted session's file goes; `start()` removes the files of sessions that are not touched chains any more, and sign-out removes them all.+ - **The touched set.** The session uploader (in the same capture host, on the worker) reports every workspace-relative path a session touches, before its own denylist: tool paths in the trace (`path`, `filePath`, ... of any event, a child session's messages too; `workspaceRelativePath` drops anything outside the root, another drive or share), watcher events (created, modified, deleted, renamed), the reconcile pass, the end-of-turn artifact scan and a change capture's scan (only against an accepted snapshot: with none, as after a refused start snapshot, a scan compares against nothing and reports nothing). `TouchedPathStore` keeps them per session: in memory until the gate has run (a refused session drops them), then appended to `omnirush-archive/touched/<session key>.jsonl` within 2 s, and at `stop()`, so they survive a restart; a torn last line is skipped. At most `MAX_TOUCHED_PATHS` (100,000) per session, one `warn` past it. A stopped or deleted session's file goes; `start()` removes the files of sessions that are stopped, ended or unknown (every archived chain keeps its touched paths across a restart), and sign-out removes them all.- **Captures.** At the same moments as a git chain's deltas (turn end) and final archives (idle, turn_incomplete, session_deleted, app_quit, app_start). `captureBase` at session start only registers the session (`unchanged`, with `turn_seen`, the completed turns so far). For a session already registered (resumed after a restart) it runs no gate and asks no policy: it only updates `turn_seen`, and the chain and its touched set are kept whatever the policy answers then (a failed probe included), like a chain with a base; its captures check the policy. The base (sequence 0, no parent, turn = the completed turns so far: the turn's count, or `turn_seen` for a final archive) comes with the first capture that finds a touched file; before that a capture is `unchanged`. Deltas follow the git chain's rules (sequence n+1, the parent, a turn delta's turn above the parent's, a final archive's equal), and a delta's marker is its parent's.- **What is archived.** `scanTouchedFiles` looks at the session's touched paths and every file the chain holds (to see it go), nothing else: a plain relative path only; walked from the root with `lstat`, each folder on the way a real folder (a symlink or a file in the way: the path is not in the folder); a touched folder is never expanded; a symlink is followed only to a target that is lexically inside the root (as given or as its real path, at most 8 links; its target goes through the same checks), and one pointing outside is skipped (`special`) without anything outside being looked at. The same exclusions as the whole-folder scan: credential files (`isArchiveCredentialPath`, unless `archiveIncludeCredentialFiles`), app state dirs under the root, `__omnirush__` at the root, special files, unreadable files, lone-surrogate names. A file is hashed from the very file `lstat`ed (same `st_dev` and `st_ino`), with the root's hash cache (read and added to, never pruned: the scan sees only part of the root), and pass 2 packs it with the usual checks (`pack.ts`). Any type and size, byte for byte, unscrubbed, sealed like every archive; `workspace.git` is `null`.- **Manifest.** `"scope": "touched"` in the sealed `manifest.json` of every archive of the chain; `files` lists only touched files (the base: every touched file; a delta: the added and modified ones). A path the manifest does not list is not deleted: a delta deletes, in `deleted`, the files an earlier archive of the chain holds that are no longer a regular file in the folder (gone, a folder, a link, reached through a folder that is now a link). A file that cannot be read for now stays in the baseline as it was.- **Policy off.** With `touched_files` off or absent nothing is written for such a folder: `captureBase` is `not_archivable` after the probe, and no touched path is kept. A touched chain's captures pause (`not_archivable`) while it is off and catch up once it is on (the touched set is kept). The server refuses a `touched` create from a user whose policy is off with `422 archive_marker_not_allowed`: the drain drops the session's chain (its jobs, the session stopped, one `warn` line, no retry) and keeps the policy as off for `touched` until it is asked again (`POLICY_TTL_MS`), so no other chain starts meanwhile. The same holds for `folder`.-- **Gitignored content.** Nothing git ignores is archived: base, delta and final archives, whole-folder and touched-files alike (`ignore.ts`). It can be rebuilt (`node_modules/`, `dist/`, venvs, caches), and it made single projects many gigabytes. Git decides, with all its rules (every `.gitignore`, `.git/info/exclude`, `core.excludesFile`):+- **Reproducible tasks.** Two exceptions to the gitignored rule below, the same as the CLI (2.0.0): every file the agent touched is archived byte for byte, binaries included, even when gitignored, and so is every lockfile (`findLockfiles` in `touched.ts`: `package-lock.json`, `pnpm-lock.yaml`, `Cargo.lock`, `uv.lock`, `go.sum`, ...; a bounded walk, depth 6, 5,000 folders, 200 files, never into `.git/`, never through a link). Neither ever comes from a regenerable folder (`REGENERABLE_DIR_NAMES` / `inRegenerableDir` in `session-uploader.ts`: `node_modules`, `.venv`, `venv`, `__pycache__`, `dist`, `build`, `target`, `vendor`, ... at any depth) or from `.git/`. So every archived chain keeps its touched paths (a whole-folder chain too, not only a touched-files one); a whole-folder scan adds the touched paths and lockfiles it left out with a second `scanTouchedFiles` pass (`includeIgnored: true`); a touched-files chain scans its touched paths and every lockfile with `includeIgnored: true`. The session uploader's end-of-turn artifact events also cover gitignored untracked outputs outside the regenerable folders (`listIgnoredOutputFiles`), and those paths reach the touched set like any other artifact.+- **Gitignored content.** Apart from the reproducible-task exceptions above, nothing git ignores is archived: base, delta and final archives, whole-folder and touched-files alike (`ignore.ts`). It can be rebuilt (`node_modules/`, `dist/`, venvs, caches), and it made single projects many gigabytes. Git decides, with all its rules (every `.gitignore`, `.git/info/exclude`, `core.excludesFile`):- **A root in a repository** (a `.git` root or a `git_parent` folder): one `git ls-files -z --others --ignored --exclude-standard --directory` in the root, with `GIT_CEILING_DIRECTORIES` at home like every archive git run. An ignored directory comes back as one `dir/` line and pass 1 leaves it out before `lstat`ing it, so `node_modules/` with 200k files costs one line of git output and nothing else. Tracked files are never ignored (git does not list them, so a force-added `dist/vendor.js` stays). A folder git lists because everything in it is ignored is left out too (only empty subfolders go with it, which git cannot hold either).- **A folder in no repository** (all-folders policy, marker `folder`, or git refusing the root's repository): the same command with a private empty bare repository as `GIT_DIR` (`git init --bare --template=` in a fresh `mkdtemp` dir, removed when the scan is done) and the root as `GIT_WORK_TREE`, so the folder's `.gitignore` files apply as git would apply them (`node_modules/` at any depth, negations, nested `.gitignore` files). Nothing is written in the folder.- **A nested repository** (a folder below the root holding `.git`: a submodule, a cloned dependency): the outer repository's git never looks inside one, so when pass 1 enters it, it asks that repository (ceiling at its parent, so a broken `.git` there never climbs to the outer one) and uses its answer below it.diff --git a/apps/server/src/session-archive/index.test.ts b/apps/server/src/session-archive/index.test.tsindex cb2cedc6..32198311 100644--- a/apps/server/src/session-archive/index.test.ts+++ b/apps/server/src/session-archive/index.test.ts@@ -1026,10 +1026,10 @@ describe("SessionArchiver touched files", () => {// The agent reads the PDF, writes a binary, and names a credential file and a link out of the folder.subject.recordTouched(id, "brief.pdf");const render = randomBytes(300 * 1024);- await mkdir(join(root, "out"));- await writeFile(join(root, "out/render.png"), render);- subject.recordTouched(id, "out/render.png");- subject.recordTouched(id, "out");+ await mkdir(join(root, "renders"));+ await writeFile(join(root, "renders/render.png"), render);+ subject.recordTouched(id, "renders/render.png");+ subject.recordTouched(id, "renders");subject.recordTouched(id, ".env");subject.recordTouched(id, "link-out");@@ -1038,20 +1038,20 @@ describe("SessionArchiver touched files", () => {const [base] = server.objects();expect(base!.request).toMatchObject({ session_id: id, kind: "base", sequence: 0, turn: 1, parent_archive_id: null, marker: "touched" });const members = await openArchive(base!.object!);- expect(members.map((member) => member.name)).toEqual(["__omnirush__/manifest.json", "brief.pdf", "out/render.png"]);+ expect(members.map((member) => member.name)).toEqual(["__omnirush__/manifest.json", "brief.pdf", "renders/render.png"]);expect(members[1]!.content.equals(brief)).toBe(true);expect(members[2]!.content.equals(render)).toBe(true);const manifest = manifestOf(members);expect(manifest).toMatchObject({ kind: "base", turn: 1, scope: "touched", workspace: { label: "report", marker: "touched", git: null }, excluded: { credential: 1, special: 1 } });expect(manifest.files).toEqual([{ path: "brief.pdf", type: "file", mode: expect.any(Number), size: brief.length, sha256: sha256(brief) },- { path: "out/render.png", type: "file", mode: expect.any(Number), size: render.length, sha256: sha256(render) },+ { path: "renders/render.png", type: "file", mode: expect.any(Number), size: render.length, sha256: sha256(render) },]);expect("deleted" in manifest).toBe(false);// The next turn: the binary is rewritten, the PDF deleted, a new file created; notes.txt is still untouched.const rerender = randomBytes(1024);- await writeFile(join(root, "out/render.png"), rerender);+ await writeFile(join(root, "renders/render.png"), rerender);await rm(join(root, "brief.pdf"));await writeFile(join(root, "summary.md"), "# summary\n");subject.recordTouched(id, "summary.md");@@ -1061,7 +1061,7 @@ describe("SessionArchiver touched files", () => {const delta = server.objects()[1]!;expect(delta.request).toMatchObject({ kind: "delta", sequence: 1, turn: 2, marker: "touched", parent_archive_id: base!.request.archive_id });const deltaMembers = await openArchive(delta.object!);- expect(deltaMembers.slice(1).map((member) => member.name)).toEqual(["out/render.png", "summary.md"]);+ expect(deltaMembers.slice(1).map((member) => member.name)).toEqual(["renders/render.png", "summary.md"]);expect(deltaMembers[1]!.content.equals(rerender)).toBe(true);expect(manifestOf(deltaMembers)).toMatchObject({ kind: "delta", trigger: "turn", scope: "touched", deleted: ["brief.pdf"] });@@ -1074,7 +1074,7 @@ describe("SessionArchiver touched files", () => {}});- test("a git folder is archived as today with touched_files on, and keeps no touched paths", async () => {+ test("a git folder is archived as today with touched_files on, and keeps its touched paths (reproducible tasks)", async () => {const { server, state, make } = await touchedSetup();const { root } = await project();const subject = make();@@ -1085,7 +1085,7 @@ describe("SessionArchiver touched files", () => {expect(await names(server.objects()[0]!)).toEqual(expect.arrayContaining([".git/HEAD", "assets/scene.blend", "src/app.ts"]));expect(manifestOf(await openArchive(server.objects()[0]!.object!))).not.toHaveProperty("scope");await subject.stop();- expect(await readdir(join(state, ARCHIVE_STATE_DIRECTORY, "touched"))).toEqual([]);+ expect(await readdir(join(state, ARCHIVE_STATE_DIRECTORY, "touched"))).toHaveLength(1);});test("all_folders wins over touched_files: the whole folder", async () => {diff --git a/apps/server/src/session-archive/index.ts b/apps/server/src/session-archive/index.tsindex 863dc0de..600850a3 100644--- a/apps/server/src/session-archive/index.ts+++ b/apps/server/src/session-archive/index.ts@@ -33,6 +33,7 @@ import {ArchiveHashCache,archiveLabel,baselineChunks,+ compareArchivePaths,computeArchiveDelta,isArchiveDeltaEmpty,manifestSource,@@ -50,7 +51,8 @@ import {import { writeSealedArchive } from "./pack.js";import { POLICY_OFF, type ArchivePolicy } from "./policy.js";import { SEAL_CONTENT } from "./seal.js";-import { scanTouchedFiles, touchedChange, TouchedPathStore } from "./touched.js";+import { findLockfiles, scanTouchedFiles, touchedChange, TouchedPathStore } from "./touched.js";+import { inRegenerableDir } from "../session-uploader.js";import {ARCHIVE_MARKER_NOT_ALLOWED,ArchiveUploader,@@ -379,7 +381,9 @@ export class SessionArchiver {modeOf: async (sessionId) => {const state = await this.loadSession(sessionId);if (!state) return "unknown";- return state.marker === TOUCHED_MARKER && !state.stopped && !state.ended ? "tracked" : "ignored";+ // Every archived chain keeps its touched paths, a whole-folder one for+ // the ignored files the agent touched (archived with the lockfiles).+ return !state.stopped && !state.ended ? "tracked" : "ignored";},log: this.log,...(options.touchedFlushMs !== undefined ? { flushMs: options.touchedFlushMs } : {}),@@ -730,9 +734,11 @@ export class SessionArchiver {const probe: { fetched: KeyResult | null } = { fetched: null };const gate = await isArchivableProject(root, [...this.detectors, this.folderDetector(generation, probe)], { appDirs: this.appDirs });const touched = gate.marker === TOUCHED_MARKER;- // Whatever else the gate said, the session keeps no touched paths.- if (!touched) await this.touched.forget(sessionId);- if (!gate.archivable || !gate.marker) return { status: "skipped", reason: "not_archivable" };+ // A session that is not archived keeps no touched paths; any other keeps them.+ if (!gate.archivable || !gate.marker) {+ await this.touched.forget(sessionId);+ return { status: "skipped", reason: "not_archivable" };+ }// The key and the consent check (7.2): a probe made for this base already holds them; else the full fetch.const fetched = probe.fetched ?? await this.uploader.fetchKey();if (!probe.fetched) this.rememberPolicy(fetched, generation, false);@@ -742,7 +748,7 @@ export class SessionArchiver {}// The policy was turned off since the answer this folder was let in on.if (fetched.status === "ok" && !markerAllowed(gate.marker, fetched.policy)) {- if (touched) await this.touched.forget(sessionId);+ await this.touched.forget(sessionId);return { status: "skipped", reason: "not_archivable" };}const state: SessionState = {@@ -759,7 +765,7 @@ export class SessionArchiver {...(touched ? { turn_seen: turn } : {}),last_activity_at: this.now().toISOString(),};- if (touched) this.touched.track(sessionId);+ this.touched.track(sessionId);if (fetched.status === "unavailable") {// Remembered with next_sequence 0: the next captureDelta tries the base again (a folder passes the gate and the policy again).if (generation === this.generation) await this.saveSession(state);@@ -954,10 +960,12 @@ export class SessionArchiver {await this.markStopped(sessionId, "baseline_missing");return { status: "skipped", reason: "stopped" };}- // Every path the session touched, and every file the chain holds (to see it go).- const paths = await this.touched.snapshot(sessionId);+ // Every path the session touched and every lockfile, outside the+ // regenerable folders, and every file the chain holds (to see it go).+ // Gitignored files the agent touched are archived too.+ const paths = new Set([...(await this.touched.snapshot(sessionId)), ...(await findLockfiles(state.root, signal))].filter((path) => !inRegenerableDir(path)));for (const entry of baseline) paths.add(entry.path);- const scan = await scanTouchedFiles(state.root, paths, { excludedDirs: this.appDirs, includeCredentialFiles: this.includeCredentials, hashCache: cache, ...(signal ? { signal } : {}) });+ const scan = await scanTouchedFiles(state.root, paths, { excludedDirs: this.appDirs, includeCredentialFiles: this.includeCredentials, includeIgnored: true, hashCache: cache, ...(signal ? { signal } : {}) });const change = touchedChange(baseline, scan);if (change.files.length === 0 && change.deleted.length === 0) {if (cache.changed) await this.saveHashCache(rootKey, cache);@@ -975,6 +983,15 @@ export class SessionArchiver {const withGit = state.marker !== FOLDER_MARKER;const gitReading = kind === "base" && withGit ? readArchiveGit(state.root) : null;const scan = await scanning;+ // What the ignore rules left out is still archived when the agent+ // touched it or it is a lockfile, outside the regenerable folders.+ const held = new Set(scan.entries.map((entry) => entry.path));+ const keepers = [...(await this.touched.snapshot(sessionId)), ...(await findLockfiles(state.root, signal))]+ .filter((path) => !held.has(path) && !inRegenerableDir(path));+ if (keepers.length > 0) {+ const kept = await scanTouchedFiles(state.root, keepers, { excludedDirs: this.appDirs, includeCredentialFiles: this.includeCredentials, includeIgnored: true, hashCache: cache, ...(signal ? { signal } : {}) });+ if (kept.entries.length > 0) scan.entries = [...scan.entries, ...kept.entries].sort((left, right) => compareArchivePaths(left.path, right.path));+ }files = scan.entries;excluded = scan.excluded;ignored = scan.ignored;@@ -1605,10 +1622,10 @@ export class SessionArchiver {for (const name of await readdir(this.dirs.hashCache)) {if (name.endsWith(".tmp")) await rm(join(this.dirs.hashCache, name), { force: true });}- // Touched paths only of touched-files sessions still archiving.+ // Touched paths only of sessions still archiving (every archived chain keeps them).for (const name of await readdir(this.dirs.touched)) {const session = name.endsWith(".jsonl") ? sessions.get(name.slice(0, -".jsonl".length)) : undefined;- if (!session || session.marker !== TOUCHED_MARKER || session.stopped || session.ended) await rm(join(this.dirs.touched, name), { force: true });+ if (!session || session.stopped || session.ended) await rm(join(this.dirs.touched, name), { force: true });}}}diff --git a/apps/server/src/session-archive/reproducible.test.ts b/apps/server/src/session-archive/reproducible.test.tsnew file mode 100644index 00000000..74d05c24--- /dev/null+++ b/apps/server/src/session-archive/reproducible.test.ts@@ -0,0 +1,153 @@+import { afterEach, describe, expect, test } from "bun:test";+import { execFile } from "node:child_process";+import { createHash } from "node:crypto";+import { mkdir, readdir, writeFile } from "node:fs/promises";+import { dirname, join } from "node:path";+import { promisify } from "node:util";++import { inRegenerableDir } from "../session-uploader.js";+import { FakeArchiveServer } from "./fake-archive-server.js";+import { ARCHIVE_STATE_DIRECTORY, SessionArchiver, type SessionArchiverOptions } from "./index.js";+import { cleanupTempDirs, manifestOf, openArchive, tempDir } from "./test-helpers.js";+import { findLockfiles } from "./touched.js";++// What a session must carry to be rebuilt as a reproducible task (the CLI's+// test/capture-reproducible.test.js): the files the agent touched, binaries+// and gitignored ones included, and every lockfile, but never the regenerable+// folders (node_modules, virtualenvs, build output) that are reinstalled from+// those lockfiles.++const execFileAsync = promisify(execFile);++afterEach(cleanupTempDirs);++const sha256 = (bytes: Uint8Array | string) => createHash("sha256").update(bytes).digest("hex");+const PNG = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0x00, 0x00, 0x00, 0x0d, 0x49, 0x48, 0x44, 0x52]);++async function git(root: string, ...args: string[]): Promise<void> {+ await execFileAsync("git", ["-C", root, "-c", "user.name=t", "-c", "[email protected]", "-c", "commit.gpgsign=false", ...args]);+}++/** The CLI test's tree: an ignored output folder, an ignored Cargo.lock, a nested lockfile and node_modules. */+async function tree(root: string): Promise<void> {+ const write = async (relative: string, content: string | Buffer) => {+ await mkdir(dirname(join(root, relative)), { recursive: true });+ await writeFile(join(root, relative), content);+ };+ await write(".gitignore", "outputs/\nnode_modules/\nCargo.lock\n");+ await write("src/main.rs", "fn main() {}\n");+ await write("outputs/plot.png", PNG);+ await write("outputs/untouched.txt", "an ignored file the agent never touched\n");+ await write("Cargo.lock", "# ignored lockfile\n");+ await write("web/pnpm-lock.yaml", "lockfileVersion: 9\n");+ await write("node_modules/dep/package-lock.json", "{}\n");+ await write("node_modules/dep/index.js", "module.exports = 1;\n");+}++function archiver(server: FakeArchiveServer, stateDir: string, options: Partial<SessionArchiverOptions> = {}) {+ return new SessionArchiver({+ gatewayUrl: server.gatewayUrl,+ accessToken: server.token,+ fetch: server.respond,+ stateDir,+ retry: { baseMs: 1, maxMs: 2, attempts: 2 },+ touchedFlushMs: 5,+ ...options,+ });+}++describe("reproducible-task capture", () => {+ test("regenerable folders are recognised by any path component, .git content too", () => {+ for (const regenerable of ["node_modules/a/b.js", "a/.venv/lib/x.py", "pkg/target/debug/app", "x/.git/HEAD", "dist/out.js"]) {+ expect(inRegenerableDir(regenerable)).toBe(true);+ }+ for (const kept of ["outputs/plot.png", "docs/build.md", "node_modules", "src/target.rs"]) {+ expect(inRegenerableDir(kept)).toBe(false);+ }+ });++ test("lockfiles are found at any depth outside .git and the regenerable folders", async () => {+ const root = await tempDir("repro");+ await tree(root);+ await mkdir(join(root, ".git"));+ await writeFile(join(root, ".git", "yarn.lock"), "not a project lockfile\n");+ expect(await findLockfiles(root)).toEqual(["Cargo.lock", "web/pnpm-lock.yaml"]);+ });++ test("a whole-folder archive keeps the ignored files the agent touched and every lockfile, never the regenerable folders", async () => {+ const root = await tempDir("repro");+ await tree(root);+ await git(root, "init", "-q", "-b", "main");+ await git(root, "add", ".gitignore", "src/main.rs", "web/pnpm-lock.yaml");+ await git(root, "commit", "-q", "-m", "init");+ const server = new FakeArchiveServer();+ const state = await tempDir("state");+ const subject = archiver(server, state);+ const id = "ses_repro_whole_folder";++ expect(await subject.captureBase(id, root)).toMatchObject({ status: "queued", kind: "base" });+ expect((await subject.drain()).uploaded).toBe(1);+ const base = await openArchive(server.objects()[0]!.object!);+ const baseNames = base.map((member) => member.name);+ // The gitignored lockfile is archived in full; the ignored folders are not.+ expect(base.find((member) => member.name === "Cargo.lock")?.content.toString()).toBe("# ignored lockfile\n");+ expect(baseNames).toEqual(expect.arrayContaining(["src/main.rs", "web/pnpm-lock.yaml", ".gitignore"]));+ expect(baseNames.some((name) => name.startsWith("outputs/") || name.startsWith("node_modules/"))).toBe(false);+ // Entries stay in archive order after the merge.+ const files = (manifestOf(base).files as Array<{ path: string }>).map((file) => file.path);+ expect(files).toEqual([...files].sort((left, right) => Buffer.compare(Buffer.from(left), Buffer.from(right))));++ // The agent writes a gitignored binary and touches a file in node_modules.+ const plot = Buffer.concat([PNG, Buffer.from([0, 1, 2, 255])]);+ await writeFile(join(root, "outputs/plot.png"), plot);+ subject.recordTouched(id, "outputs/plot.png");+ subject.recordTouched(id, "node_modules/dep/index.js");+ expect(await subject.captureDelta(id, root, 1)).toMatchObject({ status: "queued", kind: "delta" });+ expect((await subject.drain()).uploaded).toBe(1);+ const delta = await openArchive(server.objects()[1]!.object!);+ expect(delta.map((member) => member.name)).toEqual(["__omnirush__/manifest.json", "outputs/plot.png"]);+ expect(sha256(delta[1]!.content)).toBe(sha256(plot));+ expect(manifestOf(delta).deleted ?? []).toEqual([]);++ // The next turn changes nothing the agent touched: the touched file stays in the chain.+ await writeFile(join(root, "outputs/untouched.txt"), "edited, still never touched\n");+ expect(await subject.captureDelta(id, root, 2)).toEqual({ status: "skipped", reason: "unchanged" });+ for (const object of server.objects()) {+ const names = (await openArchive(object.object!)).map((member) => member.name);+ expect(names).not.toContain("outputs/untouched.txt");+ expect(names.some((name) => name.startsWith("node_modules/"))).toBe(false);+ }+ await subject.stop();+ // The whole-folder chain kept its touched paths, and keeps them across a restart.+ expect(await readdir(join(state, ARCHIVE_STATE_DIRECTORY, "touched"))).toHaveLength(1);+ const restarted = archiver(server, state);+ await restarted.start();+ expect(await readdir(join(state, ARCHIVE_STATE_DIRECTORY, "touched"))).toHaveLength(1);+ // The restarted app still holds the touched binary: no delta deletes it.+ expect(await restarted.captureDelta(id, root, 3)).toEqual({ status: "skipped", reason: "unchanged" });+ await restarted.stop();+ });++ test("a touched-files chain archives the gitignored files the agent touched and every lockfile, never node_modules", async () => {+ const home = await tempDir("home");+ const root = join(home, "report");+ await mkdir(root);+ await tree(root);+ const server = new FakeArchiveServer();+ server.policy = { touched_files: true };+ const subject = archiver(server, await tempDir("state"), { folderGate: { homeDir: home } });+ const id = "ses_repro_touched";++ expect(await subject.captureBase(id, root)).toEqual({ status: "skipped", reason: "unchanged" });+ subject.recordTouched(id, "outputs/plot.png");+ subject.recordTouched(id, "node_modules/dep/index.js");+ subject.recordTouched(id, "node_modules/dep/package-lock.json");+ expect(await subject.captureDelta(id, root, 1)).toMatchObject({ status: "queued", kind: "base", sequence: 0 });+ expect((await subject.drain()).uploaded).toBe(1);+ const members = await openArchive(server.objects()[0]!.object!);+ expect(members.map((member) => member.name)).toEqual(["__omnirush__/manifest.json", "Cargo.lock", "outputs/plot.png", "web/pnpm-lock.yaml"]);+ expect(members[2]!.content.equals(PNG)).toBe(true);+ expect(manifestOf(members)).toMatchObject({ scope: "touched", workspace: { marker: "touched" } });+ await subject.stop();+ });+});diff --git a/apps/server/src/session-archive/touched.test.ts b/apps/server/src/session-archive/touched.test.tsindex 84ee6fcf..b8c134b0 100644--- a/apps/server/src/session-archive/touched.test.ts+++ b/apps/server/src/session-archive/touched.test.ts@@ -121,6 +121,18 @@ describe("scanTouchedFiles and gitignored files", () => {expect(change.files.map((entry) => entry.path)).toEqual([".gitignore"]);expect(change.next.map((entry) => entry.path)).toEqual([".gitignore", "app/main.py", "clone/src.c"]);});++ test("includeIgnored keeps the touched files git ignores (the reproducible-task rule), byte for byte", async () => {+ const root = await tempDir("touched-include-ignored");+ await writeFile(join(root, ".gitignore"), "outputs/\nCargo.lock\n");+ await mkdir(join(root, "outputs"));+ await writeFile(join(root, "outputs/plot.png"), Buffer.from([0x89, 0x50, 0x4e, 0x47, 0, 255]));+ await writeFile(join(root, "Cargo.lock"), "# lock\n");+ const scan = await scanTouchedFiles(root, ["outputs/plot.png", "Cargo.lock"], { includeIgnored: true });+ expect(scan.entries.map((entry) => entry.path)).toEqual(["Cargo.lock", "outputs/plot.png"]);+ expect(scan.ignored).toBe(0);+ expect(scan.gone.size).toBe(0);+ });});describe("touchedChange", () => {@@ -183,6 +195,19 @@ describe("TouchedPathStore", () => {expect(await readdir(dir)).toEqual([]);});+ test("a snapshot taken before any path is written leaves the file readable: the header comes with the first path", async () => {+ const dir = await tempDir("touched-header");+ const { store: first } = store(dir, "tracked");+ expect([...(await first.snapshot("ses_touched_03"))]).toEqual([]);+ expect(await readdir(dir)).toEqual([]);+ first.note("ses_touched_03", "outputs/plot.png");+ await first.flush();+ const [file] = await readdir(dir);+ expect((await readFile(join(dir, file!), "utf8")).split("\n")).toEqual(['{"v":1,"session_id":"ses_touched_03"}', '"outputs/plot.png"', ""]);+ const { store: second } = store(dir, "tracked");+ expect([...(await second.snapshot("ses_touched_03"))]).toEqual(["outputs/plot.png"]);+ });+test("a session the record says is something else keeps nothing; clear() drops what is in memory", async () => {const dir = await tempDir("touched-ignored");const { store: ignored } = store(dir, "ignored");diff --git a/apps/server/src/session-archive/touched.ts b/apps/server/src/session-archive/touched.tsindex 9096a344..dc7e0d0a 100644--- a/apps/server/src/session-archive/touched.ts+++ b/apps/server/src/session-archive/touched.ts@@ -7,12 +7,13 @@* disk, and scanTouchedFiles turns them into archive entries at capture* time: regular files inside the root only, never through a symlinked* folder, a touched folder never expanded, with the whole-folder scan's- * exclusions, and never a file git ignores (`ignore.ts`). Nothing outside the root is ever opened or stat'ed: a path is+ * exclusions, and never a file git ignores (`ignore.ts`) unless the caller+ * keeps ignored files (includeIgnored: the archiver always does). Nothing outside the root is ever opened or stat'ed: a path is* checked lexically first, then walked from the root one lstat at a time.*/import { createHash } from "node:crypto";import type { BigIntStats } from "node:fs";-import { appendFile, lstat, open, readFile, readlink, realpath, rm, writeFile } from "node:fs/promises";+import { appendFile, lstat, open, readdir, readFile, readlink, realpath, rm, writeFile } from "node:fs/promises";import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";import {@@ -38,6 +39,7 @@ import {} from "./manifest.js";import { stateKey } from "./files.js";import { touchedIgnoredPaths } from "./ignore.js";+import { REGENERABLE_DIR_NAMES } from "../session-uploader.js";import type { ArchiveLog } from "./upload.js";/** A session keeps at most this many touched paths; later ones are not archived (one log line). */@@ -60,6 +62,8 @@ export type TouchedScanOptions = {hashCache?: ArchiveHashCache;/** Stops the scan between paths: it then rejects with the signal's reason. */signal?: AbortSignal;+ /** Gitignored files are kept (the reproducible-task rule: touched files and lockfiles are archived even when ignored). */+ includeIgnored?: boolean;/** Tests: every absolute path the scan lstats, reads a link of or opens, before it does. */onAccess?: (path: string) => void;};@@ -111,6 +115,49 @@ function linkTargetInside(path: string, target: string, roots: readonly string[]return null;}+/** Lockfiles and dependency pins, archived in full wherever they sit (outside the regenerable folders). */+const LOCKFILE_NAMES = new Set([+ "package-lock.json", "npm-shrinkwrap.json", "yarn.lock", "pnpm-lock.yaml", "bun.lock", "bun.lockb", "deno.lock",+ "Cargo.lock", "poetry.lock", "Pipfile.lock", "uv.lock", "pdm.lock", "pixi.lock", "conda-lock.yml",+ "Gemfile.lock", "composer.lock", "go.sum", "go.work.sum", "mix.lock", "pubspec.lock", "Package.resolved",+ "packages.lock.json", "gradle.lockfile", "flake.lock", "Podfile.lock", "Cartfile.resolved",+]);+const LOCKFILE_MAX_DEPTH = 6;+const LOCKFILE_MAX_DIRS = 5_000;+const LOCKFILE_MAX_FOUND = 200;++/**+ * The lockfiles under `root`, workspace-relative and portable: a bounded walk+ * (depth, folders visited, results) that never follows a symlink and skips+ * `.git/` and the regenerable folders. scanTouchedFiles checks each one again+ * before anything is archived. Never throws.+ */+export async function findLockfiles(root: string, signal?: AbortSignal): Promise<string[]> {+ const found: string[] = [];+ let visited = 0;+ const walk = async (rel: string, depth: number): Promise<void> => {+ if (depth > LOCKFILE_MAX_DEPTH || visited >= LOCKFILE_MAX_DIRS || found.length >= LOCKFILE_MAX_FOUND || signal?.aborted) return;+ visited += 1;+ let entries;+ try {+ entries = await readdir(rel ? join(root, ...rel.split("/")) : root, { withFileTypes: true });+ } catch {+ return;+ }+ entries.sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0));+ for (const entry of entries) {+ const path = rel ? `${rel}/${entry.name}` : entry.name;+ if (entry.isFile() && LOCKFILE_NAMES.has(entry.name)) {+ if (found.length < LOCKFILE_MAX_FOUND) found.push(path);+ } else if (entry.isDirectory() && entry.name !== ".git" && !REGENERABLE_DIR_NAMES.has(entry.name)) {+ await walk(path, depth + 1);+ }+ }+ };+ await walk("", 0);+ return found;+}+/*** Pass 1 of a touched-files capture: each path (and each symlink target* inside the root it leads to) walked from the root with lstat, never@@ -235,8 +282,8 @@ export async function scanTouchedFiles(root: string, paths: Iterable<string>, opawait forEachBounded([...paths], STAT_CONCURRENCY, (path) => visit(path, 0));- // Git's answer for just the files found, so the folder is never walked.- const ignoredPaths = await touchedIgnoredPaths(base, entries.map((entry) => entry.path), signal);+ // Git's answer for just the files found, so the folder is never walked (none asked when ignored files are kept).+ const ignoredPaths = options.includeIgnored === true ? new Set<string>() : await touchedIgnoredPaths(base, entries.map((entry) => entry.path), signal);signal?.throwIfAborted();if (ignoredPaths.size > 0) {let kept = 0;@@ -534,7 +581,12 @@ export class TouchedPathStore {session.torn = loaded.torn;}const known = session.known;- if (session.pending.size === 0) return;+ if (session.pending.size === 0) {+ // Nothing to write to a file that has no header yet: it is loaded (and started) again by the next write,+ // else that write would append to a file without its header, unreadable after a restart.+ if (header) session.known = null;+ return;+ }const batch = [...session.pending];session.pending = new Set();session.writing = batch;diff --git a/apps/server/src/session-uploader.test.ts b/apps/server/src/session-uploader.test.tsindex 63516033..35e6b156 100644--- a/apps/server/src/session-uploader.test.ts+++ b/apps/server/src/session-uploader.test.ts@@ -1564,11 +1564,11 @@ describe("session uploader trace additions", () => {expect(photo?.data).toEqual({ name: "photo.png", mime: "image/png", bytes: 12, sha256: sha256("png"), text: null, text_truncated: false });});- test("emits artifact events for untracked outputs a turn creates or modifies", async () => {+ test("emits artifact events for untracked outputs a turn creates or modifies, gitignored ones outside the regenerable folders too", async () => {const root = await mkdtemp(join(tmpdir(), "omnirush-upload-artifacts-"));roots.push(root);await git(root, "init", "-q");- await writeFile(join(root, ".gitignore"), "ignored/\n");+ await writeFile(join(root, ".gitignore"), "ignored/\nnode_modules/\n");await writeFile(join(root, "tracked.txt"), "tracked\n");await writeFile(join(root, "stale.txt"), "already there\n");await git(root, "add", ".gitignore", "tracked.txt");@@ -1587,13 +1587,16 @@ describe("session uploader trace additions", () => {await writeFile(join(root, "notes.md"), "generated notes\n");await mkdir(join(root, "ignored"));await writeFile(join(root, "ignored", "cache.txt"), "ignored output\n");+ await mkdir(join(root, "node_modules", "dep"), { recursive: true });+ await writeFile(join(root, "node_modules", "dep", "index.js"), "module.exports = 1;\n");sessionUploader.captureSnapshot(sessionId, "turn_completed");await sessionUploader.idle(sessionId);sessionUploader.flushTrace(sessionId);await sessionUploader.stop();const artifacts = traceEvents(uploads).filter((event) => event.type === "artifact").map((event) => event.data);- expect(artifacts.map((artifact) => artifact?.path).sort()).toEqual(["notes.md", "out/report.bin"]);+ expect(artifacts.map((artifact) => artifact?.path).sort()).toEqual(["ignored/cache.txt", "notes.md", "out/report.bin"]);+ expect(artifacts.find((artifact) => artifact?.path === "ignored/cache.txt")).toEqual({ path: "ignored/cache.txt", sha256: sha256("ignored output\n"), bytes: 15 });expect(artifacts.find((artifact) => artifact?.path === "out/report.bin")).toEqual({path: "out/report.bin",sha256: createHash("sha256").update(Buffer.from([0, 1, 2, 3, 255])).digest("hex"),diff --git a/apps/server/src/session-uploader.ts b/apps/server/src/session-uploader.tsindex 60010d3f..88c97b74 100644--- a/apps/server/src/session-uploader.ts+++ b/apps/server/src/session-uploader.ts@@ -2326,6 +2326,24 @@ async function gitPriorityPaths(root: string): Promise<Set<string>> {return new Set(parseGitStatus(status.stdout.toString("utf8"), prefix).entries.map((entry) => entry.path));}+/**+ * Folders whose content is reinstalled or rebuilt from the lockfiles (the+ * reproducible-task rule's "not needed"): a touched or locked path in one is+ * never archived for being ignored, and an ignored output in one is never an+ * artifact.+ */+export const REGENERABLE_DIR_NAMES = new Set([+ "node_modules", ".venv", "venv", "__pycache__", ".tox", ".nox", ".mypy_cache", ".pytest_cache", ".ruff_cache",+ "dist", "build", "target", "out", ".next", ".nuxt", ".svelte-kit", ".turbo", ".parcel-cache", ".cache",+ ".gradle", ".dart_tool", "Pods", "DerivedData", "vendor", "bower_components", "site-packages",+]);++/** Whether a workspace-relative portable path lies in a regenerable folder (or is `.git/` content). */+export function inRegenerableDir(path: string): boolean {+ const parts = path.split("/");+ return parts.slice(0, -1).some((part) => part === ".git" || REGENERABLE_DIR_NAMES.has(part));+}+/*** Files inside the workspace that git does not track but does not ignore* either: the outputs an agent leaves next to the source. A workspace without@@ -2344,9 +2362,48 @@ async function listUntrackedFiles(root: string): Promise<string[]> {}}+const MAX_IGNORED_OUTPUT_FILES = 5_000;++/**+ * The gitignored untracked files outside the regenerable folders (a build's+ * `outputs/plot.png`, a downloaded `data.pdf`): `git ls-files --ignored+ * --directory` names a wholly ignored folder once, and only the folders that+ * are not regenerable are listed further. Empty outside a git work tree+ * (walkFallback already lists every file there) or when git fails.+ */+async function listIgnoredOutputFiles(root: string): Promise<string[]> {+ const listed = async (args: string[]): Promise<string[]> => {+ const { stdout } = await execFileAsync("git", ["-C", root, "ls-files", "--others", "--ignored", "--exclude-standard", "-z", ...args], {+ encoding: "buffer",+ maxBuffer: 16 * 1024 * 1024,+ timeout: 15_000,+ });+ return Buffer.from(stdout).toString("utf8").split("\0").filter(Boolean);+ };+ try {+ const paths: string[] = [];+ for (const path of await listed(["--directory"])) {+ if (paths.length >= MAX_IGNORED_OUTPUT_FILES) break;+ if (!path.endsWith("/")) {+ if (!inRegenerableDir(path)) paths.push(path);+ continue;+ }+ const dir = path.slice(0, -1);+ if (inRegenerableDir(`${dir}/x`)) continue;+ for (const inner of await listed(["--", dir])) {+ if (paths.length >= MAX_IGNORED_OUTPUT_FILES) break;+ if (!inRegenerableDir(inner)) paths.push(inner);+ }+ }+ return filterListing(paths).paths;+ } catch {+ return [];+ }+}+async function artifactStats(root: string): Promise<Map<string, ArtifactStat>> {const stats = new Map<string, ArtifactStat>();- for (const path of await listUntrackedFiles(root)) {+ for (const path of [...(await listUntrackedFiles(root)), ...(await listIgnoredOutputFiles(root))]) {try {if (workspaceRelativePath(root, path) === null) continue;const absolute = resolve(root, path);@@ -4320,8 +4377,8 @@ export class SessionUploader {}/**- * Emits one "artifact" event per untracked-but-not-ignored file the turn- * created or modified. Tracked files are already covered by the change+ * Emits one "artifact" event per untracked file (gitignored ones outside the+ * regenerable folders too) the turn created or modified. Tracked files are already covered by the change* snapshot and touched_paths; this adds the outputs git would not list.*/private async captureArtifacts(state: SessionState): Promise<void> {