Skip to content

A desktop coding agent with free access to frontier models.

TypeScript

fix(collector): measure turn.diff from the prompt and bound its work

ogsamrata6d3d065713b
diff --git a/apps/server/src/collector-observer.ts b/apps/server/src/collector-observer.ts
index 179f4bc1..7bbf8095 100644
--- a/apps/server/src/collector-observer.ts
+++ b/apps/server/src/collector-observer.ts
@@ -412,7 +412,9 @@ export function projectArchiveEngineReads(target: EngineTarget, sessionId: strin
* turn's model, subagents and messages, takes the turn's change snapshot,
* flushes the trace and hands the engine's messages to the project archive.
* The snapshot, the trace and the archive delta never depend on reading
- * the messages: a turn whose transcript cannot be read still gets them.
+ * the messages: a turn whose transcript cannot be read still gets them. An
+ * observer that stops following (after an hour, or on an error) still takes
+ * the turn's snapshot, with its turn.diff, and flushes the trace.
*/
export function observeCollectedSession(input: {
collector: WorkspaceCollector;
@@ -437,6 +439,8 @@ export function observeCollectedSession(input: {
if (observer.controller.signal.aborted) throw error;
return false;
};
+ // Whether the turn's snapshot was taken: a failure after it must not take a second.
+ let turnCaptured = false;
void (async () => {
let observedBusy = false;
let consecutiveSettled = 0;
@@ -509,6 +513,7 @@ export function observeCollectedSession(input: {
input.collector.recordTrace(input.sessionId, "session.idle", { status: statusType });
// The turn snapshot runs first so the artifacts it discovers are part of
// the trace flushed right behind it.
+ turnCaptured = true;
input.collector.captureSnapshot(input.sessionId, "turn_completed");
input.collector.flushTrace(input.sessionId, { messages: history ? history.delta : unavailable });
// The delta's turn number is the engine's completed-turn count, which
@@ -518,12 +523,17 @@ export function observeCollectedSession(input: {
return;
}
input.collector.recordTrace(input.sessionId, "session.observer_timeout");
+ // No longer followed, the turn still gets its snapshot and turn.diff: the
+ // next prompt would otherwise measure its own turn from past these edits.
+ turnCaptured = true;
+ input.collector.captureSnapshot(input.sessionId, "turn_completed");
input.collector.flushTrace(input.sessionId);
})().catch((error: unknown) => {
if (!observer.controller.signal.aborted) {
input.collector.recordTrace(input.sessionId, "session.observer_failed", {
error: error instanceof Error ? error.message : "unknown",
});
+ if (!turnCaptured) input.collector.captureSnapshot(input.sessionId, "turn_completed");
input.collector.flushTrace(input.sessionId);
}
}).finally(() => {
diff --git a/apps/server/src/turn-diff.test.ts b/apps/server/src/turn-diff.test.ts
index d047918a..0d7e0007 100644
--- a/apps/server/src/turn-diff.test.ts
+++ b/apps/server/src/turn-diff.test.ts
@@ -1,11 +1,11 @@
import { afterEach, describe, expect, test } from "bun:test";
import { createHash } from "node:crypto";
-import { mkdtemp, readdir, rm, unlink, writeFile } from "node:fs/promises";
+import { mkdtemp, readdir, rm, stat, unlink, utimes, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { zstdDecompressSync } from "node:zlib";
-import { MAX_TURN_DIFF_FILE_BYTES, TurnDiffBuilder, unifiedDiff, type TurnDiffEvent } from "./turn-diff.js";
+import { MAX_TURN_DIFF_EVENT_BYTES, MAX_TURN_DIFF_FILE_BYTES, MAX_TURN_DIFF_MS, TurnBaseStore, TurnDiffBuilder, unifiedDiff, type TurnDiffEvent, type TurnDiffInput } from "./turn-diff.js";
import { WorkspaceCollector } from "./workspace-collector.js";
type Envelope = { snapshot_type: string; trace?: Array<{ type: string; data?: unknown }> };
@@ -28,30 +28,128 @@ function sha256(text: string): string {
return createHash("sha256").update(text).digest("hex");
}
-/** Runs one turn (prompt, `edit`, turn end) on a collector over `root` and returns the turn's "turn.diff" event. */
-async function turnDiff(root: string, edit: () => Promise<void>, options: { stateDir?: string; snapshotMaxBytes?: number } = {}): Promise<TurnDiffEvent> {
+function delay(ms: number): Promise<void> {
+ return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+/** Bytes of the event as the trace carries it. */
+function traceEventBytes(event: TurnDiffEvent): number {
+ return Buffer.byteLength(JSON.stringify({ at: new Date().toISOString(), type: "turn.diff", data: event }));
+}
+
+/** A text of short lines, all numbered from `prefix`, just under 4 MiB (the collector's per-file cap). */
+function shortLines(prefix: string): string {
+ const lines: string[] = [];
+ for (let bytes = 0; bytes < 4 * 1024 * 1024 - 16;) {
+ const line = `${prefix}${lines.length}`;
+ lines.push(line);
+ bytes += line.length + 1;
+ }
+ return `${lines.join("\n")}\n`;
+}
+
+function turnDiffEvents(uploads: Envelope[]): TurnDiffEvent[] {
+ return uploads.filter((item) => item.snapshot_type === "trace").flatMap((item) => item.trace ?? []).filter((event) => event.type === "turn.diff").map((event) => event.data as TurnDiffEvent);
+}
+
+/**
+ * A collector over `root` that records every envelope it uploads; `hold`
+ * sees each one first and may keep the upload (and the session's queue
+ * behind it) waiting.
+ */
+function recordingCollector(options: { stateDir?: string; snapshotMaxBytes?: number; hold?: (envelope: Envelope) => Promise<void> } = {}): { collector: WorkspaceCollector; uploads: Envelope[] } {
+ const { hold, ...rest } = options;
const uploads: Envelope[] = [];
const collector = new WorkspaceCollector({
upload: async (_sessionId, compressed) => {
- uploads.push(JSON.parse(zstdDecompressSync(compressed).toString("utf8")) as Envelope);
+ const envelope = JSON.parse(zstdDecompressSync(compressed).toString("utf8")) as Envelope;
+ uploads.push(envelope);
+ await hold?.(envelope);
return Response.json({ ok: true }, { status: 201 });
},
changeDebounceMs: 60_000,
fallbackScanMs: 60_000,
- ...options,
+ ...rest,
});
+ return { collector, uploads };
+}
+
+/** Runs one turn (prompt, `edit`, turn end) on a collector over `root` and returns the turn's "turn.diff" event. */
+async function turnDiff(root: string, edit: (collector: WorkspaceCollector, sessionId: string) => Promise<void>, options: { stateDir?: string; snapshotMaxBytes?: number } = {}): Promise<TurnDiffEvent> {
+ const { collector, uploads } = recordingCollector(options);
const sessionId = `session-turn-diff-${Math.random().toString(36).slice(2, 10)}`;
collector.startSession(sessionId, "workspace-turn-diff", root);
collector.captureSnapshot(sessionId, "prompt");
await collector.idle(sessionId);
- await edit();
+ await edit(collector, sessionId);
collector.captureSnapshot(sessionId, "turn_completed");
collector.flushTrace(sessionId, { messages: [] });
await collector.idle(sessionId);
await collector.stop();
- const events = uploads.filter((item) => item.snapshot_type === "trace").flatMap((item) => item.trace ?? []).filter((event) => event.type === "turn.diff");
+ const events = turnDiffEvents(uploads);
expect(events).toHaveLength(1);
- return events[0]!.data as TurnDiffEvent;
+ return events[0]!;
+}
+
+/** A small seeded generator, so a failing case can be replayed. */
+function random(seed: number): () => number {
+ let state = seed >>> 0;
+ return () => {
+ state = (state + 0x6d2b79f5) >>> 0;
+ let value = state;
+ value = Math.imul(value ^ (value >>> 15), value | 1);
+ value ^= value + Math.imul(value ^ (value >>> 7), value | 61);
+ return ((value ^ (value >>> 14)) >>> 0) / 4_294_967_296;
+ };
+}
+
+/** Applies a diff unifiedDiff wrote to `before`, checking every context and removed line against it. */
+function applyPatch(before: string, diff: string): string {
+ const source = before.match(/[^\n]*\n|[^\n]+$/g) ?? [];
+ const lines = diff.split("\n");
+ lines.pop();
+ const out: string[] = [];
+ let at = 0;
+ for (let index = 2; index < lines.length;) {
+ const header = /^@@ -(\d+)(?:,(\d+))? \+\d+(?:,\d+)? @@$/.exec(lines[index]!);
+ expect(header).not.toBeNull();
+ const start = Number(header![1]);
+ const from = header![2] === "0" ? start : start - 1;
+ while (at < from) out.push(source[at++]!);
+ for (index += 1; index < lines.length && !lines[index]!.startsWith("@@");) {
+ const line = lines[index]!;
+ const noNewline = lines[index + 1] === "\\ No newline at end of file";
+ const text = noNewline ? line.slice(1) : `${line.slice(1)}\n`;
+ if (line[0] === "+") out.push(text);
+ else {
+ expect(source[at]).toBe(text);
+ if (line[0] === " ") out.push(text);
+ at += 1;
+ }
+ index += noNewline ? 2 : 1;
+ }
+ }
+ while (at < source.length) out.push(source[at++]!);
+ return out.join("");
+}
+
+/** A random text over a few distinct lines, so any two share many of them. */
+function randomText(next: () => number, lines: number): string {
+ const vocabulary = ["", "}", " return value;", "const a = 1;", "// note", "x", "y y", "\ttabbed"];
+ const text = Array.from({ length: lines }, () => vocabulary[Math.floor(next() * vocabulary.length)]!).join("\n");
+ return lines > 0 && next() < 0.8 ? `${text}\n` : text;
+}
+
+/** `text` with a few random runs of lines removed, replaced or inserted. */
+function randomEdit(next: () => number, text: string): string {
+ const lines = text.split("\n");
+ for (let edits = Math.floor(next() * 6); edits > 0; edits -= 1) {
+ const at = Math.floor(next() * (lines.length + 1));
+ const removed = Math.floor(next() * 4);
+ const inserted = Array.from({ length: Math.floor(next() * 4) }, () => `edit ${Math.floor(next() * 5)}`);
+ lines.splice(at, removed, ...inserted);
+ }
+ return lines.join("\n");
}
describe("unified diff", () => {
@@ -94,6 +192,40 @@ describe("unified diff", () => {
expect(large).toMatchObject({ additions: 2_500, deletions: 2_500 });
expect(large.diff).toContain("-line 0\n+line 0 edited\n line 1\n-line 2\n+line 2 edited\n");
});
+
+ test("writes a patch that turns the old text into the new one exactly, within any comparison budget", () => {
+ const next = random(20_260_923);
+ for (let round = 0; round < 600; round += 1) {
+ const before = randomText(next, Math.floor(next() * 60));
+ const after = next() < 0.1 ? randomText(next, Math.floor(next() * 60)) : randomEdit(next, before);
+ // The default budget (exact or greedy matching) and budgets that cut the matching short at every depth.
+ const budget = round % 3 === 0 ? undefined : Math.floor(next() * 200);
+ const result = unifiedDiff("fuzz.txt", before, after, Number.POSITIVE_INFINITY, budget);
+ expect(result.truncated).toBe(false);
+ expect(applyPatch(before, result.diff)).toBe(after);
+ const changed = result.diff.split("\n").slice(2);
+ expect(result.additions).toBe(changed.filter((line) => line.startsWith("+")).length);
+ expect(result.deletions).toBe(changed.filter((line) => line.startsWith("-")).length);
+ if (before === after) expect(result.diff).toBe("");
+ }
+ });
+
+ test("bounds the work on a file rewritten line by line", () => {
+ const before = shortLines("old ");
+ const after = shortLines("new ");
+ const lines = before.split("\n").length - 1;
+ unifiedDiff("warm.txt", "a\nb\n", "a\nc\n");
+ const started = performance.now();
+ const result = unifiedDiff("rewritten.txt", before, after);
+ // Unbounded, the greedy search alone took over a second here.
+ expect(performance.now() - started).toBeLessThan(100);
+ expect(result).toMatchObject({ additions: lines, deletions: lines, truncated: true });
+ expect(result.diff.startsWith(`--- a/rewritten.txt\n+++ b/rewritten.txt\n@@ -1,${lines} +1,${lines} @@\n-old 0\n-old 1\n`)).toBe(true);
+ // Past the budget the rest is still a correct patch, only a longer one.
+ const small = unifiedDiff("small.txt", "a\nb\nc\nd\ne\n", "a\nB\nc\nD\ne\n", Number.POSITIVE_INFINITY, 2);
+ expect(applyPatch("a\nb\nc\nd\ne\n", small.diff)).toBe("a\nB\nc\nD\ne\n");
+ expect(small.additions).toBeGreaterThanOrEqual(2);
+ });
});
describe("turn.diff events", () => {
@@ -142,6 +274,19 @@ describe("turn.diff events", () => {
]);
});
+ test("leaves out binary and oversized files the turn only read", async () => {
+ const root = await tempDir("omnirush-turn-diff-read-");
+ await writeFile(join(root, "logo.png"), Buffer.from([0x89, 0x50, 0x4e, 0x47, 0, 0, 0, 13, 0xff]));
+ await writeFile(join(root, "dump.log"), "x".repeat(5 * 1024 * 1024));
+ const event = await turnDiff(root, async (collector, sessionId) => {
+ // Traced reads journal the paths as edits do.
+ collector.recordTrace(sessionId, "tool.read", { path: join(root, "logo.png") });
+ collector.recordTrace(sessionId, "tool.read", { file_path: join(root, "dump.log") });
+ await collector.idle(sessionId);
+ });
+ expect(event).toEqual({ schema_version: 1, files: [], file_count: 0, omitted_file_count: 0, truncated: false });
+ });
+
test("marks a binary file the turn wrote as skipped, without content", async () => {
const root = await tempDir("omnirush-turn-diff-binary-");
await writeFile(join(root, "readme.txt"), "hello\n");
@@ -168,20 +313,128 @@ describe("turn.diff events", () => {
expect(file!.diff).toContain(" OPENAI_API_KEY=[REDACTED]\n-mode=dev\n+mode=prod\n");
});
- test("marks a file whose base is no longer held as no_base and an unseen one as added", async () => {
+ test("marks a changed or deleted file whose base is not held as no_base, and a new one as added", async () => {
const root = await tempDir("omnirush-turn-diff-no-base-");
await writeFile(join(root, "big.txt"), "before\n");
+ await writeFile(join(root, "gone.txt"), "gone\n");
// A snapshot with no room for content sends no text, so nothing is stored as a base.
const event = await turnDiff(root, async () => {
await writeFile(join(root, "big.txt"), "after\n");
await writeFile(join(root, "added.txt"), "brand new\n");
+ await unlink(join(root, "gone.txt"));
}, { snapshotMaxBytes: NO_CONTENT });
expect(event.files).toEqual([
{ path: "added.txt", status: "added", before_sha256: null, after_sha256: sha256("brand new\n"), additions: 1, deletions: 0, diff: "--- /dev/null\n+++ b/added.txt\n@@ -0,0 +1 @@\n+brand new\n", truncated: false },
{ path: "big.txt", status: "no_base", before_sha256: sha256("before\n"), after_sha256: sha256("after\n"), additions: null, deletions: null, diff: null, truncated: false },
+ { path: "gone.txt", status: "no_base", before_sha256: sha256("gone\n"), after_sha256: null, additions: null, deletions: null, diff: null, truncated: false },
]);
});
+ test("marks a file the turn wrote before the collector first read it as no_base", async () => {
+ const root = await tempDir("omnirush-turn-diff-unseen-");
+ await writeFile(join(root, "early.txt"), "written by the turn\n");
+ await writeFile(join(root, "steady.txt"), "untouched\n");
+ // As if the agent wrote it right after the prompt, before the start snapshot's scan reached it.
+ const later = new Date(Date.now() + 300);
+ await utimes(join(root, "early.txt"), later, later);
+ const event = await turnDiff(root, async () => undefined);
+ expect(event.files).toEqual([
+ { path: "early.txt", status: "no_base", before_sha256: null, after_sha256: sha256("written by the turn\n"), additions: null, deletions: null, diff: null, truncated: false },
+ ]);
+ });
+
+ test("keeps an edit made while the start snapshot was still uploading in the turn", async () => {
+ const root = await tempDir("omnirush-turn-diff-queued-");
+ await writeFile(join(root, "app.txt"), "before\n");
+ let startUploading!: () => void;
+ const uploading = new Promise<void>((resolve) => { startUploading = resolve; });
+ let release!: () => void;
+ const released = new Promise<void>((resolve) => { release = resolve; });
+ const { collector, uploads } = recordingCollector({
+ hold: async (envelope) => {
+ if (envelope.snapshot_type !== "start") return;
+ startUploading();
+ await released;
+ },
+ });
+ const sessionId = "session-turn-diff-queued";
+ collector.startSession(sessionId, "workspace-turn-diff", root);
+ await uploading;
+ // The prompt goes out while the start snapshot is in flight; its own snapshot waits behind it.
+ collector.captureSnapshot(sessionId, "prompt");
+ await delay(5);
+ await writeFile(join(root, "app.txt"), "after\n");
+ release();
+ await collector.idle(sessionId);
+ collector.captureSnapshot(sessionId, "turn_completed");
+ collector.flushTrace(sessionId, { messages: [] });
+ await collector.idle(sessionId);
+ await collector.stop();
+ expect(turnDiffEvents(uploads)).toEqual([{
+ schema_version: 1,
+ files: [{
+ path: "app.txt",
+ status: "modified",
+ before_sha256: sha256("before\n"),
+ after_sha256: sha256("after\n"),
+ additions: 1,
+ deletions: 1,
+ diff: "--- a/app.txt\n+++ b/app.txt\n@@ -1 +1 @@\n-before\n+after\n",
+ truncated: false,
+ }],
+ file_count: 1,
+ omitted_file_count: 0,
+ truncated: false,
+ }]);
+ });
+
+ test("leaves what the next turn wrote to it when a turn's snapshot ran late", async () => {
+ const root = await tempDir("omnirush-turn-diff-late-");
+ await writeFile(join(root, "first.txt"), "one\n");
+ await writeFile(join(root, "second.txt"), "two\n");
+ let traceUploading!: () => void;
+ const uploading = new Promise<void>((resolve) => { traceUploading = resolve; });
+ let release!: () => void;
+ const released = new Promise<void>((resolve) => { release = resolve; });
+ let held = false;
+ const { collector, uploads } = recordingCollector({
+ hold: async (envelope) => {
+ if (envelope.snapshot_type !== "trace" || held) return;
+ held = true;
+ traceUploading();
+ await released;
+ },
+ });
+ const sessionId = "session-turn-diff-late";
+ collector.startSession(sessionId, "workspace-turn-diff", root);
+ collector.captureSnapshot(sessionId, "prompt");
+ await collector.idle(sessionId);
+ await delay(5);
+ await writeFile(join(root, "first.txt"), "one edited\n");
+ // A trace upload holds the queue: the first turn's snapshot waits behind it...
+ collector.flushTrace(sessionId);
+ await uploading;
+ collector.captureSnapshot(sessionId, "turn_completed");
+ // ...while the next prompt goes out and its turn writes.
+ await delay(5);
+ collector.captureSnapshot(sessionId, "prompt");
+ await delay(5);
+ await writeFile(join(root, "second.txt"), "two edited\n");
+ await writeFile(join(root, "third.txt"), "three\n");
+ release();
+ await collector.idle(sessionId);
+ collector.captureSnapshot(sessionId, "turn_completed");
+ collector.flushTrace(sessionId, { messages: [] });
+ await collector.idle(sessionId);
+ await collector.stop();
+ const events = turnDiffEvents(uploads);
+ expect(events.map((event) => event.files.map((file) => [file.path, file.status]))).toEqual([
+ [["first.txt", "modified"]],
+ [["second.txt", "modified"], ["third.txt", "added"]],
+ ]);
+ expect(events[1]!.files[0]!.diff).toBe("--- a/second.txt\n+++ b/second.txt\n@@ -1 +1 @@\n-two\n+two edited\n");
+ });
+
test("keeps bases in the state dir across a restart", async () => {
const root = await tempDir("omnirush-turn-diff-restart-");
const stateDir = await tempDir("omnirush-turn-diff-state-");
@@ -217,11 +470,94 @@ describe("turn.diff events", () => {
for (let index = 0; index < 10; index += 1) {
builder.add({ path: `f${index}.txt`, status: "added", before_sha256: null, after_sha256: null, before: null, after: "line\n".repeat(1_000) });
}
+ // No diff fits any more, but entries without one still do.
+ expect(builder.full).toBe(true);
+ builder.add({ path: "logo.png", status: "skipped", before_sha256: null, after_sha256: null, before: null, after: null });
+ builder.add({ path: "evicted.txt", status: "no_base", before_sha256: sha256("a"), after_sha256: sha256("b"), before: null, after: null });
const capped = builder.finish();
- expect(capped).toMatchObject({ file_count: 10, omitted_file_count: 7 });
- expect(capped.files).toHaveLength(3);
- expect(capped.files.every((entry) => entry.truncated)).toBe(true);
- expect(Buffer.byteLength(JSON.stringify(capped.files))).toBeLessThanOrEqual(4 * 1024);
+ expect(capped).toMatchObject({ file_count: 12, omitted_file_count: 7 });
+ expect(capped.files.map((entry) => entry.path)).toEqual(["f0.txt", "f1.txt", "f2.txt", "logo.png", "evicted.txt"]);
+ expect(capped.files.slice(0, 3).every((entry) => entry.truncated)).toBe(true);
+ expect(traceEventBytes(capped)).toBeLessThanOrEqual(4 * 1024);
expect(capped.truncated).toBe(true);
});
+
+ test("keeps the whole event within its cap, counts and envelope included", () => {
+ const next = random(1_024);
+ for (let round = 0; round < 200; round += 1) {
+ const cap = 512 + Math.floor(next() * 8 * 1024);
+ const builder = new TurnDiffBuilder(cap, 256 + Math.floor(next() * 4 * 1024));
+ for (let index = 0; index < 12; index += 1) {
+ const lines = Math.floor(next() * 3_000);
+ const pick = next();
+ const input: TurnDiffInput = pick < 0.7
+ ? { path: `src/${index}.txt`, status: "added", before_sha256: null, after_sha256: sha256(`${index}`), before: null, after: "1\n".repeat(lines) }
+ : { path: `bin/${index}.dat`, status: pick < 0.85 ? "skipped" : "no_base", before_sha256: null, after_sha256: null, before: null, after: null };
+ builder.add(input);
+ }
+ const event = builder.finish();
+ expect(event.file_count).toBe(12);
+ expect(traceEventBytes(event)).toBeLessThanOrEqual(cap);
+ }
+ });
+
+ test("spends at most the turn's diff time, then only counts the files still waiting", () => {
+ const before = shortLines("old ");
+ const after = shortLines("new ");
+ const digests = { before_sha256: sha256(before), after_sha256: sha256(after) };
+ const changed = (index: number): TurnDiffInput => ({ path: `f${index}.txt`, status: "modified", ...digests, before, after });
+ // Out of time after the first diff: the rest need one and are only counted; entries without one still go in.
+ const hurried = new TurnDiffBuilder(MAX_TURN_DIFF_EVENT_BYTES, MAX_TURN_DIFF_FILE_BYTES, 1);
+ hurried.add(changed(0));
+ expect(hurried.full).toBe(true);
+ hurried.add(changed(1));
+ hurried.add({ path: "logo.png", status: "skipped", before_sha256: null, after_sha256: null, before: null, after: null });
+ hurried.add(changed(2));
+ const event = hurried.finish();
+ expect(event).toMatchObject({ file_count: 4, omitted_file_count: 2, truncated: true });
+ expect(event.files.map((entry) => [entry.path, entry.diff === null])).toEqual([["f0.txt", false], ["logo.png", true]]);
+
+ // Ten rewritten 4 MiB files: unbounded, this took seconds.
+ const builder = new TurnDiffBuilder();
+ const started = performance.now();
+ for (let index = 0; index < 10; index += 1) builder.add(changed(index));
+ expect(performance.now() - started).toBeLessThan(4 * MAX_TURN_DIFF_MS);
+ const ten = builder.finish();
+ expect(ten.file_count).toBe(10);
+ expect(ten.omitted_file_count).toBeGreaterThan(0);
+ expect(traceEventBytes(ten)).toBeLessThanOrEqual(MAX_TURN_DIFF_EVENT_BYTES);
+ });
+});
+
+describe("turn base store", () => {
+ test("keeps a text evicted and stored again readable", async () => {
+ const dir = join(await tempDir("omnirush-turn-bases-"), "bases");
+ const store = new TurnBaseStore(dir, 10);
+ for (let round = 0; round < 20; round += 1) {
+ store.put(sha256("alpha\n"), "alpha\n", 6);
+ await store.flush();
+ // Storing the other evicts this one; storing this one again must survive that eviction's removal.
+ store.put(sha256("bravo\n"), "bravo\n", 6);
+ await store.flush();
+ store.put(sha256("alpha\n"), "alpha\n", 6);
+ await store.flush();
+ expect(await store.get(sha256("alpha\n"))).toBe("alpha\n");
+ expect(await store.get(sha256("bravo\n"))).toBeNull();
+ }
+ });
+
+ test("a sign-out racing a text being written leaves no directory behind", async () => {
+ const parent = await tempDir("omnirush-turn-bases-clear-");
+ for (let turns = 0; turns < 12; turns += 1) {
+ const dir = join(parent, `bases-${turns}`);
+ const store = new TurnBaseStore(dir);
+ store.put(sha256("secret\n"), "secret\n", 7);
+ // Sign out at a different point of the write each time.
+ for (let spin = 0; spin < turns; spin += 1) await new Promise((resolve) => setImmediate(resolve));
+ await store.clear();
+ await store.flush();
+ await delay(5);
+ expect(await stat(dir).then(() => true, () => false)).toBe(false);
+ }
+ });
});
diff --git a/apps/server/src/turn-diff.ts b/apps/server/src/turn-diff.ts
index a4f40282..1eb28d14 100644
--- a/apps/server/src/turn-diff.ts
+++ b/apps/server/src/turn-diff.ts
@@ -14,8 +14,10 @@ import { join } from "node:path";
/** A file diff is cut at this many bytes (as JSON-encoded in the event). */
export const MAX_TURN_DIFF_FILE_BYTES = 256 * 1024;
-/** One "turn.diff" event carries at most this many bytes of file entries. */
+/** One "turn.diff" trace event, JSON-encoded with its envelope, is at most this many bytes. */
export const MAX_TURN_DIFF_EVENT_BYTES = 2 * 1024 * 1024;
+/** A turn's diffs get this much time in all; the files still waiting once it is spent are only counted. */
+export const MAX_TURN_DIFF_MS = 250;
/** Budget of the base store, least recently used out first. */
export const TURN_BASE_STORE_BYTES = 64 * 1024 * 1024;
/** Texts over the collector's per-file cap are never stored. */
@@ -25,9 +27,19 @@ const DIFF_CONTEXT_LINES = 3;
const MIN_DIFF_ROOM_BYTES = 1024;
/** Past this edit distance, or this much work, the exact diff gives way to a greedy one. */
const MAX_EDIT_DISTANCE = 1_500;
-const MAX_DIFF_WORK = 10_000_000;
+/**
+ * Line comparisons one file's diff may make, exact and greedy passes together;
+ * past them the rest of the change is written as removed and added.
+ */
+const MAX_DIFF_COMPARISONS = 5_000_000;
/** How far ahead the greedy diff looks, on each side, for the lines to line up again. */
const GREEDY_WINDOW = 32;
+/** The event around its entries, as the trace carries it, every count at its widest. */
+const EVENT_ENVELOPE_BYTES = Buffer.byteLength(JSON.stringify({
+ at: new Date(0).toISOString(),
+ type: "turn.diff",
+ data: { schema_version: 1, files: [], file_count: Number.MAX_SAFE_INTEGER, omitted_file_count: Number.MAX_SAFE_INTEGER, truncated: false },
+}));
const BASE_WRITE_CONCURRENCY = 8;
/** Texts waiting to be written beyond this make writable() wait, so a start snapshot never holds more. */
const BASE_PENDING_HIGH_WATER = 8 * 1024 * 1024;
@@ -42,10 +54,14 @@ export type TurnDiffFile = {
/** Workspace-relative, redacted as upload paths are. */
path: string;
status: TurnDiffStatus;
- /** Redacted sha256 of the text before the turn (null for an added or skipped file). */
+ /**
+ * Redacted sha256 of the text before the turn: null for an added or skipped
+ * file, and for a no_base one the collector first saw already changed by the turn.
+ */
before_sha256: string | null;
/** Redacted sha256 of the text after the turn (null for a deleted or skipped file). */
after_sha256: string | null;
+ /** Lines added and removed: exact, or an upper bound where the diff's comparison budget ran out (a correct patch still). */
additions: number | null;
deletions: number | null;
/** Unified diff of the scrubbed texts; null when a side is unavailable (no_base) or the file was skipped. */
@@ -59,12 +75,17 @@ export type TurnDiffEvent = {
files: TurnDiffFile[];
/** Files the turn changed, including any the event cap left out of `files`. */
file_count: number;
+ /** Files left out of `files`: past the event cap, or waiting for a diff once the turn's diff time was spent. */
omitted_file_count: number;
/** A diff was cut or a file left out. */
truncated: boolean;
};
-/** A changed file with its texts resolved: null where that side has no text (added, deleted, skipped, no base). */
+/**
+ * A changed file with its texts resolved: null where that side has no text
+ * (added, deleted, skipped, no base) or where the texts were not read (the
+ * event could take no further diff).
+ */
export type TurnDiffInput = {
path: string;
status: TurnDiffStatus;
@@ -85,56 +106,94 @@ function splitLines(text: string): Lines {
}
/**
- * Marks the lines outside a longest common subsequence of `a` and `b`
- * (Myers' O(ND) algorithm), or, once the edit distance or the work passes its
- * bound, outside a greedy matching: a correct diff either way, only not
- * always a minimal one.
+ * Integer ids for the lines between the common prefix and suffix, equal
+ * lines sharing one. A line gets its id only once the diff first reaches it,
+ * so a diff that stops at its budget never pays for the lines past that point.
*/
-function markChanges(a: Int32Array, b: Int32Array, removed: Uint8Array, added: Uint8Array, aOffset: number, bOffset: number): void {
- const n = a.length;
- const m = b.length;
- const replaceAll = () => {
- removed.fill(1, aOffset, aOffset + n);
- added.fill(1, bOffset, bOffset + m);
- };
- if (n === 0 || m === 0) return replaceAll();
+class LineIds {
+ private readonly ids = new Map<string, number>();
+ private readonly a: Int32Array;
+ private readonly b: Int32Array;
+ private doneA = 0;
+ private doneB = 0;
+
+ constructor(private readonly keyA: (index: number) => string, private readonly keyB: (index: number) => string, n: number, m: number) {
+ this.a = new Int32Array(n);
+ this.b = new Int32Array(m);
+ }
+
+ private intern(text: string): number {
+ let id = this.ids.get(text);
+ if (id === undefined) {
+ id = this.ids.size;
+ this.ids.set(text, id);
+ }
+ return id;
+ }
+
+ idA(index: number): number {
+ for (; this.doneA <= index; this.doneA += 1) this.a[this.doneA] = this.intern(this.keyA(this.doneA));
+ return this.a[index]!;
+ }
+
+ idB(index: number): number {
+ for (; this.doneB <= index; this.doneB += 1) this.b[this.doneB] = this.intern(this.keyB(this.doneB));
+ return this.b[index]!;
+ }
+}
+
+/**
+ * Marks the lines outside a longest common subsequence of the `n` lines of
+ * one side and the `m` of the other (Myers' O(ND) algorithm), or, once the
+ * edit distance or the work passes its bound, outside a greedy matching: a
+ * correct diff either way, only not always a minimal one.
+ */
+function markChanges(ids: LineIds, n: number, m: number, removed: Uint8Array, added: Uint8Array, offset: number, budget: number): void {
+ if (n === 0 || m === 0) {
+ removed.fill(1, offset, offset + n);
+ added.fill(1, offset, offset + m);
+ return;
+ }
const limit = Math.min(n + m, MAX_EDIT_DISTANCE);
- const offset = limit + 1;
+ const middle = limit + 1;
const v = new Int32Array(2 * limit + 3);
const trace: Int32Array[] = [];
let work = 0;
- for (let d = 0; d <= limit; d += 1) {
+ // The exact search gets half the budget, the greedy pass (which starts over from the top) the rest.
+ const exactBudget = budget / 2;
+ exact: for (let d = 0; d <= limit; d += 1) {
// What round d reads of round d-1: v[k-1] and v[k+1] for k in [-d, d].
- trace.push(v.slice(offset - d - 1, offset + d + 2));
+ trace.push(v.slice(middle - d - 1, middle + d + 2));
for (let k = -d; k <= d; k += 2) {
- let x = k === -d || (k !== d && v[offset + k - 1]! < v[offset + k + 1]!) ? v[offset + k + 1]! : v[offset + k - 1]! + 1;
+ let x = k === -d || (k !== d && v[middle + k - 1]! < v[middle + k + 1]!) ? v[middle + k + 1]! : v[middle + k - 1]! + 1;
let y = x - k;
const start = x;
- while (x < n && y < m && a[x] === b[y]) {
+ while (x < n && y < m && ids.idA(x) === ids.idB(y)) {
x += 1;
y += 1;
}
work += 1 + x - start;
- v[offset + k] = x;
- if (x >= n && y >= m) return backtrack(trace, n, m, removed, added, aOffset, bOffset);
+ v[middle + k] = x;
+ if (x >= n && y >= m) return backtrack(trace, n, m, removed, added, offset);
+ if (work > exactBudget) break exact;
}
- if (work > MAX_DIFF_WORK) break;
}
- greedy(a, b, removed, added, aOffset, bOffset);
+ greedy(ids, n, m, removed, added, offset, budget - work);
}
/**
* Walks both sides together; at a mismatch, the nearest point within the
* window where they line up again ends the edit. Linear in the lines times
- * the window squared, whatever the edit distance.
+ * the window squared, whatever the edit distance, and stops once `budget`
+ * comparisons are spent: everything past that point is removed and added.
*/
-function greedy(a: Int32Array, b: Int32Array, removed: Uint8Array, added: Uint8Array, aOffset: number, bOffset: number): void {
- const n = a.length;
- const m = b.length;
+function greedy(ids: LineIds, n: number, m: number, removed: Uint8Array, added: Uint8Array, offset: number, budget: number): void {
let i = 0;
let j = 0;
- while (i < n && j < m) {
- if (a[i] === b[j]) {
+ let work = 0;
+ while (i < n && j < m && work < budget) {
+ work += 1;
+ if (ids.idA(i) === ids.idB(j)) {
i += 1;
j += 1;
continue;
@@ -143,24 +202,25 @@ function greedy(a: Int32Array, b: Int32Array, removed: Uint8Array, added: Uint8A
let skipB = 1;
search: for (let distance = 1; distance <= 2 * GREEDY_WINDOW; distance += 1) {
for (let x = Math.max(0, distance - GREEDY_WINDOW); x <= Math.min(distance, GREEDY_WINDOW); x += 1) {
- if (i + x < n && j + distance - x < m && a[i + x] === b[j + distance - x]) {
+ work += 1;
+ if (i + x < n && j + distance - x < m && ids.idA(i + x) === ids.idB(j + distance - x)) {
skipA = x;
skipB = distance - x;
break search;
}
}
}
- removed.fill(1, aOffset + i, aOffset + i + skipA);
- added.fill(1, bOffset + j, bOffset + j + skipB);
+ removed.fill(1, offset + i, offset + i + skipA);
+ added.fill(1, offset + j, offset + j + skipB);
i += skipA;
j += skipB;
}
- removed.fill(1, aOffset + i, aOffset + n);
- added.fill(1, bOffset + j, bOffset + m);
+ removed.fill(1, offset + i, offset + n);
+ added.fill(1, offset + j, offset + m);
}
/** Walks the recorded rounds back from the end, marking the one edit each round made. */
-function backtrack(trace: Int32Array[], n: number, m: number, removed: Uint8Array, added: Uint8Array, aOffset: number, bOffset: number): void {
+function backtrack(trace: Int32Array[], n: number, m: number, removed: Uint8Array, added: Uint8Array, offset: number): void {
let x = n;
let y = m;
for (let d = trace.length - 1; d > 0; d -= 1) {
@@ -170,8 +230,8 @@ function backtrack(trace: Int32Array[], n: number, m: number, removed: Uint8Arra
const down = k === -d || (k !== d && at(k - 1) < at(k + 1));
const previousX = at(down ? k + 1 : k - 1);
const previousY = previousX - (down ? k + 1 : k - 1);
- if (down) added[bOffset + previousY] = 1;
- else removed[aOffset + previousX] = 1;
+ if (down) added[offset + previousY] = 1;
+ else removed[offset + previousX] = 1;
x = previousX;
y = previousY;
}
@@ -184,11 +244,12 @@ function lineCost(line: string): number {
/**
* A unified diff (3 lines of context) of `before` into `after`, a null side
- * standing for /dev/null. The counts cover the whole change; the text stops
- * at the last whole line within `maxBytes` (measured JSON-encoded), marked
- * truncated.
+ * standing for /dev/null. The counts cover the whole change (exact, or an
+ * upper bound once `maxComparisons` cut the line matching short); the text
+ * stops at the last whole line within `maxBytes` (measured JSON-encoded),
+ * marked truncated. The patch always turns `before` into `after`.
*/
-export function unifiedDiff(path: string, before: string | null, after: string | null, maxBytes = MAX_TURN_DIFF_FILE_BYTES): { diff: string; additions: number; deletions: number; truncated: boolean; bytes: number } {
+export function unifiedDiff(path: string, before: string | null, after: string | null, maxBytes = MAX_TURN_DIFF_FILE_BYTES, maxComparisons = MAX_DIFF_COMPARISONS): { diff: string; additions: number; deletions: number; truncated: boolean; bytes: number } {
const a = splitLines(before ?? "");
const b = splitLines(after ?? "");
// A last line without a newline differs from the same line with one.
@@ -199,23 +260,12 @@ export function unifiedDiff(path: string, before: string | null, after: string |
while (prefix < nA && prefix < nB && key(a, prefix) === key(b, prefix)) prefix += 1;
let suffix = 0;
while (suffix < nA - prefix && suffix < nB - prefix && key(a, nA - 1 - suffix) === key(b, nB - 1 - suffix)) suffix += 1;
- const ids = new Map<string, number>();
- const idsOf = (side: Lines, count: number) => {
- const out = new Int32Array(count);
- for (let index = 0; index < count; index += 1) {
- const text = key(side, prefix + index);
- let id = ids.get(text);
- if (id === undefined) {
- id = ids.size;
- ids.set(text, id);
- }
- out[index] = id;
- }
- return out;
- };
+ const n = nA - prefix - suffix;
+ const m = nB - prefix - suffix;
+ const ids = new LineIds((index) => key(a, prefix + index), (index) => key(b, prefix + index), n, m);
const removed = new Uint8Array(nA);
const added = new Uint8Array(nB);
- markChanges(idsOf(a, nA - prefix - suffix), idsOf(b, nB - prefix - suffix), removed, added, prefix, prefix);
+ markChanges(ids, n, m, removed, added, prefix, maxComparisons);
type Block = { aStart: number; aEnd: number; bStart: number; bEnd: number };
const blocks: Block[] = [];
@@ -287,20 +337,29 @@ export function unifiedDiff(path: string, before: string | null, after: string |
/**
* Assembles one "turn.diff" event under the event cap: each file's diff gets
- * at most MAX_TURN_DIFF_FILE_BYTES and what is left of the event's budget;
- * once the budget is spent, further files are only counted.
+ * at most MAX_TURN_DIFF_FILE_BYTES and what is left of the event's budget.
+ * Once no further diff fits, in the event or in the turn's diff time, a file
+ * that needs one is only counted, while an entry without diff text (skipped,
+ * no_base) still goes in as long as it fits.
*/
export class TurnDiffBuilder {
private readonly files: TurnDiffFile[] = [];
- private used = 0;
+ /** Bytes of the event so far, its envelope reserved from the start. */
+ private used = EVENT_ENVELOPE_BYTES;
private omitted = 0;
private truncated = false;
+ /** Time spent diffing so far. */
+ private diffMs = 0;
- constructor(private readonly maxEventBytes = MAX_TURN_DIFF_EVENT_BYTES, private readonly maxFileBytes = MAX_TURN_DIFF_FILE_BYTES) {}
+ constructor(
+ private readonly maxEventBytes = MAX_TURN_DIFF_EVENT_BYTES,
+ private readonly maxFileBytes = MAX_TURN_DIFF_FILE_BYTES,
+ private readonly maxDiffMs = MAX_TURN_DIFF_MS,
+ ) {}
- /** No further diff fits: the caller need not resolve its texts. */
+ /** No further diff fits: the caller need not resolve texts. */
get full(): boolean {
- return this.used + MIN_DIFF_ROOM_BYTES > this.maxEventBytes;
+ return this.used + MIN_DIFF_ROOM_BYTES > this.maxEventBytes || this.diffMs >= this.maxDiffMs;
}
omit(): void {
@@ -309,33 +368,38 @@ export class TurnDiffBuilder {
}
add(input: TurnDiffInput): void {
+ const diffed = input.status === "added" || input.status === "modified" || input.status === "deleted";
+ if (diffed && this.full) return this.omit();
+ const before = input.status === "added" ? null : input.before;
+ const after = input.status === "deleted" ? null : input.after;
+ const hasText = diffed && (before !== null || after !== null);
const file: TurnDiffFile = {
path: input.path,
status: input.status,
before_sha256: input.before_sha256,
after_sha256: input.after_sha256,
- additions: null,
- deletions: null,
+ // At their widest while the entry is measured: a count never exceeds its text's length.
+ additions: hasText ? after?.length ?? 0 : null,
+ deletions: hasText ? before?.length ?? 0 : null,
diff: null,
truncated: false,
};
// The entry without its diff, plus the comma between entries.
const overhead = Buffer.byteLength(JSON.stringify(file)) + 1;
const room = this.maxEventBytes - this.used - overhead;
- const hasText = (input.status === "added" || input.status === "modified" || input.status === "deleted") && (input.before !== null || input.after !== null);
if (room < (hasText ? MIN_DIFF_ROOM_BYTES : 0)) return this.omit();
- let diffBytes = 0;
if (hasText) {
- const result = unifiedDiff(input.path, input.status === "added" ? null : input.before, input.status === "deleted" ? null : input.after, Math.min(this.maxFileBytes, room));
+ const started = performance.now();
+ const result = unifiedDiff(input.path, before, after, Math.min(this.maxFileBytes, room));
+ this.diffMs += performance.now() - started;
file.additions = result.additions;
file.deletions = result.deletions;
file.diff = result.diff;
file.truncated = result.truncated;
- diffBytes = result.bytes;
if (result.truncated) this.truncated = true;
}
this.files.push(file);
- this.used += overhead + (file.diff === null ? 0 : diffBytes + 2 - 4);
+ this.used += Buffer.byteLength(JSON.stringify(file)) + 1;
}
finish(): TurnDiffEvent {
@@ -376,6 +440,10 @@ export class TurnBaseStore {
private saveTail: Promise<void> = Promise.resolve();
private readonly ready: Promise<void>;
private waiters: Array<{ limit: number; resolve: () => void }> = [];
+ /** Removals of evicted texts still under way, by digest: a write of the same digest waits for its removal. */
+ private readonly removals = new Map<string, Promise<void>>();
+ /** Writes under way: clear() lets them settle before it removes the directory. */
+ private readonly writing = new Set<Promise<void>>();
constructor(private readonly dir: string, private readonly budget = TURN_BASE_STORE_BYTES) {
this.ready = this.load(dir);
@@ -416,10 +484,20 @@ export class TurnBaseStore {
if (this.used <= this.budget) break;
this.index.delete(sha256);
this.used -= bytes;
- void rm(join(this.dir, sha256), { force: true }).catch(() => undefined);
+ const removal: Promise<void> = rm(join(this.dir, sha256), { force: true }).catch(() => undefined).finally(() => {
+ if (this.removals.get(sha256) === removal) this.removals.delete(sha256);
+ });
+ this.removals.set(sha256, removal);
}
}
+ /** Whether a text is held under `sha256`, without reading it back. */
+ async has(sha256: string): Promise<boolean> {
+ if (this.pending.has(sha256)) return true;
+ await this.ready;
+ return this.index.has(sha256);
+ }
+
/** Remembers `text` (the scrubbed text whose redacted sha256 is `sha256`, `bytes` long). */
put(sha256: string, text: string, bytes: number): void {
if (bytes > MAX_BASE_TEXT_BYTES) return;
@@ -436,7 +514,8 @@ export class TurnBaseStore {
while (this.active < BASE_WRITE_CONCURRENCY && this.queue.length > 0) {
const sha256 = this.queue.shift()!;
this.active += 1;
- void this.write(sha256).finally(() => {
+ const writing: Promise<void> = this.write(sha256).finally(() => {
+ this.writing.delete(writing);
this.active -= 1;
const held = this.pending.get(sha256);
if (held) {
@@ -446,6 +525,7 @@ export class TurnBaseStore {
this.pump();
this.wake();
});
+ this.writing.add(writing);
}
}
@@ -477,6 +557,8 @@ export class TurnBaseStore {
const dir = this.dir;
try {
await this.ready;
+ // An evicted copy of the same digest is removed first, or its late removal would take this one.
+ await this.removals.get(sha256);
if (!held || generation !== this.generation) return;
const { text, bytes } = held;
if (this.index.has(sha256)) return this.touch(sha256, bytes);
@@ -558,15 +640,21 @@ export class TurnBaseStore {
clearTimeout(this.saveTimer);
this.saveTimer = null;
}
+ // Queued texts are never written. One already being written sees the new
+ // generation and removes itself, and settles before the directory goes:
+ // nothing it does can recreate the directory afterwards.
+ this.queue.length = 0;
+ const writing = [...this.writing];
// After the load, which would otherwise fill the index again.
await this.ready;
this.index.clear();
- this.queue.length = 0;
this.pending.clear();
this.pendingBytes = 0;
this.used = 0;
this.dirty = false;
this.wake();
+ await Promise.all(writing);
+ await Promise.all(this.removals.values());
await this.saveTail;
await rm(this.dir, { recursive: true, force: true });
}
diff --git a/apps/server/src/workspace-collector.ts b/apps/server/src/workspace-collector.ts
index a0a7e054..3a685d9d 100644
--- a/apps/server/src/workspace-collector.ts
+++ b/apps/server/src/workspace-collector.ts
@@ -110,6 +110,11 @@ const REDACTED_TEXT_CACHE_BYTES = 32 * 1024 * 1024;
const SPOOL_DIRECTORY = "omnirush-collector-spool";
/** The scrubbed texts "turn.diff" events are measured against (turn-diff.ts). */
const BASE_DIRECTORY = "omnirush-collector-bases";
+/**
+ * An mtime this far ahead of the clock (an unpacked archive, a share on
+ * another machine) is no write of the current turn's.
+ */
+const TURN_CLOCK_SLACK_MS = 1_000;
const MAX_SPOOL_BYTES = 128 * 1024 * 1024;
const MAX_SPOOL_ENTRIES = 200;
const MAX_SPOOL_ATTEMPTS = 24;
@@ -316,10 +321,15 @@ type SessionState = {
ignoredCache: Map<string, boolean>;
/** The last accepted manifest: what the next change snapshot is measured against. */
manifest: Manifest | null;
- /** The accepted manifest as the current turn began: what its "turn.diff" is measured against. */
- turnManifest: Manifest | null;
- /** Files the journal saw this turn write that may be binary or over the size cap: "skipped" in its "turn.diff". */
- turnSkipped: Set<string>;
+ /** The workspace as the current turn began: what its "turn.diff" is measured against. */
+ turnManifest: TurnBaseline | null;
+ /** When the current turn's prompt was sent (null before the first). */
+ turnStartedAt: number | null;
+ /**
+ * Files the journal saw the turn write that may be binary or over the size
+ * cap, with their mtime: "skipped" in its "turn.diff".
+ */
+ turnSkipped: Map<string, number>;
/** The per-root hash cache, shared with every other session open on the same root. */
cache: Map<string, HashCacheEntry>;
listing: { denied: number; truncated: boolean };
@@ -2430,8 +2440,62 @@ type SnapshotCandidate = {
priority: boolean;
};
-/** A file whose digest moved over a turn: null where it is not in that manifest. */
-type TurnChange = { path: string; before: HashCacheEntry | null; after: HashCacheEntry | null };
+/**
+ * The workspace as a turn began, by path: null for a file the collector only
+ * ever saw once the turn had written it (it was there, but what it held
+ * before the turn is unknown).
+ */
+type TurnBaseline = ReadonlyMap<string, HashCacheEntry | null>;
+
+/**
+ * A file whose digest moved over a turn: null where it is not in that
+ * manifest; `unseen` when it was there before the turn, in a form never seen.
+ */
+type TurnChange = { path: string; before: HashCacheEntry | null; after: HashCacheEntry | null; unseen: boolean };
+
+/** Whether a file last modified at `mtimeMs` was written at or after `since` (and not in the future). */
+function writtenSince(mtimeMs: number, since: number, now = Date.now()): boolean {
+ return mtimeMs >= since && mtimeMs <= now + TURN_CLOCK_SLACK_MS;
+}
+
+/**
+ * `manifest` as the workspace stood when the turn prompted at `since` began.
+ * A file written since then (the agent's edit, landing while an earlier
+ * upload held the queue and this snapshot waited) gives way to what
+ * `earlier` (newest first) last knew of it from before: its entry there, its
+ * absence (a file the turn created), or null when every one saw it only once
+ * written, so the edit stays in the turn.
+ */
+function turnBaseline(manifest: Manifest, since: number | null, earlier: ReadonlyArray<TurnBaseline | null>): TurnBaseline {
+ if (since === null) return manifest;
+ const now = Date.now();
+ let baseline: Map<string, HashCacheEntry | null> | null = null;
+ for (const [path, entry] of manifest) {
+ if (!writtenSince(entry.mtimeMs, since, now)) continue;
+ baseline ??= new Map(manifest);
+ let known: HashCacheEntry | null | undefined = null;
+ for (const source of earlier) {
+ if (!source) continue;
+ const seen = source.get(path);
+ if (seen === null || (seen && writtenSince(seen.mtimeMs, since, now))) continue;
+ known = seen;
+ break;
+ }
+ if (known === undefined) baseline.delete(path);
+ else baseline.set(path, known);
+ }
+ return baseline ?? manifest;
+}
+
+/**
+ * Whether the file `file` describes is a write of the current turn's (a
+ * traced read or an attachment journals a path too): written since the
+ * turn's prompt, or, before any prompt, moved since the cache last saw it.
+ */
+function writtenThisTurn(turnStartedAt: number | null, cached: HashCacheEntry | undefined, file: { size: number; mtimeMs: number }): boolean {
+ if (turnStartedAt !== null) return writtenSince(file.mtimeMs, turnStartedAt);
+ return cached !== undefined && (cached.size !== file.size || cached.mtimeMs !== file.mtimeMs);
+}
/** Whether the first 8 KiB of a file read as binary (false when it cannot be read). */
async function startsBinary(absolute: string): Promise<boolean> {
@@ -3061,7 +3125,8 @@ export class WorkspaceCollector {
ignoredCache: new Map(),
manifest: null,
turnManifest: null,
- turnSkipped: new Set(),
+ turnStartedAt: null,
+ turnSkipped: new Map(),
cache: this.acquireCache(root),
listing: { denied: 0, truncated: false },
dirty: new Set(),
@@ -3639,63 +3704,81 @@ export class WorkspaceCollector {
// The artifact baseline must reflect the workspace as the turn begins, not
// once the queued start snapshot has finished uploading.
if (trigger === "prompt") state.artifactBaseline = artifactStats(state.root);
+ // So must the turn's start: what the agent writes while an earlier upload
+ // still holds the queue belongs to this turn. It is the first whole
+ // millisecond after the prompt: Date.now() rounds down, and a file written
+ // just before may carry a later fraction of the same millisecond.
+ if (trigger === "prompt") state.turnStartedAt = Date.now() + 1;
+ const startedAt = state.turnStartedAt;
if (state.changeTimer) {
clearTimeout(state.changeTimer);
state.changeTimer = null;
state.changeTrigger = null;
}
this.enqueue(state, async () => {
+ const previous = state.manifest;
await this.captureChange(state, trigger);
- if (trigger === "turn_completed") return this.recordTurnDiff(state);
+ const current = state.manifest;
+ if (trigger === "turn_completed") {
+ // A prompt sent since the turn ended (this job waited behind an
+ // upload) began the next turn: what it wrote belongs to that one.
+ const next = state.turnStartedAt !== startedAt ? state.turnStartedAt : null;
+ if (!current || next === null) return this.recordTurnDiff(state, current, null);
+ const taken = turnBaseline(current, next, [previous, state.turnManifest]);
+ // A file nothing saw from before the next turn keeps its entry here.
+ return this.recordTurnDiff(state, new Map([...taken].map(([path, entry]) => [path, entry ?? current.get(path)!])), next);
+ }
// The turn is measured from the workspace as its prompt found it.
- state.turnManifest = state.manifest;
- state.turnSkipped = new Set();
+ state.turnManifest = current && turnBaseline(current, startedAt, [previous, state.turnManifest]);
+ for (const [path, at] of state.turnSkipped) if (startedAt !== null && at < startedAt) state.turnSkipped.delete(path);
});
}
/**
* Appends the "turn.diff" event of the turn that just completed: a unified
* diff of the scrubbed texts of every file whose redacted digest moved
- * between the manifest the turn began on and the last accepted one (added,
- * modified, deleted, or no_base when the text sent before the turn is no
- * longer held), and each binary or oversized file the turn wrote as
- * skipped, without content. The next turn is measured from here.
+ * between the workspace the turn began on and `after` (added, modified,
+ * deleted, or no_base when the text from before the turn is not held or
+ * was never seen), and each binary or oversized file the turn wrote as
+ * skipped, without content. The next turn is measured from here; the
+ * skipped files of a next turn already under way (written since `next`,
+ * its prompt) are left to it.
*/
- private async recordTurnDiff(state: SessionState): Promise<void> {
+ private async recordTurnDiff(state: SessionState, after: Manifest | null, next: number | null): Promise<void> {
const before = state.turnManifest;
- const after = state.manifest;
const skipped = state.turnSkipped;
state.turnManifest = after;
- state.turnSkipped = new Set();
+ state.turnSkipped = new Map(next === null ? [] : [...skipped].filter(([, at]) => at >= next));
if (state.finished || !before || !after) return;
const changes: TurnChange[] = [];
if (before !== after) {
for (const [path, entry] of after) {
const previous = before.get(path);
- if (previous?.sha256 !== entry.sha256) changes.push({ path, before: previous ?? null, after: entry });
+ if (previous?.sha256 !== entry.sha256) changes.push({ path, before: previous ?? null, after: entry, unseen: previous === null });
}
- for (const [path, entry] of before) if (!after.has(path)) changes.push({ path, before: entry, after: null });
+ for (const [path, entry] of before) if (!after.has(path)) changes.push({ path, before: entry, after: null, unseen: false });
+ }
+ for (const [path, at] of skipped) {
+ if ((next === null || at < next) && !after.has(path) && !before.has(path)) changes.push({ path, before: null, after: null, unseen: false });
}
- for (const path of skipped) if (!after.has(path) && !before.has(path)) changes.push({ path, before: null, after: null });
changes.sort((left, right) => (left.path < right.path ? -1 : left.path > right.path ? 1 : 0));
const builder = new TurnDiffBuilder();
const pool = new ReadPool(1);
const ancestors: AncestorCache = new Map();
const yielder = new LoopYielder();
for (const change of changes) {
- if (builder.full) {
- builder.omit();
- continue;
- }
- const input = await this.turnDiffInput(state, change, pool, ancestors);
+ // Once no further diff fits (the event's cap, or the turn's diff time),
+ // no text is read: an entry without one still goes in while it fits.
+ const input = await this.turnDiffInput(state, change, pool, ancestors, !builder.full);
if (input) builder.add(input);
await yielder.pause();
}
this.appendTrace(state, "turn.diff", builder.finish());
}
- private async turnDiffInput(state: SessionState, change: TurnChange, pool: ReadPool, ancestors: AncestorCache): Promise<TurnDiffInput | null> {
- const { path, before, after } = change;
+ /** The entry `change` makes, with its texts when `texts` is set (without, it only needs a status). */
+ private async turnDiffInput(state: SessionState, change: TurnChange, pool: ReadPool, ancestors: AncestorCache, texts: boolean): Promise<TurnDiffInput | null> {
+ const { path, before, after, unseen } = change;
const input = {
path: (after ?? before)?.uploadPath ?? collectorPathForUpload(path),
before_sha256: before?.sha256 ?? null,
@@ -3704,6 +3787,10 @@ export class WorkspaceCollector {
after: null,
};
if (after) {
+ // Changed by the turn from a form the collector never saw: nothing to diff against.
+ if (unseen) return { ...input, status: "no_base" };
+ if (before && !(await this.bases.has(before.sha256))) return { ...input, status: "no_base" };
+ if (!texts) return { ...input, status: before ? "modified" : "added" };
const base = before ? await this.bases.get(before.sha256) : null;
if (before && base === null) return { ...input, status: "no_base" };
const text = await this.bases.get(after.sha256) ?? await this.rereadSentText(state, path, after, pool, ancestors);
@@ -3713,7 +3800,13 @@ export class WorkspaceCollector {
// Gone from the manifest (or never in it): deleted, or a binary or oversized file now.
const absolute = resolve(state.root, path);
const file = await hasRealAncestors(state.root, path, ancestors) ? await lstat(absolute).catch(() => null) : null;
- if (!file) return before ? { ...input, status: "deleted", before: await this.bases.get(before.sha256) } : null;
+ if (!file) {
+ if (!before) return null;
+ if (!(await this.bases.has(before.sha256))) return { ...input, status: "no_base" };
+ if (!texts) return { ...input, status: "deleted" };
+ const base = await this.bases.get(before.sha256);
+ return base === null ? { ...input, status: "no_base" } : { ...input, status: "deleted", before: base };
+ }
if (!file.isFile() || file.isSymbolicLink()) return null;
if (file.size <= MAX_COLLECTOR_FILE_BYTES && !state.cache.get(path)?.binary && !(await startsBinary(absolute))) return null;
return { ...input, after_sha256: null, status: "skipped" };
@@ -3935,7 +4028,8 @@ export class WorkspaceCollector {
let entry: ChangeJournalEntry;
try {
const file = await lstat(absolute);
- if (file.isFile() && !file.isSymbolicLink() && file.size > MAX_CHANGE_JOURNAL_BYTES) state.turnSkipped.add(path);
+ const written = writtenThisTurn(state.turnStartedAt, state.cache.get(path), file);
+ if (file.isFile() && !file.isSymbolicLink() && file.size > MAX_CHANGE_JOURNAL_BYTES && written) state.turnSkipped.set(path, file.mtimeMs);
if (!file.isFile() || file.isSymbolicLink() || file.size > MAX_COLLECTOR_FILE_BYTES) {
entry = { path, at: new Date().toISOString(), status: "skipped" };
} else if (file.size > MAX_CHANGE_JOURNAL_BYTES) {
@@ -3947,7 +4041,7 @@ export class WorkspaceCollector {
const buffer = await readFile(absolute);
this.metrics.fileReads += 1;
if (buffer.length > MAX_COLLECTOR_FILE_BYTES || isBinary(buffer)) {
- state.turnSkipped.add(path);
+ if (written) state.turnSkipped.set(path, file.mtimeMs);
entry = { path, at: new Date().toISOString(), status: "skipped" };
} else {
// The journal's read doubles as the snapshot's: the cache learns the
@@ -4155,7 +4249,7 @@ export class WorkspaceCollector {
// next change snapshot reports exactly what moved since this one.
accepted = true;
state.manifest = manifest;
- state.turnManifest ??= manifest;
+ state.turnManifest ??= turnBaseline(manifest, state.turnStartedAt, [previous]);
state.listing = { denied: scan.deniedCount, truncated: scan.manifestTruncated };
state.lastHead = git?.commit ?? null;
if (journal.length > 0) this.acknowledgeJournal(state, journal);
diff --git a/docs/workspace-collector-privacy.md b/docs/workspace-collector-privacy.md
index e508d7c9..db9a0f04 100644
--- a/docs/workspace-collector-privacy.md
+++ b/docs/workspace-collector-privacy.md
@@ -21,4 +21,4 @@ The desktop collector keeps its footprint bounded while producing these snapshot
## Per-turn diffs: the `turn.diff` trace event
-At the end of each turn the trace gets one `turn.diff` event (`apps/server/src/turn-diff.ts`), pairing the turn with what it changed in the workspace: every file whose redacted sha256 moved between the accepted manifest the turn began on (as its prompt found the workspace) and the last accepted one. Each entry carries the workspace-relative redacted path, a `status` (`added`, `modified`, `deleted`, `skipped` for a binary or over-4-MiB file the turn wrote, sent without content, or `no_base` when the text sent before the turn is no longer held), both redacted digests, the `+`/`-` line counts and a unified diff of the scrubbed text sent before the turn against the scrubbed text sent after it. A file diff is cut at 256 KiB and the event at 2 MiB (JSON-encoded), both marked `truncated`; files past the event cap are only counted in `omitted_file_count`. The diff is built from text the collector already sent, so the denylist and scrubber apply unchanged, and the trace is scrubbed again as a whole like every other event. To have the "before" side, the collector keeps the scrubbed texts it sent (start snapshots seed it) in a content-addressed store under the app state directory (`omnirush-collector-bases`, owner-only files, 64 MiB at most, least recently used out first), so bases survive a restart; a sign-out deletes it with the spool.
+At the end of each turn the trace gets one `turn.diff` event (`apps/server/src/turn-diff.ts`), pairing the turn with what it changed in the workspace: every file whose redacted sha256 moved between the workspace as the turn's prompt found it and the last accepted manifest. The turn begins when its prompt is sent, not when the queued prompt snapshot runs: a file written after that moment (an edit landing while an earlier upload still holds the queue) is measured from what the collector last saw of it before the prompt, and one it only ever saw already written by the turn is reported as `no_base` without a "before" digest. Likewise, when the next prompt goes out before a turn's own snapshot has run, what the next turn wrote is left to it. A turn the observer stops following (after an hour, or on an error) still gets its snapshot and `turn.diff`. Each entry carries the workspace-relative redacted path, a `status` (`added`, `modified`, `deleted`, `skipped` for a binary or over-4-MiB file the turn wrote, sent without content, or `no_base`, for a changed or deleted file, when there is no text from before the turn to diff against: no longer held, or never seen), both redacted digests, the `+`/`-` line counts and a unified diff of the scrubbed text sent before the turn against the scrubbed text sent after it. A binary or oversized file the turn only read (a traced read, an attachment) is not an entry: only one written since the prompt counts. The diff work is bounded: one file's line matching stops after 5 million comparisons and writes the rest of its change as removed and added lines (still a patch that turns the old text into the new one, its counts then an upper bound), and a turn's diffs get 250 ms of work in all, after which the files still waiting for a diff are only counted in `omitted_file_count`. A file diff is cut at 256 KiB and the whole event, envelope included, at 2 MiB (JSON-encoded), both marked `truncated`; files past the event cap are only counted in `omitted_file_count`, while entries without diff text (`skipped`, `no_base`) still go in as long as they fit. The diff is built from text the collector already sent, so the denylist and scrubber apply unchanged, and the trace is scrubbed again as a whole like every other event. To have the "before" side, the collector keeps the scrubbed texts it sent (start snapshots seed it) in a content-addressed store under the app state directory (`omnirush-collector-bases`, owner-only files, 64 MiB at most, least recently used out first), so bases survive a restart; a sign-out deletes it with the spool. Version 1.0.10 and earlier neither read nor delete that directory: after a downgrade it stays on disk, up to 64 MiB of scrubbed workspace text, until it is removed by hand or a later version signs out.