A local-first multi-agent app development platform, built on PenguinHarness
fix(core): single-write Trace appends and crash-torn tail healing (#249)
Yaowei Zheng3b493df37d58
diff --git a/packages/core/src/trace/writer.ts b/packages/core/src/trace/writer.tsindex c14d6e80..2afcd344 100644--- a/packages/core/src/trace/writer.ts+++ b/packages/core/src/trace/writer.ts@@ -16,7 +16,7 @@* context_engine).* - Path convention: `<tracesDir>/<yyyy-mm-dd>/<sessionId>_<index3>.jsonl`.*/-import { appendFile, mkdir, readFile } from "node:fs/promises";+import { mkdir, open, readFile } from "node:fs/promises";import { dirname, join } from "node:path";import {@@ -68,22 +68,37 @@ function isRecordable(msg: OmniMessage): boolean {/*** append-only JSONL Trace writer.*- * Every write uses `appendFile` (O_APPEND) rather than caching a file handle and seeking to- * write, avoiding overwriting existing content; this also removes the need for an explicit- * close.+ * Every append opens the file with O_APPEND, issues **one** `write(2)` for the whole record,+ * and closes the handle — no cached handle, no seeking, so existing content can never be+ * overwritten. `fs.appendFile` is deliberately NOT used: it splits payloads larger than+ * 512 KiB into multiple underlying writes (Node's internal `kWriteFileMaxChunkSize`), so a+ * process death between chunks leaves exactly the first 524288 bytes of a big record (a huge+ * tool_call's arguments, a base64 image Data URL) on disk — and every record appended after+ * it glues onto that torn line, turning one crash into an unparseable line that swallows+ * later records too. A single-write append can only be cut short by the OS itself (power+ * loss, ENOSPC short write), which the tail-heal below and the readers' torn-tail tolerance+ * (parseTraceLines) are built for.+ *+ * Tail healing: before the first append this instance makes to a **pre-existing** shard file+ * (Session resumption continues the original file), the last byte is probed — a file not+ * ending in `\n` carries a crash-torn tail, and the next record is prefixed with `\n` so it+ * starts on a fresh line instead of merging into the torn one (the torn line itself is+ * skipped by the tolerant readers; the records after it stay readable). A mid-record append+ * failure inside this process (ENOSPC, forced short write) marks the tail torn and heals the+ * same way on the next append. A spurious heal costs one blank line, which every reader+ * ignores.** Concurrency: one live Session has exactly one Writer instance, but that instance receives* appends from **multiple async producers in the same process** — the engine's LLM stream- * driver and each parallel tool execution write independently. `appendFile` splits large- * payloads (multi-MB records such as base64 image Data URLs) into multiple underlying writes,- * so two overlapping appends can interleave mid-record and corrupt the JSONL (#215). All- * mutating operations (`write`, `rotate`) are therefore serialized through one per-instance- * promise chain: each record lands as one uninterrupted append, and a rotation cannot land in- * the middle of an append. Cross-instance/file concurrency does not occur by design **within a- * single server/CLI process**: a Session allows one active run at a time, child sessions write- * their own files, and server-side Trace import only ever creates brand-new files (`wx`). Two+ * driver and each parallel tool execution write independently (#215). All mutating operations+ * (`write`, `rotate`) are therefore serialized through one per-instance promise chain: each+ * record lands as one uninterrupted append, and a rotation cannot land in the middle of an+ * append. Cross-instance/file concurrency does not occur by design **within a single+ * server/CLI process**: a Session allows one active run at a time, child sessions write their+ * own files, and server-side Trace import only ever creates brand-new files (`wx`). Two* processes pointed at the same agent data directory are outside this contract (and outside- * supported usage) — the chain cannot cover them.+ * supported usage), though the single-write append keeps even that case tear-free on local+ * filesystems that make O_APPEND writes atomic.** Error semantics: a failed operation rejects **that caller's** returned promise only; the* chain itself absorbs the failure so subsequent operations still run (a transient disk error@@ -99,8 +114,10 @@ export class Writer {private readonly dateDir: string;/** Current Trace index, starting at 1; incremented by `rotate()`. */private index = 1;- /** Set true once the date directory has been created for the current file, to avoid a redundant mkdir. */- private ensuredDirForIndex = -1;+ /** Index whose file is prepared (date directory created, pre-existing tail probed); avoids redoing either per append. */+ private preparedIndex = -1;+ /** True while the current shard may end mid-record — assigned per shard by the tail probe, set by a mid-record append failure, cleared once a record fully lands. The next record then starts on a fresh line. */+ private tornTail = false;/** Serialization chain: mutating operations run strictly in submission order (see class docs). */private chain: Promise<void> = Promise.resolve();@@ -133,7 +150,8 @@ export class Writer {/*** Appends one message. Only written if it's a recordable message; streaming `partial_*` is- * skipped. `mkdir -p`s the date directory on the first write to the current file.+ * skipped. The first write to the current file `mkdir -p`s the date directory and probes a+ * pre-existing file for a crash-torn tail (see `probeTornTail`).** The append is serialized on the instance chain, so the record lands as one uninterrupted* JSONL line even when other writes (or a rotation) are submitted concurrently; the target@@ -145,14 +163,74 @@ export class Writer {if (!isRecordable(msg)) return;return this.enqueue(async () => {const path = this.currentPath();- if (this.ensuredDirForIndex !== this.index) {+ if (this.preparedIndex !== this.index) {await mkdir(dirname(path), { recursive: true });- this.ensuredDirForIndex = this.index;+ this.tornTail = await this.probeTornTail(path);+ this.preparedIndex = this.index;}- await appendFile(path, `${JSON.stringify(msg)}\n`, "utf8");+ const record = `${JSON.stringify(msg)}\n`;+ await this.appendRecord(path, this.tornTail ? `\n${record}` : record);+ this.tornTail = false; // the record (and any heal prefix) fully reached the file});}+ /**+ * Whether a pre-existing shard file ends mid-record (resumption continues the original+ * file): a non-empty file whose last byte is not `\n` carries a crash-torn tail, and the+ * next append must not merge into that line. A missing or empty file is simply fresh; any+ * other probe error surfaces to the caller like an append failure (and is retried with the+ * next write, since the index stays unprepared).+ */+ private async probeTornTail(path: string): Promise<boolean> {+ let fh;+ try {+ fh = await open(path, "r");+ } catch (err) {+ if ((err as NodeJS.ErrnoException).code === "ENOENT") return false;+ throw err;+ }+ try {+ const { size } = await fh.stat();+ if (size === 0) return false;+ const { bytesRead, buffer } = await fh.read(Buffer.alloc(1), 0, 1, size - 1);+ return bytesRead !== 1 || buffer[0] !== 0x0a;+ } finally {+ await fh.close();+ }+ }++ /**+ * Appends one record through a single `write(2)` on an O_APPEND handle (see the class docs+ * for why `appendFile`'s 512 KiB chunking is unacceptable here). The short-write loop is a+ * safety net only — one iteration is the norm; if a retry is ever needed and then fails,+ * part of the record is on disk, so the tail is marked torn for the next append to heal.+ */+ private async appendRecord(path: string, data: string): Promise<void> {+ const buf = Buffer.from(data, "utf8");+ const fh = await open(path, "a");+ try {+ let written = 0;+ while (written < buf.length) {+ let bytesWritten: number;+ try {+ ({ bytesWritten } = await fh.write(buf, written, buf.length - written));+ } catch (err) {+ if (written > 0) this.tornTail = true;+ throw err;+ }+ if (bytesWritten <= 0) {+ // A zero-progress write (out-of-space filesystems can report this instead of+ // throwing): bail out rather than loop forever.+ if (written > 0) this.tornTail = true;+ throw new Error(`Trace append made no progress at byte ${written} of ${buf.length}`);+ }+ written += bytesWritten;+ }+ } finally {+ await fh.close();+ }+ }+/** Writes multiple messages in sequence. */async writeAll(msgs: OmniMessage[]): Promise<void> {for (const msg of msgs) {@@ -183,6 +261,8 @@ export class Writer {*/async rotate(): Promise<void> {return this.enqueue(async () => {+ // No torn-tail bookkeeping here: the first write to the new index re-probes and+ // reassigns the flag, so the previous shard's state cannot leak into the next file.this.index += 1;});}diff --git a/packages/core/test/trace.test.ts b/packages/core/test/trace.test.tsindex 6c50c202..2a19ffbe 100644--- a/packages/core/test/trace.test.ts+++ b/packages/core/test/trace.test.ts@@ -1,8 +1,8 @@-import { mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";+import { mkdir, mkdtemp, open, readFile, rm, stat, writeFile } from "node:fs/promises";import { tmpdir } from "node:os";import { join, sep } from "node:path";-import { afterEach, beforeEach, describe, expect, it } from "vitest";+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";import {assistantText,@@ -15,10 +15,18 @@ import {withOrigin,} from "../src/omnimessage/index.js";import type { OmniMessage } from "../src/omnimessage/index.js";-import { Writer, readTrace } from "../src/trace/index.js";+import { Writer, parseTraceLines, readTrace } from "../src/trace/index.js";const SESSION_ID = "sess_abc";+/** FileHandle's prototype, grabbed from a throwaway handle (the class is not exported). */+async function fileHandleProto(dir: string): Promise<{ write: (...args: unknown[]) => unknown }> {+ const probe = await open(join(dir, "probe.tmp"), "w");+ const proto = Object.getPrototypeOf(probe) as { write: (...args: unknown[]) => unknown };+ await probe.close();+ return proto;+}+function meta() {return sessionMeta({session_id: SESSION_ID,@@ -219,6 +227,34 @@ describe("Writer", () => {expect(secondLines).toEqual(shardTwo.map((msg) => JSON.stringify(msg)));});+ it("appends a multi-MB record with a single underlying write (no 512 KiB chunk tear window)", async () => {+ // fs.appendFile would split this into three writes (Node chunks at 512 KiB = 524288+ // bytes); a crash between chunks used to leave exactly the first 524288 bytes of the+ // record, and the next session's appends then glued onto that torn line. The Writer+ // issues one write(2) per record instead, so no crash can split a record mid-file.+ const writer = new Writer({+ tracesDir,+ sessionId: SESSION_ID,+ date: new Date(2026, 0, 9),+ });+ const head = meta();+ const big = imageUrlMessage(`data:image/png;base64,${"C".repeat(1200 * 1024)}`);+ const bigRecordBytes = Buffer.byteLength(`${JSON.stringify(big)}\n`);++ const writeSpy = vi.spyOn(await fileHandleProto(tracesDir), "write");+ try {+ await writer.writeAll([head, big]);+ // One write() per record — the big one lands whole, never in 524288-byte chunks+ // (>= so a chunk of exactly 524288 bytes would be caught too).+ const lengths = writeSpy.mock.calls.map((call) => call[2]);+ expect(lengths).toContain(bigRecordBytes);+ expect(lengths.filter((len) => (len as number) >= 512 * 1024)).toEqual([bigRecordBytes]);+ } finally {+ writeSpy.mockRestore();+ }+ expect(await readTrace(writer.currentPath())).toEqual([head, big]);+ });+it("a failed write rejects its own caller without wedging subsequent writes", async () => {const writer = new Writer({tracesDir,@@ -243,4 +279,149 @@ describe("Writer", () => {expect(rows).toEqual([recovered]);});});++ // The failure branches of the single-write append: a partial write leaves a torn tail that+ // the next append must heal; a zero-progress write must bail out instead of looping.+ describe("append failure semantics", () => {+ it("marks the tail torn after a partial append failure and heals on the next write", async () => {+ const writer = new Writer({+ tracesDir,+ sessionId: SESSION_ID,+ date: new Date(2026, 0, 9),+ });+ const first = assistantText("intact record");+ await writer.write(first); // the shard exists with a clean tail before the failure++ const torn = assistantText("torn mid-write");+ const tornData = Buffer.from(`${JSON.stringify(torn)}\n`, "utf8");+ const half = Math.floor(tornData.length / 2);+ const proto = await fileHandleProto(tracesDir);+ const originalWrite = proto.write;+ const writeSpy = vi.spyOn(proto, "write");+ try {+ // First syscall lands only half the record, the second fails: ENOSPC mid-record.+ writeSpy+ .mockImplementationOnce(function (this: unknown, ...args: unknown[]) {+ const [buf, offset] = args as [Buffer, number, number];+ return originalWrite.call(this, buf, offset, half);+ })+ .mockImplementationOnce(() => {+ return Promise.reject(+ Object.assign(new Error("ENOSPC: no space left on device, write"), {+ code: "ENOSPC",+ }),+ );+ });+ await expect(writer.write(torn)).rejects.toThrow(/ENOSPC/);+ } finally {+ writeSpy.mockRestore();+ }++ const healed = assistantText("after the failure");+ await writer.write(healed);++ const raw = await readFile(writer.currentPath(), "utf8");+ expect(raw).toBe(+ `${JSON.stringify(first)}\n` ++ `${tornData.subarray(0, half).toString("utf8")}\n` ++ `${JSON.stringify(healed)}\n`,+ );+ // The torn half-record is skipped as malformed; both intact records survive.+ expect(parseTraceLines(raw)).toEqual([first, healed]);+ });++ it("bails out of a zero-progress write and appends cleanly afterwards (nothing reached the file)", async () => {+ const writer = new Writer({+ tracesDir,+ sessionId: SESSION_ID,+ date: new Date(2026, 0, 9),+ });+ const writeSpy = vi.spyOn(await fileHandleProto(tracesDir), "write");+ try {+ writeSpy.mockImplementationOnce((...args: unknown[]) =>+ Promise.resolve({ bytesWritten: 0, buffer: args[0] }),+ );+ await expect(writer.write(assistantText("stuck"))).rejects.toThrow(/made no progress/);+ } finally {+ writeSpy.mockRestore();+ }++ // Zero bytes landed, so the tail is intact: the next append gets no heal prefix.+ const next = assistantText("clean append");+ await writer.write(next);+ expect(await readFile(writer.currentPath(), "utf8")).toBe(`${JSON.stringify(next)}\n`);+ });+ });++ // A process death mid-append can only truncate the LAST record (single-write appends), but+ // resumption keeps writing to the same file — without healing, the first post-resume record+ // would glue onto the torn line and be swallowed with it.+ describe("crash-torn tail healing on resume", () => {+ const DATE_DIR = "2026-01-09";++ /** A Writer resuming the session's existing 001 shard (dateDir pins the original file). */+ function resumingWriter(): Writer {+ return new Writer({ tracesDir, sessionId: SESSION_ID, dateDir: DATE_DIR, startIndex: 1 });+ }++ async function seedShard(content: string): Promise<string> {+ await mkdir(join(tracesDir, DATE_DIR), { recursive: true });+ const path = join(tracesDir, DATE_DIR, `${SESSION_ID}_001.jsonl`);+ await writeFile(path, content, "utf8");+ return path;+ }++ it("starts the next record on a fresh line after a torn tail", async () => {+ const head = meta();+ const intact = `${JSON.stringify(head)}\n`;+ const torn = JSON.stringify(assistantText("giant record")).slice(0, 40); // no trailing \n+ const path = await seedShard(intact + torn);++ const afterCrash = assistantText("after crash");+ await resumingWriter().write(afterCrash);++ const raw = await readFile(path, "utf8");+ expect(raw).toBe(`${intact + torn}\n${JSON.stringify(afterCrash)}\n`);+ // The torn line is skipped as malformed; both intact records survive.+ expect(parseTraceLines(raw)).toEqual([head, afterCrash]);+ });++ it("appends without a heal prefix when the resumed file ends cleanly", async () => {+ const intact = `${JSON.stringify(meta())}\n`;+ const path = await seedShard(intact);++ const next = assistantText("resumed");+ await resumingWriter().write(next);++ expect(await readFile(path, "utf8")).toBe(`${intact}${JSON.stringify(next)}\n`);+ });++ it("treats an empty pre-existing file as clean", async () => {+ const path = await seedShard("");++ const next = assistantText("first record");+ await resumingWriter().write(next);++ expect(await readFile(path, "utf8")).toBe(`${JSON.stringify(next)}\n`);+ });++ it("probes once per shard: a healed file gets no second prefix, and rotation resets the state", async () => {+ const torn = `{"timestamp":"x","type":"model_msg","payload":{"type":"text","role":"assist`;+ const path = await seedShard(torn);++ const writer = resumingWriter();+ const first = assistantText("heals");+ const second = assistantText("plain append");+ await writer.writeAll([first, second]);+ expect(await readFile(path, "utf8")).toBe(+ `${torn}\n${JSON.stringify(first)}\n${JSON.stringify(second)}\n`,+ );++ // The next shard is a brand-new file: no heal prefix may leak into it.+ await writer.rotate();+ const fresh = assistantText("next shard");+ await writer.write(fresh);+ expect(await readFile(writer.currentPath(), "utf8")).toBe(`${JSON.stringify(fresh)}\n`);+ });+ });});diff --git a/packages/docs/content/sessions-and-traces.en.md b/packages/docs/content/sessions-and-traces.en.mdindex f26dd912..c8f73071 100644--- a/packages/docs/content/sessions-and-traces.en.md+++ b/packages/docs/content/sessions-and-traces.en.md@@ -55,6 +55,7 @@ A Trace is an append-only JSON Lines file; each line is one OmniMessage envelope- Not recorded: streaming `partial_*` fragments (the producer appends the complete message once the segment ends), and nested messages tagged with `origin` — a subagent's messages go to the child Session's own Trace, while the parent Trace keeps a single `subagent` pointer event at the spawn site recording the child Session id.- `request_begin` and `request_end(status)` come in pairs delimiting one Request; replay uses `request_end.status === "completed"` as the commit criterion for that turn.- Appends are serialized inside the writer: records from concurrent producers (the model stream, parallel tool executions) land strictly one after another, each as a single uninterrupted line — a multi-megabyte record such as a base64 image Data URL can never be torn apart by a concurrent append, and file rotation never splits a record.+- Each record is appended with a single `write(2)` (not `fs.appendFile`, which splits payloads larger than 512 KiB into multiple underlying writes), so an abnormal process exit can at most truncate the last record — it can never tear one apart in the middle. Before Session resumption continues an existing file, the writer probes the file tail: if a previous crash left a torn line (no trailing newline), the next record is preceded by a newline, so the torn line never swallows the records appended after it.See `packages/core/src/trace/writer.ts` for the implementation.diff --git a/packages/docs/content/sessions-and-traces.zh.md b/packages/docs/content/sessions-and-traces.zh.mdindex ac61e115..b15a4f02 100644--- a/packages/docs/content/sessions-and-traces.zh.md+++ b/packages/docs/content/sessions-and-traces.zh.md@@ -53,6 +53,7 @@ Trace 是 append-only 的 JSON Lines 文件,每行一个 OmniMessage 信封(- 不记录的消息:流式 `partial_*` 分片(片段结束后由生产方补写完整消息);带 `origin` 标记的嵌套消息——子 Agent 的消息写入子 Session 自己的 Trace,父 Trace 只在派生位置保留一个 `subagent` 指针事件,记录子 Session id。- `request_begin` 与 `request_end(status)` 成对出现,界定一轮 Request;回放以 `request_end.status === "completed"` 作为该轮已提交的判据。- 追加在写入器内部串行执行:并发生产者(模型流、并行工具执行)的记录严格逐条落盘,每条都是一行完整内容——base64 图片 Data URL 之类的多 MB 大记录不会被并发追加撕裂,文件轮转也不会切断任何记录。+- 每条记录以单次 `write(2)` 追加(不用 `fs.appendFile`——它会把超过 512 KiB 的负载拆成多次底层写入),因此进程异常退出至多截断最后一条记录,不会把某条记录从中间撕开。Session 恢复续写既有文件前会探测文件尾:若上次异常退出留下了残行(末尾不是换行符),下一条记录会先补换行再落盘,残行不会吞掉后续记录。实现见 `packages/core/src/trace/writer.ts`。