Skip to content

A local-first multi-agent app development platform, built on PenguinHarness

fix(core): single-write Trace appends and crash-torn tail healing (#249)

Yaowei Zheng3b493df37d58
diff --git a/packages/core/src/trace/writer.ts b/packages/core/src/trace/writer.ts
index c14d6e80..2afcd344 100644
--- a/packages/core/src/trace/writer.ts
+++ b/packages/core/src/trace/writer.ts
@@ -16,7 +16,7 @@
* context_engine).
* - Path convention: `<tracesDir>/<yyyy-mm-dd>/<sessionId>_<index3>.jsonl`.
*/
-import { appendFile, mkdir, readFile } from "node:fs/promises";
+import { mkdir, open, readFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import {
@@ -68,22 +68,37 @@ function isRecordable(msg: OmniMessage): boolean {
/**
* append-only JSONL Trace writer.
*
- * Every write uses `appendFile` (O_APPEND) rather than caching a file handle and seeking to
- * write, avoiding overwriting existing content; this also removes the need for an explicit
- * close.
+ * Every append opens the file with O_APPEND, issues **one** `write(2)` for the whole record,
+ * and closes the handle — no cached handle, no seeking, so existing content can never be
+ * overwritten. `fs.appendFile` is deliberately NOT used: it splits payloads larger than
+ * 512 KiB into multiple underlying writes (Node's internal `kWriteFileMaxChunkSize`), so a
+ * process death between chunks leaves exactly the first 524288 bytes of a big record (a huge
+ * tool_call's arguments, a base64 image Data URL) on disk — and every record appended after
+ * it glues onto that torn line, turning one crash into an unparseable line that swallows
+ * later records too. A single-write append can only be cut short by the OS itself (power
+ * loss, ENOSPC short write), which the tail-heal below and the readers' torn-tail tolerance
+ * (parseTraceLines) are built for.
+ *
+ * Tail healing: before the first append this instance makes to a **pre-existing** shard file
+ * (Session resumption continues the original file), the last byte is probed — a file not
+ * ending in `\n` carries a crash-torn tail, and the next record is prefixed with `\n` so it
+ * starts on a fresh line instead of merging into the torn one (the torn line itself is
+ * skipped by the tolerant readers; the records after it stay readable). A mid-record append
+ * failure inside this process (ENOSPC, forced short write) marks the tail torn and heals the
+ * same way on the next append. A spurious heal costs one blank line, which every reader
+ * ignores.
*
* Concurrency: one live Session has exactly one Writer instance, but that instance receives
* appends from **multiple async producers in the same process** — the engine's LLM stream
- * driver and each parallel tool execution write independently. `appendFile` splits large
- * payloads (multi-MB records such as base64 image Data URLs) into multiple underlying writes,
- * so two overlapping appends can interleave mid-record and corrupt the JSONL (#215). All
- * mutating operations (`write`, `rotate`) are therefore serialized through one per-instance
- * promise chain: each record lands as one uninterrupted append, and a rotation cannot land in
- * the middle of an append. Cross-instance/file concurrency does not occur by design **within a
- * single server/CLI process**: a Session allows one active run at a time, child sessions write
- * their own files, and server-side Trace import only ever creates brand-new files (`wx`). Two
+ * driver and each parallel tool execution write independently (#215). All mutating operations
+ * (`write`, `rotate`) are therefore serialized through one per-instance promise chain: each
+ * record lands as one uninterrupted append, and a rotation cannot land in the middle of an
+ * append. Cross-instance/file concurrency does not occur by design **within a single
+ * server/CLI process**: a Session allows one active run at a time, child sessions write their
+ * own files, and server-side Trace import only ever creates brand-new files (`wx`). Two
* processes pointed at the same agent data directory are outside this contract (and outside
- * supported usage) — the chain cannot cover them.
+ * supported usage), though the single-write append keeps even that case tear-free on local
+ * filesystems that make O_APPEND writes atomic.
*
* Error semantics: a failed operation rejects **that caller's** returned promise only; the
* chain itself absorbs the failure so subsequent operations still run (a transient disk error
@@ -99,8 +114,10 @@ export class Writer {
private readonly dateDir: string;
/** Current Trace index, starting at 1; incremented by `rotate()`. */
private index = 1;
- /** Set true once the date directory has been created for the current file, to avoid a redundant mkdir. */
- private ensuredDirForIndex = -1;
+ /** Index whose file is prepared (date directory created, pre-existing tail probed); avoids redoing either per append. */
+ private preparedIndex = -1;
+ /** True while the current shard may end mid-record — assigned per shard by the tail probe, set by a mid-record append failure, cleared once a record fully lands. The next record then starts on a fresh line. */
+ private tornTail = false;
/** Serialization chain: mutating operations run strictly in submission order (see class docs). */
private chain: Promise<void> = Promise.resolve();
@@ -133,7 +150,8 @@ export class Writer {
/**
* Appends one message. Only written if it's a recordable message; streaming `partial_*` is
- * skipped. `mkdir -p`s the date directory on the first write to the current file.
+ * skipped. The first write to the current file `mkdir -p`s the date directory and probes a
+ * pre-existing file for a crash-torn tail (see `probeTornTail`).
*
* The append is serialized on the instance chain, so the record lands as one uninterrupted
* JSONL line even when other writes (or a rotation) are submitted concurrently; the target
@@ -145,14 +163,74 @@ export class Writer {
if (!isRecordable(msg)) return;
return this.enqueue(async () => {
const path = this.currentPath();
- if (this.ensuredDirForIndex !== this.index) {
+ if (this.preparedIndex !== this.index) {
await mkdir(dirname(path), { recursive: true });
- this.ensuredDirForIndex = this.index;
+ this.tornTail = await this.probeTornTail(path);
+ this.preparedIndex = this.index;
}
- await appendFile(path, `${JSON.stringify(msg)}\n`, "utf8");
+ const record = `${JSON.stringify(msg)}\n`;
+ await this.appendRecord(path, this.tornTail ? `\n${record}` : record);
+ this.tornTail = false; // the record (and any heal prefix) fully reached the file
});
}
+ /**
+ * Whether a pre-existing shard file ends mid-record (resumption continues the original
+ * file): a non-empty file whose last byte is not `\n` carries a crash-torn tail, and the
+ * next append must not merge into that line. A missing or empty file is simply fresh; any
+ * other probe error surfaces to the caller like an append failure (and is retried with the
+ * next write, since the index stays unprepared).
+ */
+ private async probeTornTail(path: string): Promise<boolean> {
+ let fh;
+ try {
+ fh = await open(path, "r");
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException).code === "ENOENT") return false;
+ throw err;
+ }
+ try {
+ const { size } = await fh.stat();
+ if (size === 0) return false;
+ const { bytesRead, buffer } = await fh.read(Buffer.alloc(1), 0, 1, size - 1);
+ return bytesRead !== 1 || buffer[0] !== 0x0a;
+ } finally {
+ await fh.close();
+ }
+ }
+
+ /**
+ * Appends one record through a single `write(2)` on an O_APPEND handle (see the class docs
+ * for why `appendFile`'s 512 KiB chunking is unacceptable here). The short-write loop is a
+ * safety net only — one iteration is the norm; if a retry is ever needed and then fails,
+ * part of the record is on disk, so the tail is marked torn for the next append to heal.
+ */
+ private async appendRecord(path: string, data: string): Promise<void> {
+ const buf = Buffer.from(data, "utf8");
+ const fh = await open(path, "a");
+ try {
+ let written = 0;
+ while (written < buf.length) {
+ let bytesWritten: number;
+ try {
+ ({ bytesWritten } = await fh.write(buf, written, buf.length - written));
+ } catch (err) {
+ if (written > 0) this.tornTail = true;
+ throw err;
+ }
+ if (bytesWritten <= 0) {
+ // A zero-progress write (out-of-space filesystems can report this instead of
+ // throwing): bail out rather than loop forever.
+ if (written > 0) this.tornTail = true;
+ throw new Error(`Trace append made no progress at byte ${written} of ${buf.length}`);
+ }
+ written += bytesWritten;
+ }
+ } finally {
+ await fh.close();
+ }
+ }
+
/** Writes multiple messages in sequence. */
async writeAll(msgs: OmniMessage[]): Promise<void> {
for (const msg of msgs) {
@@ -183,6 +261,8 @@ export class Writer {
*/
async rotate(): Promise<void> {
return this.enqueue(async () => {
+ // No torn-tail bookkeeping here: the first write to the new index re-probes and
+ // reassigns the flag, so the previous shard's state cannot leak into the next file.
this.index += 1;
});
}
diff --git a/packages/core/test/trace.test.ts b/packages/core/test/trace.test.ts
index 6c50c202..2a19ffbe 100644
--- a/packages/core/test/trace.test.ts
+++ b/packages/core/test/trace.test.ts
@@ -1,8 +1,8 @@
-import { mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
+import { mkdir, mkdtemp, open, readFile, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, sep } from "node:path";
-import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import {
assistantText,
@@ -15,10 +15,18 @@ import {
withOrigin,
} from "../src/omnimessage/index.js";
import type { OmniMessage } from "../src/omnimessage/index.js";
-import { Writer, readTrace } from "../src/trace/index.js";
+import { Writer, parseTraceLines, readTrace } from "../src/trace/index.js";
const SESSION_ID = "sess_abc";
+/** FileHandle's prototype, grabbed from a throwaway handle (the class is not exported). */
+async function fileHandleProto(dir: string): Promise<{ write: (...args: unknown[]) => unknown }> {
+ const probe = await open(join(dir, "probe.tmp"), "w");
+ const proto = Object.getPrototypeOf(probe) as { write: (...args: unknown[]) => unknown };
+ await probe.close();
+ return proto;
+}
+
function meta() {
return sessionMeta({
session_id: SESSION_ID,
@@ -219,6 +227,34 @@ describe("Writer", () => {
expect(secondLines).toEqual(shardTwo.map((msg) => JSON.stringify(msg)));
});
+ it("appends a multi-MB record with a single underlying write (no 512 KiB chunk tear window)", async () => {
+ // fs.appendFile would split this into three writes (Node chunks at 512 KiB = 524288
+ // bytes); a crash between chunks used to leave exactly the first 524288 bytes of the
+ // record, and the next session's appends then glued onto that torn line. The Writer
+ // issues one write(2) per record instead, so no crash can split a record mid-file.
+ const writer = new Writer({
+ tracesDir,
+ sessionId: SESSION_ID,
+ date: new Date(2026, 0, 9),
+ });
+ const head = meta();
+ const big = imageUrlMessage(`data:image/png;base64,${"C".repeat(1200 * 1024)}`);
+ const bigRecordBytes = Buffer.byteLength(`${JSON.stringify(big)}\n`);
+
+ const writeSpy = vi.spyOn(await fileHandleProto(tracesDir), "write");
+ try {
+ await writer.writeAll([head, big]);
+ // One write() per record — the big one lands whole, never in 524288-byte chunks
+ // (>= so a chunk of exactly 524288 bytes would be caught too).
+ const lengths = writeSpy.mock.calls.map((call) => call[2]);
+ expect(lengths).toContain(bigRecordBytes);
+ expect(lengths.filter((len) => (len as number) >= 512 * 1024)).toEqual([bigRecordBytes]);
+ } finally {
+ writeSpy.mockRestore();
+ }
+ expect(await readTrace(writer.currentPath())).toEqual([head, big]);
+ });
+
it("a failed write rejects its own caller without wedging subsequent writes", async () => {
const writer = new Writer({
tracesDir,
@@ -243,4 +279,149 @@ describe("Writer", () => {
expect(rows).toEqual([recovered]);
});
});
+
+ // The failure branches of the single-write append: a partial write leaves a torn tail that
+ // the next append must heal; a zero-progress write must bail out instead of looping.
+ describe("append failure semantics", () => {
+ it("marks the tail torn after a partial append failure and heals on the next write", async () => {
+ const writer = new Writer({
+ tracesDir,
+ sessionId: SESSION_ID,
+ date: new Date(2026, 0, 9),
+ });
+ const first = assistantText("intact record");
+ await writer.write(first); // the shard exists with a clean tail before the failure
+
+ const torn = assistantText("torn mid-write");
+ const tornData = Buffer.from(`${JSON.stringify(torn)}\n`, "utf8");
+ const half = Math.floor(tornData.length / 2);
+ const proto = await fileHandleProto(tracesDir);
+ const originalWrite = proto.write;
+ const writeSpy = vi.spyOn(proto, "write");
+ try {
+ // First syscall lands only half the record, the second fails: ENOSPC mid-record.
+ writeSpy
+ .mockImplementationOnce(function (this: unknown, ...args: unknown[]) {
+ const [buf, offset] = args as [Buffer, number, number];
+ return originalWrite.call(this, buf, offset, half);
+ })
+ .mockImplementationOnce(() => {
+ return Promise.reject(
+ Object.assign(new Error("ENOSPC: no space left on device, write"), {
+ code: "ENOSPC",
+ }),
+ );
+ });
+ await expect(writer.write(torn)).rejects.toThrow(/ENOSPC/);
+ } finally {
+ writeSpy.mockRestore();
+ }
+
+ const healed = assistantText("after the failure");
+ await writer.write(healed);
+
+ const raw = await readFile(writer.currentPath(), "utf8");
+ expect(raw).toBe(
+ `${JSON.stringify(first)}\n` +
+ `${tornData.subarray(0, half).toString("utf8")}\n` +
+ `${JSON.stringify(healed)}\n`,
+ );
+ // The torn half-record is skipped as malformed; both intact records survive.
+ expect(parseTraceLines(raw)).toEqual([first, healed]);
+ });
+
+ it("bails out of a zero-progress write and appends cleanly afterwards (nothing reached the file)", async () => {
+ const writer = new Writer({
+ tracesDir,
+ sessionId: SESSION_ID,
+ date: new Date(2026, 0, 9),
+ });
+ const writeSpy = vi.spyOn(await fileHandleProto(tracesDir), "write");
+ try {
+ writeSpy.mockImplementationOnce((...args: unknown[]) =>
+ Promise.resolve({ bytesWritten: 0, buffer: args[0] }),
+ );
+ await expect(writer.write(assistantText("stuck"))).rejects.toThrow(/made no progress/);
+ } finally {
+ writeSpy.mockRestore();
+ }
+
+ // Zero bytes landed, so the tail is intact: the next append gets no heal prefix.
+ const next = assistantText("clean append");
+ await writer.write(next);
+ expect(await readFile(writer.currentPath(), "utf8")).toBe(`${JSON.stringify(next)}\n`);
+ });
+ });
+
+ // A process death mid-append can only truncate the LAST record (single-write appends), but
+ // resumption keeps writing to the same file — without healing, the first post-resume record
+ // would glue onto the torn line and be swallowed with it.
+ describe("crash-torn tail healing on resume", () => {
+ const DATE_DIR = "2026-01-09";
+
+ /** A Writer resuming the session's existing 001 shard (dateDir pins the original file). */
+ function resumingWriter(): Writer {
+ return new Writer({ tracesDir, sessionId: SESSION_ID, dateDir: DATE_DIR, startIndex: 1 });
+ }
+
+ async function seedShard(content: string): Promise<string> {
+ await mkdir(join(tracesDir, DATE_DIR), { recursive: true });
+ const path = join(tracesDir, DATE_DIR, `${SESSION_ID}_001.jsonl`);
+ await writeFile(path, content, "utf8");
+ return path;
+ }
+
+ it("starts the next record on a fresh line after a torn tail", async () => {
+ const head = meta();
+ const intact = `${JSON.stringify(head)}\n`;
+ const torn = JSON.stringify(assistantText("giant record")).slice(0, 40); // no trailing \n
+ const path = await seedShard(intact + torn);
+
+ const afterCrash = assistantText("after crash");
+ await resumingWriter().write(afterCrash);
+
+ const raw = await readFile(path, "utf8");
+ expect(raw).toBe(`${intact + torn}\n${JSON.stringify(afterCrash)}\n`);
+ // The torn line is skipped as malformed; both intact records survive.
+ expect(parseTraceLines(raw)).toEqual([head, afterCrash]);
+ });
+
+ it("appends without a heal prefix when the resumed file ends cleanly", async () => {
+ const intact = `${JSON.stringify(meta())}\n`;
+ const path = await seedShard(intact);
+
+ const next = assistantText("resumed");
+ await resumingWriter().write(next);
+
+ expect(await readFile(path, "utf8")).toBe(`${intact}${JSON.stringify(next)}\n`);
+ });
+
+ it("treats an empty pre-existing file as clean", async () => {
+ const path = await seedShard("");
+
+ const next = assistantText("first record");
+ await resumingWriter().write(next);
+
+ expect(await readFile(path, "utf8")).toBe(`${JSON.stringify(next)}\n`);
+ });
+
+ it("probes once per shard: a healed file gets no second prefix, and rotation resets the state", async () => {
+ const torn = `{"timestamp":"x","type":"model_msg","payload":{"type":"text","role":"assist`;
+ const path = await seedShard(torn);
+
+ const writer = resumingWriter();
+ const first = assistantText("heals");
+ const second = assistantText("plain append");
+ await writer.writeAll([first, second]);
+ expect(await readFile(path, "utf8")).toBe(
+ `${torn}\n${JSON.stringify(first)}\n${JSON.stringify(second)}\n`,
+ );
+
+ // The next shard is a brand-new file: no heal prefix may leak into it.
+ await writer.rotate();
+ const fresh = assistantText("next shard");
+ await writer.write(fresh);
+ expect(await readFile(writer.currentPath(), "utf8")).toBe(`${JSON.stringify(fresh)}\n`);
+ });
+ });
});
diff --git a/packages/docs/content/sessions-and-traces.en.md b/packages/docs/content/sessions-and-traces.en.md
index f26dd912..c8f73071 100644
--- a/packages/docs/content/sessions-and-traces.en.md
+++ b/packages/docs/content/sessions-and-traces.en.md
@@ -55,6 +55,7 @@ A Trace is an append-only JSON Lines file; each line is one OmniMessage envelope
- Not recorded: streaming `partial_*` fragments (the producer appends the complete message once the segment ends), and nested messages tagged with `origin` — a subagent's messages go to the child Session's own Trace, while the parent Trace keeps a single `subagent` pointer event at the spawn site recording the child Session id.
- `request_begin` and `request_end(status)` come in pairs delimiting one Request; replay uses `request_end.status === "completed"` as the commit criterion for that turn.
- Appends are serialized inside the writer: records from concurrent producers (the model stream, parallel tool executions) land strictly one after another, each as a single uninterrupted line — a multi-megabyte record such as a base64 image Data URL can never be torn apart by a concurrent append, and file rotation never splits a record.
+- Each record is appended with a single `write(2)` (not `fs.appendFile`, which splits payloads larger than 512 KiB into multiple underlying writes), so an abnormal process exit can at most truncate the last record — it can never tear one apart in the middle. Before Session resumption continues an existing file, the writer probes the file tail: if a previous crash left a torn line (no trailing newline), the next record is preceded by a newline, so the torn line never swallows the records appended after it.
See `packages/core/src/trace/writer.ts` for the implementation.
diff --git a/packages/docs/content/sessions-and-traces.zh.md b/packages/docs/content/sessions-and-traces.zh.md
index ac61e115..b15a4f02 100644
--- a/packages/docs/content/sessions-and-traces.zh.md
+++ b/packages/docs/content/sessions-and-traces.zh.md
@@ -53,6 +53,7 @@ Trace 是 append-only 的 JSON Lines 文件,每行一个 OmniMessage 信封(
- 不记录的消息:流式 `partial_*` 分片(片段结束后由生产方补写完整消息);带 `origin` 标记的嵌套消息——子 Agent 的消息写入子 Session 自己的 Trace,父 Trace 只在派生位置保留一个 `subagent` 指针事件,记录子 Session id。
- `request_begin` 与 `request_end(status)` 成对出现,界定一轮 Request;回放以 `request_end.status === "completed"` 作为该轮已提交的判据。
- 追加在写入器内部串行执行:并发生产者(模型流、并行工具执行)的记录严格逐条落盘,每条都是一行完整内容——base64 图片 Data URL 之类的多 MB 大记录不会被并发追加撕裂,文件轮转也不会切断任何记录。
+- 每条记录以单次 `write(2)` 追加(不用 `fs.appendFile`——它会把超过 512 KiB 的负载拆成多次底层写入),因此进程异常退出至多截断最后一条记录,不会把某条记录从中间撕开。Session 恢复续写既有文件前会探测文件尾:若上次异常退出留下了残行(末尾不是换行符),下一条记录会先补换行再落盘,残行不会吞掉后续记录。
实现见 `packages/core/src/trace/writer.ts`。