Skip to content

Node-based terminal manager for AI coding agents — tmux-backed terminals and parallel agent sessions as draggable nodes on an infinite pan/zoom canvas. macOS, Linux, and a browser Server Edition.

fix(durable-state): review of #1054 — sender told at boot, byte budgets, no stale overwrite

review9212315f82d4
diff --git a/CLAUDE.md b/CLAUDE.md
index 853b8932e..8d3c06a96 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -7469,10 +7469,29 @@ spec, never a fourth copy of the read / sanitize / write / flush code.
lists are capped, and a file that is not JSON / too large (16 MB) / not an envelope is set aside as
`<file>.corrupt` (one copy) and the fact starts EMPTY with a warning. **Loading never throws** — a
bad orchestration file must not take the boot with it.
-- **Writes** go through `writeFileAtomic` (the fs-atomic guard applies), coalesced over 50 ms, one
- in flight, latest snapshot wins; `flushAllDurableFactsSync()` runs on the desktop's second
- `before-quit` pass, `hookServer.stop()` flushes the ledger, and the Server Edition's canvas-control
- `stop()` disposes its two files. A **crash** inside the 50 ms window loses that window.
+- **Writes** are a unique `wx` temp then `renameAtomicSync` (the fs-atomic guard applies), coalesced
+ over 50 ms, one in flight, latest snapshot wins; `flushAllDurableFactsSync()` runs on the desktop's
+ second `before-quit` pass, `hookServer.stop()` flushes the ledger, and the Server Edition's
+ canvas-control `stop()` disposes its three files (queue, reports, holds). A **crash** inside the
+ 50 ms window loses that window. **A sync flush is never overwritten by an older async write**
+ (review of #1054, reproduced: `save([1])`, `flush()`, `save([2])`, `flushSync()` left `[1]` on
+ disk): `flushSync` bumps a generation, and the async path checks it and renames in ONE synchronous
+ step, dropping its temp when stale.
+- **Every owner budgets its bytes under the 16 MB load limit**, because a file past it is set aside
+ WHOLE (reproduced in review: 80 queued 250K-char bodies wrote 20 MB and 0 of 80 came back). The
+ queue writes full entries up to 8 MB of JSON and the rest REDUCED (no body, short fields only —
+ `bodyOmitted`, which restore turns into an expiry the sender hears about); the ledger writes
+ replies up to 8 MB and the rest as UNKNOWN rows (still refused, never re-run).
+- **Only the instance that owns the hook endpoint owns the facts.** The ledger is loaded by
+ `hookServer.start()`, which fails for a second instance; the queue, reports and holds follow the
+ same rule (`standDown()` when `startForApp()` returned a warning — desktop `main/index.ts`, Server
+ Edition `ownsDurableState`), so a second instance on the same userData neither expires messages
+ the live one holds nor overwrites its files.
+- **The desktop queue restore waits for the workspace INDEX** (`restoreDeliveryQueue(…, {ready:
+ workspaceStore.load({sideline:false})})`): an entry that lapsed during the downtime is expired
+ there, and its sender leg resolves the sender's project and board log through the index, which
+ nothing else has loaded at that point of boot (review of #1054: the expiry otherwise reached only
+ the in-memory trace ring). The Server Edition already awaited the load before canvas control.
- **Measured** (this Linux dev host, the ledger at its 4096-row cap with realistic open replies):
377 bytes a row, a 1.5 MB file, 11.5 ms per full write, 16.6 ms to load at boot; a 20-row ledger
writes in 0.41 ms. The queue is bounded far lower (16 per target), the reports at 1000.
@@ -7529,8 +7548,10 @@ What a restart MEANS, per fact — decided beside each fact, stated in its heade
refused, never re-run, and nothing can settle it any more because its late answer died with the
old process; a reply over 64 KB is written as UNKNOWN rather than dropped (a missing row would let
the retry run).
-- **Known limit:** two instances sharing one userData (a dev sandbox) are last-writer-wins on these
- files, and both would expire the same lapsed messages at boot. Nothing locks them.
+- **Delivery is at most once across a crash:** a flush writes the entry OFF disk before its
+ attempt (claim before effect), so a crash mid-delivery loses that one message rather than typing
+ it twice after the next boot. Lapsed entries at restore are never inserted into the live lists
+ (so no flush can deliver one while the expiries are reported) and take no capacity slot.
## The test suite never touches a live tmux server
diff --git a/src/core/agents/agent-messaging.ts b/src/core/agents/agent-messaging.ts
index 875aa92e2..dc234ec23 100644
--- a/src/core/agents/agent-messaging.ts
+++ b/src/core/agents/agent-messaging.ts
@@ -408,9 +408,19 @@ export function createDeliveryQueue(
*/
export async function restoreDeliveryQueue(
queue: DeliveryQueue,
- file: Pick<DurableFactFile<PersistedQueueEntry>, 'load'>
+ file: Pick<DurableFactFile<PersistedQueueEntry>, 'load'>,
+ opts: {
+ /**
+ * Resolves once the stores an expiry's sender leg reads are loaded — on the desktop the
+ * workspace INDEX (`projects()` and the board-log routes resolve nothing before it, so an entry
+ * expired at restore would reach only the in-memory trace ring and its sender would never hear).
+ * A rejection is waited out, not propagated: the restore still runs.
+ */
+ ready?: Promise<unknown>
+ } = {}
): Promise<void> {
try {
+ if (opts.ready) await opts.ready.catch(() => undefined)
await queue.restore(file.load())
} catch (e) {
console.warn(`[agent-messaging] could not restore the delivery queue (${String(e)})`)
diff --git a/src/core/agents/delivery-queue.persist.test.ts b/src/core/agents/delivery-queue.persist.test.ts
index 0ac0ecc7c..3207440bf 100644
--- a/src/core/agents/delivery-queue.persist.test.ts
+++ b/src/core/agents/delivery-queue.persist.test.ts
@@ -4,6 +4,7 @@ import {
DELIVERY_QUEUE_TTL_MS,
QUEUE_FACT,
QUEUE_PERSIST_BODY_MAX,
+ QUEUE_PERSIST_BYTES_BUDGET,
restoredBindingVerdict,
sanitizePersistedQueueEntry,
type CancelTimer,
@@ -13,7 +14,9 @@ import {
type QueuedDeliveryRequest
} from './delivery-queue'
import type { AgentMessageOutcome } from './agent-message-decide'
-import { DurableFactFile } from '../durable-state'
+import { DURABLE_STATE_MAX_BYTES, DurableFactFile } from '../durable-state'
+import { restoreDeliveryQueue } from './agent-messaging'
+import fs from 'node:fs'
import { testTmpDir } from '../test-tmp'
/**
@@ -24,7 +27,13 @@ import { testTmpDir } from '../test-tmp'
* for; a board comment and a station notice are never replayed into a pane.
*/
-function instance(opts: { now: () => number; binding: (id: string) => QueueBinding | undefined; outcome?: AgentMessageOutcome }) {
+function instance(opts: {
+ now: () => number
+ binding: (id: string) => QueueBinding | undefined
+ outcome?: AgentMessageOutcome
+ trace?: DeliveryQueueDeps['trace']
+ onDeliver?: (req: QueuedDeliveryRequest) => void
+}) {
const delivered: QueuedDeliveryRequest[] = []
const expired: { req: QueuedDeliveryRequest; queuedForMs: number }[] = []
const flushed: { req: QueuedDeliveryRequest; outcome: AgentMessageOutcome }[] = []
@@ -35,13 +44,16 @@ function instance(opts: { now: () => number; binding: (id: string) => QueueBindi
const deps: DeliveryQueueDeps = {
now: opts.now,
deliver: async (req) => {
+ opts.onDeliver?.(req)
delivered.push(req)
return opts.outcome ?? { kind: 'delivered', traceId: 'd', traced: 'memory', receipt: 'observed', signal: 'newTurn' }
},
- trace: async (input) => {
- traced.push(input.outcome)
- return { traceId: `t${traced.length}`, traced: 'memory' }
- },
+ trace:
+ opts.trace ??
+ (async (input) => {
+ traced.push(input.outcome)
+ return { traceId: `t${traced.length}`, traced: 'memory' }
+ }),
onExpired: (req, info) => expired.push({ req, queuedForMs: info.queuedForMs }),
onFlushed: (req, outcome) => flushed.push({ req, outcome }),
onQueued: (req) => queued.push(req),
@@ -198,4 +210,92 @@ describe('delivery queue across a restart', () => {
await b.queue.onTargetIdle('st1')
expect(b.delivered.map((r) => r.body)).toEqual(['next task'])
})
+
+ it('the written file stays under the load limit however much is queued; nothing is silently dropped (review repro)', async () => {
+ const dir = testTmpDir('nt-queue-budget-')
+ const file = new DurableFactFile(QUEUE_FACT, { userDataDir: dir, debounceMs: 1 })
+ const a = instance({ now: () => 1000, binding: () => ({ sessionId: 's' }) })
+ for (let t = 0; t < 5; t++)
+ for (let i = 0; i < 16; i++)
+ await a.queue.enqueue(req({ targetNodeId: `st${t}`, body: `${t}-${i}-` + 'x'.repeat(250_000) }))
+ file.save(a.saved())
+ await file.flush()
+ expect(fs.statSync(file.path).size).toBeLessThan(DURABLE_STATE_MAX_BYTES)
+ const loaded = new DurableFactFile(QUEUE_FACT, { userDataDir: dir }).load()
+ expect(loaded).toHaveLength(80)
+ const full = loaded.filter((e) => !e.bodyOmitted)
+ expect(full.length).toBeGreaterThan(0)
+ expect(full.reduce((n, e) => n + JSON.stringify(e).length, 0)).toBeLessThanOrEqual(QUEUE_PERSIST_BYTES_BUDGET)
+ // Every reduced entry is ENDED loudly at restore, the full ones wait for their target.
+ const b = instance({ now: () => 2000, binding: () => ({ sessionId: 's' }) })
+ await b.queue.restore(loaded)
+ expect(b.expired).toHaveLength(80 - full.length)
+ expect(b.expired.length + [0, 1, 2, 3, 4].reduce((n, t) => n + b.queue.depth(`st${t}`), 0)).toBe(80)
+ })
+
+ it('lapsed entries never enter the live lists: a flush during an expiry\'s await delivers nothing of them', async () => {
+ const a = instance({ now: () => 1000, binding: () => ({ sessionId: 's' }) })
+ await a.queue.enqueue(req({ verb: 'board-comment', sourceNodeId: 'board-comment:c1', projectId: 'p1' }))
+ await a.queue.enqueue(req({ body: 'x'.repeat(QUEUE_PERSIST_BODY_MAX + 1) }))
+ let release = (): void => {}
+ let b!: ReturnType<typeof instance>
+ const stalled = new Promise<void>((r) => (release = r))
+ b = instance({
+ now: () => 2000,
+ binding: () => ({ sessionId: 's' }),
+ trace: async () => {
+ await b.queue.onTargetIdle('st1') // a `done` arriving while the expiry is being reported
+ await stalled
+ return { traceId: 't', traced: 'memory' }
+ }
+ })
+ const restoring = b.queue.restore(onDisk(a.saved()))
+ await new Promise((r) => setTimeout(r, 0))
+ release()
+ await restoring
+ expect(b.delivered).toEqual([])
+ expect(b.expired).toHaveLength(2)
+ })
+
+ it('lapsed entries take no capacity: 16 deliverable ones all come back beside an expired one', async () => {
+ const entries: PersistedQueueEntry[] = [
+ { req: req({ verb: 'board-comment', sourceNodeId: 'board-comment:c1' }), enqueuedAt: 1000, ttlMs: DELIVERY_QUEUE_TTL_MS, queuedTraceId: 't' },
+ ...Array.from({ length: 16 }, (_, i) => ({
+ req: req({ body: `m${i}` }),
+ enqueuedAt: 1000,
+ ttlMs: DELIVERY_QUEUE_TTL_MS,
+ queuedTraceId: `t${i}`,
+ binding: { sessionId: 's' }
+ }))
+ ]
+ const b = instance({ now: () => 2000, binding: () => ({ sessionId: 's' }) })
+ await b.queue.restore(onDisk(entries))
+ expect(b.queue.depth('st1')).toBe(16)
+ expect(b.expired.map((e) => e.req.verb)).toEqual(['board-comment'])
+ })
+
+ it('an entry is written off disk BEFORE its delivery attempt (at most once across a crash)', async () => {
+ let onDiskDuringDelivery: PersistedQueueEntry[] | null = null
+ let a!: ReturnType<typeof instance>
+ a = instance({ now: () => 1000, binding: () => ({ sessionId: 's' }), onDeliver: () => (onDiskDuringDelivery = a.saved()) })
+ await a.queue.enqueue(req())
+ expect(a.saved()).toHaveLength(1)
+ await a.queue.onTargetIdle('st1')
+ expect(onDiskDuringDelivery).toEqual([])
+ })
+
+ it('restoreDeliveryQueue reports no expiry until `ready` (the desktop workspace index) has resolved', async () => {
+ const a = instance({ now: () => 1000, binding: () => ({ sessionId: 's' }) })
+ await a.queue.enqueue(req())
+ const b = instance({ now: () => 1000 + DELIVERY_QUEUE_TTL_MS + 1, binding: () => ({ sessionId: 's' }) })
+ let loaded = (): void => {}
+ const ready = new Promise<void>((r) => (loaded = r))
+ const disk = onDisk(a.saved())
+ const done = restoreDeliveryQueue(b.queue, { load: () => disk }, { ready })
+ await new Promise((r) => setTimeout(r, 5))
+ expect(b.expired).toEqual([])
+ loaded()
+ await done
+ expect(b.expired).toHaveLength(1)
+ })
})
diff --git a/src/core/agents/delivery-queue.ts b/src/core/agents/delivery-queue.ts
index 00e471518..3a4d81e1c 100644
--- a/src/core/agents/delivery-queue.ts
+++ b/src/core/agents/delivery-queue.ts
@@ -190,6 +190,38 @@ export interface PersistedQueueEntry {
/** A body larger than this is not written; its entry is expired at restore (sender told). */
export const QUEUE_PERSIST_BODY_MAX = 256 * 1024
+/** JSON bytes of FULL entries a snapshot writes before the rest go out reduced (`snapshot`). Half
+ * the file limit: a reduced entry is a few hundred bytes, so 1024 of them fit in the other half. */
+export const QUEUE_PERSIST_BYTES_BUDGET = 8 * 1024 * 1024
+
+/** A reduced entry keeps only what its expiry needs to be routed and traced. */
+const REDUCED_STRING_MAX = 200
+const REDUCED_EXTRAS_MAX = 4
+
+function clip(v: string): string {
+ return v.length > REDUCED_STRING_MAX ? v.slice(0, REDUCED_STRING_MAX) : v
+}
+
+/** The request of an entry written without its body: short fields only, never a message text. */
+function reducedRequest(req: QueuedDeliveryRequest): QueuedDeliveryRequest {
+ const out: QueuedDeliveryRequest = {
+ verb: typeof req.verb === 'string' ? clip(req.verb) : req.verb,
+ sourceNodeId: clip(req.sourceNodeId),
+ targetNodeId: req.targetNodeId,
+ sourceTitle: clip(req.sourceTitle),
+ body: ''
+ }
+ let extras = 0
+ for (const [k, v] of Object.entries(req)) {
+ if (k in out || extras >= REDUCED_EXTRAS_MAX) continue
+ if (typeof v === 'number' || typeof v === 'boolean' || (typeof v === 'string' && v.length <= REDUCED_STRING_MAX)) {
+ out[k] = v
+ extras++
+ }
+ }
+ return out
+}
+
/** At most this many entries on disk. `DELIVERY_QUEUE_CAPACITY` per target bounds it in practice. */
export const QUEUE_PERSIST_MAX = 1024
/** The longest TTL a restored entry may claim — a hand-edited `ttlMs` must not keep one forever. */
@@ -383,6 +415,9 @@ export class DeliveryQueue {
// must not flush the same entry twice. It goes back on failure, at the FRONT, preserving order.
list.shift()
entry.cancelTimer()
+ // Written off disk BEFORE the attempt (claim before effect): a crash mid-delivery then loses
+ // this one message rather than typing it twice after the next boot. At most once.
+ this.persist()
// A restored entry goes only into the session it was queued for (see the header).
const verdict = entry.restored
? restoredBindingVerdict(entry.binding, this.deps.bindingOf?.(nodeId))
@@ -453,6 +488,11 @@ export class DeliveryQueue {
if (list.length === 0) this.queues.delete(nodeId)
entry.cancelTimer()
this.persist()
+ await this.reportExpired(entry)
+ }
+
+ /** Trace `expired` and tell the sender — the two legs every expiry owes. */
+ private async reportExpired(entry: QueueEntry): Promise<void> {
const queuedForMs = this.deps.now() - entry.enqueuedAt
const t = await this.deps.trace({
sourceNodeId: entry.req.sourceNodeId,
@@ -467,17 +507,29 @@ export class DeliveryQueue {
/** Every queued entry as it is written to disk, oldest first per target. */
snapshot(): PersistedQueueEntry[] {
const out: PersistedQueueEntry[] = []
+ // The file is set aside WHOLE at load past DURABLE_STATE_MAX_BYTES, so the write is budgeted:
+ // once the full entries reach QUEUE_PERSIST_BYTES_BUDGET (JSON bytes, oldest first per target),
+ // the rest are written REDUCED — no body, only short fields — which restore turns into an
+ // expiry the sender hears about. Never a file the next boot throws away with every message in it.
+ let used = 0
for (const list of this.queues.values()) {
for (const e of list) {
- const omit = e.req.body.length > QUEUE_PERSIST_BODY_MAX
- out.push({
- req: omit ? { ...e.req, body: '' } : e.req,
+ const base = {
enqueuedAt: e.enqueuedAt,
ttlMs: e.ttlMs,
queuedTraceId: e.queuedTraceId,
- ...(e.binding ? { binding: e.binding } : {}),
- ...(omit ? { bodyOmitted: true as const } : {})
- })
+ ...(e.binding ? { binding: e.binding } : {})
+ }
+ const full: PersistedQueueEntry = { req: e.req, ...base }
+ const size = e.req.body.length > QUEUE_PERSIST_BODY_MAX ? Infinity : JSON.stringify(full).length
+ if (used + size <= QUEUE_PERSIST_BYTES_BUDGET) {
+ used += size
+ out.push(full)
+ continue
+ }
+ const reduced: PersistedQueueEntry = { req: reducedRequest(e.req), ...base, bodyOmitted: true }
+ used += JSON.stringify(reduced).length
+ out.push(reduced)
}
}
return out
@@ -494,7 +546,6 @@ export class DeliveryQueue {
const now = this.deps.now()
const lapsed: QueueEntry[] = []
for (const p of entries) {
- const list = this.queues.get(p.req.targetNodeId) ?? []
// A clock that went backwards must not stretch the wait past one full TTL.
const age = Math.max(0, now - p.enqueuedAt)
const remaining = Math.min(p.ttlMs, p.ttlMs - age)
@@ -507,20 +558,28 @@ export class DeliveryQueue {
restored: true,
...(p.binding ? { binding: p.binding } : {})
}
- list.push(entry)
- this.queues.set(p.req.targetNodeId, list)
this.deps.onQueued?.(entry.req)
+ // Capacity counts only what is really re-queued: a lapsed entry never takes a slot.
const deliverable =
remaining > 0 &&
!p.bodyOmitted &&
RESTORABLE_VERBS.has(String(p.req.verb)) &&
- list.length <= this.capacity
- if (deliverable) {
- entry.cancelTimer = this.schedule(remaining, () => void this.expire(p.req.targetNodeId, entry))
- } else lapsed.push(entry)
+ this.depth(p.req.targetNodeId) < this.capacity
+ if (!deliverable) {
+ // Never inserted into the live lists: a flush running during one of the expiries below
+ // must not be able to deliver a board comment or an empty (body-omitted) entry.
+ lapsed.push(entry)
+ continue
+ }
+ const list = this.queues.get(p.req.targetNodeId) ?? []
+ list.push(entry)
+ this.queues.set(p.req.targetNodeId, list)
+ entry.cancelTimer = this.schedule(remaining, () => void this.expire(p.req.targetNodeId, entry))
}
+ // The lapsed entries' ends are reported BEFORE the file drops them: a crash in between reports
+ // one twice at the next boot, never not at all.
+ for (const entry of lapsed) await this.reportExpired(entry)
this.persist()
- for (const entry of lapsed) await this.expire(entry.req.targetNodeId, entry)
}
private persist(): void {
diff --git a/src/core/control-request-ledger.persist.test.ts b/src/core/control-request-ledger.persist.test.ts
index f3e8619c9..ee7362335 100644
--- a/src/core/control-request-ledger.persist.test.ts
+++ b/src/core/control-request-ledger.persist.test.ts
@@ -1,5 +1,7 @@
import { describe, it, expect } from 'vitest'
+import { DURABLE_STATE_MAX_BYTES } from './durable-state'
import {
+ CONTROL_REQUEST_REPLIES_BUDGET,
CONTROL_REQUEST_REPLY_MAX_BYTES,
ControlRequestLedger,
REQUEST_LEDGER_TTL_MS,
@@ -72,4 +74,24 @@ describe('ControlRequestLedger persistence', () => {
expect(sanitizeLedgerRow({ ...base, reply: undefined })).toMatchObject({ state: 'unknown' })
expect(sanitizeLedgerRow(null)).toBeNull()
})
+
+ it('stays under the file limit when every row carries a maximal reply: the rows past the budget are UNKNOWN', () => {
+ const a = new ControlRequestLedger({ now: () => 0 })
+ const big = 'y'.repeat(CONTROL_REQUEST_REPLY_MAX_BYTES - 200)
+ for (let i = 0; i < 4096; i++) {
+ const d = a.begin(`n${i % 16}`, `r${i}`, fp)
+ if (d.kind === 'run') d.claim.settle({ ok: true, message: big })
+ }
+ const rows = a.exportRows()
+ expect(rows).toHaveLength(4096)
+ expect(JSON.stringify(rows).length).toBeLessThan(DURABLE_STATE_MAX_BYTES)
+ const settled = rows.filter((r) => r.state === 'settled')
+ expect(settled.length).toBeGreaterThan(0)
+ expect(JSON.stringify(settled.map((r) => r.reply)).length).toBeLessThanOrEqual(CONTROL_REQUEST_REPLIES_BUDGET + settled.length)
+ // The oldest keep their reply; a row past the budget is still refused after a restart.
+ expect(rows[0].state).toBe('settled')
+ const last = rows[rows.length - 1]
+ expect(last.state).toBe('unknown')
+ expect(restarted([last], 1).begin(last.caller, last.requestId, fp)).toMatchObject({ outcome: 'request-outcome-unknown' })
+ })
})
diff --git a/src/core/control-request-ledger.ts b/src/core/control-request-ledger.ts
index 3e71cf922..9bc6e40c8 100644
--- a/src/core/control-request-ledger.ts
+++ b/src/core/control-request-ledger.ts
@@ -331,6 +331,10 @@ export interface PersistedLedgerRow {
* few hundred bytes; the bound only keeps a pathological `result` out of the file. */
export const CONTROL_REQUEST_REPLY_MAX_BYTES = 64 * 1024
+/** JSON bytes of replies one ledger file carries; the rows past it are written UNKNOWN. Rows
+ * themselves are ~300 bytes, so 4096 of them fit beside it under DURABLE_STATE_MAX_BYTES. */
+export const CONTROL_REQUEST_REPLIES_BUDGET = 8 * 1024 * 1024
+
const FINGERPRINT_RE = /^[0-9a-f]{64}$/
/** Re-check one row read from disk (hand-editable input). `null` drops it. */
@@ -469,6 +473,10 @@ export class ControlRequestLedger {
*/
exportRows(): PersistedLedgerRow[] {
const out: PersistedLedgerRow[] = []
+ // Budgeted, oldest claim first: 4096 rows of 64 KB replies would pass the file limit and the
+ // whole file would be set aside at load. Past the budget a settled row is written UNKNOWN (no
+ // reply) — still refused, never re-run.
+ let used = 0
for (const row of this.rows.values()) {
const rec: PersistedLedgerRow = {
caller: row.caller,
@@ -480,8 +488,11 @@ export class ControlRequestLedger {
}
if (row.state === 'settled' && row.reply) {
const reply = sanitizeReply(row.reply)
- if (reply) rec.reply = reply
- else rec.state = 'unknown'
+ const size = reply ? JSON.stringify(reply).length : 0
+ if (reply && used + size <= CONTROL_REQUEST_REPLIES_BUDGET) {
+ rec.reply = reply
+ used += size
+ } else rec.state = 'unknown'
}
out.push(rec)
}
diff --git a/src/core/durable-state.test.ts b/src/core/durable-state.test.ts
index 3cfa24a2e..195feb5ac 100644
--- a/src/core/durable-state.test.ts
+++ b/src/core/durable-state.test.ts
@@ -144,4 +144,31 @@ describe('DurableFactFile', () => {
await a.flush()
expect(h.make().load().map((r) => r.n)).toEqual([3, 4, 5])
})
+
+ it('a synchronous flush is never overwritten by an OLDER async write still in flight (review repro)', async () => {
+ const h = harness()
+ const a = h.make()
+ a.save([{ id: 'old', n: 1 }])
+ const inFlight = a.flush() // async write of [old] started, not yet renamed
+ a.save([{ id: 'new', n: 2 }])
+ a.flushSync() // quit: [new] lands synchronously
+ await inFlight
+ expect(h.make().load()).toEqual([{ id: 'new', n: 2 }])
+ // No temp litter left by the dropped rename.
+ expect(fs.readdirSync(path.dirname(h.file)).filter((f) => f.endsWith('.tmp'))).toEqual([])
+ })
+
+ it('a stood-down file neither reads nor writes (a second instance that does not own the fact)', async () => {
+ const h = harness()
+ const owner = h.make()
+ owner.save([{ id: 'owner', n: 1 }])
+ await owner.flush()
+ const second = h.make()
+ second.standDown()
+ expect(second.load()).toEqual([])
+ second.save([{ id: 'intruder', n: 2 }])
+ second.flushSync()
+ await second.flush()
+ expect(h.make().load()).toEqual([{ id: 'owner', n: 1 }])
+ })
})
diff --git a/src/core/durable-state.ts b/src/core/durable-state.ts
index f1cdeeb9e..dfbe2f7f9 100644
--- a/src/core/durable-state.ts
+++ b/src/core/durable-state.ts
@@ -1,15 +1,16 @@
// Orchestration facts that must outlive the process that learned them.
//
-// WHY THIS EXISTS. Three facts a canvas-control orchestration leans on lived only in process
+// WHY THIS EXISTS. Four facts a canvas-control orchestration leans on lived only in process
// memory, so an app (or Server Edition) restart erased them while the thing they described kept
// going:
// - a `send` answered `queued` (the sender was told it would be delivered) was silently lost;
// - a station's `report-outcome` vanished, so every `--after-success` dependent read BLOCKED;
-// - the `--request-id` ledger emptied, so a retry after a restart re-ran an open that happened.
+// - the `--request-id` ledger emptied, so a retry after a restart re-ran an open that happened;
+// - the plain-`--after` hand-over hold was forgotten, so a dependent fired on a stale `done`.
// Each owner keeps its own rules (what a restart MEANS for that fact is decided next to the fact —
-// delivery-queue.ts, station-outcome-store.ts, control-request-ledger.ts); this module is only the
-// storage all of them share, so a new fact is one more `DurableFactSpec`, not a fourth copy of the
-// read / sanitize / write / flush dance.
+// delivery-queue.ts, station-outcome-store.ts, station-handover.ts, control-request-ledger.ts);
+// this module is only the storage all of them share, so a new fact is one more `DurableFactSpec`,
+// not another copy of the read / sanitize / write / flush dance.
//
// THE RULES, each for a reason:
// - One file per fact kind under `<userData>/orchestration-state/`, MACHINE-LOCAL. Never
@@ -24,9 +25,16 @@
// is not JSON, too large, or not an envelope is set aside as `<file>.corrupt` (one copy, replaced
// each time) and the fact starts empty with a warning. Loading NEVER throws: a boot that dies on
// a bad orchestration file would take the whole app with it.
-// - Written through `writeFileAtomic` (and `renameAtomicSync` for the synchronous flush at
-// quit) — never a bare rename, see fs-atomic.ts. Mode 0600: a queued message's body is the
-// user's text.
+// - Written as a unique `wx` temp then `renameAtomicSync` — never a bare rename, see
+// fs-atomic.ts. Mode 0600: a queued message's body is the user's text.
+// - A synchronous flush (quit) is never overwritten by an OLDER async write still in flight: it
+// bumps a generation, and the async path checks the generation and renames in one synchronous
+// step, dropping its temp when it is stale.
+// - Every OWNER must keep its serialized file under `DURABLE_STATE_MAX_BYTES` — a file past it is
+// set aside WHOLE at load. The queue and the ledger enforce a byte budget on what they write
+// (bodies / replies past it are written in their refusal-safe reduced form).
+// - `standDown()` for a process that does not own the fact (a second instance on the same
+// userData that lost the hook endpoint): it neither reads nor writes the file.
// - Saves are COALESCED (`debounceMs`, a short window — every control call can touch the
// ledger), at most one write in flight, and the LATEST snapshot always wins. `flushSync` writes
// whatever is pending before the process exits; a crash inside the window loses that window,
@@ -35,12 +43,13 @@
// Pure `src/core`: no electron. The clock and the timer are injectable so tests drive it.
import { closeSync, fstatSync, mkdirSync, openSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'
-import { renameAtomicSync, tempNameFor, writeFileAtomic } from './fs-atomic'
+import { promises as fsp } from 'node:fs'
+import { renameAtomicSync, tempNameFor } from './fs-atomic'
/** Where every durable orchestration fact lives, relative to userData. */
export const DURABLE_STATE_DIR = 'orchestration-state'
-/** A file larger than this is not ours (every owner caps its records far below it). */
+/** A file larger than this is set aside at load. Every owner budgets its writes below it. */
export const DURABLE_STATE_MAX_BYTES = 16 * 1024 * 1024
/** How long saves are coalesced. Short: a claimed request-id row should reach disk before the call
@@ -95,6 +104,9 @@ export class DurableFactFile<T> {
private cancelTimer: (() => void) | null = null
private writing: Promise<void> | null = null
private disposed = false
+ /** Bumped by every synchronous flush; an async write started before it drops its rename. */
+ private generation = 0
+ private stoodDown = false
constructor(
private readonly spec: DurableFactSpec<T>,
@@ -118,6 +130,7 @@ export class DurableFactFile<T> {
/** Read and sanitize the file. Synchronous (boot), never throws; absent ⇒ `[]` silently. */
load(): T[] {
+ if (this.stoodDown) return []
let text: string
let fd: number | undefined
try {
@@ -209,6 +222,8 @@ export class DurableFactFile<T> {
const snap = this.pending
if (!snap) return
this.pending = null
+ // Any async write still in flight carries an OLDER snapshot: it must not rename over this one.
+ this.generation++
try {
this.ensureDir()
const tmp = tempNameFor(this.path)
@@ -224,6 +239,23 @@ export class DurableFactFile<T> {
}
}
+ /**
+ * This process does not own the fact (a second instance on the same userData that lost the hook
+ * endpoint to the first): never read or write the file again. `load()` answers `[]` from here on,
+ * so it neither restores (and expires) messages the owning instance holds nor overwrites its file.
+ */
+ standDown(): void {
+ if (this.cancelTimer) {
+ this.cancelTimer()
+ this.cancelTimer = null
+ }
+ this.pending = null
+ this.disposed = true
+ this.stoodDown = true
+ this.generation++
+ liveFiles.delete(this)
+ }
+
/** Stop accepting saves after flushing (shutdown). */
dispose(): void {
this.flushSync()
@@ -237,11 +269,23 @@ export class DurableFactFile<T> {
const snap = this.pending
if (!snap) return
this.pending = null
+ const generation = this.generation
this.writing = (async (): Promise<void> => {
+ const tmp = tempNameFor(this.path)
try {
this.ensureDir()
- await writeFileAtomic(this.path, this.serialize(snap), { mode: 0o600 })
+ // Same shape as `writeFileAtomic` (unique `wx` temp, then the retrying rename), split so the
+ // generation check and the rename happen in ONE synchronous step: a `flushSync` that ran
+ // while the temp was being written has put a NEWER snapshot on disk, and this older one
+ // must not land on top of it. Checked and renamed with nothing able to run in between.
+ await fsp.writeFile(tmp, this.serialize(snap), { encoding: 'utf-8', flag: 'wx', mode: 0o600 })
+ if (generation !== this.generation) {
+ rmSync(tmp, { force: true })
+ return
+ }
+ renameAtomicSync(tmp, this.path)
} catch (e) {
+ rmSync(tmp, { force: true })
this.warn(`[durable-state] ${this.spec.kind}: save failed (${String(e)})`)
}
})()
diff --git a/src/main/index.ts b/src/main/index.ts
index 757d38b34..bfdbaf3e6 100644
--- a/src/main/index.ts
+++ b/src/main/index.ts
@@ -1969,10 +1969,11 @@ app.whenReady().then(async () => {
// renderer reload does not lose it; pushed whole to the window on every change.
// Durable across an app restart too (station-outcomes.json), each report bound to the session
// that made it; `loadFromDisk()` runs below, after the status mirror it compares against.
+ const stationOutcomesFile = new DurableFactFile(OUTCOME_FACT, { userDataDir: corePlatform.userDataDir })
const stationOutcomes = new StationOutcomeStore(
(records) => sendToMain(IPC.stationOutcomeChanged, records),
{
- durable: new DurableFactFile(OUTCOME_FACT, { userDataDir: corePlatform.userDataDir }),
+ durable: stationOutcomesFile,
sessionOf: (id) => {
const m = mirrorEntry(id)
return m ? { sessionId: m.sessionId, agentId: m.agentId } : undefined
@@ -1987,10 +1988,11 @@ app.whenReady().then(async () => {
// handed new work is not "done" until a turn that started after the hand-over has ended. Pushed
// whole to the window, whose launch loop reads it. Durable (station-handovers.json), loaded below
// between the reports and the queue.
+ const stationHandoversFile = new DurableFactFile(HANDOVER_FACT, { userDataDir: corePlatform.userDataDir })
const stationHandovers = new StationHandoverTracker(
(records) => sendToMain(IPC.stationHandoverChanged, records),
Date.now,
- new DurableFactFile(HANDOVER_FACT, { userDataDir: corePlatform.userDataDir })
+ stationHandoversFile
)
registerStationHandoverIpc(corePlatform, () => stationHandovers)
messagingDeps.onHandover = (ev) => {
@@ -2016,6 +2018,13 @@ app.whenReady().then(async () => {
// listeners (setListener/setRawListener/setControlHandler) attach later, which the server
// tolerates — early hook POSTs are simply dropped, never mis-routed.
const hookStartupWarning = await hookServer.startForApp()
+ // The durable orchestration facts belong to the instance that owns the hook endpoint, exactly
+ // like the request ledger (which `hookServer.start()` only loads when it wins the endpoint). A
+ // second instance on the same userData that lost it must neither restore them — it would expire,
+ // and tell senders about, messages the owning instance still holds — nor overwrite their files.
+ if (hookStartupWarning) {
+ for (const f of [deliveryQueueFile, stationOutcomesFile, stationHandoversFile]) f.standDown()
+ }
// ---- Node identity (src/core/agents/node-auth-secret.ts) ------------------------------------
// One secret does two jobs: it arms the hook server's per-node capability (closing the "shared
// bearer can name any sibling node" hole) and it signs the codex thread → node records the hook
@@ -2118,7 +2127,13 @@ app.whenReady().then(async () => {
// sender told, every message whose TTL ran out while the app was down.
stationOutcomes.loadFromDisk()
stationHandovers.loadFromDisk()
- void restoreDeliveryQueue(messagingDeps.queue, deliveryQueueFile)
+ // The queue waits for the workspace INDEX: an entry that lapsed while the app was down is expired
+ // here, and its sender leg (board-log line in the sender's project) resolves projects through the
+ // index, which nothing else has loaded yet at this point of boot (the renderer's own
+ // `workspace.load` comes later). Read-only load (`sideline: false`), like the relay path's.
+ void restoreDeliveryQueue(messagingDeps.queue, deliveryQueueFile, {
+ ready: workspaceStore.load({ sideline: false })
+ })
/** The one display-title rule for everything the HOST sends out (push alerts, Live Activity
* updates, the notch capsule): the live session name unless the node was hand-renamed. */
diff --git a/src/main/station-outcome-wiring.test.ts b/src/main/station-outcome-wiring.test.ts
index 60a59b220..6b8068450 100644
--- a/src/main/station-outcome-wiring.test.ts
+++ b/src/main/station-outcome-wiring.test.ts
@@ -55,22 +55,32 @@ describe('desktop main', () => {
})
it('persists the store and the delivery queue, and loads them after the status mirror (durable state)', () => {
- expect(main).toMatch(/durable: new DurableFactFile\(OUTCOME_FACT/)
+ expect(main).toMatch(/const stationOutcomesFile = new DurableFactFile\(OUTCOME_FACT/)
+ expect(main).toContain("durable: stationOutcomesFile")
expect(main).toMatch(/createDeliveryQueue\(messagingDeps, \{ durable: deliveryQueueFile \}\)/)
const mirror = main.indexOf('initAgentStatusMirror()\n')
const load = main.indexOf('stationOutcomes.loadFromDisk()')
const holds = main.indexOf('stationHandovers.loadFromDisk()')
- const queue = main.indexOf('restoreDeliveryQueue(messagingDeps.queue, deliveryQueueFile)')
+ const queue = main.indexOf('restoreDeliveryQueue(messagingDeps.queue, deliveryQueueFile, {')
expect(mirror).toBeGreaterThan(-1)
expect(load).toBeGreaterThan(mirror)
expect(holds).toBeGreaterThan(load)
expect(queue).toBeGreaterThan(holds)
- expect(main).toMatch(/new DurableFactFile\(HANDOVER_FACT/)
+ expect(main).toMatch(/const stationHandoversFile = new DurableFactFile\(HANDOVER_FACT/)
// A station's new session withdraws its report BEFORE the renderer hears the event.
const withdraw = main.indexOf('stationOutcomes.onAgentEvent(enriched)')
expect(withdraw).toBeGreaterThan(-1)
expect(withdraw).toBeLessThan(main.indexOf('sendToMain(IPC.agentStatus, enriched)'))
expect(main).toContain('flushAllDurableFactsSync()')
+ // The restore waits for the workspace index, or a restore-time expiry reaches no sender.
+ expect(main.slice(queue, queue + 200)).toContain('ready: workspaceStore.load({ sideline: false })')
+ // A second instance that lost the hook endpoint stands every durable file down.
+ const gate = main.indexOf('const hookStartupWarning = await hookServer.startForApp()')
+ expect(gate).toBeGreaterThan(-1)
+ expect(gate).toBeLessThan(load)
+ expect(main.slice(gate, gate + 900)).toMatch(
+ /if \(hookStartupWarning\) \{\s*for \(const f of \[deliveryQueueFile, stationOutcomesFile, stationHandoversFile\]\) f\.standDown\(\)/
+ )
})
})
@@ -110,6 +120,8 @@ describe('Server Edition', () => {
expect(serverControl).toContain('queueFile.dispose()')
expect(serverControl).toContain('outcomesFile.dispose()')
expect(serverControl).toContain('handoversFile.dispose()')
+ expect(serverControl).toContain('if (deps.ownsDurableState === false) queueFile.standDown()')
+ expect(serverIndex).toContain('ownsDurableState: hookStartupWarning === null')
expect(serverControl.indexOf('stationHandovers.loadFromDisk()')).toBeLessThan(
serverControl.indexOf('await restoreDeliveryQueue(queue, queueFile)')
)
diff --git a/src/server/canvas-control.ts b/src/server/canvas-control.ts
index 16e9ebd1f..5be9c4f1a 100644
--- a/src/server/canvas-control.ts
+++ b/src/server/canvas-control.ts
@@ -91,6 +91,13 @@ export interface ServerCanvasControlDeps {
* specifically exercising the install and has redirected `HOME` to a scratch directory first.
*/
installAgentIntegrations: boolean
+ /**
+ * Does this process own the hook endpoint (`hookServer.startForApp()` returned no warning)? The
+ * durable orchestration facts (queue, station reports, hand-over holds) belong to the owning
+ * instance, like the request ledger: a second instance on the same data dir must neither restore
+ * them nor overwrite their files. Absent = owns (every test, and a caller that did not ask).
+ */
+ ownsDurableState?: boolean
}
export interface ServerCanvasControl {
@@ -221,6 +228,10 @@ export async function initServerCanvasControl(
Date.now,
handoversFile
)
+ if (deps.ownsDurableState === false) {
+ outcomesFile.standDown()
+ handoversFile.standDown()
+ }
stationOutcomes.loadFromDisk()
stationHandovers.loadFromDisk()
const factory = new HeadlessNodeFactory({
@@ -287,6 +298,7 @@ export async function initServerCanvasControl(
// this edition's creator ledger is process-local, so a restored message whose caller→target proof
// did not survive the restart is refused `caller-not-owner` at flush — with its sender told.
const queueFile = new DurableFactFile(QUEUE_FACT, { userDataDir: platform().userDataDir })
+ if (deps.ownsDurableState === false) queueFile.standDown()
const queue = createDeliveryQueue(messaging, { durable: queueFile })
messaging.queue = queue
diff --git a/src/server/index.ts b/src/server/index.ts
index 6ffd40921..f1b61ee7e 100644
--- a/src/server/index.ts
+++ b/src/server/index.ts
@@ -730,7 +730,9 @@ export async function startServer(
// answer the issue lane gets from the GitHub host controller.
issueRepository: (projectId) =>
github.controller.status(projectId).then((view) => view.project?.repository ?? null),
- installAgentIntegrations: config.installHooks !== false
+ installAgentIntegrations: config.installHooks !== false,
+ // The durable orchestration facts follow hook-endpoint ownership, like the request ledger.
+ ownsDurableState: hookStartupWarning === null
})
hookServer.setControlHandler(canvasControl.handler)
} catch (error) {
diff --git a/src/shared/board-comment.ts b/src/shared/board-comment.ts
index 4c459d02c..fcc39e89b 100644
--- a/src/shared/board-comment.ts
+++ b/src/shared/board-comment.ts
@@ -37,8 +37,9 @@ export const BOARD_COMMENT_BODY_MAX = 16_384
/** How long after a `queued` outcome a comment row stops presenting it as live: the core's
* deliver-on-idle TTL (`DELIVERY_QUEUE_TTL_MS`, 5 min — pinned by a core test) plus a minute. The
- * queue is process memory; an app that quit while a comment waited never writes its end, and a
- * row reading "queued" forever would be a silent lie. */
+ * queue is durable and a clean restart expires a waiting comment at boot (writing its end), but a
+ * crash inside the queue's save window, or a second instance that does not own the queue, still
+ * never writes one — and a row reading "queued" forever would be a silent lie. */
export const BOARD_COMMENT_QUEUE_STALE_MS = 6 * 60_000
/** The trace/queue identity of a delivery started by a board comment: `board-comment:<commentId>`.