Node-based terminal manager for AI coding agents — tmux-backed terminals and parallel agent sessions as draggable nodes on an infinite pan/zoom canvas. macOS, Linux, and a browser Server Edition.
TypeScript adhdagent-orchestrationai-agentscanvasclaude-codecodexcoding-agentsdeveloper-toolselectrongeminilinuxmacosmultiplexeropencodeparallel-agentsterminalterminal-emulatorterminal-multiplexertmuxworkspace-manager
fix(durable-state): review of #1054 — sender told at boot, byte budgets, no stale overwrite
review9212315f82d4
diff --git a/CLAUDE.md b/CLAUDE.mdindex 853b8932e..8d3c06a96 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -7469,10 +7469,29 @@ spec, never a fourth copy of the read / sanitize / write / flush code.lists are capped, and a file that is not JSON / too large (16 MB) / not an envelope is set aside as`<file>.corrupt` (one copy) and the fact starts EMPTY with a warning. **Loading never throws** — abad orchestration file must not take the boot with it.-- **Writes** go through `writeFileAtomic` (the fs-atomic guard applies), coalesced over 50 ms, one- in flight, latest snapshot wins; `flushAllDurableFactsSync()` runs on the desktop's second- `before-quit` pass, `hookServer.stop()` flushes the ledger, and the Server Edition's canvas-control- `stop()` disposes its two files. A **crash** inside the 50 ms window loses that window.+- **Writes** are a unique `wx` temp then `renameAtomicSync` (the fs-atomic guard applies), coalesced+ over 50 ms, one in flight, latest snapshot wins; `flushAllDurableFactsSync()` runs on the desktop's+ second `before-quit` pass, `hookServer.stop()` flushes the ledger, and the Server Edition's+ canvas-control `stop()` disposes its three files (queue, reports, holds). A **crash** inside the+ 50 ms window loses that window. **A sync flush is never overwritten by an older async write**+ (review of #1054, reproduced: `save([1])`, `flush()`, `save([2])`, `flushSync()` left `[1]` on+ disk): `flushSync` bumps a generation, and the async path checks it and renames in ONE synchronous+ step, dropping its temp when stale.+- **Every owner budgets its bytes under the 16 MB load limit**, because a file past it is set aside+ WHOLE (reproduced in review: 80 queued 250K-char bodies wrote 20 MB and 0 of 80 came back). The+ queue writes full entries up to 8 MB of JSON and the rest REDUCED (no body, short fields only —+ `bodyOmitted`, which restore turns into an expiry the sender hears about); the ledger writes+ replies up to 8 MB and the rest as UNKNOWN rows (still refused, never re-run).+- **Only the instance that owns the hook endpoint owns the facts.** The ledger is loaded by+ `hookServer.start()`, which fails for a second instance; the queue, reports and holds follow the+ same rule (`standDown()` when `startForApp()` returned a warning — desktop `main/index.ts`, Server+ Edition `ownsDurableState`), so a second instance on the same userData neither expires messages+ the live one holds nor overwrites its files.+- **The desktop queue restore waits for the workspace INDEX** (`restoreDeliveryQueue(…, {ready:+ workspaceStore.load({sideline:false})})`): an entry that lapsed during the downtime is expired+ there, and its sender leg resolves the sender's project and board log through the index, which+ nothing else has loaded at that point of boot (review of #1054: the expiry otherwise reached only+ the in-memory trace ring). The Server Edition already awaited the load before canvas control.- **Measured** (this Linux dev host, the ledger at its 4096-row cap with realistic open replies):377 bytes a row, a 1.5 MB file, 11.5 ms per full write, 16.6 ms to load at boot; a 20-row ledgerwrites in 0.41 ms. The queue is bounded far lower (16 per target), the reports at 1000.@@ -7529,8 +7548,10 @@ What a restart MEANS, per fact — decided beside each fact, stated in its headerefused, never re-run, and nothing can settle it any more because its late answer died with theold process; a reply over 64 KB is written as UNKNOWN rather than dropped (a missing row would letthe retry run).-- **Known limit:** two instances sharing one userData (a dev sandbox) are last-writer-wins on these- files, and both would expire the same lapsed messages at boot. Nothing locks them.+- **Delivery is at most once across a crash:** a flush writes the entry OFF disk before its+ attempt (claim before effect), so a crash mid-delivery loses that one message rather than typing+ it twice after the next boot. Lapsed entries at restore are never inserted into the live lists+ (so no flush can deliver one while the expiries are reported) and take no capacity slot.## The test suite never touches a live tmux serverdiff --git a/src/core/agents/agent-messaging.ts b/src/core/agents/agent-messaging.tsindex 875aa92e2..dc234ec23 100644--- a/src/core/agents/agent-messaging.ts+++ b/src/core/agents/agent-messaging.ts@@ -408,9 +408,19 @@ export function createDeliveryQueue(*/export async function restoreDeliveryQueue(queue: DeliveryQueue,- file: Pick<DurableFactFile<PersistedQueueEntry>, 'load'>+ file: Pick<DurableFactFile<PersistedQueueEntry>, 'load'>,+ opts: {+ /**+ * Resolves once the stores an expiry's sender leg reads are loaded — on the desktop the+ * workspace INDEX (`projects()` and the board-log routes resolve nothing before it, so an entry+ * expired at restore would reach only the in-memory trace ring and its sender would never hear).+ * A rejection is waited out, not propagated: the restore still runs.+ */+ ready?: Promise<unknown>+ } = {}): Promise<void> {try {+ if (opts.ready) await opts.ready.catch(() => undefined)await queue.restore(file.load())} catch (e) {console.warn(`[agent-messaging] could not restore the delivery queue (${String(e)})`)diff --git a/src/core/agents/delivery-queue.persist.test.ts b/src/core/agents/delivery-queue.persist.test.tsindex 0ac0ecc7c..3207440bf 100644--- a/src/core/agents/delivery-queue.persist.test.ts+++ b/src/core/agents/delivery-queue.persist.test.ts@@ -4,6 +4,7 @@ import {DELIVERY_QUEUE_TTL_MS,QUEUE_FACT,QUEUE_PERSIST_BODY_MAX,+ QUEUE_PERSIST_BYTES_BUDGET,restoredBindingVerdict,sanitizePersistedQueueEntry,type CancelTimer,@@ -13,7 +14,9 @@ import {type QueuedDeliveryRequest} from './delivery-queue'import type { AgentMessageOutcome } from './agent-message-decide'-import { DurableFactFile } from '../durable-state'+import { DURABLE_STATE_MAX_BYTES, DurableFactFile } from '../durable-state'+import { restoreDeliveryQueue } from './agent-messaging'+import fs from 'node:fs'import { testTmpDir } from '../test-tmp'/**@@ -24,7 +27,13 @@ import { testTmpDir } from '../test-tmp'* for; a board comment and a station notice are never replayed into a pane.*/-function instance(opts: { now: () => number; binding: (id: string) => QueueBinding | undefined; outcome?: AgentMessageOutcome }) {+function instance(opts: {+ now: () => number+ binding: (id: string) => QueueBinding | undefined+ outcome?: AgentMessageOutcome+ trace?: DeliveryQueueDeps['trace']+ onDeliver?: (req: QueuedDeliveryRequest) => void+}) {const delivered: QueuedDeliveryRequest[] = []const expired: { req: QueuedDeliveryRequest; queuedForMs: number }[] = []const flushed: { req: QueuedDeliveryRequest; outcome: AgentMessageOutcome }[] = []@@ -35,13 +44,16 @@ function instance(opts: { now: () => number; binding: (id: string) => QueueBindiconst deps: DeliveryQueueDeps = {now: opts.now,deliver: async (req) => {+ opts.onDeliver?.(req)delivered.push(req)return opts.outcome ?? { kind: 'delivered', traceId: 'd', traced: 'memory', receipt: 'observed', signal: 'newTurn' }},- trace: async (input) => {- traced.push(input.outcome)- return { traceId: `t${traced.length}`, traced: 'memory' }- },+ trace:+ opts.trace ??+ (async (input) => {+ traced.push(input.outcome)+ return { traceId: `t${traced.length}`, traced: 'memory' }+ }),onExpired: (req, info) => expired.push({ req, queuedForMs: info.queuedForMs }),onFlushed: (req, outcome) => flushed.push({ req, outcome }),onQueued: (req) => queued.push(req),@@ -198,4 +210,92 @@ describe('delivery queue across a restart', () => {await b.queue.onTargetIdle('st1')expect(b.delivered.map((r) => r.body)).toEqual(['next task'])})++ it('the written file stays under the load limit however much is queued; nothing is silently dropped (review repro)', async () => {+ const dir = testTmpDir('nt-queue-budget-')+ const file = new DurableFactFile(QUEUE_FACT, { userDataDir: dir, debounceMs: 1 })+ const a = instance({ now: () => 1000, binding: () => ({ sessionId: 's' }) })+ for (let t = 0; t < 5; t++)+ for (let i = 0; i < 16; i++)+ await a.queue.enqueue(req({ targetNodeId: `st${t}`, body: `${t}-${i}-` + 'x'.repeat(250_000) }))+ file.save(a.saved())+ await file.flush()+ expect(fs.statSync(file.path).size).toBeLessThan(DURABLE_STATE_MAX_BYTES)+ const loaded = new DurableFactFile(QUEUE_FACT, { userDataDir: dir }).load()+ expect(loaded).toHaveLength(80)+ const full = loaded.filter((e) => !e.bodyOmitted)+ expect(full.length).toBeGreaterThan(0)+ expect(full.reduce((n, e) => n + JSON.stringify(e).length, 0)).toBeLessThanOrEqual(QUEUE_PERSIST_BYTES_BUDGET)+ // Every reduced entry is ENDED loudly at restore, the full ones wait for their target.+ const b = instance({ now: () => 2000, binding: () => ({ sessionId: 's' }) })+ await b.queue.restore(loaded)+ expect(b.expired).toHaveLength(80 - full.length)+ expect(b.expired.length + [0, 1, 2, 3, 4].reduce((n, t) => n + b.queue.depth(`st${t}`), 0)).toBe(80)+ })++ it('lapsed entries never enter the live lists: a flush during an expiry\'s await delivers nothing of them', async () => {+ const a = instance({ now: () => 1000, binding: () => ({ sessionId: 's' }) })+ await a.queue.enqueue(req({ verb: 'board-comment', sourceNodeId: 'board-comment:c1', projectId: 'p1' }))+ await a.queue.enqueue(req({ body: 'x'.repeat(QUEUE_PERSIST_BODY_MAX + 1) }))+ let release = (): void => {}+ let b!: ReturnType<typeof instance>+ const stalled = new Promise<void>((r) => (release = r))+ b = instance({+ now: () => 2000,+ binding: () => ({ sessionId: 's' }),+ trace: async () => {+ await b.queue.onTargetIdle('st1') // a `done` arriving while the expiry is being reported+ await stalled+ return { traceId: 't', traced: 'memory' }+ }+ })+ const restoring = b.queue.restore(onDisk(a.saved()))+ await new Promise((r) => setTimeout(r, 0))+ release()+ await restoring+ expect(b.delivered).toEqual([])+ expect(b.expired).toHaveLength(2)+ })++ it('lapsed entries take no capacity: 16 deliverable ones all come back beside an expired one', async () => {+ const entries: PersistedQueueEntry[] = [+ { req: req({ verb: 'board-comment', sourceNodeId: 'board-comment:c1' }), enqueuedAt: 1000, ttlMs: DELIVERY_QUEUE_TTL_MS, queuedTraceId: 't' },+ ...Array.from({ length: 16 }, (_, i) => ({+ req: req({ body: `m${i}` }),+ enqueuedAt: 1000,+ ttlMs: DELIVERY_QUEUE_TTL_MS,+ queuedTraceId: `t${i}`,+ binding: { sessionId: 's' }+ }))+ ]+ const b = instance({ now: () => 2000, binding: () => ({ sessionId: 's' }) })+ await b.queue.restore(onDisk(entries))+ expect(b.queue.depth('st1')).toBe(16)+ expect(b.expired.map((e) => e.req.verb)).toEqual(['board-comment'])+ })++ it('an entry is written off disk BEFORE its delivery attempt (at most once across a crash)', async () => {+ let onDiskDuringDelivery: PersistedQueueEntry[] | null = null+ let a!: ReturnType<typeof instance>+ a = instance({ now: () => 1000, binding: () => ({ sessionId: 's' }), onDeliver: () => (onDiskDuringDelivery = a.saved()) })+ await a.queue.enqueue(req())+ expect(a.saved()).toHaveLength(1)+ await a.queue.onTargetIdle('st1')+ expect(onDiskDuringDelivery).toEqual([])+ })++ it('restoreDeliveryQueue reports no expiry until `ready` (the desktop workspace index) has resolved', async () => {+ const a = instance({ now: () => 1000, binding: () => ({ sessionId: 's' }) })+ await a.queue.enqueue(req())+ const b = instance({ now: () => 1000 + DELIVERY_QUEUE_TTL_MS + 1, binding: () => ({ sessionId: 's' }) })+ let loaded = (): void => {}+ const ready = new Promise<void>((r) => (loaded = r))+ const disk = onDisk(a.saved())+ const done = restoreDeliveryQueue(b.queue, { load: () => disk }, { ready })+ await new Promise((r) => setTimeout(r, 5))+ expect(b.expired).toEqual([])+ loaded()+ await done+ expect(b.expired).toHaveLength(1)+ })})diff --git a/src/core/agents/delivery-queue.ts b/src/core/agents/delivery-queue.tsindex 00e471518..3a4d81e1c 100644--- a/src/core/agents/delivery-queue.ts+++ b/src/core/agents/delivery-queue.ts@@ -190,6 +190,38 @@ export interface PersistedQueueEntry {/** A body larger than this is not written; its entry is expired at restore (sender told). */export const QUEUE_PERSIST_BODY_MAX = 256 * 1024+/** JSON bytes of FULL entries a snapshot writes before the rest go out reduced (`snapshot`). Half+ * the file limit: a reduced entry is a few hundred bytes, so 1024 of them fit in the other half. */+export const QUEUE_PERSIST_BYTES_BUDGET = 8 * 1024 * 1024++/** A reduced entry keeps only what its expiry needs to be routed and traced. */+const REDUCED_STRING_MAX = 200+const REDUCED_EXTRAS_MAX = 4++function clip(v: string): string {+ return v.length > REDUCED_STRING_MAX ? v.slice(0, REDUCED_STRING_MAX) : v+}++/** The request of an entry written without its body: short fields only, never a message text. */+function reducedRequest(req: QueuedDeliveryRequest): QueuedDeliveryRequest {+ const out: QueuedDeliveryRequest = {+ verb: typeof req.verb === 'string' ? clip(req.verb) : req.verb,+ sourceNodeId: clip(req.sourceNodeId),+ targetNodeId: req.targetNodeId,+ sourceTitle: clip(req.sourceTitle),+ body: ''+ }+ let extras = 0+ for (const [k, v] of Object.entries(req)) {+ if (k in out || extras >= REDUCED_EXTRAS_MAX) continue+ if (typeof v === 'number' || typeof v === 'boolean' || (typeof v === 'string' && v.length <= REDUCED_STRING_MAX)) {+ out[k] = v+ extras+++ }+ }+ return out+}+/** At most this many entries on disk. `DELIVERY_QUEUE_CAPACITY` per target bounds it in practice. */export const QUEUE_PERSIST_MAX = 1024/** The longest TTL a restored entry may claim — a hand-edited `ttlMs` must not keep one forever. */@@ -383,6 +415,9 @@ export class DeliveryQueue {// must not flush the same entry twice. It goes back on failure, at the FRONT, preserving order.list.shift()entry.cancelTimer()+ // Written off disk BEFORE the attempt (claim before effect): a crash mid-delivery then loses+ // this one message rather than typing it twice after the next boot. At most once.+ this.persist()// A restored entry goes only into the session it was queued for (see the header).const verdict = entry.restored? restoredBindingVerdict(entry.binding, this.deps.bindingOf?.(nodeId))@@ -453,6 +488,11 @@ export class DeliveryQueue {if (list.length === 0) this.queues.delete(nodeId)entry.cancelTimer()this.persist()+ await this.reportExpired(entry)+ }++ /** Trace `expired` and tell the sender — the two legs every expiry owes. */+ private async reportExpired(entry: QueueEntry): Promise<void> {const queuedForMs = this.deps.now() - entry.enqueuedAtconst t = await this.deps.trace({sourceNodeId: entry.req.sourceNodeId,@@ -467,17 +507,29 @@ export class DeliveryQueue {/** Every queued entry as it is written to disk, oldest first per target. */snapshot(): PersistedQueueEntry[] {const out: PersistedQueueEntry[] = []+ // The file is set aside WHOLE at load past DURABLE_STATE_MAX_BYTES, so the write is budgeted:+ // once the full entries reach QUEUE_PERSIST_BYTES_BUDGET (JSON bytes, oldest first per target),+ // the rest are written REDUCED — no body, only short fields — which restore turns into an+ // expiry the sender hears about. Never a file the next boot throws away with every message in it.+ let used = 0for (const list of this.queues.values()) {for (const e of list) {- const omit = e.req.body.length > QUEUE_PERSIST_BODY_MAX- out.push({- req: omit ? { ...e.req, body: '' } : e.req,+ const base = {enqueuedAt: e.enqueuedAt,ttlMs: e.ttlMs,queuedTraceId: e.queuedTraceId,- ...(e.binding ? { binding: e.binding } : {}),- ...(omit ? { bodyOmitted: true as const } : {})- })+ ...(e.binding ? { binding: e.binding } : {})+ }+ const full: PersistedQueueEntry = { req: e.req, ...base }+ const size = e.req.body.length > QUEUE_PERSIST_BODY_MAX ? Infinity : JSON.stringify(full).length+ if (used + size <= QUEUE_PERSIST_BYTES_BUDGET) {+ used += size+ out.push(full)+ continue+ }+ const reduced: PersistedQueueEntry = { req: reducedRequest(e.req), ...base, bodyOmitted: true }+ used += JSON.stringify(reduced).length+ out.push(reduced)}}return out@@ -494,7 +546,6 @@ export class DeliveryQueue {const now = this.deps.now()const lapsed: QueueEntry[] = []for (const p of entries) {- const list = this.queues.get(p.req.targetNodeId) ?? []// A clock that went backwards must not stretch the wait past one full TTL.const age = Math.max(0, now - p.enqueuedAt)const remaining = Math.min(p.ttlMs, p.ttlMs - age)@@ -507,20 +558,28 @@ export class DeliveryQueue {restored: true,...(p.binding ? { binding: p.binding } : {})}- list.push(entry)- this.queues.set(p.req.targetNodeId, list)this.deps.onQueued?.(entry.req)+ // Capacity counts only what is really re-queued: a lapsed entry never takes a slot.const deliverable =remaining > 0 &&!p.bodyOmitted &&RESTORABLE_VERBS.has(String(p.req.verb)) &&- list.length <= this.capacity- if (deliverable) {- entry.cancelTimer = this.schedule(remaining, () => void this.expire(p.req.targetNodeId, entry))- } else lapsed.push(entry)+ this.depth(p.req.targetNodeId) < this.capacity+ if (!deliverable) {+ // Never inserted into the live lists: a flush running during one of the expiries below+ // must not be able to deliver a board comment or an empty (body-omitted) entry.+ lapsed.push(entry)+ continue+ }+ const list = this.queues.get(p.req.targetNodeId) ?? []+ list.push(entry)+ this.queues.set(p.req.targetNodeId, list)+ entry.cancelTimer = this.schedule(remaining, () => void this.expire(p.req.targetNodeId, entry))}+ // The lapsed entries' ends are reported BEFORE the file drops them: a crash in between reports+ // one twice at the next boot, never not at all.+ for (const entry of lapsed) await this.reportExpired(entry)this.persist()- for (const entry of lapsed) await this.expire(entry.req.targetNodeId, entry)}private persist(): void {diff --git a/src/core/control-request-ledger.persist.test.ts b/src/core/control-request-ledger.persist.test.tsindex f3e8619c9..ee7362335 100644--- a/src/core/control-request-ledger.persist.test.ts+++ b/src/core/control-request-ledger.persist.test.ts@@ -1,5 +1,7 @@import { describe, it, expect } from 'vitest'+import { DURABLE_STATE_MAX_BYTES } from './durable-state'import {+ CONTROL_REQUEST_REPLIES_BUDGET,CONTROL_REQUEST_REPLY_MAX_BYTES,ControlRequestLedger,REQUEST_LEDGER_TTL_MS,@@ -72,4 +74,24 @@ describe('ControlRequestLedger persistence', () => {expect(sanitizeLedgerRow({ ...base, reply: undefined })).toMatchObject({ state: 'unknown' })expect(sanitizeLedgerRow(null)).toBeNull()})++ it('stays under the file limit when every row carries a maximal reply: the rows past the budget are UNKNOWN', () => {+ const a = new ControlRequestLedger({ now: () => 0 })+ const big = 'y'.repeat(CONTROL_REQUEST_REPLY_MAX_BYTES - 200)+ for (let i = 0; i < 4096; i++) {+ const d = a.begin(`n${i % 16}`, `r${i}`, fp)+ if (d.kind === 'run') d.claim.settle({ ok: true, message: big })+ }+ const rows = a.exportRows()+ expect(rows).toHaveLength(4096)+ expect(JSON.stringify(rows).length).toBeLessThan(DURABLE_STATE_MAX_BYTES)+ const settled = rows.filter((r) => r.state === 'settled')+ expect(settled.length).toBeGreaterThan(0)+ expect(JSON.stringify(settled.map((r) => r.reply)).length).toBeLessThanOrEqual(CONTROL_REQUEST_REPLIES_BUDGET + settled.length)+ // The oldest keep their reply; a row past the budget is still refused after a restart.+ expect(rows[0].state).toBe('settled')+ const last = rows[rows.length - 1]+ expect(last.state).toBe('unknown')+ expect(restarted([last], 1).begin(last.caller, last.requestId, fp)).toMatchObject({ outcome: 'request-outcome-unknown' })+ })})diff --git a/src/core/control-request-ledger.ts b/src/core/control-request-ledger.tsindex 3e71cf922..9bc6e40c8 100644--- a/src/core/control-request-ledger.ts+++ b/src/core/control-request-ledger.ts@@ -331,6 +331,10 @@ export interface PersistedLedgerRow {* few hundred bytes; the bound only keeps a pathological `result` out of the file. */export const CONTROL_REQUEST_REPLY_MAX_BYTES = 64 * 1024+/** JSON bytes of replies one ledger file carries; the rows past it are written UNKNOWN. Rows+ * themselves are ~300 bytes, so 4096 of them fit beside it under DURABLE_STATE_MAX_BYTES. */+export const CONTROL_REQUEST_REPLIES_BUDGET = 8 * 1024 * 1024+const FINGERPRINT_RE = /^[0-9a-f]{64}$//** Re-check one row read from disk (hand-editable input). `null` drops it. */@@ -469,6 +473,10 @@ export class ControlRequestLedger {*/exportRows(): PersistedLedgerRow[] {const out: PersistedLedgerRow[] = []+ // Budgeted, oldest claim first: 4096 rows of 64 KB replies would pass the file limit and the+ // whole file would be set aside at load. Past the budget a settled row is written UNKNOWN (no+ // reply) — still refused, never re-run.+ let used = 0for (const row of this.rows.values()) {const rec: PersistedLedgerRow = {caller: row.caller,@@ -480,8 +488,11 @@ export class ControlRequestLedger {}if (row.state === 'settled' && row.reply) {const reply = sanitizeReply(row.reply)- if (reply) rec.reply = reply- else rec.state = 'unknown'+ const size = reply ? JSON.stringify(reply).length : 0+ if (reply && used + size <= CONTROL_REQUEST_REPLIES_BUDGET) {+ rec.reply = reply+ used += size+ } else rec.state = 'unknown'}out.push(rec)}diff --git a/src/core/durable-state.test.ts b/src/core/durable-state.test.tsindex 3cfa24a2e..195feb5ac 100644--- a/src/core/durable-state.test.ts+++ b/src/core/durable-state.test.ts@@ -144,4 +144,31 @@ describe('DurableFactFile', () => {await a.flush()expect(h.make().load().map((r) => r.n)).toEqual([3, 4, 5])})++ it('a synchronous flush is never overwritten by an OLDER async write still in flight (review repro)', async () => {+ const h = harness()+ const a = h.make()+ a.save([{ id: 'old', n: 1 }])+ const inFlight = a.flush() // async write of [old] started, not yet renamed+ a.save([{ id: 'new', n: 2 }])+ a.flushSync() // quit: [new] lands synchronously+ await inFlight+ expect(h.make().load()).toEqual([{ id: 'new', n: 2 }])+ // No temp litter left by the dropped rename.+ expect(fs.readdirSync(path.dirname(h.file)).filter((f) => f.endsWith('.tmp'))).toEqual([])+ })++ it('a stood-down file neither reads nor writes (a second instance that does not own the fact)', async () => {+ const h = harness()+ const owner = h.make()+ owner.save([{ id: 'owner', n: 1 }])+ await owner.flush()+ const second = h.make()+ second.standDown()+ expect(second.load()).toEqual([])+ second.save([{ id: 'intruder', n: 2 }])+ second.flushSync()+ await second.flush()+ expect(h.make().load()).toEqual([{ id: 'owner', n: 1 }])+ })})diff --git a/src/core/durable-state.ts b/src/core/durable-state.tsindex f1cdeeb9e..dfbe2f7f9 100644--- a/src/core/durable-state.ts+++ b/src/core/durable-state.ts@@ -1,15 +1,16 @@// Orchestration facts that must outlive the process that learned them.//-// WHY THIS EXISTS. Three facts a canvas-control orchestration leans on lived only in process+// WHY THIS EXISTS. Four facts a canvas-control orchestration leans on lived only in process// memory, so an app (or Server Edition) restart erased them while the thing they described kept// going:// - a `send` answered `queued` (the sender was told it would be delivered) was silently lost;// - a station's `report-outcome` vanished, so every `--after-success` dependent read BLOCKED;-// - the `--request-id` ledger emptied, so a retry after a restart re-ran an open that happened.+// - the `--request-id` ledger emptied, so a retry after a restart re-ran an open that happened;+// - the plain-`--after` hand-over hold was forgotten, so a dependent fired on a stale `done`.// Each owner keeps its own rules (what a restart MEANS for that fact is decided next to the fact —-// delivery-queue.ts, station-outcome-store.ts, control-request-ledger.ts); this module is only the-// storage all of them share, so a new fact is one more `DurableFactSpec`, not a fourth copy of the-// read / sanitize / write / flush dance.+// delivery-queue.ts, station-outcome-store.ts, station-handover.ts, control-request-ledger.ts);+// this module is only the storage all of them share, so a new fact is one more `DurableFactSpec`,+// not another copy of the read / sanitize / write / flush dance.//// THE RULES, each for a reason:// - One file per fact kind under `<userData>/orchestration-state/`, MACHINE-LOCAL. Never@@ -24,9 +25,16 @@// is not JSON, too large, or not an envelope is set aside as `<file>.corrupt` (one copy, replaced// each time) and the fact starts empty with a warning. Loading NEVER throws: a boot that dies on// a bad orchestration file would take the whole app with it.-// - Written through `writeFileAtomic` (and `renameAtomicSync` for the synchronous flush at-// quit) — never a bare rename, see fs-atomic.ts. Mode 0600: a queued message's body is the-// user's text.+// - Written as a unique `wx` temp then `renameAtomicSync` — never a bare rename, see+// fs-atomic.ts. Mode 0600: a queued message's body is the user's text.+// - A synchronous flush (quit) is never overwritten by an OLDER async write still in flight: it+// bumps a generation, and the async path checks the generation and renames in one synchronous+// step, dropping its temp when it is stale.+// - Every OWNER must keep its serialized file under `DURABLE_STATE_MAX_BYTES` — a file past it is+// set aside WHOLE at load. The queue and the ledger enforce a byte budget on what they write+// (bodies / replies past it are written in their refusal-safe reduced form).+// - `standDown()` for a process that does not own the fact (a second instance on the same+// userData that lost the hook endpoint): it neither reads nor writes the file.// - Saves are COALESCED (`debounceMs`, a short window — every control call can touch the// ledger), at most one write in flight, and the LATEST snapshot always wins. `flushSync` writes// whatever is pending before the process exits; a crash inside the window loses that window,@@ -35,12 +43,13 @@// Pure `src/core`: no electron. The clock and the timer are injectable so tests drive it.import { closeSync, fstatSync, mkdirSync, openSync, readFileSync, rmSync, writeFileSync } from 'node:fs'import { join } from 'node:path'-import { renameAtomicSync, tempNameFor, writeFileAtomic } from './fs-atomic'+import { promises as fsp } from 'node:fs'+import { renameAtomicSync, tempNameFor } from './fs-atomic'/** Where every durable orchestration fact lives, relative to userData. */export const DURABLE_STATE_DIR = 'orchestration-state'-/** A file larger than this is not ours (every owner caps its records far below it). */+/** A file larger than this is set aside at load. Every owner budgets its writes below it. */export const DURABLE_STATE_MAX_BYTES = 16 * 1024 * 1024/** How long saves are coalesced. Short: a claimed request-id row should reach disk before the call@@ -95,6 +104,9 @@ export class DurableFactFile<T> {private cancelTimer: (() => void) | null = nullprivate writing: Promise<void> | null = nullprivate disposed = false+ /** Bumped by every synchronous flush; an async write started before it drops its rename. */+ private generation = 0+ private stoodDown = falseconstructor(private readonly spec: DurableFactSpec<T>,@@ -118,6 +130,7 @@ export class DurableFactFile<T> {/** Read and sanitize the file. Synchronous (boot), never throws; absent ⇒ `[]` silently. */load(): T[] {+ if (this.stoodDown) return []let text: stringlet fd: number | undefinedtry {@@ -209,6 +222,8 @@ export class DurableFactFile<T> {const snap = this.pendingif (!snap) returnthis.pending = null+ // Any async write still in flight carries an OLDER snapshot: it must not rename over this one.+ this.generation++try {this.ensureDir()const tmp = tempNameFor(this.path)@@ -224,6 +239,23 @@ export class DurableFactFile<T> {}}+ /**+ * This process does not own the fact (a second instance on the same userData that lost the hook+ * endpoint to the first): never read or write the file again. `load()` answers `[]` from here on,+ * so it neither restores (and expires) messages the owning instance holds nor overwrites its file.+ */+ standDown(): void {+ if (this.cancelTimer) {+ this.cancelTimer()+ this.cancelTimer = null+ }+ this.pending = null+ this.disposed = true+ this.stoodDown = true+ this.generation+++ liveFiles.delete(this)+ }+/** Stop accepting saves after flushing (shutdown). */dispose(): void {this.flushSync()@@ -237,11 +269,23 @@ export class DurableFactFile<T> {const snap = this.pendingif (!snap) returnthis.pending = null+ const generation = this.generationthis.writing = (async (): Promise<void> => {+ const tmp = tempNameFor(this.path)try {this.ensureDir()- await writeFileAtomic(this.path, this.serialize(snap), { mode: 0o600 })+ // Same shape as `writeFileAtomic` (unique `wx` temp, then the retrying rename), split so the+ // generation check and the rename happen in ONE synchronous step: a `flushSync` that ran+ // while the temp was being written has put a NEWER snapshot on disk, and this older one+ // must not land on top of it. Checked and renamed with nothing able to run in between.+ await fsp.writeFile(tmp, this.serialize(snap), { encoding: 'utf-8', flag: 'wx', mode: 0o600 })+ if (generation !== this.generation) {+ rmSync(tmp, { force: true })+ return+ }+ renameAtomicSync(tmp, this.path)} catch (e) {+ rmSync(tmp, { force: true })this.warn(`[durable-state] ${this.spec.kind}: save failed (${String(e)})`)}})()diff --git a/src/main/index.ts b/src/main/index.tsindex 757d38b34..bfdbaf3e6 100644--- a/src/main/index.ts+++ b/src/main/index.ts@@ -1969,10 +1969,11 @@ app.whenReady().then(async () => {// renderer reload does not lose it; pushed whole to the window on every change.// Durable across an app restart too (station-outcomes.json), each report bound to the session// that made it; `loadFromDisk()` runs below, after the status mirror it compares against.+ const stationOutcomesFile = new DurableFactFile(OUTCOME_FACT, { userDataDir: corePlatform.userDataDir })const stationOutcomes = new StationOutcomeStore((records) => sendToMain(IPC.stationOutcomeChanged, records),{- durable: new DurableFactFile(OUTCOME_FACT, { userDataDir: corePlatform.userDataDir }),+ durable: stationOutcomesFile,sessionOf: (id) => {const m = mirrorEntry(id)return m ? { sessionId: m.sessionId, agentId: m.agentId } : undefined@@ -1987,10 +1988,11 @@ app.whenReady().then(async () => {// handed new work is not "done" until a turn that started after the hand-over has ended. Pushed// whole to the window, whose launch loop reads it. Durable (station-handovers.json), loaded below// between the reports and the queue.+ const stationHandoversFile = new DurableFactFile(HANDOVER_FACT, { userDataDir: corePlatform.userDataDir })const stationHandovers = new StationHandoverTracker((records) => sendToMain(IPC.stationHandoverChanged, records),Date.now,- new DurableFactFile(HANDOVER_FACT, { userDataDir: corePlatform.userDataDir })+ stationHandoversFile)registerStationHandoverIpc(corePlatform, () => stationHandovers)messagingDeps.onHandover = (ev) => {@@ -2016,6 +2018,13 @@ app.whenReady().then(async () => {// listeners (setListener/setRawListener/setControlHandler) attach later, which the server// tolerates — early hook POSTs are simply dropped, never mis-routed.const hookStartupWarning = await hookServer.startForApp()+ // The durable orchestration facts belong to the instance that owns the hook endpoint, exactly+ // like the request ledger (which `hookServer.start()` only loads when it wins the endpoint). A+ // second instance on the same userData that lost it must neither restore them — it would expire,+ // and tell senders about, messages the owning instance still holds — nor overwrite their files.+ if (hookStartupWarning) {+ for (const f of [deliveryQueueFile, stationOutcomesFile, stationHandoversFile]) f.standDown()+ }// ---- Node identity (src/core/agents/node-auth-secret.ts) ------------------------------------// One secret does two jobs: it arms the hook server's per-node capability (closing the "shared// bearer can name any sibling node" hole) and it signs the codex thread → node records the hook@@ -2118,7 +2127,13 @@ app.whenReady().then(async () => {// sender told, every message whose TTL ran out while the app was down.stationOutcomes.loadFromDisk()stationHandovers.loadFromDisk()- void restoreDeliveryQueue(messagingDeps.queue, deliveryQueueFile)+ // The queue waits for the workspace INDEX: an entry that lapsed while the app was down is expired+ // here, and its sender leg (board-log line in the sender's project) resolves projects through the+ // index, which nothing else has loaded yet at this point of boot (the renderer's own+ // `workspace.load` comes later). Read-only load (`sideline: false`), like the relay path's.+ void restoreDeliveryQueue(messagingDeps.queue, deliveryQueueFile, {+ ready: workspaceStore.load({ sideline: false })+ })/** The one display-title rule for everything the HOST sends out (push alerts, Live Activity* updates, the notch capsule): the live session name unless the node was hand-renamed. */diff --git a/src/main/station-outcome-wiring.test.ts b/src/main/station-outcome-wiring.test.tsindex 60a59b220..6b8068450 100644--- a/src/main/station-outcome-wiring.test.ts+++ b/src/main/station-outcome-wiring.test.ts@@ -55,22 +55,32 @@ describe('desktop main', () => {})it('persists the store and the delivery queue, and loads them after the status mirror (durable state)', () => {- expect(main).toMatch(/durable: new DurableFactFile\(OUTCOME_FACT/)+ expect(main).toMatch(/const stationOutcomesFile = new DurableFactFile\(OUTCOME_FACT/)+ expect(main).toContain("durable: stationOutcomesFile")expect(main).toMatch(/createDeliveryQueue\(messagingDeps, \{ durable: deliveryQueueFile \}\)/)const mirror = main.indexOf('initAgentStatusMirror()\n')const load = main.indexOf('stationOutcomes.loadFromDisk()')const holds = main.indexOf('stationHandovers.loadFromDisk()')- const queue = main.indexOf('restoreDeliveryQueue(messagingDeps.queue, deliveryQueueFile)')+ const queue = main.indexOf('restoreDeliveryQueue(messagingDeps.queue, deliveryQueueFile, {')expect(mirror).toBeGreaterThan(-1)expect(load).toBeGreaterThan(mirror)expect(holds).toBeGreaterThan(load)expect(queue).toBeGreaterThan(holds)- expect(main).toMatch(/new DurableFactFile\(HANDOVER_FACT/)+ expect(main).toMatch(/const stationHandoversFile = new DurableFactFile\(HANDOVER_FACT/)// A station's new session withdraws its report BEFORE the renderer hears the event.const withdraw = main.indexOf('stationOutcomes.onAgentEvent(enriched)')expect(withdraw).toBeGreaterThan(-1)expect(withdraw).toBeLessThan(main.indexOf('sendToMain(IPC.agentStatus, enriched)'))expect(main).toContain('flushAllDurableFactsSync()')+ // The restore waits for the workspace index, or a restore-time expiry reaches no sender.+ expect(main.slice(queue, queue + 200)).toContain('ready: workspaceStore.load({ sideline: false })')+ // A second instance that lost the hook endpoint stands every durable file down.+ const gate = main.indexOf('const hookStartupWarning = await hookServer.startForApp()')+ expect(gate).toBeGreaterThan(-1)+ expect(gate).toBeLessThan(load)+ expect(main.slice(gate, gate + 900)).toMatch(+ /if \(hookStartupWarning\) \{\s*for \(const f of \[deliveryQueueFile, stationOutcomesFile, stationHandoversFile\]\) f\.standDown\(\)/+ )})})@@ -110,6 +120,8 @@ describe('Server Edition', () => {expect(serverControl).toContain('queueFile.dispose()')expect(serverControl).toContain('outcomesFile.dispose()')expect(serverControl).toContain('handoversFile.dispose()')+ expect(serverControl).toContain('if (deps.ownsDurableState === false) queueFile.standDown()')+ expect(serverIndex).toContain('ownsDurableState: hookStartupWarning === null')expect(serverControl.indexOf('stationHandovers.loadFromDisk()')).toBeLessThan(serverControl.indexOf('await restoreDeliveryQueue(queue, queueFile)'))diff --git a/src/server/canvas-control.ts b/src/server/canvas-control.tsindex 16e9ebd1f..5be9c4f1a 100644--- a/src/server/canvas-control.ts+++ b/src/server/canvas-control.ts@@ -91,6 +91,13 @@ export interface ServerCanvasControlDeps {* specifically exercising the install and has redirected `HOME` to a scratch directory first.*/installAgentIntegrations: boolean+ /**+ * Does this process own the hook endpoint (`hookServer.startForApp()` returned no warning)? The+ * durable orchestration facts (queue, station reports, hand-over holds) belong to the owning+ * instance, like the request ledger: a second instance on the same data dir must neither restore+ * them nor overwrite their files. Absent = owns (every test, and a caller that did not ask).+ */+ ownsDurableState?: boolean}export interface ServerCanvasControl {@@ -221,6 +228,10 @@ export async function initServerCanvasControl(Date.now,handoversFile)+ if (deps.ownsDurableState === false) {+ outcomesFile.standDown()+ handoversFile.standDown()+ }stationOutcomes.loadFromDisk()stationHandovers.loadFromDisk()const factory = new HeadlessNodeFactory({@@ -287,6 +298,7 @@ export async function initServerCanvasControl(// this edition's creator ledger is process-local, so a restored message whose caller→target proof// did not survive the restart is refused `caller-not-owner` at flush — with its sender told.const queueFile = new DurableFactFile(QUEUE_FACT, { userDataDir: platform().userDataDir })+ if (deps.ownsDurableState === false) queueFile.standDown()const queue = createDeliveryQueue(messaging, { durable: queueFile })messaging.queue = queuediff --git a/src/server/index.ts b/src/server/index.tsindex 6ffd40921..f1b61ee7e 100644--- a/src/server/index.ts+++ b/src/server/index.ts@@ -730,7 +730,9 @@ export async function startServer(// answer the issue lane gets from the GitHub host controller.issueRepository: (projectId) =>github.controller.status(projectId).then((view) => view.project?.repository ?? null),- installAgentIntegrations: config.installHooks !== false+ installAgentIntegrations: config.installHooks !== false,+ // The durable orchestration facts follow hook-endpoint ownership, like the request ledger.+ ownsDurableState: hookStartupWarning === null})hookServer.setControlHandler(canvasControl.handler)} catch (error) {diff --git a/src/shared/board-comment.ts b/src/shared/board-comment.tsindex 4c459d02c..fcc39e89b 100644--- a/src/shared/board-comment.ts+++ b/src/shared/board-comment.ts@@ -37,8 +37,9 @@ export const BOARD_COMMENT_BODY_MAX = 16_384/** How long after a `queued` outcome a comment row stops presenting it as live: the core's* deliver-on-idle TTL (`DELIVERY_QUEUE_TTL_MS`, 5 min — pinned by a core test) plus a minute. The- * queue is process memory; an app that quit while a comment waited never writes its end, and a- * row reading "queued" forever would be a silent lie. */+ * queue is durable and a clean restart expires a waiting comment at boot (writing its end), but a+ * crash inside the queue's save window, or a second instance that does not own the queue, still+ * never writes one — and a row reading "queued" forever would be a silent lie. */export const BOARD_COMMENT_QUEUE_STALE_MS = 6 * 60_000/** The trace/queue identity of a delivery started by a board comment: `board-comment:<commentId>`.