Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

Validate daemon lifetime with sustained soak and service chaos (#1009)

Dee7305140e0352
diff --git a/cli/e2e/endurance.e2e.ts b/cli/e2e/endurance.e2e.ts
new file mode 100644
index 000000000..985a74b82
--- /dev/null
+++ b/cli/e2e/endurance.e2e.ts
@@ -0,0 +1,456 @@
+/**
+ * The daemon over time, every service in its own process: what no single end-to-end test shows (opt-in:
+ * `SOAK=1`; about 90 minutes at its defaults).
+ *
+ * - **Soak** (`SOAK_MINUTES`, 60): Claude Code and Codex agents (`SOAK_AGENTS`, 6) take turns without a
+ * break, with interrupts, questions, tool calls and floods of terminal output among them; windows come
+ * and go, open an agent's terminal and close it; search, the Devices tab, the edge host's readers, the
+ * models and the Store are asked as the apps ask them; a fake dial is plugged in and out. Every harnessd
+ * process (master, core, each service's) is sampled every `SOAK_SAMPLE_MS` (30 s): resident memory,
+ * footprint and open files. Growth rates are measurements by default. Set `SOAK_MAX_MIB_PER_HOUR`
+ * and `SOAK_MAX_FDS_PER_HOUR` only against an established baseline for the same workload.
+ * - **Chaos** (`CHAOS_MINUTES`, 30), while the agents go on: a service process (search, viewers, the edge
+ * host, the gateway, models, the devices, and the experiments once woken) is killed, two at once, frozen
+ * until the master finds it hung, or killed again as it starts. Every time: the core never restarts; a
+ * request to the service is answered at once (`SERVICE_UNAVAILABLE` or its answer), never left hanging;
+ * the service comes back and serves again. At the end every turn sent was started once, in the order
+ * sent, and no tmux client or process of the daemon outlives it.
+ *
+ * `SOAK_OUT=<folder>` keeps the samples (CSV) and the report (JSON). `SOAK_SEED` picks the run.
+ */
+import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'
+import { LocalClient, type Frame } from './harness/client.js'
+import { IsolatedDaemon, until } from './harness/daemon.js'
+import { Desk } from './harness/desk.js'
+import { alive, everyPid, forget, harnessdProcesses, slopes, startSampler, strays, tmuxClients, TurnLedger, type Sample } from './harness/endurance.js'
+
+const ON = process.env.SOAK === '1'
+function knob(name: string, fallback: number, min: number, max: number): number {
+ const n = Number(process.env[name] ?? fallback)
+ if (!Number.isFinite(n) || n < min || n > max) throw new Error(`${name} must be between ${min} and ${max}`)
+ return n
+}
+const SOAK_MINUTES = knob('SOAK_MINUTES', 60, 0.1, 240)
+const CHAOS_MINUTES = knob('CHAOS_MINUTES', 30, 0.1, 240)
+const AGENTS = Math.floor(knob('SOAK_AGENTS', 6, 2, 24))
+const SAMPLE_MS = knob('SOAK_SAMPLE_MS', 30_000, 1000, 120_000)
+// Growth limits require a baseline for the same workload. Otherwise report measurements only.
+const MAX_MIB_PER_HOUR = process.env.SOAK_MAX_MIB_PER_HOUR === undefined ? null : knob('SOAK_MAX_MIB_PER_HOUR', 20, 0, 10000)
+const MAX_FDS_PER_HOUR = process.env.SOAK_MAX_FDS_PER_HOUR === undefined ? null : knob('SOAK_MAX_FDS_PER_HOUR', 30, 0, 10000)
+const SEED = knob('SOAK_SEED', 20261007, 0, 0xffffffff)
+const OUT = process.env.SOAK_OUT ?? (ON ? mkdtempSync(join(tmpdir(), 'harness-soak-')) : '')
+/** How long a request to a service that is down may take to be answered: at once, not after its wait. */
+const FAST_MS = 5_000
+const PROTOCOL = 3
+const MARK = /soak-[a-z0-9]+-\d+/
+
+/** mulberry32: small, seeded, and the same sequence on every machine. */
+function random(seed: number): () => number {
+ let a = seed >>> 0
+ return () => {
+ a = (a + 0x6d2b79f5) >>> 0
+ let t = a
+ t = Math.imul(t ^ (t >>> 15), t | 1)
+ t ^= t + Math.imul(t ^ (t >>> 7), t | 61)
+ return ((t ^ (t >>> 14)) >>> 0) / 4294967296
+ }
+}
+const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms))
+async function pause(ms: number, deadline: () => boolean): Promise<void> {
+ const end = Date.now() + ms
+ while (!deadline() && Date.now() < end) await sleep(Math.min(1000, end - Date.now()))
+}
+const isTurn = (type: string, agentId: string) => (frame: Frame) => frame.type === type && frame.agentId === agentId
+
+/** A request each service's process answers, as the apps ask it: answered by the service when it is up. */
+const PROBES: Record<string, { type: string; payload: Record<string, unknown> }> = {
+ gateway: { type: 'e2ee_pairings_list', payload: {} },
+ sharing: { type: 'harness_share_list', payload: {} },
+ search: { type: 'session_search', payload: { query: 'soak' } },
+ viewers: { type: 'dsh_list', payload: {} },
+ edge: { type: 'machine_resources', payload: {} },
+ models: { type: 'models_list', payload: {} },
+ devices: { type: 'harness_devices_list', payload: {} },
+ orchestrator: { type: 'orchestrator', payload: { action: 'list' } },
+ teams: { type: 'team', payload: { action: 'capabilities' } },
+ commandBar: { type: 'command_bar', payload: { request: { prompt: '' } } },
+}
+/** The service names a process's comings and goings are logged under (`[services] <name> connected`). */
+const LINKS: Record<string, string> = { search: 'search', viewers: 'viewers', edge: 'monitor', gateway: 'gateway', models: 'models', devices: 'devices', orchestrator: 'orchestrator', teams: 'teams', commandBar: 'commandBar', sharing: 'sharing' }
+const down = (answer: Record<string, unknown>) => answer.error === 'SERVICE_UNAVAILABLE' || answer.error === 'SERVICE_FAILED' || answer.error === 'GATEWAY_UNAVAILABLE'
+
+const acceptable = (name: string, answer: Record<string, unknown>): boolean => answer.error === undefined
+ || (name === 'commandBar' && answer.error === 'INVALID_REQUEST')
+ || (name === 'sharing' && answer.error === 'HARNESS_NOT_FOUND')
+async function wakeServices(client: LocalClient): Promise<void> {
+ for (const [name, probe] of Object.entries(PROBES)) {
+ await until(`${name} to answer its probe`, async () => {
+ const answer = await client.request(probe.type, probe.payload, 30_000)
+ if (down(answer)) return null
+ if (!acceptable(name, answer)) throw new Error(`${name}: ${JSON.stringify(answer)}`)
+ return true
+ }, 60_000, 200)
+ }
+}
+
+interface Findings { lines: string[] }
+
+/** An agent's pane as it is now, kept beside the report (`SOAK_OUT`) once per agent and token. */
+async function keepPane(d: IsolatedDaemon, agentId: string, name: string): Promise<void> {
+ let client: LocalClient | undefined
+ try {
+ client = await LocalClient.connect(d)
+ const row = ((await client.request('agents_list', {}, 15_000)).agents as Array<Record<string, any>>).find((agent) => agent.id === agentId)
+ if (row?.tmuxPane) {
+ writeFileSync(join(OUT, `pane-${name}.txt`), await d.capture(String(row.tmuxPane)))
+ // Input and question readers include scrollback. Keeping only the visible pane hid the old
+ // dialog they still read after its answer, while the screen already showed the next composer.
+ const capture = await d.tmux.run('capture-pane', '-p', '-e', '-J', '-S', '-100', '-t', String(row.tmuxPane))
+ writeFileSync(join(OUT, `pane-${name}-history.txt`), capture)
+ }
+ } catch { /* a pane that cannot be read: the finding stands without it */ }
+ finally { client?.close() }
+}
+
+/** One agent's turns, one after another until `until`: plain, a tool call, a question answered, an
+ * interrupt, a flood of output; each carries a token the ledger checks. */
+async function agentLoop(d: IsolatedDaemon, agentId: string, tag: string, ledger: TurnLedger, deadline: () => boolean, rand: () => number, findings: Findings): Promise<number> {
+ const client = await LocalClient.connect(d)
+ let n = 0
+ try {
+ while (!deadline()) {
+ const token = `soak-${tag}-${n++}`
+ const pick = rand()
+ const kind = pick < 0.5 ? 'plain' : pick < 0.65 ? 'tool' : pick < 0.78 ? 'ask' : pick < 0.93 ? 'interrupt' : 'flood'
+ const content = kind === 'plain' ? `${token} hello` : kind === 'tool' ? `!tool echo ${token}` : kind === 'ask' ? `!ask ${token}`
+ : kind === 'interrupt' ? `!holdtool sleep-${token}` : `!flood ${token}`
+ const ended = kind === 'interrupt' ? null : client.next(isTurn('turn_ended', agentId), 120_000, `turn_ended (${token})`)
+ const asked = kind === 'ask' ? client.next((frame) => frame.type === 'commander_question' && frame.agentId === agentId, 60_000, `question (${token})`) : null
+ ledger.send(agentId, token)
+ client.send('message', { agentId, content })
+ try {
+ if (asked) {
+ const question = (await asked).payload ?? {}
+ const shaped = (question.questions as Array<{ q: string }> | undefined)?.[0]
+ client.send('question_response', { requestId: question.requestId, agentId, answers: shaped ? { [shaped.q]: 'Coffee' } : {} })
+ }
+ if (kind === 'interrupt') {
+ await client.next((frame) => frame.type === 'turn_started' && frame.agentId === agentId, 60_000, `turn_started (${token})`)
+ await sleep(800 + rand() * 1_500)
+ // Told, not asked; and a turn interrupted ends with no turn_ended (core/turns/cancel.ts): every window
+ // reads the agent as idle.
+ const idle = client.next((frame) => frame.type === 'agent_activity' && frame.agentId === agentId && frame.payload?.activity?.state === 'idle', 60_000, `idle after its interrupt (${token})`)
+ client.send('cancel', { agentId })
+ await idle
+ // The engine's own end of the interrupted turn, written as the pane takes the interrupt.
+ await sleep(1_000)
+ }
+ await ended
+ } catch (error) {
+ findings.lines.push(`${new Date().toISOString()} ${tag} ${kind} ${token}: ${error instanceof Error ? error.message : String(error)}`)
+ // What the agent's pane showed then: what the daemon read, to tell its mistake from the engine's.
+ await keepPane(d, agentId, `${tag}-${token}`)
+ await ended?.catch(() => {})
+ }
+ forget(client)
+ await sleep(rand() * 1_500)
+ }
+ } finally { client.close() }
+ return n
+}
+
+/** Windows coming and going: each says its tabs and focus, opens an agent's terminal, keeps it a while,
+ * and closes it or simply goes; and the apps' requests to the services, now and then. */
+async function deskLoop(d: IsolatedDaemon, agentIds: string[], deadline: () => boolean, rand: () => number, findings: Findings): Promise<number> {
+ let windows = 0
+ const ask = async (client: LocalClient, service: string): Promise<void> => {
+ const probe = PROBES[service]
+ const answer = await client.request(probe.type, probe.payload, 60_000).catch((error) => ({ error: String(error) }))
+ if (!acceptable(service, answer) && !down(answer)) findings.lines.push(`${new Date().toISOString()} desk ${probe.type}: ${JSON.stringify(answer).slice(0, 200)}`)
+ }
+ while (!deadline()) {
+ windows++
+ const window = await LocalClient.connect(d)
+ try {
+ const shown = agentIds.filter(() => rand() < 0.6)
+ window.send('app_swarms', { active: 't1', swarms: [{ id: 't1', name: 'Tab', agentIds: shown, panes: shown.length }], tiles: [] })
+ window.send('app_panes', { agentIds: shown, foreground: rand() < 0.8 })
+ const agentId = agentIds[Math.floor(rand() * agentIds.length)]
+ window.send('app_focus', { agentId })
+ const requestId = `open-${windows}`
+ const opened = window.next((frame) => (frame.type === 'terminal_ready' || frame.type === 'terminal_error') && frame.payload?.requestId === requestId, 30_000, 'terminal_ready')
+ window.send('terminal_open', { requestId, protocolVersion: PROTOCOL, agentId, cols: 100, rows: 30, takeover: false })
+ const ready = await opened.catch((error) => ({ type: 'timeout', payload: { error: String(error) } }) as Frame)
+ if (ready.type !== 'terminal_ready') findings.lines.push(`${new Date().toISOString()} desk terminal_open: ${JSON.stringify(ready.payload).slice(0, 200)}`)
+ const streamId = ready.payload?.streamId as string | undefined
+ const holdFor = 3_000 + rand() * 12_000
+ const until = Date.now() + holdFor
+ while (Date.now() < until && !deadline()) {
+ if (streamId) {
+ window.send('terminal_alive', { streamId })
+ const drawn = window.binaries.filter((frame) => frame.streamId === streamId)
+ if (drawn.length) window.send('terminal_ack', { streamId, lastSeq: drawn[drawn.length - 1].seq })
+ }
+ const service = Object.keys(PROBES)[Math.floor(rand() * Object.keys(PROBES).length)]
+ if (rand() < 0.4 && !['orchestrator', 'teams', 'commandBar'].includes(service)) await ask(window, service)
+ forget(window)
+ await sleep(2_000)
+ }
+ if (streamId && rand() < 0.7) window.send('terminal_close', { streamId })
+ } finally {
+ window.close()
+ }
+ await sleep(rand() * 3_000)
+ }
+ return windows
+}
+
+/** A dial plugged in and taken out now and then. */
+async function dialLoop(desk: Desk, deadline: () => boolean, rand: () => number): Promise<number> {
+ let plugs = 0
+ while (!deadline()) {
+ await pause(60_000 + rand() * 120_000, deadline)
+ if (deadline()) break
+ plugs++
+ await desk.plug('SOAK-DIAL', 'e2:e0:00:00:00:5a')
+ await pause(20_000 + rand() * 40_000, deadline)
+ await desk.unplug('SOAK-DIAL')
+ }
+ return plugs
+}
+
+async function createAgents(d: IsolatedDaemon, count: number): Promise<string[]> {
+ const client = await LocalClient.connect(d)
+ const ids: string[] = []
+ try {
+ for (let i = 0; i < count; i++) {
+ const cwd = join(d.projectsDir, `soak-${i}`)
+ mkdirSync(cwd, { recursive: true })
+ const created = await client.request('agent_create', { engine: i % 2 ? 'codex' : 'claude', cwd, bypassPermission: true }, 90_000)
+ expect(created.error, JSON.stringify(created)).toBeUndefined()
+ ids.push(created.agent.id)
+ }
+ for (const id of ids) {
+ await until(`${id.slice(0, 8)} to bind`, async () => ((await client.request('agents_list', {})).agents as Array<Record<string, unknown>>)
+ .find((agent) => agent.id === id)?.sessionId || null, 90_000, 500)
+ }
+ } finally { client.close() }
+ return ids
+}
+
+describe.skipIf(!ON)('the daemon over time, every service in its own process', () => {
+ let daemon: IsolatedDaemon | undefined
+ let desk: Desk | undefined
+ const cleanup: Array<() => void | Promise<void>> = []
+ afterEach(async () => {
+ const ended = await Promise.allSettled(cleanup.splice(0).reverse().map(stop => Promise.resolve().then(stop)))
+ await desk?.close()
+ await daemon?.close()
+ daemon = undefined
+ desk = undefined
+ for (const result of ended) if (result.status === 'rejected') throw result.reason
+ })
+
+ const fresh = async (env: Record<string, string> = {}): Promise<{ d: IsolatedDaemon; desk: Desk }> => {
+ const d = await IsolatedDaemon.create({ env: { CABLE_DISABLE: 'false', ...env } })
+ daemon = d
+ const file = join(d.root, 'dials.json')
+ writeFileSync(file, '[]')
+ d.env.HARNESSD_TEST_DIAL_PORT = file
+ desk = new Desk(file)
+ onTestFailed(() => {
+ writeFileSync(join(OUT, `daemon-${d.port}.log`), d.log())
+ console.log(`---- daemon log (whole log in ${OUT})\n${d.log().split('\n').slice(-120).join('\n')}`)
+ })
+ await d.start()
+ return { d, desk: desk! }
+ }
+
+ it(`soak: ${AGENTS} agents for ${SOAK_MINUTES} min; measure every process; preserve every turn`, async () => {
+ mkdirSync(OUT, { recursive: true })
+ const { d, desk } = await fresh()
+ const agentIds = await createAgents(d, AGENTS)
+ const observer = await LocalClient.connect(d)
+ cleanup.push(() => observer.close())
+ const ledger = new TurnLedger(observer, MARK)
+ const reading = setInterval(() => ledger.drain(), 1_000)
+ cleanup.push(() => clearInterval(reading))
+ await wakeServices(observer)
+ const samples: Sample[] = []
+ const stopSampling = startSampler(d, SAMPLE_MS, join(OUT, 'soak-samples.csv'), samples)
+ cleanup.push(stopSampling)
+ const findings: Findings = { lines: [] }
+ const ends = Date.now() + SOAK_MINUTES * 60_000
+ let stopping = false
+ cleanup.push(() => { stopping = true })
+ const deadline = () => stopping || Date.now() >= ends
+ const [turns, windows, plugs] = await Promise.all([
+ Promise.all(agentIds.map((id, i) => agentLoop(d, id, `a${i}`, ledger, deadline, random(SEED + i + 1), findings))),
+ deskLoop(d, agentIds, deadline, random(SEED + 100), findings),
+ dialLoop(desk, deadline, random(SEED + 200)),
+ ])
+ await stopSampling()
+ clearInterval(reading)
+ await sleep(3_000)
+ const differences = ledger.differences()
+ observer.close()
+ const warmup = Math.min(10 * 60_000, (SOAK_MINUTES * 60_000) / 4)
+ const growth = slopes(samples, warmup, Math.max(SAMPLE_MS * 4, 2 * 60_000))
+ const report = { minutes: SOAK_MINUTES, agents: AGENTS, turns: turns.reduce((sum, n) => sum + n, 0), windows, plugs, ledger: ledger.totals(), differences, findings: findings.lines, slopes: growth, coresStarted: d.coresStarted() }
+ writeFileSync(join(OUT, 'soak-report.json'), JSON.stringify(report, null, 2))
+ console.log(`soak report: ${join(OUT, 'soak-report.json')}\n${growth.map((s) => `${s.name}: ${s.first.toFixed(0)} → ${s.last.toFixed(0)} MiB, ${s.perHour?.toFixed(1) ?? 'unmeasured'} MiB/h; fds ${s.fdsFirst} → ${s.fdsLast}, ${s.fdsPerHour?.toFixed(1)}/h; restarts ${s.restarts}`).join('\n')}`)
+ expect(d.coresStarted()).toBe(1)
+ expect(differences).toEqual([])
+ expect(findings.lines).toEqual([])
+ expect(growth.map(p => p.name).sort()).toEqual(['master', 'core', ...Object.keys(PROBES)].sort())
+ for (const process of growth) {
+ expect(process.samples, `${process.name} sample count`).toBeGreaterThanOrEqual(2)
+ expect(process.restarts, `${process.name} restarted`).toBe(0)
+ if (MAX_MIB_PER_HOUR !== null) {
+ expect(process.perHour, `${process.name}: insufficient windows for a growth limit`).not.toBeNull()
+ expect(process.perHour).toBeLessThan(MAX_MIB_PER_HOUR)
+ }
+ if (MAX_FDS_PER_HOUR !== null) {
+ expect(process.fdsPerHour, `${process.name}: insufficient descriptor measurements`).not.toBeNull()
+ expect(process.fdsPerHour).toBeLessThan(MAX_FDS_PER_HOUR)
+ }
+ }
+ }, (SOAK_MINUTES + 15) * 60_000)
+
+ it(`chaos: services killed, frozen and killed as they start, for ${CHAOS_MINUTES} min, while ${AGENTS} agents work`, async () => {
+ mkdirSync(OUT, { recursive: true })
+ const { d, desk } = await fresh({
+ // Frozen, a service is found hung in seconds; killed, it is back in under two; and it is never parked:
+ // the run kills each one far more often than a crash loop would.
+ HARNESSD_SERVICE_HEARTBEAT_TIMEOUT_MS: '4000',
+ HARNESSD_SERVICE_INITIAL_BACKOFF_MS: '200',
+ HARNESSD_SERVICE_MAX_BACKOFF_MS: '2000',
+ HARNESSD_SERVICE_PARK_CRASHES: '100000',
+ })
+ const rand = random(SEED + 1_000)
+ const agentIds = await createAgents(d, AGENTS)
+ const observer = await LocalClient.connect(d)
+ cleanup.push(() => observer.close())
+ const ledger = new TurnLedger(observer, MARK)
+ const reading = setInterval(() => ledger.drain(), 1_000)
+ cleanup.push(() => clearInterval(reading))
+ const findings: Findings = { lines: [] }
+ // The experiments and the devices, woken as the apps wake them, so their processes are in the run too.
+ const waker = await LocalClient.connect(d)
+ await desk.plug('CHAOS-DIAL', 'e2:e0:00:00:00:5b')
+ cleanup.push(() => waker.close())
+ await wakeServices(waker)
+ const samples: Sample[] = []
+ const stopSampling = startSampler(d, SAMPLE_MS, join(OUT, 'chaos-samples.csv'), samples)
+ cleanup.push(stopSampling)
+ const ends = Date.now() + CHAOS_MINUTES * 60_000
+ let stopping = false
+ cleanup.push(() => { stopping = true })
+ const deadline = () => stopping || Date.now() >= ends
+ const events: Array<Record<string, unknown>> = []
+ const starts = (name: string) => [...d.log().matchAll(new RegExp(`\\[harnessd\\] service ${name} started \\(pid (\\d+)\\)`, 'g'))].map((match) => Number(match[1]))
+ const linked = (name: string) => d.log().split(`[services] ${LINKS[name]} connected`).length - 1
+
+ /** Asked while it is down: answered at once. Frozen: answered by the time the master has found it hung
+ * and killed it, never left waiting. Back: answered by it again. */
+ const probe = async (name: string, phase: 'down' | 'frozen' | 'back'): Promise<void> => {
+ const spec = PROBES[name]
+ if (!spec) return
+ const asked = Date.now()
+ const answer = await waker.request(spec.type, spec.payload, 60_000).catch((error) => ({ error: `no answer: ${String(error)}` }))
+ const ms = Date.now() - asked
+ if (phase === 'down' && ms > FAST_MS) findings.lines.push(`${name} down: ${spec.type} answered after ${ms} ms: ${JSON.stringify(answer).slice(0, 160)}`)
+ if (!acceptable(name, answer) && (phase === 'back' || !down(answer))) findings.lines.push(`${name} ${phase}: ${spec.type}: ${JSON.stringify(answer).slice(0, 160)}`)
+ events[events.length - 1][`${phase}Ms`] = ms
+ }
+ /** The master started it again and it connected to the core. */
+ const comesBack = async (name: string, startsBefore: number, linksBefore: number): Promise<boolean> => {
+ try {
+ await until(`${name} to be started again`, () => starts(name).length > startsBefore || null, 60_000, 100)
+ await until(`${name} to connect again`, () => linked(name) > linksBefore || null, 60_000, 100)
+ return true
+ } catch (error) {
+ findings.lines.push(`${name} never came back: ${error instanceof Error ? error.message : String(error)}`)
+ return false
+ }
+ }
+
+ const actions = ['kill', 'kill two', 'freeze', 'kill as it starts'] as const
+ const chaos = (async () => {
+ let cycle = 0
+ while (!deadline()) {
+ await pause(5000 + rand() * 5000, deadline)
+ if (deadline()) break
+ const running = [...harnessdProcesses(d)].filter(([name, pid]) => name !== 'master' && name !== 'core' && alive(pid))
+ if (!running.length) continue
+ const names = Object.keys(PROBES)
+ const name = names[cycle % names.length]
+ const action = actions[Math.floor(cycle / names.length) % actions.length]
+ cycle++
+ const primary = running.find(([role]) => role === name)
+ if (!primary) { findings.lines.push(`${name} was not running for ${action}`); continue }
+ const second = running.find(([role]) => role !== name)
+ const targets = action === 'kill two' && second ? [primary, second] : [primary]
+ const before = targets.map(([name]) => ({ name, starts: starts(name).length, links: linked(name) }))
+ const hung = (): number => d.log().split(`[harnessd] service ${targets[0][0]} sent no heartbeat`).length
+ const hungBefore = hung()
+ events.push({ at: new Date().toISOString(), action, targets: targets.map(([name]) => name) })
+ for (const [, pid] of targets) process.kill(pid, action === 'freeze' ? 'SIGSTOP' : 'SIGKILL')
+ if (action === 'kill as it starts') {
+ const [{ name, starts: count }] = before
+ await until(`${name} to be started`, () => starts(name).length > count || null, 30_000, 20).catch(error => { findings.lines.push(String(error)) })
+ const next = starts(name)[count]
+ if (next && alive(next)) process.kill(next, 'SIGKILL')
+ before[0].starts = count + 1
+ }
+ if (action === 'freeze') {
+ const frozen = probe(targets[0][0], 'frozen')
+ await until(`${targets[0][0]} to be found hung`, () => hung() > hungBefore || null, 30_000, 100).catch(error => { findings.lines.push(String(error)) })
+ await frozen
+ } else await probe(targets[0][0], 'down')
+ for (const { name, starts: count, links } of before) {
+ if (await comesBack(name, count, links)) await probe(name, 'back')
+ }
+ if (d.coresStarted() !== 1) findings.lines.push(`the core restarted after ${action} ${targets.map(([name]) => name).join(', ')}`)
+ }
+ })()
+ const [turns] = await Promise.all([
+ Promise.all(agentIds.map((id, i) => agentLoop(d, id, `c${i}`, ledger, deadline, random(SEED + 2_000 + i), findings))),
+ deskLoop(d, agentIds, deadline, random(SEED + 3_000), findings),
+ chaos,
+ ])
+ await stopSampling()
+ clearInterval(reading)
+ await sleep(3_000)
+ const differences = ledger.differences()
+ observer.close()
+ waker.close()
+ await desk.unplug('CHAOS-DIAL')
+ // Every window has gone: no terminal stream, so no tmux client, is left behind.
+ const clients = await until('every tmux client to be let go', async () => ((await tmuxClients(d)) === 0 ? true : null), 30_000, 500).catch(async () => `still ${await tmuxClients(d)}`)
+ const pids = everyPid(d)
+ const growth = slopes(samples, 0, Math.max(SAMPLE_MS * 4, 2 * 60_000))
+ const report = { minutes: CHAOS_MINUTES, agents: AGENTS, turns: turns.reduce((sum, n) => sum + n, 0), events, ledger: ledger.totals(), differences, findings: findings.lines, slopes: growth, coresStarted: d.coresStarted(), tmuxClients: clients }
+ writeFileSync(join(OUT, 'chaos-report.json'), JSON.stringify(report, null, 2))
+ console.log(`chaos report: ${join(OUT, 'chaos-report.json')}: ${events.length} events, ${report.turns} turns`)
+ expect(d.coresStarted()).toBe(1)
+ expect(differences).toEqual([])
+ expect(findings.lines).toEqual([])
+ expect(events.length, 'no chaos event exercised').toBeGreaterThan(0)
+ if (CHAOS_MINUTES >= 15) {
+ expect(new Set(events.map(event => event.action))).toEqual(new Set(actions))
+ expect(new Set(events.flatMap(event => event.targets as string[]))).toEqual(new Set(Object.keys(PROBES)))
+ expect(new Set(events.flatMap(event => (event.targets as string[]).map(name => `${event.action}:${name}`))))
+ .toEqual(new Set(actions.flatMap(action => Object.keys(PROBES).map(name => `${action}:${name}`))))
+ }
+ expect(clients).toBe(true)
+ // Nothing of the daemon's outlives it: every process its master started, its engines and its servers.
+ await daemon!.close()
+ daemon = undefined
+ await until('every process of the daemon to be gone', () => pids.every((pid) => !alive(pid)) || null, 30_000, 250)
+ expect(await strays(d)).toEqual([])
+ }, (CHAOS_MINUTES + 15) * 60_000)
+})
diff --git a/cli/e2e/enduranceHarness.e2e.ts b/cli/e2e/enduranceHarness.e2e.ts
new file mode 100644
index 000000000..49fa86cdf
--- /dev/null
+++ b/cli/e2e/enduranceHarness.e2e.ts
@@ -0,0 +1,25 @@
+/** The long-run harness must not turn missing evidence into a passing result. */
+import { describe, expect, it } from 'vitest'
+import type { LocalClient } from './harness/client.js'
+import { slopes, TurnLedger, type Sample } from './harness/endurance.js'
+
+describe('endurance evidence', () => {
+ it('requires three independent windows before reporting memory growth', () => {
+ const sample = (at: number, rssMiB: number, pid = 1): Sample => ({ at, rssMiB, name: 'core', pid, footprintMiB: null, fds: 10 })
+ expect(slopes([], 0, 1000)).toEqual([])
+ expect(slopes([sample(0, 10), sample(500, 10)], 0, 1000)[0]).toMatchObject({ perHour: null, windows: 1 })
+ expect(slopes([sample(0, 10), sample(1000, 11), sample(2000, 12)], 0, 1000)[0]).toMatchObject({ perHour: 3600, windows: 3, fdsPerHour: 0 })
+ expect(slopes([sample(0, 10), sample(1000, 11), sample(2000, 1, 2)], 0, 1000)[0]).toMatchObject({ perHour: null, restarts: 1, samples: 1 })
+ })
+ it('finds lost, duplicate, out-of-order and unsolicited turns including unknown agents', () => {
+ const frames: LocalClient['frames'] = []
+ const ledger = new TurnLedger({ frames } as LocalClient, /soak-[a-z]+-\d+/)
+ const seen = (agentId: string, token: string) => frames.push({ type: 'turn_started', agentId, payload: { userMessage: token } })
+ ledger.send('a', 'soak-a-1'); ledger.send('a', 'soak-a-2'); ledger.send('a', 'soak-a-3')
+ seen('a', 'soak-a-2'); seen('a', 'soak-a-1'); seen('unknown', 'soak-x-1')
+ expect(ledger.differences()).toEqual(['a: never seen soak-a-3 (1 of 3)', 'a: out of order', 'unknown: never sent soak-x-1'])
+ seen('a', 'soak-a-2')
+ expect(ledger.differences()).toContain('a: seen twice soak-a-2')
+ expect(ledger.totals()).toEqual({ sent: 3, seen: 4 })
+ })
+})
diff --git a/cli/e2e/harness/endurance.ts b/cli/e2e/harness/endurance.ts
new file mode 100644
index 000000000..55f19ef03
--- /dev/null
+++ b/cli/e2e/harness/endurance.ts
@@ -0,0 +1,203 @@
+/**
+ * What the soak and chaos run (e2e/endurance.e2e.ts) reads of a daemon over time: every harnessd process
+ * it runs (the master, the core and each service's process) with its memory and open files, sampled on a
+ * period; the slope of each, over the run; and the ledger of every turn the run sent, against what a
+ * window saw.
+ */
+import { execFile } from 'node:child_process'
+import { appendFileSync, readdirSync, readFileSync } from 'node:fs'
+import { promisify } from 'node:util'
+import type { Frame, LocalClient } from './client.js'
+import type { IsolatedDaemon } from './daemon.js'
+
+const exec = promisify(execFile)
+
+/** This daemon's harnessd processes now, by name: `master`, `core`, and each service process by its name. */
+export function harnessdProcesses(d: IsolatedDaemon): Map<string, number> {
+ const found = new Map<string, number>()
+ if (d.pid) found.set('master', d.pid)
+ const core = d.corePid()
+ if (core) found.set('core', core)
+ for (const match of d.log().matchAll(/\[harnessd\] service (\S+) started \(pid (\d+)\)/g)) found.set(match[1], Number(match[2]))
+ return found
+}
+
+/** Every pid this daemon's master ever said it started, cores and services: none may outlive it. */
+export function everyPid(d: IsolatedDaemon): number[] {
+ return [...(d.pid ? [d.pid] : []), ...[...d.log().matchAll(/\[harnessd\] (?:core|service \S+) started \(pid (\d+)\)/g)].map((match) => Number(match[1]))]
+}
+
+export function alive(pid: number): boolean {
+ try { process.kill(pid, 0); return true } catch { return false }
+}
+
+export interface Sample { at: number; name: string; pid: number; rssMiB: number; footprintMiB: number | null; fds: number | null }
+
+/** Resident memory, footprint (macOS's own measure of what a process costs) and open files of one process. */
+export async function measure(pid: number): Promise<Omit<Sample, 'at' | 'name' | 'pid'> | null> {
+ if (process.platform === 'linux') {
+ try {
+ const status = readFileSync(`/proc/${pid}/status`, 'utf8')
+ const rss = Number(/VmRSS:\s+(\d+)/.exec(status)?.[1] ?? 0) / 1024
+ return { rssMiB: rss, footprintMiB: null, fds: readdirSync(`/proc/${pid}/fd`).length }
+ } catch { return null }
+ }
+ const rss = await exec('ps', ['-o', 'rss=', '-p', String(pid)], { timeout: 3000 }).then(({ stdout }) => Number(stdout.trim()) / 1024, () => 0)
+ if (!rss) return null
+ const footprint = await exec('footprint', ['-p', String(pid)], { timeout: 3000 }).then(({ stdout }) => {
+ const match = /Footprint:\s+([\d.]+)\s+(KB|MB|GB)/.exec(stdout)
+ if (!match) return null
+ const value = Number(match[1])
+ return match[2] === 'KB' ? value / 1024 : match[2] === 'GB' ? value * 1024 : value
+ }, () => null)
+ const fds = await exec('lsof', ['-n', '-P', '-a', '-p', String(pid), '-F', 'f'], { timeout: 3000, maxBuffer: 16 * 1024 * 1024 })
+ .then(({ stdout }) => stdout.split('\n').filter(line => /^f\d+$/.test(line)).length, () => null)
+ return { rssMiB: rss, footprintMiB: footprint, fds }
+}
+
+/** Every harnessd process of a daemon, every `everyMs`, into `samples` and a CSV file. */
+export function startSampler(d: IsolatedDaemon, everyMs: number, csv: string, samples: Sample[]): () => Promise<void> {
+ appendFileSync(csv, 'at,name,pid,rssMiB,footprintMiB,fds\n')
+ let stopped = false
+ let failure: unknown
+ let running: Promise<void> = Promise.resolve()
+ let timer: ReturnType<typeof setTimeout> | undefined
+ const round = async (): Promise<void> => {
+ const at = Date.now()
+ for (const [name, pid] of harnessdProcesses(d)) {
+ if (stopped) break
+ if (!alive(pid)) continue
+ const measured = await measure(pid)
+ if (!measured) continue
+ const sample = { at, name, pid, ...measured }
+ samples.push(sample)
+ appendFileSync(csv, `${at},${name},${pid},${measured.rssMiB.toFixed(1)},${measured.footprintMiB?.toFixed(1) ?? ''},${measured.fds ?? ''}\n`)
+ }
+ }
+ const tick = (): void => {
+ running = round().catch(error => { failure = error; stopped = true }).finally(() => { if (!stopped) timer = setTimeout(tick, everyMs) })
+ }
+ tick()
+ return async () => { stopped = true; clearTimeout(timer); await running; if (failure) throw failure }
+}
+
+export interface Slope { name: string; samples: number; first: number; last: number; perHour: number | null; windows: number; fdsFirst: number | null; fdsLast: number | null; fdsPerHour: number | null; restarts: number }
+
+/** Least squares over (minutes, value). */
+function fit(points: Array<[number, number]>): number {
+ const n = points.length
+ if (n < 2) return 0
+ const mx = points.reduce((sum, [x]) => sum + x, 0) / n
+ const my = points.reduce((sum, [, y]) => sum + y, 0) / n
+ const sxx = points.reduce((sum, [x]) => sum + (x - mx) ** 2, 0)
+ if (!sxx) return 0
+ return points.reduce((sum, [x, y]) => sum + (x - mx) * (y - my), 0) / sxx
+}
+
+/**
+ * Each process's growth per hour, past the warm-up: the line through the lowest sample of each window of
+ * `windowMs`, so a collection that has not run yet is not read as growth (the lowest is what the process
+ * holds), and only the last pid of each name (a process restarted starts over).
+ */
+export function slopes(samples: Sample[], warmupMs: number, windowMs: number): Slope[] {
+ if (!samples.length) return []
+ const start = samples[0].at
+ const names = [...new Set(samples.map((sample) => sample.name))]
+ return names.map((name) => {
+ const all = samples.filter((sample) => sample.name === name)
+ const pids = [...new Set(all.map((sample) => sample.pid))]
+ const lastPid = pids[pids.length - 1]
+ const mine = all.filter((sample) => sample.pid === lastPid && sample.at - start >= warmupMs)
+ const windows = new Map<number, Sample[]>()
+ for (const sample of mine) {
+ const key = Math.floor((sample.at - start) / windowMs)
+ windows.set(key, [...(windows.get(key) ?? []), sample])
+ }
+ const lows = [...windows.values()].map((group) => group.reduce((low, sample) => (sample.rssMiB < low.rssMiB ? sample : low)))
+ const fdLows = [...windows.values()].map((group) => group.filter((sample) => sample.fds !== null).reduce<Sample | null>((low, sample) => (!low || sample.fds! < low.fds! ? sample : low), null))
+ .filter((sample): sample is Sample => !!sample)
+ const minutes = (sample: Sample): number => (sample.at - start) / 60_000
+ return {
+ name,
+ samples: mine.length,
+ first: mine[0]?.rssMiB ?? 0,
+ last: mine[mine.length - 1]?.rssMiB ?? 0,
+ perHour: lows.length >= 3 ? fit(lows.map((sample) => [minutes(sample), sample.rssMiB])) * 60 : null,
+ windows: lows.length,
+ fdsFirst: fdLows[0]?.fds ?? null,
+ fdsLast: fdLows[fdLows.length - 1]?.fds ?? null,
+ fdsPerHour: fdLows.length >= 2 ? fit(fdLows.map((sample) => [minutes(sample), sample.fds!])) * 60 : null,
+ restarts: pids.length - 1,
+ }
+ })
+}
+
+/**
+ * Every turn the run sends, by agent, against what a window saw: each started once, in the order sent.
+ * The window's frames are read as they come and then let go, so a run of an hour does not hold them all.
+ */
+export class TurnLedger {
+ private readonly sent = new Map<string, string[]>()
+ private readonly seen = new Map<string, string[]>()
+ private read = 0
+
+ constructor(private readonly window: LocalClient, private readonly marker: RegExp) {}
+
+ send(agentId: string, token: string): void {
+ this.sent.set(agentId, [...(this.sent.get(agentId) ?? []), token])
+ }
+
+ /** What the window has heard since the last look; its frames are then dropped. */
+ drain(): void {
+ const frames: Frame[] = this.window.frames
+ for (; this.read < frames.length; this.read++) {
+ const frame = frames[this.read]
+ if (frame.type !== 'turn_started' || typeof frame.agentId !== 'string') continue
+ const token = this.marker.exec(String(frame.payload?.userMessage ?? ''))?.[0]
+ if (token) this.seen.set(frame.agentId, [...(this.seen.get(frame.agentId) ?? []), token])
+ }
+ if (frames.length > 2_000) { frames.length = 0; this.read = 0 }
+ }
+
+ /** Each agent's difference: a token sent and never seen, seen twice, or seen out of the order sent. */
+ differences(): string[] {
+ this.drain()
+ const out: string[] = []
+ for (const agentId of new Set([...this.sent.keys(), ...this.seen.keys()])) {
+ const sent = this.sent.get(agentId) ?? []
+ const seen = this.seen.get(agentId) ?? []
+ const unexpected = seen.filter(token => !sent.includes(token))
+ if (unexpected.length) out.push(`${agentId.slice(0, 8)}: never sent ${unexpected.slice(0, 5).join(' ')}`)
+ const twice = seen.filter((token, at) => seen.indexOf(token) !== at)
+ if (twice.length) out.push(`${agentId.slice(0, 8)}: seen twice ${twice.slice(0, 5).join(' ')}`)
+ const missing = sent.filter((token) => !seen.includes(token))
+ if (missing.length) out.push(`${agentId.slice(0, 8)}: never seen ${missing.slice(0, 5).join(' ')} (${missing.length} of ${sent.length})`)
+ const order = seen.filter((token) => sent.includes(token))
+ const expected = sent.filter((token) => order.includes(token))
+ if (order.join(' ') !== [...new Set(expected)].join(' ') && !twice.length) out.push(`${agentId.slice(0, 8)}: out of order`)
+ }
+ return out
+ }
+
+ totals(): { sent: number; seen: number } {
+ return { sent: [...this.sent.values()].reduce((sum, list) => sum + list.length, 0), seen: [...this.seen.values()].reduce((sum, list) => sum + list.length, 0) }
+ }
+}
+
+/** Frames and terminal frames a long-lived client no longer needs, let go. */
+export function forget(client: LocalClient): void {
+ if (client.frames.length > 2_000) client.frames.length = 0
+ if (client.binaries.length > 2_000) client.binaries.length = 0
+}
+
+/** tmux clients attached on the daemon's private server: one per open terminal stream. */
+export async function tmuxClients(d: IsolatedDaemon): Promise<number> {
+ const listed = await d.tmux.run('list-clients')
+ return listed.trim() ? listed.trim().split('\n').length : 0
+}
+
+/** Processes still running whose command line names this daemon's root: its engines, viewer servers, workers. */
+export async function strays(d: IsolatedDaemon): Promise<string[]> {
+ const { stdout } = await exec('ps', ['-A', '-o', 'pid=,command='], { timeout: 5000 })
+ return stdout.split('\n').filter((line) => line.includes(d.root) && !line.includes(' ps -A')).map((line) => line.trim())
+}
diff --git a/docs/validation-and-release.md b/docs/validation-and-release.md
index 25c28f704..bd486b7cb 100644
--- a/docs/validation-and-release.md
+++ b/docs/validation-and-release.md
@@ -113,6 +113,18 @@ GitHub report a PR unstable, which `make merge-pr` refuses to merge. Once it has
green, run it on pull requests, make `e2e-summary` required and add it to the evidence
collector.
+For changes to daemon lifetime or process boundaries, select an opt-in soak and chaos workload
+(`cli/e2e/endurance.e2e.ts`). On a Mac, run `SOAK=1 E2E_BUNDLE=1 SOAK_OUT=<folder> npx vitest run
+--config vitest.e2e.config.ts e2e/endurance.e2e.ts` in `cli/`. The defaults are 60 minutes of soak and
+30 minutes of chaos with six agents; `SOAK_MINUTES`, `CHAOS_MINUTES` and `SOAK_AGENTS` select a
+smaller validation workload. Runs of at least 15 chaos minutes require every fault kind and service.
+Windows, terminal streams, all service processes and a fake dial are exercised. Every turn must be
+seen once, in order; the core must never restart; failed services must answer promptly and recover.
+`SOAK_OUT` keeps process samples and JSON reports. Memory/descriptor growth is reported, with null
+for insufficient measurement windows. Set `SOAK_MAX_MIB_PER_HOUR` or `SOAK_MAX_FDS_PER_HOUR` only
+with a measured baseline for the same workload; there is no invented default growth limit. Short
+smoke runs test the harness and failure paths, not long-term memory stability.
+
For repository process tooling only, `scope=process` runs its Python regression
tests without installing or building unrelated components. It does not validate
application changes. Workflow edits also need `actionlint` and a run exercising