The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
Validate daemon lifetime with sustained soak and service chaos (#1009)
Dee7305140e0352
diff --git a/cli/e2e/endurance.e2e.ts b/cli/e2e/endurance.e2e.tsnew file mode 100644index 000000000..985a74b82--- /dev/null+++ b/cli/e2e/endurance.e2e.ts@@ -0,0 +1,456 @@+/**+ * The daemon over time, every service in its own process: what no single end-to-end test shows (opt-in:+ * `SOAK=1`; about 90 minutes at its defaults).+ *+ * - **Soak** (`SOAK_MINUTES`, 60): Claude Code and Codex agents (`SOAK_AGENTS`, 6) take turns without a+ * break, with interrupts, questions, tool calls and floods of terminal output among them; windows come+ * and go, open an agent's terminal and close it; search, the Devices tab, the edge host's readers, the+ * models and the Store are asked as the apps ask them; a fake dial is plugged in and out. Every harnessd+ * process (master, core, each service's) is sampled every `SOAK_SAMPLE_MS` (30 s): resident memory,+ * footprint and open files. Growth rates are measurements by default. Set `SOAK_MAX_MIB_PER_HOUR`+ * and `SOAK_MAX_FDS_PER_HOUR` only against an established baseline for the same workload.+ * - **Chaos** (`CHAOS_MINUTES`, 30), while the agents go on: a service process (search, viewers, the edge+ * host, the gateway, models, the devices, and the experiments once woken) is killed, two at once, frozen+ * until the master finds it hung, or killed again as it starts. Every time: the core never restarts; a+ * request to the service is answered at once (`SERVICE_UNAVAILABLE` or its answer), never left hanging;+ * the service comes back and serves again. At the end every turn sent was started once, in the order+ * sent, and no tmux client or process of the daemon outlives it.+ *+ * `SOAK_OUT=<folder>` keeps the samples (CSV) and the report (JSON). `SOAK_SEED` picks the run.+ */+import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'+import { tmpdir } from 'node:os'+import { join } from 'node:path'+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'+import { LocalClient, type Frame } from './harness/client.js'+import { IsolatedDaemon, until } from './harness/daemon.js'+import { Desk } from './harness/desk.js'+import { alive, everyPid, forget, harnessdProcesses, slopes, startSampler, strays, tmuxClients, TurnLedger, type Sample } from './harness/endurance.js'++const ON = process.env.SOAK === '1'+function knob(name: string, fallback: number, min: number, max: number): number {+ const n = Number(process.env[name] ?? fallback)+ if (!Number.isFinite(n) || n < min || n > max) throw new Error(`${name} must be between ${min} and ${max}`)+ return n+}+const SOAK_MINUTES = knob('SOAK_MINUTES', 60, 0.1, 240)+const CHAOS_MINUTES = knob('CHAOS_MINUTES', 30, 0.1, 240)+const AGENTS = Math.floor(knob('SOAK_AGENTS', 6, 2, 24))+const SAMPLE_MS = knob('SOAK_SAMPLE_MS', 30_000, 1000, 120_000)+// Growth limits require a baseline for the same workload. Otherwise report measurements only.+const MAX_MIB_PER_HOUR = process.env.SOAK_MAX_MIB_PER_HOUR === undefined ? null : knob('SOAK_MAX_MIB_PER_HOUR', 20, 0, 10000)+const MAX_FDS_PER_HOUR = process.env.SOAK_MAX_FDS_PER_HOUR === undefined ? null : knob('SOAK_MAX_FDS_PER_HOUR', 30, 0, 10000)+const SEED = knob('SOAK_SEED', 20261007, 0, 0xffffffff)+const OUT = process.env.SOAK_OUT ?? (ON ? mkdtempSync(join(tmpdir(), 'harness-soak-')) : '')+/** How long a request to a service that is down may take to be answered: at once, not after its wait. */+const FAST_MS = 5_000+const PROTOCOL = 3+const MARK = /soak-[a-z0-9]+-\d+/++/** mulberry32: small, seeded, and the same sequence on every machine. */+function random(seed: number): () => number {+ let a = seed >>> 0+ return () => {+ a = (a + 0x6d2b79f5) >>> 0+ let t = a+ t = Math.imul(t ^ (t >>> 15), t | 1)+ t ^= t + Math.imul(t ^ (t >>> 7), t | 61)+ return ((t ^ (t >>> 14)) >>> 0) / 4294967296+ }+}+const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms))+async function pause(ms: number, deadline: () => boolean): Promise<void> {+ const end = Date.now() + ms+ while (!deadline() && Date.now() < end) await sleep(Math.min(1000, end - Date.now()))+}+const isTurn = (type: string, agentId: string) => (frame: Frame) => frame.type === type && frame.agentId === agentId++/** A request each service's process answers, as the apps ask it: answered by the service when it is up. */+const PROBES: Record<string, { type: string; payload: Record<string, unknown> }> = {+ gateway: { type: 'e2ee_pairings_list', payload: {} },+ sharing: { type: 'harness_share_list', payload: {} },+ search: { type: 'session_search', payload: { query: 'soak' } },+ viewers: { type: 'dsh_list', payload: {} },+ edge: { type: 'machine_resources', payload: {} },+ models: { type: 'models_list', payload: {} },+ devices: { type: 'harness_devices_list', payload: {} },+ orchestrator: { type: 'orchestrator', payload: { action: 'list' } },+ teams: { type: 'team', payload: { action: 'capabilities' } },+ commandBar: { type: 'command_bar', payload: { request: { prompt: '' } } },+}+/** The service names a process's comings and goings are logged under (`[services] <name> connected`). */+const LINKS: Record<string, string> = { search: 'search', viewers: 'viewers', edge: 'monitor', gateway: 'gateway', models: 'models', devices: 'devices', orchestrator: 'orchestrator', teams: 'teams', commandBar: 'commandBar', sharing: 'sharing' }+const down = (answer: Record<string, unknown>) => answer.error === 'SERVICE_UNAVAILABLE' || answer.error === 'SERVICE_FAILED' || answer.error === 'GATEWAY_UNAVAILABLE'++const acceptable = (name: string, answer: Record<string, unknown>): boolean => answer.error === undefined+ || (name === 'commandBar' && answer.error === 'INVALID_REQUEST')+ || (name === 'sharing' && answer.error === 'HARNESS_NOT_FOUND')+async function wakeServices(client: LocalClient): Promise<void> {+ for (const [name, probe] of Object.entries(PROBES)) {+ await until(`${name} to answer its probe`, async () => {+ const answer = await client.request(probe.type, probe.payload, 30_000)+ if (down(answer)) return null+ if (!acceptable(name, answer)) throw new Error(`${name}: ${JSON.stringify(answer)}`)+ return true+ }, 60_000, 200)+ }+}++interface Findings { lines: string[] }++/** An agent's pane as it is now, kept beside the report (`SOAK_OUT`) once per agent and token. */+async function keepPane(d: IsolatedDaemon, agentId: string, name: string): Promise<void> {+ let client: LocalClient | undefined+ try {+ client = await LocalClient.connect(d)+ const row = ((await client.request('agents_list', {}, 15_000)).agents as Array<Record<string, any>>).find((agent) => agent.id === agentId)+ if (row?.tmuxPane) {+ writeFileSync(join(OUT, `pane-${name}.txt`), await d.capture(String(row.tmuxPane)))+ // Input and question readers include scrollback. Keeping only the visible pane hid the old+ // dialog they still read after its answer, while the screen already showed the next composer.+ const capture = await d.tmux.run('capture-pane', '-p', '-e', '-J', '-S', '-100', '-t', String(row.tmuxPane))+ writeFileSync(join(OUT, `pane-${name}-history.txt`), capture)+ }+ } catch { /* a pane that cannot be read: the finding stands without it */ }+ finally { client?.close() }+}++/** One agent's turns, one after another until `until`: plain, a tool call, a question answered, an+ * interrupt, a flood of output; each carries a token the ledger checks. */+async function agentLoop(d: IsolatedDaemon, agentId: string, tag: string, ledger: TurnLedger, deadline: () => boolean, rand: () => number, findings: Findings): Promise<number> {+ const client = await LocalClient.connect(d)+ let n = 0+ try {+ while (!deadline()) {+ const token = `soak-${tag}-${n++}`+ const pick = rand()+ const kind = pick < 0.5 ? 'plain' : pick < 0.65 ? 'tool' : pick < 0.78 ? 'ask' : pick < 0.93 ? 'interrupt' : 'flood'+ const content = kind === 'plain' ? `${token} hello` : kind === 'tool' ? `!tool echo ${token}` : kind === 'ask' ? `!ask ${token}`+ : kind === 'interrupt' ? `!holdtool sleep-${token}` : `!flood ${token}`+ const ended = kind === 'interrupt' ? null : client.next(isTurn('turn_ended', agentId), 120_000, `turn_ended (${token})`)+ const asked = kind === 'ask' ? client.next((frame) => frame.type === 'commander_question' && frame.agentId === agentId, 60_000, `question (${token})`) : null+ ledger.send(agentId, token)+ client.send('message', { agentId, content })+ try {+ if (asked) {+ const question = (await asked).payload ?? {}+ const shaped = (question.questions as Array<{ q: string }> | undefined)?.[0]+ client.send('question_response', { requestId: question.requestId, agentId, answers: shaped ? { [shaped.q]: 'Coffee' } : {} })+ }+ if (kind === 'interrupt') {+ await client.next((frame) => frame.type === 'turn_started' && frame.agentId === agentId, 60_000, `turn_started (${token})`)+ await sleep(800 + rand() * 1_500)+ // Told, not asked; and a turn interrupted ends with no turn_ended (core/turns/cancel.ts): every window+ // reads the agent as idle.+ const idle = client.next((frame) => frame.type === 'agent_activity' && frame.agentId === agentId && frame.payload?.activity?.state === 'idle', 60_000, `idle after its interrupt (${token})`)+ client.send('cancel', { agentId })+ await idle+ // The engine's own end of the interrupted turn, written as the pane takes the interrupt.+ await sleep(1_000)+ }+ await ended+ } catch (error) {+ findings.lines.push(`${new Date().toISOString()} ${tag} ${kind} ${token}: ${error instanceof Error ? error.message : String(error)}`)+ // What the agent's pane showed then: what the daemon read, to tell its mistake from the engine's.+ await keepPane(d, agentId, `${tag}-${token}`)+ await ended?.catch(() => {})+ }+ forget(client)+ await sleep(rand() * 1_500)+ }+ } finally { client.close() }+ return n+}++/** Windows coming and going: each says its tabs and focus, opens an agent's terminal, keeps it a while,+ * and closes it or simply goes; and the apps' requests to the services, now and then. */+async function deskLoop(d: IsolatedDaemon, agentIds: string[], deadline: () => boolean, rand: () => number, findings: Findings): Promise<number> {+ let windows = 0+ const ask = async (client: LocalClient, service: string): Promise<void> => {+ const probe = PROBES[service]+ const answer = await client.request(probe.type, probe.payload, 60_000).catch((error) => ({ error: String(error) }))+ if (!acceptable(service, answer) && !down(answer)) findings.lines.push(`${new Date().toISOString()} desk ${probe.type}: ${JSON.stringify(answer).slice(0, 200)}`)+ }+ while (!deadline()) {+ windows+++ const window = await LocalClient.connect(d)+ try {+ const shown = agentIds.filter(() => rand() < 0.6)+ window.send('app_swarms', { active: 't1', swarms: [{ id: 't1', name: 'Tab', agentIds: shown, panes: shown.length }], tiles: [] })+ window.send('app_panes', { agentIds: shown, foreground: rand() < 0.8 })+ const agentId = agentIds[Math.floor(rand() * agentIds.length)]+ window.send('app_focus', { agentId })+ const requestId = `open-${windows}`+ const opened = window.next((frame) => (frame.type === 'terminal_ready' || frame.type === 'terminal_error') && frame.payload?.requestId === requestId, 30_000, 'terminal_ready')+ window.send('terminal_open', { requestId, protocolVersion: PROTOCOL, agentId, cols: 100, rows: 30, takeover: false })+ const ready = await opened.catch((error) => ({ type: 'timeout', payload: { error: String(error) } }) as Frame)+ if (ready.type !== 'terminal_ready') findings.lines.push(`${new Date().toISOString()} desk terminal_open: ${JSON.stringify(ready.payload).slice(0, 200)}`)+ const streamId = ready.payload?.streamId as string | undefined+ const holdFor = 3_000 + rand() * 12_000+ const until = Date.now() + holdFor+ while (Date.now() < until && !deadline()) {+ if (streamId) {+ window.send('terminal_alive', { streamId })+ const drawn = window.binaries.filter((frame) => frame.streamId === streamId)+ if (drawn.length) window.send('terminal_ack', { streamId, lastSeq: drawn[drawn.length - 1].seq })+ }+ const service = Object.keys(PROBES)[Math.floor(rand() * Object.keys(PROBES).length)]+ if (rand() < 0.4 && !['orchestrator', 'teams', 'commandBar'].includes(service)) await ask(window, service)+ forget(window)+ await sleep(2_000)+ }+ if (streamId && rand() < 0.7) window.send('terminal_close', { streamId })+ } finally {+ window.close()+ }+ await sleep(rand() * 3_000)+ }+ return windows+}++/** A dial plugged in and taken out now and then. */+async function dialLoop(desk: Desk, deadline: () => boolean, rand: () => number): Promise<number> {+ let plugs = 0+ while (!deadline()) {+ await pause(60_000 + rand() * 120_000, deadline)+ if (deadline()) break+ plugs+++ await desk.plug('SOAK-DIAL', 'e2:e0:00:00:00:5a')+ await pause(20_000 + rand() * 40_000, deadline)+ await desk.unplug('SOAK-DIAL')+ }+ return plugs+}++async function createAgents(d: IsolatedDaemon, count: number): Promise<string[]> {+ const client = await LocalClient.connect(d)+ const ids: string[] = []+ try {+ for (let i = 0; i < count; i++) {+ const cwd = join(d.projectsDir, `soak-${i}`)+ mkdirSync(cwd, { recursive: true })+ const created = await client.request('agent_create', { engine: i % 2 ? 'codex' : 'claude', cwd, bypassPermission: true }, 90_000)+ expect(created.error, JSON.stringify(created)).toBeUndefined()+ ids.push(created.agent.id)+ }+ for (const id of ids) {+ await until(`${id.slice(0, 8)} to bind`, async () => ((await client.request('agents_list', {})).agents as Array<Record<string, unknown>>)+ .find((agent) => agent.id === id)?.sessionId || null, 90_000, 500)+ }+ } finally { client.close() }+ return ids+}++describe.skipIf(!ON)('the daemon over time, every service in its own process', () => {+ let daemon: IsolatedDaemon | undefined+ let desk: Desk | undefined+ const cleanup: Array<() => void | Promise<void>> = []+ afterEach(async () => {+ const ended = await Promise.allSettled(cleanup.splice(0).reverse().map(stop => Promise.resolve().then(stop)))+ await desk?.close()+ await daemon?.close()+ daemon = undefined+ desk = undefined+ for (const result of ended) if (result.status === 'rejected') throw result.reason+ })++ const fresh = async (env: Record<string, string> = {}): Promise<{ d: IsolatedDaemon; desk: Desk }> => {+ const d = await IsolatedDaemon.create({ env: { CABLE_DISABLE: 'false', ...env } })+ daemon = d+ const file = join(d.root, 'dials.json')+ writeFileSync(file, '[]')+ d.env.HARNESSD_TEST_DIAL_PORT = file+ desk = new Desk(file)+ onTestFailed(() => {+ writeFileSync(join(OUT, `daemon-${d.port}.log`), d.log())+ console.log(`---- daemon log (whole log in ${OUT})\n${d.log().split('\n').slice(-120).join('\n')}`)+ })+ await d.start()+ return { d, desk: desk! }+ }++ it(`soak: ${AGENTS} agents for ${SOAK_MINUTES} min; measure every process; preserve every turn`, async () => {+ mkdirSync(OUT, { recursive: true })+ const { d, desk } = await fresh()+ const agentIds = await createAgents(d, AGENTS)+ const observer = await LocalClient.connect(d)+ cleanup.push(() => observer.close())+ const ledger = new TurnLedger(observer, MARK)+ const reading = setInterval(() => ledger.drain(), 1_000)+ cleanup.push(() => clearInterval(reading))+ await wakeServices(observer)+ const samples: Sample[] = []+ const stopSampling = startSampler(d, SAMPLE_MS, join(OUT, 'soak-samples.csv'), samples)+ cleanup.push(stopSampling)+ const findings: Findings = { lines: [] }+ const ends = Date.now() + SOAK_MINUTES * 60_000+ let stopping = false+ cleanup.push(() => { stopping = true })+ const deadline = () => stopping || Date.now() >= ends+ const [turns, windows, plugs] = await Promise.all([+ Promise.all(agentIds.map((id, i) => agentLoop(d, id, `a${i}`, ledger, deadline, random(SEED + i + 1), findings))),+ deskLoop(d, agentIds, deadline, random(SEED + 100), findings),+ dialLoop(desk, deadline, random(SEED + 200)),+ ])+ await stopSampling()+ clearInterval(reading)+ await sleep(3_000)+ const differences = ledger.differences()+ observer.close()+ const warmup = Math.min(10 * 60_000, (SOAK_MINUTES * 60_000) / 4)+ const growth = slopes(samples, warmup, Math.max(SAMPLE_MS * 4, 2 * 60_000))+ const report = { minutes: SOAK_MINUTES, agents: AGENTS, turns: turns.reduce((sum, n) => sum + n, 0), windows, plugs, ledger: ledger.totals(), differences, findings: findings.lines, slopes: growth, coresStarted: d.coresStarted() }+ writeFileSync(join(OUT, 'soak-report.json'), JSON.stringify(report, null, 2))+ console.log(`soak report: ${join(OUT, 'soak-report.json')}\n${growth.map((s) => `${s.name}: ${s.first.toFixed(0)} → ${s.last.toFixed(0)} MiB, ${s.perHour?.toFixed(1) ?? 'unmeasured'} MiB/h; fds ${s.fdsFirst} → ${s.fdsLast}, ${s.fdsPerHour?.toFixed(1)}/h; restarts ${s.restarts}`).join('\n')}`)+ expect(d.coresStarted()).toBe(1)+ expect(differences).toEqual([])+ expect(findings.lines).toEqual([])+ expect(growth.map(p => p.name).sort()).toEqual(['master', 'core', ...Object.keys(PROBES)].sort())+ for (const process of growth) {+ expect(process.samples, `${process.name} sample count`).toBeGreaterThanOrEqual(2)+ expect(process.restarts, `${process.name} restarted`).toBe(0)+ if (MAX_MIB_PER_HOUR !== null) {+ expect(process.perHour, `${process.name}: insufficient windows for a growth limit`).not.toBeNull()+ expect(process.perHour).toBeLessThan(MAX_MIB_PER_HOUR)+ }+ if (MAX_FDS_PER_HOUR !== null) {+ expect(process.fdsPerHour, `${process.name}: insufficient descriptor measurements`).not.toBeNull()+ expect(process.fdsPerHour).toBeLessThan(MAX_FDS_PER_HOUR)+ }+ }+ }, (SOAK_MINUTES + 15) * 60_000)++ it(`chaos: services killed, frozen and killed as they start, for ${CHAOS_MINUTES} min, while ${AGENTS} agents work`, async () => {+ mkdirSync(OUT, { recursive: true })+ const { d, desk } = await fresh({+ // Frozen, a service is found hung in seconds; killed, it is back in under two; and it is never parked:+ // the run kills each one far more often than a crash loop would.+ HARNESSD_SERVICE_HEARTBEAT_TIMEOUT_MS: '4000',+ HARNESSD_SERVICE_INITIAL_BACKOFF_MS: '200',+ HARNESSD_SERVICE_MAX_BACKOFF_MS: '2000',+ HARNESSD_SERVICE_PARK_CRASHES: '100000',+ })+ const rand = random(SEED + 1_000)+ const agentIds = await createAgents(d, AGENTS)+ const observer = await LocalClient.connect(d)+ cleanup.push(() => observer.close())+ const ledger = new TurnLedger(observer, MARK)+ const reading = setInterval(() => ledger.drain(), 1_000)+ cleanup.push(() => clearInterval(reading))+ const findings: Findings = { lines: [] }+ // The experiments and the devices, woken as the apps wake them, so their processes are in the run too.+ const waker = await LocalClient.connect(d)+ await desk.plug('CHAOS-DIAL', 'e2:e0:00:00:00:5b')+ cleanup.push(() => waker.close())+ await wakeServices(waker)+ const samples: Sample[] = []+ const stopSampling = startSampler(d, SAMPLE_MS, join(OUT, 'chaos-samples.csv'), samples)+ cleanup.push(stopSampling)+ const ends = Date.now() + CHAOS_MINUTES * 60_000+ let stopping = false+ cleanup.push(() => { stopping = true })+ const deadline = () => stopping || Date.now() >= ends+ const events: Array<Record<string, unknown>> = []+ const starts = (name: string) => [...d.log().matchAll(new RegExp(`\\[harnessd\\] service ${name} started \\(pid (\\d+)\\)`, 'g'))].map((match) => Number(match[1]))+ const linked = (name: string) => d.log().split(`[services] ${LINKS[name]} connected`).length - 1++ /** Asked while it is down: answered at once. Frozen: answered by the time the master has found it hung+ * and killed it, never left waiting. Back: answered by it again. */+ const probe = async (name: string, phase: 'down' | 'frozen' | 'back'): Promise<void> => {+ const spec = PROBES[name]+ if (!spec) return+ const asked = Date.now()+ const answer = await waker.request(spec.type, spec.payload, 60_000).catch((error) => ({ error: `no answer: ${String(error)}` }))+ const ms = Date.now() - asked+ if (phase === 'down' && ms > FAST_MS) findings.lines.push(`${name} down: ${spec.type} answered after ${ms} ms: ${JSON.stringify(answer).slice(0, 160)}`)+ if (!acceptable(name, answer) && (phase === 'back' || !down(answer))) findings.lines.push(`${name} ${phase}: ${spec.type}: ${JSON.stringify(answer).slice(0, 160)}`)+ events[events.length - 1][`${phase}Ms`] = ms+ }+ /** The master started it again and it connected to the core. */+ const comesBack = async (name: string, startsBefore: number, linksBefore: number): Promise<boolean> => {+ try {+ await until(`${name} to be started again`, () => starts(name).length > startsBefore || null, 60_000, 100)+ await until(`${name} to connect again`, () => linked(name) > linksBefore || null, 60_000, 100)+ return true+ } catch (error) {+ findings.lines.push(`${name} never came back: ${error instanceof Error ? error.message : String(error)}`)+ return false+ }+ }++ const actions = ['kill', 'kill two', 'freeze', 'kill as it starts'] as const+ const chaos = (async () => {+ let cycle = 0+ while (!deadline()) {+ await pause(5000 + rand() * 5000, deadline)+ if (deadline()) break+ const running = [...harnessdProcesses(d)].filter(([name, pid]) => name !== 'master' && name !== 'core' && alive(pid))+ if (!running.length) continue+ const names = Object.keys(PROBES)+ const name = names[cycle % names.length]+ const action = actions[Math.floor(cycle / names.length) % actions.length]+ cycle+++ const primary = running.find(([role]) => role === name)+ if (!primary) { findings.lines.push(`${name} was not running for ${action}`); continue }+ const second = running.find(([role]) => role !== name)+ const targets = action === 'kill two' && second ? [primary, second] : [primary]+ const before = targets.map(([name]) => ({ name, starts: starts(name).length, links: linked(name) }))+ const hung = (): number => d.log().split(`[harnessd] service ${targets[0][0]} sent no heartbeat`).length+ const hungBefore = hung()+ events.push({ at: new Date().toISOString(), action, targets: targets.map(([name]) => name) })+ for (const [, pid] of targets) process.kill(pid, action === 'freeze' ? 'SIGSTOP' : 'SIGKILL')+ if (action === 'kill as it starts') {+ const [{ name, starts: count }] = before+ await until(`${name} to be started`, () => starts(name).length > count || null, 30_000, 20).catch(error => { findings.lines.push(String(error)) })+ const next = starts(name)[count]+ if (next && alive(next)) process.kill(next, 'SIGKILL')+ before[0].starts = count + 1+ }+ if (action === 'freeze') {+ const frozen = probe(targets[0][0], 'frozen')+ await until(`${targets[0][0]} to be found hung`, () => hung() > hungBefore || null, 30_000, 100).catch(error => { findings.lines.push(String(error)) })+ await frozen+ } else await probe(targets[0][0], 'down')+ for (const { name, starts: count, links } of before) {+ if (await comesBack(name, count, links)) await probe(name, 'back')+ }+ if (d.coresStarted() !== 1) findings.lines.push(`the core restarted after ${action} ${targets.map(([name]) => name).join(', ')}`)+ }+ })()+ const [turns] = await Promise.all([+ Promise.all(agentIds.map((id, i) => agentLoop(d, id, `c${i}`, ledger, deadline, random(SEED + 2_000 + i), findings))),+ deskLoop(d, agentIds, deadline, random(SEED + 3_000), findings),+ chaos,+ ])+ await stopSampling()+ clearInterval(reading)+ await sleep(3_000)+ const differences = ledger.differences()+ observer.close()+ waker.close()+ await desk.unplug('CHAOS-DIAL')+ // Every window has gone: no terminal stream, so no tmux client, is left behind.+ const clients = await until('every tmux client to be let go', async () => ((await tmuxClients(d)) === 0 ? true : null), 30_000, 500).catch(async () => `still ${await tmuxClients(d)}`)+ const pids = everyPid(d)+ const growth = slopes(samples, 0, Math.max(SAMPLE_MS * 4, 2 * 60_000))+ const report = { minutes: CHAOS_MINUTES, agents: AGENTS, turns: turns.reduce((sum, n) => sum + n, 0), events, ledger: ledger.totals(), differences, findings: findings.lines, slopes: growth, coresStarted: d.coresStarted(), tmuxClients: clients }+ writeFileSync(join(OUT, 'chaos-report.json'), JSON.stringify(report, null, 2))+ console.log(`chaos report: ${join(OUT, 'chaos-report.json')}: ${events.length} events, ${report.turns} turns`)+ expect(d.coresStarted()).toBe(1)+ expect(differences).toEqual([])+ expect(findings.lines).toEqual([])+ expect(events.length, 'no chaos event exercised').toBeGreaterThan(0)+ if (CHAOS_MINUTES >= 15) {+ expect(new Set(events.map(event => event.action))).toEqual(new Set(actions))+ expect(new Set(events.flatMap(event => event.targets as string[]))).toEqual(new Set(Object.keys(PROBES)))+ expect(new Set(events.flatMap(event => (event.targets as string[]).map(name => `${event.action}:${name}`))))+ .toEqual(new Set(actions.flatMap(action => Object.keys(PROBES).map(name => `${action}:${name}`))))+ }+ expect(clients).toBe(true)+ // Nothing of the daemon's outlives it: every process its master started, its engines and its servers.+ await daemon!.close()+ daemon = undefined+ await until('every process of the daemon to be gone', () => pids.every((pid) => !alive(pid)) || null, 30_000, 250)+ expect(await strays(d)).toEqual([])+ }, (CHAOS_MINUTES + 15) * 60_000)+})diff --git a/cli/e2e/enduranceHarness.e2e.ts b/cli/e2e/enduranceHarness.e2e.tsnew file mode 100644index 000000000..49fa86cdf--- /dev/null+++ b/cli/e2e/enduranceHarness.e2e.ts@@ -0,0 +1,25 @@+/** The long-run harness must not turn missing evidence into a passing result. */+import { describe, expect, it } from 'vitest'+import type { LocalClient } from './harness/client.js'+import { slopes, TurnLedger, type Sample } from './harness/endurance.js'++describe('endurance evidence', () => {+ it('requires three independent windows before reporting memory growth', () => {+ const sample = (at: number, rssMiB: number, pid = 1): Sample => ({ at, rssMiB, name: 'core', pid, footprintMiB: null, fds: 10 })+ expect(slopes([], 0, 1000)).toEqual([])+ expect(slopes([sample(0, 10), sample(500, 10)], 0, 1000)[0]).toMatchObject({ perHour: null, windows: 1 })+ expect(slopes([sample(0, 10), sample(1000, 11), sample(2000, 12)], 0, 1000)[0]).toMatchObject({ perHour: 3600, windows: 3, fdsPerHour: 0 })+ expect(slopes([sample(0, 10), sample(1000, 11), sample(2000, 1, 2)], 0, 1000)[0]).toMatchObject({ perHour: null, restarts: 1, samples: 1 })+ })+ it('finds lost, duplicate, out-of-order and unsolicited turns including unknown agents', () => {+ const frames: LocalClient['frames'] = []+ const ledger = new TurnLedger({ frames } as LocalClient, /soak-[a-z]+-\d+/)+ const seen = (agentId: string, token: string) => frames.push({ type: 'turn_started', agentId, payload: { userMessage: token } })+ ledger.send('a', 'soak-a-1'); ledger.send('a', 'soak-a-2'); ledger.send('a', 'soak-a-3')+ seen('a', 'soak-a-2'); seen('a', 'soak-a-1'); seen('unknown', 'soak-x-1')+ expect(ledger.differences()).toEqual(['a: never seen soak-a-3 (1 of 3)', 'a: out of order', 'unknown: never sent soak-x-1'])+ seen('a', 'soak-a-2')+ expect(ledger.differences()).toContain('a: seen twice soak-a-2')+ expect(ledger.totals()).toEqual({ sent: 3, seen: 4 })+ })+})diff --git a/cli/e2e/harness/endurance.ts b/cli/e2e/harness/endurance.tsnew file mode 100644index 000000000..55f19ef03--- /dev/null+++ b/cli/e2e/harness/endurance.ts@@ -0,0 +1,203 @@+/**+ * What the soak and chaos run (e2e/endurance.e2e.ts) reads of a daemon over time: every harnessd process+ * it runs (the master, the core and each service's process) with its memory and open files, sampled on a+ * period; the slope of each, over the run; and the ledger of every turn the run sent, against what a+ * window saw.+ */+import { execFile } from 'node:child_process'+import { appendFileSync, readdirSync, readFileSync } from 'node:fs'+import { promisify } from 'node:util'+import type { Frame, LocalClient } from './client.js'+import type { IsolatedDaemon } from './daemon.js'++const exec = promisify(execFile)++/** This daemon's harnessd processes now, by name: `master`, `core`, and each service process by its name. */+export function harnessdProcesses(d: IsolatedDaemon): Map<string, number> {+ const found = new Map<string, number>()+ if (d.pid) found.set('master', d.pid)+ const core = d.corePid()+ if (core) found.set('core', core)+ for (const match of d.log().matchAll(/\[harnessd\] service (\S+) started \(pid (\d+)\)/g)) found.set(match[1], Number(match[2]))+ return found+}++/** Every pid this daemon's master ever said it started, cores and services: none may outlive it. */+export function everyPid(d: IsolatedDaemon): number[] {+ return [...(d.pid ? [d.pid] : []), ...[...d.log().matchAll(/\[harnessd\] (?:core|service \S+) started \(pid (\d+)\)/g)].map((match) => Number(match[1]))]+}++export function alive(pid: number): boolean {+ try { process.kill(pid, 0); return true } catch { return false }+}++export interface Sample { at: number; name: string; pid: number; rssMiB: number; footprintMiB: number | null; fds: number | null }++/** Resident memory, footprint (macOS's own measure of what a process costs) and open files of one process. */+export async function measure(pid: number): Promise<Omit<Sample, 'at' | 'name' | 'pid'> | null> {+ if (process.platform === 'linux') {+ try {+ const status = readFileSync(`/proc/${pid}/status`, 'utf8')+ const rss = Number(/VmRSS:\s+(\d+)/.exec(status)?.[1] ?? 0) / 1024+ return { rssMiB: rss, footprintMiB: null, fds: readdirSync(`/proc/${pid}/fd`).length }+ } catch { return null }+ }+ const rss = await exec('ps', ['-o', 'rss=', '-p', String(pid)], { timeout: 3000 }).then(({ stdout }) => Number(stdout.trim()) / 1024, () => 0)+ if (!rss) return null+ const footprint = await exec('footprint', ['-p', String(pid)], { timeout: 3000 }).then(({ stdout }) => {+ const match = /Footprint:\s+([\d.]+)\s+(KB|MB|GB)/.exec(stdout)+ if (!match) return null+ const value = Number(match[1])+ return match[2] === 'KB' ? value / 1024 : match[2] === 'GB' ? value * 1024 : value+ }, () => null)+ const fds = await exec('lsof', ['-n', '-P', '-a', '-p', String(pid), '-F', 'f'], { timeout: 3000, maxBuffer: 16 * 1024 * 1024 })+ .then(({ stdout }) => stdout.split('\n').filter(line => /^f\d+$/.test(line)).length, () => null)+ return { rssMiB: rss, footprintMiB: footprint, fds }+}++/** Every harnessd process of a daemon, every `everyMs`, into `samples` and a CSV file. */+export function startSampler(d: IsolatedDaemon, everyMs: number, csv: string, samples: Sample[]): () => Promise<void> {+ appendFileSync(csv, 'at,name,pid,rssMiB,footprintMiB,fds\n')+ let stopped = false+ let failure: unknown+ let running: Promise<void> = Promise.resolve()+ let timer: ReturnType<typeof setTimeout> | undefined+ const round = async (): Promise<void> => {+ const at = Date.now()+ for (const [name, pid] of harnessdProcesses(d)) {+ if (stopped) break+ if (!alive(pid)) continue+ const measured = await measure(pid)+ if (!measured) continue+ const sample = { at, name, pid, ...measured }+ samples.push(sample)+ appendFileSync(csv, `${at},${name},${pid},${measured.rssMiB.toFixed(1)},${measured.footprintMiB?.toFixed(1) ?? ''},${measured.fds ?? ''}\n`)+ }+ }+ const tick = (): void => {+ running = round().catch(error => { failure = error; stopped = true }).finally(() => { if (!stopped) timer = setTimeout(tick, everyMs) })+ }+ tick()+ return async () => { stopped = true; clearTimeout(timer); await running; if (failure) throw failure }+}++export interface Slope { name: string; samples: number; first: number; last: number; perHour: number | null; windows: number; fdsFirst: number | null; fdsLast: number | null; fdsPerHour: number | null; restarts: number }++/** Least squares over (minutes, value). */+function fit(points: Array<[number, number]>): number {+ const n = points.length+ if (n < 2) return 0+ const mx = points.reduce((sum, [x]) => sum + x, 0) / n+ const my = points.reduce((sum, [, y]) => sum + y, 0) / n+ const sxx = points.reduce((sum, [x]) => sum + (x - mx) ** 2, 0)+ if (!sxx) return 0+ return points.reduce((sum, [x, y]) => sum + (x - mx) * (y - my), 0) / sxx+}++/**+ * Each process's growth per hour, past the warm-up: the line through the lowest sample of each window of+ * `windowMs`, so a collection that has not run yet is not read as growth (the lowest is what the process+ * holds), and only the last pid of each name (a process restarted starts over).+ */+export function slopes(samples: Sample[], warmupMs: number, windowMs: number): Slope[] {+ if (!samples.length) return []+ const start = samples[0].at+ const names = [...new Set(samples.map((sample) => sample.name))]+ return names.map((name) => {+ const all = samples.filter((sample) => sample.name === name)+ const pids = [...new Set(all.map((sample) => sample.pid))]+ const lastPid = pids[pids.length - 1]+ const mine = all.filter((sample) => sample.pid === lastPid && sample.at - start >= warmupMs)+ const windows = new Map<number, Sample[]>()+ for (const sample of mine) {+ const key = Math.floor((sample.at - start) / windowMs)+ windows.set(key, [...(windows.get(key) ?? []), sample])+ }+ const lows = [...windows.values()].map((group) => group.reduce((low, sample) => (sample.rssMiB < low.rssMiB ? sample : low)))+ const fdLows = [...windows.values()].map((group) => group.filter((sample) => sample.fds !== null).reduce<Sample | null>((low, sample) => (!low || sample.fds! < low.fds! ? sample : low), null))+ .filter((sample): sample is Sample => !!sample)+ const minutes = (sample: Sample): number => (sample.at - start) / 60_000+ return {+ name,+ samples: mine.length,+ first: mine[0]?.rssMiB ?? 0,+ last: mine[mine.length - 1]?.rssMiB ?? 0,+ perHour: lows.length >= 3 ? fit(lows.map((sample) => [minutes(sample), sample.rssMiB])) * 60 : null,+ windows: lows.length,+ fdsFirst: fdLows[0]?.fds ?? null,+ fdsLast: fdLows[fdLows.length - 1]?.fds ?? null,+ fdsPerHour: fdLows.length >= 2 ? fit(fdLows.map((sample) => [minutes(sample), sample.fds!])) * 60 : null,+ restarts: pids.length - 1,+ }+ })+}++/**+ * Every turn the run sends, by agent, against what a window saw: each started once, in the order sent.+ * The window's frames are read as they come and then let go, so a run of an hour does not hold them all.+ */+export class TurnLedger {+ private readonly sent = new Map<string, string[]>()+ private readonly seen = new Map<string, string[]>()+ private read = 0++ constructor(private readonly window: LocalClient, private readonly marker: RegExp) {}++ send(agentId: string, token: string): void {+ this.sent.set(agentId, [...(this.sent.get(agentId) ?? []), token])+ }++ /** What the window has heard since the last look; its frames are then dropped. */+ drain(): void {+ const frames: Frame[] = this.window.frames+ for (; this.read < frames.length; this.read++) {+ const frame = frames[this.read]+ if (frame.type !== 'turn_started' || typeof frame.agentId !== 'string') continue+ const token = this.marker.exec(String(frame.payload?.userMessage ?? ''))?.[0]+ if (token) this.seen.set(frame.agentId, [...(this.seen.get(frame.agentId) ?? []), token])+ }+ if (frames.length > 2_000) { frames.length = 0; this.read = 0 }+ }++ /** Each agent's difference: a token sent and never seen, seen twice, or seen out of the order sent. */+ differences(): string[] {+ this.drain()+ const out: string[] = []+ for (const agentId of new Set([...this.sent.keys(), ...this.seen.keys()])) {+ const sent = this.sent.get(agentId) ?? []+ const seen = this.seen.get(agentId) ?? []+ const unexpected = seen.filter(token => !sent.includes(token))+ if (unexpected.length) out.push(`${agentId.slice(0, 8)}: never sent ${unexpected.slice(0, 5).join(' ')}`)+ const twice = seen.filter((token, at) => seen.indexOf(token) !== at)+ if (twice.length) out.push(`${agentId.slice(0, 8)}: seen twice ${twice.slice(0, 5).join(' ')}`)+ const missing = sent.filter((token) => !seen.includes(token))+ if (missing.length) out.push(`${agentId.slice(0, 8)}: never seen ${missing.slice(0, 5).join(' ')} (${missing.length} of ${sent.length})`)+ const order = seen.filter((token) => sent.includes(token))+ const expected = sent.filter((token) => order.includes(token))+ if (order.join(' ') !== [...new Set(expected)].join(' ') && !twice.length) out.push(`${agentId.slice(0, 8)}: out of order`)+ }+ return out+ }++ totals(): { sent: number; seen: number } {+ return { sent: [...this.sent.values()].reduce((sum, list) => sum + list.length, 0), seen: [...this.seen.values()].reduce((sum, list) => sum + list.length, 0) }+ }+}++/** Frames and terminal frames a long-lived client no longer needs, let go. */+export function forget(client: LocalClient): void {+ if (client.frames.length > 2_000) client.frames.length = 0+ if (client.binaries.length > 2_000) client.binaries.length = 0+}++/** tmux clients attached on the daemon's private server: one per open terminal stream. */+export async function tmuxClients(d: IsolatedDaemon): Promise<number> {+ const listed = await d.tmux.run('list-clients')+ return listed.trim() ? listed.trim().split('\n').length : 0+}++/** Processes still running whose command line names this daemon's root: its engines, viewer servers, workers. */+export async function strays(d: IsolatedDaemon): Promise<string[]> {+ const { stdout } = await exec('ps', ['-A', '-o', 'pid=,command='], { timeout: 5000 })+ return stdout.split('\n').filter((line) => line.includes(d.root) && !line.includes(' ps -A')).map((line) => line.trim())+}diff --git a/docs/validation-and-release.md b/docs/validation-and-release.mdindex 25c28f704..bd486b7cb 100644--- a/docs/validation-and-release.md+++ b/docs/validation-and-release.md@@ -113,6 +113,18 @@ GitHub report a PR unstable, which `make merge-pr` refuses to merge. Once it hasgreen, run it on pull requests, make `e2e-summary` required and add it to the evidencecollector.+For changes to daemon lifetime or process boundaries, select an opt-in soak and chaos workload+(`cli/e2e/endurance.e2e.ts`). On a Mac, run `SOAK=1 E2E_BUNDLE=1 SOAK_OUT=<folder> npx vitest run+--config vitest.e2e.config.ts e2e/endurance.e2e.ts` in `cli/`. The defaults are 60 minutes of soak and+30 minutes of chaos with six agents; `SOAK_MINUTES`, `CHAOS_MINUTES` and `SOAK_AGENTS` select a+smaller validation workload. Runs of at least 15 chaos minutes require every fault kind and service.+Windows, terminal streams, all service processes and a fake dial are exercised. Every turn must be+seen once, in order; the core must never restart; failed services must answer promptly and recover.+`SOAK_OUT` keeps process samples and JSON reports. Memory/descriptor growth is reported, with null+for insufficient measurement windows. Set `SOAK_MAX_MIB_PER_HOUR` or `SOAK_MAX_FDS_PER_HOUR` only+with a measured baseline for the same workload; there is no invented default growth limit. Short+smoke runs test the harness and failure paths, not long-term memory stability.+For repository process tooling only, `scope=process` runs its Python regressiontests without installing or building unrelated components. It does not validateapplication changes. Workflow edits also need `actionlint` and a run exercising