Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

harnessd: round 20, an engine that freezes; the night's findings and handoff (#769)

Deeeaf81b40c99a
diff --git a/cli/e2e/frozen.e2e.ts b/cli/e2e/frozen.e2e.ts
new file mode 100644
index 000000000..b891158ad
--- /dev/null
+++ b/cli/e2e/frozen.e2e.ts
@@ -0,0 +1,141 @@
+/**
+ * An engine that freezes rather than exits, for Claude Code and Codex: the process is stopped
+ * (SIGSTOP, as a debugger, a suspended laptop's swap storm or a stuck network call can leave it), and
+ * its pane takes keystrokes it cannot read. A message sent to it must be taken once, not twice, when
+ * it wakes; a cancel must leave it able to work; a stop must not wait on it forever; and every other
+ * agent must go on as if nothing were wrong.
+ */
+import { execFileSync } from 'node:child_process'
+import { mkdirSync } from 'node:fs'
+import { join } from 'node:path'
+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'
+import { LocalClient, type Frame } from './harness/client.js'
+import { IsolatedDaemon, until } from './harness/daemon.js'
+
+type Engine = 'claude' | 'codex'
+type Row = Record<string, any>
+
+const rows = async (client: LocalClient): Promise<Row[]> =>
+ (await client.request<{ agents: Row[] }>('agents_list', { includeStopped: true }, 30_000)).agents
+const row = async (client: LocalClient, agentId: string) => (await rows(client)).find((agent) => agent.id === agentId)
+async function create(d: IsolatedDaemon, client: LocalClient, engine: Engine, folder: string): Promise<Row> {
+ const cwd = join(d.projectsDir, folder)
+ mkdirSync(cwd, { recursive: true })
+ const created = await client.request('agent_create', { engine, cwd, bypassPermission: true }, 90_000)
+ expect(created.error, `${folder}: ${JSON.stringify(created)}`).toBeUndefined()
+ return until(`${folder} to bind its conversation`, async () => {
+ const agent = await row(client, created.agent.id)
+ return agent?.sessionId && agent.status === 'active' ? agent : null
+ }, 60_000, 500)
+}
+const isTurn = (type: string, agentId: string) => (frame: Frame) => frame.type === type && frame.agentId === agentId
+async function turn(client: LocalClient, agentId: string, content: string): Promise<void> {
+ const started = client.next(isTurn('turn_started', agentId), 45_000, `turn_started (${content})`)
+ const ended = client.next(isTurn('turn_ended', agentId), 45_000, `turn_ended (${content})`)
+ client.send('message', { agentId, content })
+ expect((await started).payload?.userMessage).toBe(content)
+ await ended
+}
+
+/** The engine's own process in the agent's pane: the fake engine's wrapper, found under the pane. */
+async function enginePid(d: IsolatedDaemon, agent: Row): Promise<number> {
+ const pane = String(agent.terminal?.runtimes?.[0]?.paneId ?? agent.tmuxPane)
+ const root = Number((await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_pid}')).trim())
+ const table = execFileSync('ps', ['-A', '-o', 'pid=,ppid=,command='], { encoding: 'utf8' })
+ .split('\n').map((line) => /^\s*(\d+)\s+(\d+)\s+(.*)$/.exec(line)).filter(Boolean)
+ .map((m) => ({ pid: Number(m![1]), ppid: Number(m![2]), command: m![3] }))
+ const wrapper = join(d.root, 'bin', agent.engine)
+ const under = new Set([root])
+ for (let grew = true; grew;) {
+ grew = false
+ for (const p of table) if (under.has(p.ppid) && !under.has(p.pid)) { under.add(p.pid); grew = true }
+ }
+ const engine = table.find((p) => under.has(p.pid) && p.command.includes(wrapper))
+ if (!engine) throw new Error(`no ${agent.engine} process under pane ${pane}`)
+ return engine.pid
+}
+const alive = (pid: number): boolean => { try { process.kill(pid, 0); return true } catch { return false } }
+
+describe('an engine that freezes', () => {
+ let daemon: IsolatedDaemon | undefined
+ const frozen: number[] = []
+ afterEach(async () => {
+ // A stopped process left behind would outlive the test.
+ for (const pid of frozen.splice(0)) { try { process.kill(pid, 'SIGCONT') } catch { /* gone */ } }
+ await daemon?.close(); daemon = undefined
+ })
+ const fresh = async () => {
+ const d = await IsolatedDaemon.create()
+ daemon = d
+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-150).join('\n')}`) })
+ await d.start()
+ return d
+ }
+
+ it.each(['claude', 'codex'] as const)('%s: a message to a frozen engine is taken once when it wakes, and the others go on meanwhile', async (engine) => {
+ const d = await fresh()
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, engine, `frozen-${engine}`)
+ const other = await create(d, client, engine === 'claude' ? 'codex' : 'claude', `awake-${engine}`)
+ await turn(client, agent.id, 'before it froze')
+
+ const pid = await enginePid(d, agent)
+ process.kill(pid, 'SIGSTOP')
+ frozen.push(pid)
+ const from = client.frames.length
+ client.send('message', { agentId: agent.id, content: 'sent while frozen' })
+ // Long enough for every check and retry the daemon makes on a message it cannot see taken.
+ await new Promise((resolve) => setTimeout(resolve, 15_000))
+ // Meanwhile the other agent works as if nothing were wrong.
+ await turn(client, other.id, 'while the other one is frozen')
+
+ process.kill(pid, 'SIGCONT')
+ await client.waitFor((frame) => isTurn('turn_ended', agent.id)(frame) && client.frames.indexOf(frame) >= from, 45_000, 'the woken engine to finish the message')
+ await new Promise((resolve) => setTimeout(resolve, 5_000))
+ const starts = client.frames.slice(from).filter((frame) => isTurn('turn_started', agent.id)(frame))
+ expect(starts.map((frame) => frame.payload?.userMessage)).toEqual(['sent while frozen'])
+ await turn(client, agent.id, 'after it woke')
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ })
+
+ it('a cancel sent to a frozen engine mid-turn leaves it able to work when it wakes', async () => {
+ const d = await fresh()
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, 'claude', 'cancel-frozen')
+ client.send('message', { agentId: agent.id, content: '!slow 6000' })
+ await client.next(isTurn('turn_started', agent.id), 30_000, 'turn_started')
+ const pid = await enginePid(d, agent)
+ process.kill(pid, 'SIGSTOP')
+ frozen.push(pid)
+ client.send('cancel', { agentId: agent.id })
+ await new Promise((resolve) => setTimeout(resolve, 5_000))
+ process.kill(pid, 'SIGCONT')
+ // The turn ends (cancelled or finished), and the agent is idle and takes the next message.
+ await until('the agent to be idle again', async () => (await row(client, agent.id))?.activity?.state !== 'working' || null, 45_000, 500)
+ await turn(client, agent.id, 'after the cancel')
+ client.close()
+ })
+
+ it.each(['claude', 'codex'] as const)('%s: a stop does not wait on a frozen engine forever: it stops, and the process goes', async (engine) => {
+ const d = await fresh()
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, engine, `stop-frozen-${engine}`)
+ await turn(client, agent.id, 'before it froze')
+ const pid = await enginePid(d, agent)
+ process.kill(pid, 'SIGSTOP')
+ frozen.push(pid)
+ const startedAt = Date.now()
+ const stopped = await client.request('agent_delete', { agentId: agent.id }, 120_000)
+ const tookMs = Date.now() - startedAt
+ // Stopped, or refused with a reason the app can show: never left hanging.
+ if (stopped.error) expect(typeof stopped.error).toBe('string')
+ else {
+ await until('the agent to read as stopped', async () => (await row(client, agent.id))?.status === 'stopped' || null, 60_000, 500)
+ await until('the frozen process to be gone', () => !alive(pid) || null, 60_000, 500)
+ }
+ expect(tookMs).toBeLessThan(90_000)
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ })
+})
diff --git a/cli/src/watcher/watcher.spec.ts b/cli/src/watcher/watcher.spec.ts
index d8e217662..f8b7659d9 100644
--- a/cli/src/watcher/watcher.spec.ts
+++ b/cli/src/watcher/watcher.spec.ts
@@ -461,8 +461,11 @@ describe('Watcher.hold and release', () => {
await appendFile(transcriptPath, '{"n":2}\n')
await reading
const hold = (await held)!
- expect(hold.offset).toBe(8)
- expect(lines).toEqual(['{"n":1}'])
+ // The read in progress stops at the hold: everything before the hold's offset has been delivered,
+ // and nothing after it. Where that is depends on whether the read had already reached the second
+ // line (on a slow disk it has; CI saw it once), and either way nothing is lost or sent twice.
+ expect([8, 16]).toContain(hold.offset)
+ expect(Buffer.byteLength(lines.map((line) => `${line}\n`).join(''))).toBe(hold.offset)
hold.release()
await watcher.pollSession('s1')
expect(lines).toEqual(['{"n":1}', '{"n":2}'])
diff --git a/docs/design/2026-10-03-harnessd.md b/docs/design/2026-10-03-harnessd.md
index 8b0bd561c..b1d058fe2 100644
--- a/docs/design/2026-10-03-harnessd.md
+++ b/docs/design/2026-10-03-harnessd.md
@@ -737,6 +737,49 @@ Found while mapping, and fixed on their own rather than inside a move:
- purged, after which search no longer finds it and its transcripts are gone.
No defect was found.
+ - Scale (`e2e/scale.e2e.ts`, `SCALE_AGENTS`). Many agents, half of each engine, are created at once,
+ take a turn each, go through a restart and come back.
+ - 24 agents: the core went from 219 MiB empty to 230 MiB after the restart, and all were back in 4.6 s.
+ - 50 agents, the design's bar: the core went from 222 MiB to 268 MiB, and all were back in 8.3 s.
+ - Part 22, found by the seventeenth round (`e2e/machine.e2e.ts`): what the machine does to the
+ daemon. Two defects:
+ - A dead tmux server went unnoticed. tmux says "no server" two ways. One of them, `error
+ connecting to <socket> (No such file or directory)`, is what `kill-server`, a tmux crash and a
+ reboot all leave, and it was read as "tmux unavailable". So the daemon never took an agent off,
+ and the apps showed dead agents as active until a new server happened to start.
+ - A restore replayed recent turns live. After a daemon restart, a restored agent whose transcript
+ was under ten minutes old had its turns sent out again as live, with their notifications. The
+ first-turn rule excused a resume but not a restore. An attach that takes a relaunch mark now
+ never replays live.
+
+ After a reboot, the daemon brings every agent back on its own, with its conversation.
+ - Part 23, found by the eighteenth round (`e2e/diskfull.e2e.ts`, `DISKFULL=1`): the data folder on
+ a 64 MB disk image of its own, filled while agents work. Two defects:
+ - A rename the daemon answered while the disk was full was lost to a restart that came before the
+ next rename. Names were written only on a rename; a failed names write is now tried again on
+ every save.
+ - Saving the resume record threw out of a binding after the agent was bound, so the windows never
+ heard of it. The record is best effort there; a stop still saves first and refuses on failure.
+
+ The core never restarted, and agents kept taking turns throughout.
+ - The nineteenth round (`e2e/folders.e2e.ts`), for both engines:
+ - folders named with spaces, quotes, `$HOME` and backticks, accents and emoji: each holds an
+ agent that works, comes back after a restart and is closed;
+ - a folder deleted under a running agent, or while the daemon is down: it costs that agent at
+ most.
+
+ No defect was found.
+ - The twentieth round (`e2e/frozen.e2e.ts`): an engine stopped (SIGSTOP) rather than exited.
+ - A message sent to it is taken once when it wakes, and the other agents go on meanwhile.
+ - A cancel leaves it able to work.
+ - A stop does not wait on it forever.
+
+ No defect was found.
+ - Every request the apps send is answered: all 39 of the desktop's, all 21 of the phone's and all 20
+ of hn's, checked against the daemon's sources.
+ - The desktop's `lessons` is answered `UNSUPPORTED` on purpose, for a feature not yet built.
+ - hn's `terminal_remain` is a feature of hn's own local shells.
+ - hn's `terminal_info` had never been answered, and is now (part of the comparison above).
## Testing
diff --git a/docs/design/2026-10-04-harnessd-before-after.md b/docs/design/2026-10-04-harnessd-before-after.md
index bf1bcc999..2635deaf2 100644
--- a/docs/design/2026-10-04-harnessd-before-after.md
+++ b/docs/design/2026-10-04-harnessd-before-after.md
@@ -126,7 +126,7 @@ that grows past its budget. Each failure message says where the code belongs.
## What the end-to-end suite has found
-15 real bugs, each fixed with a test that fails without the fix. They include:
+20 real bugs, each fixed with a test that fails without the fix. They include:
- a corrupt registry that blocked every agent create;
- a pane killed from outside that stayed "active" forever;
@@ -136,7 +136,14 @@ that grows past its budget. Each failure message says where the code belongs.
- a turn taken just after a resume or restart that no window ever saw;
- a Stop pressed during start that was refused, and the ghost agent it could leave;
- a compaction that showed every window its turn starting twice;
-- a Codex fork that never bound its conversation after a daemon restart.
+- a Codex fork that never bound its conversation after a daemon restart;
+- hn's `terminal_info`, which no build had ever answered (found by comparing with v0.3.57);
+- a tmux server that died, after which dead agents showed as active;
+- turns replayed live after a daemon restart;
+- on a full disk: a rename lost to a restart, and a binding the windows never heard of.
+
+Comparing every answer with the released v0.3.57 found no other difference across 90 answers. A
+desk of 50 agents runs in 268 MiB, and all of them are back 8.3 s after a restart.
The full list is in the design doc's "Found while mapping".
diff --git a/docs/research/2026-10-05-harnessd-overnight.md b/docs/research/2026-10-05-harnessd-overnight.md
new file mode 100644
index 000000000..5b09a21e2
--- /dev/null
+++ b/docs/research/2026-10-05-harnessd-overnight.md
@@ -0,0 +1,81 @@
+# harnessd overnight, 2026-10-04 to 10-05
+
+Everything below is merged to main. Nothing is released.
+
+## Merged
+
+| PR | What |
+|---|---|
+| #757 | Services can run in their own processes (`HARNESSD_SERVICES=search`, opt-in), supervised like the core: a killed, hung, leaking or crash-looping service costs only itself. Rules for coding agents (`AGENTS.md` per layer, `CLAUDE.md` importing them) and `architecture.spec.ts`, which fails the build when code crosses a wall. |
+| #758 | Services answer their own requests. A feature adds a request without touching `backendSocket.ts` or `cli.ts`. The Store is a service. |
+| #760 | A 64 MB floor under every engine's whole transcript reads. A test that compares every answer with the released build (nothing differs). Round 16, how agents end. The devices plan. |
+| #762 | hn's `terminal_info` is answered; no build had ever answered it. A full desk end to end: 50 agents. |
+| #767 | Dead agents stop showing as active when tmux's server dies. No turns replayed live after a restart. Round 17. |
+| #768 | A full disk loses nothing the daemon answered. Rounds 18 (full disk) and 19 (awkward and vanishing folders). |
+| next | Round 20 (an engine that freezes), and the docs. |
+
+## Found and fixed (5 tonight, 20 in all)
+
+1. hn's `terminal_info` was never answered. The terminal streams pass it over, and the socket
+ dropped it.
+2. A tmux server that died went unnoticed, so dead agents showed as active. tmux's "no server"
+ message for a missing socket file was read as "tmux unavailable".
+3. A restore after a daemon restart replayed turns under ten minutes old as live, with their
+ notifications.
+4. A rename answered while the disk was full was lost to a restart that came before the next rename.
+5. On a full disk, a binding the windows never heard of. Saving the resume record threw out of it.
+
+Each fix has a test that fails without it.
+
+## Verified on main
+
+- **Compared with v0.3.57** (`e2e/compat.e2e.ts`): the same scenario on the released build and this
+ one gives the same answers. That covers 90 answers, every frame a turn, a question and a permission
+ push, and every malformed request's refusal. The only changes are on purpose: hn's `terminal_info`
+ is answered now, and `session_search` adds a field.
+- **Upgrading from v0.3.57** (`e2e/migration.e2e.ts`): agents at work come through the released
+ build's own update, then through the master's first start.
+- **The full end-to-end suite** passes: 190 tests in 27 files, on a fake Claude Code and Codex, a
+ private tmux server and a throwaway home.
+- **The unit suite** passes: 8,839 tests. The core and harnessd coverage gates are at 100%.
+- **Scale:** 24 agents run in 230 MiB and 50 in 268 MiB, all back after a restart in 4.6 s and 8.3 s.
+- **Soak:** 1,200 turns over 4 agents; memory went 297 → 265 MiB, open files 33 → 33.
+- **Chaos:** seeds 12 to 16, 150 random operations each, all pass.
+- **The apps:** all 39 requests the desktop sends, 21 from the phone and 20 from hn are answered.
+
+## Before a release
+
+1. Dogfood. On a computer you use: `git checkout main && (cd cli && npm ci) && make install-cli`,
+ then the desktop app as usual. `harness version` should print `0.3.57-dev.<sha>`.
+2. Build the last release's bundle (`node build-bundle.mjs` in a checkout of its tag) and run:
+ - `COMPAT_FROM=<it>/dist/cli.js npm run test:e2e -- compat`
+ - `MIGRATION_FROM=<it>/dist/cli.js npm run test:e2e -- migration`
+3. `make release-cli`.
+
+## The tests that only run when asked
+
+| Test | How | Why it is not in CI |
+|---|---|---|
+| Compared with a release | `COMPAT_FROM=<bundle>` | needs an old release built |
+| Upgrade from a release | `MIGRATION_FROM=<bundle>` | the same |
+| Full disk | `DISKFULL=1` (macOS) | mounts a 64 MB disk image |
+| Scale | `SCALE_AGENTS=50` | a few GB of memory |
+| Soak | `SOAK_ROUNDS=300` | minutes |
+| Chaos | `CHAOS_SEED=<n> CHAOS_OPS=150` | minutes; the default seed runs in CI |
+
+## Known limits and next steps
+
+- **Devices**: the plan is in the design doc (D0 the fleet on its own, D1 the dial behind the
+ boundary, D2 its own process, D3 the Wi-Fi device with the relay). It was not started tonight on
+ purpose: a deep change in code the end-to-end suite cannot reach (multi-machine) is the wrong risk
+ right before a release.
+- **Models' requests** (`grid_*`, `models_list`) are still in the socket's switch. That is a
+ teammate's active area (#755); they move to `services/models.ts` when the team is ready.
+- **Search in its own process** is opt-in until it has been dogfooded.
+- **Cursor's watcher** still re-reads Cursor's transcript whole on each change. It needs pages of
+ its own.
+- **Engine flags:** the cache of what an engine's `--help` supports keeps a flag until the daemon
+ restarts, even if an engine update drops it. Rare, and loud when it happens (the engine refuses
+ the flag).
+- **hn's `terminal_remain`** (drawing a dead pane's message) is not done by the daemon, only by hn's
+ own local shells.