The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
harnessd: round 20, an engine that freezes; the night's findings and handoff (#769)
Deeeaf81b40c99a
diff --git a/cli/e2e/frozen.e2e.ts b/cli/e2e/frozen.e2e.tsnew file mode 100644index 000000000..b891158ad--- /dev/null+++ b/cli/e2e/frozen.e2e.ts@@ -0,0 +1,141 @@+/**+ * An engine that freezes rather than exits, for Claude Code and Codex: the process is stopped+ * (SIGSTOP, as a debugger, a suspended laptop's swap storm or a stuck network call can leave it), and+ * its pane takes keystrokes it cannot read. A message sent to it must be taken once, not twice, when+ * it wakes; a cancel must leave it able to work; a stop must not wait on it forever; and every other+ * agent must go on as if nothing were wrong.+ */+import { execFileSync } from 'node:child_process'+import { mkdirSync } from 'node:fs'+import { join } from 'node:path'+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'+import { LocalClient, type Frame } from './harness/client.js'+import { IsolatedDaemon, until } from './harness/daemon.js'++type Engine = 'claude' | 'codex'+type Row = Record<string, any>++const rows = async (client: LocalClient): Promise<Row[]> =>+ (await client.request<{ agents: Row[] }>('agents_list', { includeStopped: true }, 30_000)).agents+const row = async (client: LocalClient, agentId: string) => (await rows(client)).find((agent) => agent.id === agentId)+async function create(d: IsolatedDaemon, client: LocalClient, engine: Engine, folder: string): Promise<Row> {+ const cwd = join(d.projectsDir, folder)+ mkdirSync(cwd, { recursive: true })+ const created = await client.request('agent_create', { engine, cwd, bypassPermission: true }, 90_000)+ expect(created.error, `${folder}: ${JSON.stringify(created)}`).toBeUndefined()+ return until(`${folder} to bind its conversation`, async () => {+ const agent = await row(client, created.agent.id)+ return agent?.sessionId && agent.status === 'active' ? agent : null+ }, 60_000, 500)+}+const isTurn = (type: string, agentId: string) => (frame: Frame) => frame.type === type && frame.agentId === agentId+async function turn(client: LocalClient, agentId: string, content: string): Promise<void> {+ const started = client.next(isTurn('turn_started', agentId), 45_000, `turn_started (${content})`)+ const ended = client.next(isTurn('turn_ended', agentId), 45_000, `turn_ended (${content})`)+ client.send('message', { agentId, content })+ expect((await started).payload?.userMessage).toBe(content)+ await ended+}++/** The engine's own process in the agent's pane: the fake engine's wrapper, found under the pane. */+async function enginePid(d: IsolatedDaemon, agent: Row): Promise<number> {+ const pane = String(agent.terminal?.runtimes?.[0]?.paneId ?? agent.tmuxPane)+ const root = Number((await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_pid}')).trim())+ const table = execFileSync('ps', ['-A', '-o', 'pid=,ppid=,command='], { encoding: 'utf8' })+ .split('\n').map((line) => /^\s*(\d+)\s+(\d+)\s+(.*)$/.exec(line)).filter(Boolean)+ .map((m) => ({ pid: Number(m![1]), ppid: Number(m![2]), command: m![3] }))+ const wrapper = join(d.root, 'bin', agent.engine)+ const under = new Set([root])+ for (let grew = true; grew;) {+ grew = false+ for (const p of table) if (under.has(p.ppid) && !under.has(p.pid)) { under.add(p.pid); grew = true }+ }+ const engine = table.find((p) => under.has(p.pid) && p.command.includes(wrapper))+ if (!engine) throw new Error(`no ${agent.engine} process under pane ${pane}`)+ return engine.pid+}+const alive = (pid: number): boolean => { try { process.kill(pid, 0); return true } catch { return false } }++describe('an engine that freezes', () => {+ let daemon: IsolatedDaemon | undefined+ const frozen: number[] = []+ afterEach(async () => {+ // A stopped process left behind would outlive the test.+ for (const pid of frozen.splice(0)) { try { process.kill(pid, 'SIGCONT') } catch { /* gone */ } }+ await daemon?.close(); daemon = undefined+ })+ const fresh = async () => {+ const d = await IsolatedDaemon.create()+ daemon = d+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-150).join('\n')}`) })+ await d.start()+ return d+ }++ it.each(['claude', 'codex'] as const)('%s: a message to a frozen engine is taken once when it wakes, and the others go on meanwhile', async (engine) => {+ const d = await fresh()+ const client = await LocalClient.connect(d)+ const agent = await create(d, client, engine, `frozen-${engine}`)+ const other = await create(d, client, engine === 'claude' ? 'codex' : 'claude', `awake-${engine}`)+ await turn(client, agent.id, 'before it froze')++ const pid = await enginePid(d, agent)+ process.kill(pid, 'SIGSTOP')+ frozen.push(pid)+ const from = client.frames.length+ client.send('message', { agentId: agent.id, content: 'sent while frozen' })+ // Long enough for every check and retry the daemon makes on a message it cannot see taken.+ await new Promise((resolve) => setTimeout(resolve, 15_000))+ // Meanwhile the other agent works as if nothing were wrong.+ await turn(client, other.id, 'while the other one is frozen')++ process.kill(pid, 'SIGCONT')+ await client.waitFor((frame) => isTurn('turn_ended', agent.id)(frame) && client.frames.indexOf(frame) >= from, 45_000, 'the woken engine to finish the message')+ await new Promise((resolve) => setTimeout(resolve, 5_000))+ const starts = client.frames.slice(from).filter((frame) => isTurn('turn_started', agent.id)(frame))+ expect(starts.map((frame) => frame.payload?.userMessage)).toEqual(['sent while frozen'])+ await turn(client, agent.id, 'after it woke')+ expect(d.coresStarted()).toBe(1)+ client.close()+ })++ it('a cancel sent to a frozen engine mid-turn leaves it able to work when it wakes', async () => {+ const d = await fresh()+ const client = await LocalClient.connect(d)+ const agent = await create(d, client, 'claude', 'cancel-frozen')+ client.send('message', { agentId: agent.id, content: '!slow 6000' })+ await client.next(isTurn('turn_started', agent.id), 30_000, 'turn_started')+ const pid = await enginePid(d, agent)+ process.kill(pid, 'SIGSTOP')+ frozen.push(pid)+ client.send('cancel', { agentId: agent.id })+ await new Promise((resolve) => setTimeout(resolve, 5_000))+ process.kill(pid, 'SIGCONT')+ // The turn ends (cancelled or finished), and the agent is idle and takes the next message.+ await until('the agent to be idle again', async () => (await row(client, agent.id))?.activity?.state !== 'working' || null, 45_000, 500)+ await turn(client, agent.id, 'after the cancel')+ client.close()+ })++ it.each(['claude', 'codex'] as const)('%s: a stop does not wait on a frozen engine forever: it stops, and the process goes', async (engine) => {+ const d = await fresh()+ const client = await LocalClient.connect(d)+ const agent = await create(d, client, engine, `stop-frozen-${engine}`)+ await turn(client, agent.id, 'before it froze')+ const pid = await enginePid(d, agent)+ process.kill(pid, 'SIGSTOP')+ frozen.push(pid)+ const startedAt = Date.now()+ const stopped = await client.request('agent_delete', { agentId: agent.id }, 120_000)+ const tookMs = Date.now() - startedAt+ // Stopped, or refused with a reason the app can show: never left hanging.+ if (stopped.error) expect(typeof stopped.error).toBe('string')+ else {+ await until('the agent to read as stopped', async () => (await row(client, agent.id))?.status === 'stopped' || null, 60_000, 500)+ await until('the frozen process to be gone', () => !alive(pid) || null, 60_000, 500)+ }+ expect(tookMs).toBeLessThan(90_000)+ expect(d.coresStarted()).toBe(1)+ client.close()+ })+})diff --git a/cli/src/watcher/watcher.spec.ts b/cli/src/watcher/watcher.spec.tsindex d8e217662..f8b7659d9 100644--- a/cli/src/watcher/watcher.spec.ts+++ b/cli/src/watcher/watcher.spec.ts@@ -461,8 +461,11 @@ describe('Watcher.hold and release', () => {await appendFile(transcriptPath, '{"n":2}\n')await readingconst hold = (await held)!- expect(hold.offset).toBe(8)- expect(lines).toEqual(['{"n":1}'])+ // The read in progress stops at the hold: everything before the hold's offset has been delivered,+ // and nothing after it. Where that is depends on whether the read had already reached the second+ // line (on a slow disk it has; CI saw it once), and either way nothing is lost or sent twice.+ expect([8, 16]).toContain(hold.offset)+ expect(Buffer.byteLength(lines.map((line) => `${line}\n`).join(''))).toBe(hold.offset)hold.release()await watcher.pollSession('s1')expect(lines).toEqual(['{"n":1}', '{"n":2}'])diff --git a/docs/design/2026-10-03-harnessd.md b/docs/design/2026-10-03-harnessd.mdindex 8b0bd561c..b1d058fe2 100644--- a/docs/design/2026-10-03-harnessd.md+++ b/docs/design/2026-10-03-harnessd.md@@ -737,6 +737,49 @@ Found while mapping, and fixed on their own rather than inside a move:- purged, after which search no longer finds it and its transcripts are gone.No defect was found.+ - Scale (`e2e/scale.e2e.ts`, `SCALE_AGENTS`). Many agents, half of each engine, are created at once,+ take a turn each, go through a restart and come back.+ - 24 agents: the core went from 219 MiB empty to 230 MiB after the restart, and all were back in 4.6 s.+ - 50 agents, the design's bar: the core went from 222 MiB to 268 MiB, and all were back in 8.3 s.+ - Part 22, found by the seventeenth round (`e2e/machine.e2e.ts`): what the machine does to the+ daemon. Two defects:+ - A dead tmux server went unnoticed. tmux says "no server" two ways. One of them, `error+ connecting to <socket> (No such file or directory)`, is what `kill-server`, a tmux crash and a+ reboot all leave, and it was read as "tmux unavailable". So the daemon never took an agent off,+ and the apps showed dead agents as active until a new server happened to start.+ - A restore replayed recent turns live. After a daemon restart, a restored agent whose transcript+ was under ten minutes old had its turns sent out again as live, with their notifications. The+ first-turn rule excused a resume but not a restore. An attach that takes a relaunch mark now+ never replays live.++ After a reboot, the daemon brings every agent back on its own, with its conversation.+ - Part 23, found by the eighteenth round (`e2e/diskfull.e2e.ts`, `DISKFULL=1`): the data folder on+ a 64 MB disk image of its own, filled while agents work. Two defects:+ - A rename the daemon answered while the disk was full was lost to a restart that came before the+ next rename. Names were written only on a rename; a failed names write is now tried again on+ every save.+ - Saving the resume record threw out of a binding after the agent was bound, so the windows never+ heard of it. The record is best effort there; a stop still saves first and refuses on failure.++ The core never restarted, and agents kept taking turns throughout.+ - The nineteenth round (`e2e/folders.e2e.ts`), for both engines:+ - folders named with spaces, quotes, `$HOME` and backticks, accents and emoji: each holds an+ agent that works, comes back after a restart and is closed;+ - a folder deleted under a running agent, or while the daemon is down: it costs that agent at+ most.++ No defect was found.+ - The twentieth round (`e2e/frozen.e2e.ts`): an engine stopped (SIGSTOP) rather than exited.+ - A message sent to it is taken once when it wakes, and the other agents go on meanwhile.+ - A cancel leaves it able to work.+ - A stop does not wait on it forever.++ No defect was found.+ - Every request the apps send is answered: all 39 of the desktop's, all 21 of the phone's and all 20+ of hn's, checked against the daemon's sources.+ - The desktop's `lessons` is answered `UNSUPPORTED` on purpose, for a feature not yet built.+ - hn's `terminal_remain` is a feature of hn's own local shells.+ - hn's `terminal_info` had never been answered, and is now (part of the comparison above).## Testingdiff --git a/docs/design/2026-10-04-harnessd-before-after.md b/docs/design/2026-10-04-harnessd-before-after.mdindex bf1bcc999..2635deaf2 100644--- a/docs/design/2026-10-04-harnessd-before-after.md+++ b/docs/design/2026-10-04-harnessd-before-after.md@@ -126,7 +126,7 @@ that grows past its budget. Each failure message says where the code belongs.## What the end-to-end suite has found-15 real bugs, each fixed with a test that fails without the fix. They include:+20 real bugs, each fixed with a test that fails without the fix. They include:- a corrupt registry that blocked every agent create;- a pane killed from outside that stayed "active" forever;@@ -136,7 +136,14 @@ that grows past its budget. Each failure message says where the code belongs.- a turn taken just after a resume or restart that no window ever saw;- a Stop pressed during start that was refused, and the ghost agent it could leave;- a compaction that showed every window its turn starting twice;-- a Codex fork that never bound its conversation after a daemon restart.+- a Codex fork that never bound its conversation after a daemon restart;+- hn's `terminal_info`, which no build had ever answered (found by comparing with v0.3.57);+- a tmux server that died, after which dead agents showed as active;+- turns replayed live after a daemon restart;+- on a full disk: a rename lost to a restart, and a binding the windows never heard of.++Comparing every answer with the released v0.3.57 found no other difference across 90 answers. A+desk of 50 agents runs in 268 MiB, and all of them are back 8.3 s after a restart.The full list is in the design doc's "Found while mapping".diff --git a/docs/research/2026-10-05-harnessd-overnight.md b/docs/research/2026-10-05-harnessd-overnight.mdnew file mode 100644index 000000000..5b09a21e2--- /dev/null+++ b/docs/research/2026-10-05-harnessd-overnight.md@@ -0,0 +1,81 @@+# harnessd overnight, 2026-10-04 to 10-05++Everything below is merged to main. Nothing is released.++## Merged++| PR | What |+|---|---|+| #757 | Services can run in their own processes (`HARNESSD_SERVICES=search`, opt-in), supervised like the core: a killed, hung, leaking or crash-looping service costs only itself. Rules for coding agents (`AGENTS.md` per layer, `CLAUDE.md` importing them) and `architecture.spec.ts`, which fails the build when code crosses a wall. |+| #758 | Services answer their own requests. A feature adds a request without touching `backendSocket.ts` or `cli.ts`. The Store is a service. |+| #760 | A 64 MB floor under every engine's whole transcript reads. A test that compares every answer with the released build (nothing differs). Round 16, how agents end. The devices plan. |+| #762 | hn's `terminal_info` is answered; no build had ever answered it. A full desk end to end: 50 agents. |+| #767 | Dead agents stop showing as active when tmux's server dies. No turns replayed live after a restart. Round 17. |+| #768 | A full disk loses nothing the daemon answered. Rounds 18 (full disk) and 19 (awkward and vanishing folders). |+| next | Round 20 (an engine that freezes), and the docs. |++## Found and fixed (5 tonight, 20 in all)++1. hn's `terminal_info` was never answered. The terminal streams pass it over, and the socket+ dropped it.+2. A tmux server that died went unnoticed, so dead agents showed as active. tmux's "no server"+ message for a missing socket file was read as "tmux unavailable".+3. A restore after a daemon restart replayed turns under ten minutes old as live, with their+ notifications.+4. A rename answered while the disk was full was lost to a restart that came before the next rename.+5. On a full disk, a binding the windows never heard of. Saving the resume record threw out of it.++Each fix has a test that fails without it.++## Verified on main++- **Compared with v0.3.57** (`e2e/compat.e2e.ts`): the same scenario on the released build and this+ one gives the same answers. That covers 90 answers, every frame a turn, a question and a permission+ push, and every malformed request's refusal. The only changes are on purpose: hn's `terminal_info`+ is answered now, and `session_search` adds a field.+- **Upgrading from v0.3.57** (`e2e/migration.e2e.ts`): agents at work come through the released+ build's own update, then through the master's first start.+- **The full end-to-end suite** passes: 190 tests in 27 files, on a fake Claude Code and Codex, a+ private tmux server and a throwaway home.+- **The unit suite** passes: 8,839 tests. The core and harnessd coverage gates are at 100%.+- **Scale:** 24 agents run in 230 MiB and 50 in 268 MiB, all back after a restart in 4.6 s and 8.3 s.+- **Soak:** 1,200 turns over 4 agents; memory went 297 → 265 MiB, open files 33 → 33.+- **Chaos:** seeds 12 to 16, 150 random operations each, all pass.+- **The apps:** all 39 requests the desktop sends, 21 from the phone and 20 from hn are answered.++## Before a release++1. Dogfood. On a computer you use: `git checkout main && (cd cli && npm ci) && make install-cli`,+ then the desktop app as usual. `harness version` should print `0.3.57-dev.<sha>`.+2. Build the last release's bundle (`node build-bundle.mjs` in a checkout of its tag) and run:+ - `COMPAT_FROM=<it>/dist/cli.js npm run test:e2e -- compat`+ - `MIGRATION_FROM=<it>/dist/cli.js npm run test:e2e -- migration`+3. `make release-cli`.++## The tests that only run when asked++| Test | How | Why it is not in CI |+|---|---|---|+| Compared with a release | `COMPAT_FROM=<bundle>` | needs an old release built |+| Upgrade from a release | `MIGRATION_FROM=<bundle>` | the same |+| Full disk | `DISKFULL=1` (macOS) | mounts a 64 MB disk image |+| Scale | `SCALE_AGENTS=50` | a few GB of memory |+| Soak | `SOAK_ROUNDS=300` | minutes |+| Chaos | `CHAOS_SEED=<n> CHAOS_OPS=150` | minutes; the default seed runs in CI |++## Known limits and next steps++- **Devices**: the plan is in the design doc (D0 the fleet on its own, D1 the dial behind the+ boundary, D2 its own process, D3 the Wi-Fi device with the relay). It was not started tonight on+ purpose: a deep change in code the end-to-end suite cannot reach (multi-machine) is the wrong risk+ right before a release.+- **Models' requests** (`grid_*`, `models_list`) are still in the socket's switch. That is a+ teammate's active area (#755); they move to `services/models.ts` when the team is ready.+- **Search in its own process** is opt-in until it has been dogfooded.+- **Cursor's watcher** still re-reads Cursor's transcript whole on each change. It needs pages of+ its own.+- **Engine flags:** the cache of what an engine's `--help` supports keeps a flag until the daemon+ restarts, even if an engine update drops it. Rare, and loud when it happens (the engine refuses+ the flag).+- **hn's `terminal_remain`** (drawing a dead pane's message) is not done by the daemon, only by hn's+ own local shells.