The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
The updater leaves the core for a process the master runs, and the core never downloads a build (#925)
Dee08c80d76dd21
diff --git a/cli/AGENTS.md b/cli/AGENTS.mdindex 0a76e4bc1..dc3272571 100644--- a/cli/AGENTS.md+++ b/cli/AGENTS.md@@ -18,9 +18,12 @@ SERVICES src/services/ everything else: search, viewers, models, workspaces,`src/core/main.ts` `runForeground()` is the composition root: it creates the modules and wires themtogether. It is the core's own entry (`harness __run`), which the master starts; `src/cli.ts` is the CLI,and calls in for `__run`. `harness start -f` runs the master in the foreground, as launchd and systemd do.-A core with no master (`HARNESS_NO_MASTER=1`, or one an older release's handoff started) hands each update-to a master that judges it (`src/core/updateHandoff.ts`). `src/backendSocket.ts` is the transport: it-receives frames and dispatches them.+The updater is the master's, in a process of its own (`src/services/updaterProcess.ts`): the core never+downloads a build. When it stages one, the master has the core hand over (`harnessd:update`) and judges the+new build. A core with no master gets no updates: one an older release's own handoff started hands itself+to a master once that release has gone, and one `HARNESS_NO_MASTER=1` asked for runs as it is+(`src/core/updateHandoff.ts`). `src/backendSocket.ts` is the transport: it receives frames and dispatches+them.`src/gateway/` is the relay: the backend link, the E2EE sessions and keys, and every rule about what aremote client may send and how what it is sent is sealed. It runs in a process of its own(`src/gateway/gatewayProcess.ts`; the core's side is `src/core/gatewayLink.ts`), or in the core's withdiff --git a/cli/e2e/harness/daemon.ts b/cli/e2e/harness/daemon.tsindex af07a87b4..8b21fb48d 100644--- a/cli/e2e/harness/daemon.ts+++ b/cli/e2e/harness/daemon.ts@@ -57,6 +57,8 @@ export interface DaemonOptions {noMaster?: boolean/** Start as a supervisor does, `harness start -f`: the master in the foreground, the core its child. */foreground?: boolean+ /** Start with these arguments to node instead (a launcher the test provides), in the daemon's environment. */+ launch?: string[]/** Run this bundle (an installed `cli.js`) instead of the checkout's source. */scriptPath?: string/** Keep the daemon's data folder here instead of under the throwaway root (a test volume). The fake@@ -308,7 +310,7 @@ export class IsolatedDaemon {// A test's own bundle (an old release), the run's bundle (`E2E_BUNDLE`, see bundle.ts), or the sources.const bundle = this.options.scriptPath ?? process.env.E2E_BUNDLE_PATHconst script = bundle ? [bundle] : ['--import', 'tsx', 'src/cli.ts']- const child = spawn(process.execPath, [...heap, ...script, ...entry], {+ const child = spawn(process.execPath, this.options.launch ?? [...heap, ...script, ...entry], {cwd: CLI_ROOT, env: this.env, stdio: ['ignore', 'pipe', 'pipe'],})this.child = childdiff --git a/cli/e2e/harness/olderReleaseHandoff.mjs b/cli/e2e/harness/olderReleaseHandoff.mjsnew file mode 100644index 000000000..49dc3cd45--- /dev/null+++ b/cli/e2e/harness/olderReleaseHandoff.mjs@@ -0,0 +1,21 @@+// What an older release's own update handoff does, for e2e/updateOnItsOwn.e2e.ts: it spawns the new build's+// core (`cli.js __run`, detached, told ADAPTER_UPDATED_TO), waits for it to claim the pid file, judges it+// a while longer (the release waits for its connection), and leaves. Releases before harnessd's master did+// exactly this, and so do later ones run without a master; the core it leaves behind has no master.+//+// Usage: node olderReleaseHandoff.mjs <cli.js> <pid file> <file whose presence lets it leave>+import { spawn } from 'node:child_process'+import { existsSync, readFileSync } from 'node:fs'++const [cli, pidFile, mayLeave] = process.argv.slice(2)+const core = spawn(process.execPath, [cli, '__run'], {+ detached: true, stdio: 'inherit', env: { ...process.env, ADAPTER_UPDATED_TO: process.env.HANDED_OVER_VERSION ?? '0.0.0' },+})+core.unref()+const deadline = Date.now() + 120_000+const judged = setInterval(() => {+ let claimed = false+ try { claimed = Number(readFileSync(pidFile, 'utf8').trim()) === core.pid } catch { /* not yet */ }+ if (claimed && existsSync(mayLeave)) { clearInterval(judged); process.exit(0) }+ if (Date.now() > deadline) process.exit(1)+}, 200)diff --git a/cli/e2e/migration.e2e.ts b/cli/e2e/migration.e2e.tsindex 563de4193..e96cccbbe 100644--- a/cli/e2e/migration.e2e.ts+++ b/cli/e2e/migration.e2e.ts@@ -3,8 +3,9 @@* does when a release ships. A daemon of the released build (`MIGRATION_FROM`, its bundled `cli.js`)* runs with agents at work, finds this checkout's build in its update manifest, and hands over to it* the way that release hands over: it spawns the new `cli.js __run` and exits. The new core comes up- * on its own, with no master, keeps every agent and its conversation, and the next start brings up- * harnessd's master over the same agents.+ * on its own, with no master, keeps every agent and its conversation, and once that release has gone it+ * hands the machine to harnessd's master, which runs the updater (core/updateHandoff.ts). Then the next+ * start brings up the master over the same agents again.** Skipped unless MIGRATION_FROM names a bundle, so CI does not build old releases. Before a release:* build the last released tag's bundle (`node build-bundle.mjs` in a checkout of it) and run@@ -103,8 +104,17 @@ describe.skipIf(!FROM)('upgrading a machine from a released build', () => {expect((await health())?.version).toBe(fromVersion)}, 300_000)+ /** The master the handed-over core started, if it is running: no child of the test's. */+ const handedTo = async (): Promise<number | null> => {+ const status = await fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json() as Promise<Record<string, any>>).catch(() => null)+ const pid = status?.harnessd?.masterPid+ return typeof pid === 'number' && pid !== daemon!.pid && IsolatedDaemon.alive(pid) ? pid : null+ }+afterAll(async () => {- // Whatever runs the port now — the detached successor included — goes with the test.+ // Whatever runs the port now — a master the handed-over core started included — goes with the test.+ const master = daemon ? await handedTo() : null+ if (master) { try { process.kill(master, 'SIGTERM') } catch { /* gone */ } }for (const pid of listeners()) { try { process.kill(pid, 'SIGTERM') } catch { /* gone */ } }await daemon?.close()await new Promise<void>((done) => server ? server.close(() => done()) : done())@@ -135,21 +145,26 @@ describe.skipIf(!FROM)('upgrading a machine from a released build', () => {offered = 'next'await until(`the machine to run ${NEXT}`, async () => (await health())?.version === NEXT || null, 120_000, 500)client.close()++ // Once the released build has gone, the core it left with no master hands the machine to one, which+ // runs the updater: a second restart, seconds after the first, as any update's is. The agents go on.+ const master = await until('the handed-over core to hand the machine to a master', () => handedTo(), 120_000, 500)client = await LocalClient.connect(d)for (const agent of agents) {- await until(`${agent.id.slice(0, 8)} back on ${NEXT}`, async () => {+ await until(`${agent.id.slice(0, 8)} back under the master the core started`, async () => {const now = (await rows(client)).find((one) => one.id === agent.id)return now?.status === 'active' && now.sessionId === agent.sessionId ? now : null}, 60_000, 500)- await turn(client, agent.id, `on ${NEXT}, handed over`)+ await turn(client, agent.id, `on ${NEXT}, under the master the core started`)}client.close()// The next `harness start` (a reboot, the desktop app) starts the new build as harnessd: master and core.// The released build's own process may still be about, waiting on its successor; it goes first.await d.stop()+ process.kill(master, 'SIGTERM')for (const pid of listeners()) process.kill(pid, 'SIGTERM')- await until('the handed-over core to stop', () => listeners().length === 0 || null, 30_000, 250)+ await until('the daemon to stop', () => (listeners().length === 0 && !IsolatedDaemon.alive(master)) || null, 30_000, 250);(d.options as { noMaster?: boolean }).noMaster = falseawait d.start()expect((await health())?.version).toBe(NEXT)diff --git a/cli/e2e/reexec.e2e.ts b/cli/e2e/reexec.e2e.tsindex b5108c955..488a641c0 100644--- a/cli/e2e/reexec.e2e.ts+++ b/cli/e2e/reexec.e2e.ts@@ -260,10 +260,13 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\let scratch = ''let server: Server | undefinedlet daemon: IsolatedDaemon | undefined- let offered: 'released' | 'next' = 'released'+ let offered: 'released' | 'next' | 'later' = 'released'let releasedVersion = ''let releasedCanReexec = false- const bundles = new Map<'released' | 'next', { cli: Buffer; notify: Buffer; version: string }>()+ const bundles = new Map<'released' | 'next' | 'later', { cli: Buffer; notify: Buffer; version: string }>()+ const LATER = '0.99.1'+ /** The newest build this machine has kept: NEXT, or LATER once a released master's core updated to it. */+ let newest = NEXTconst cliDir = () => join(scratch, 'cli')const status = async (): Promise<Record<string, any> | null> =>fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json()).catch(() => null)@@ -273,6 +276,7 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\const out = join(scratch, 'build')execFileSync(process.execPath, ['build-bundle.mjs'], { cwd: CLI_ROOT, env: { ...process.env, ADAPTER_VERSION: NEXT, BUNDLE_OUT_DIR: out }, stdio: 'pipe' })bundles.set('next', { cli: readFileSync(join(out, 'cli.js')), notify: readFileSync(join(out, 'notify.mjs')), version: NEXT })+ bundles.set('later', { cli: Buffer.from(atVersion(readFileSync(join(out, 'cli.js'), 'utf8'), NEXT, LATER)), notify: readFileSync(join(out, 'notify.mjs')), version: LATER })releasedVersion = execFileSync(process.execPath, [RELEASED!, 'version'], { encoding: 'utf8' }).trim()bundles.set('released', { cli: readFileSync(RELEASED!), notify: readFileSync(join(dirname(RELEASED!), 'notify.mjs')), version: releasedVersion })server = createServer((request, response) => {@@ -339,8 +343,18 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\expect(daemon!.log()).not.toContain('re-executing')}await until('the released master to keep the update', () => daemon!.log().includes('the update stayed up — keeping it'), 30_000)+ if (!releasedCanReexec) {+ // A master from before the updater left the core runs none, and could not become this build's: the+ // core runs it beside itself (core/updaterBeside.ts), so the next build still arrives.+ expect(daemon!.log()).toContain('[update] this core\'s master runs no updater — running it beside this core')+ offered = 'later'+ await until(`the released master to run ${LATER}`, async () => (await status())?.version === LATER || null, 90_000, 250)+ expect((await status())?.harnessd.masterPid).toBe(master)+ await until('the released master to keep it too', () => daemon!.log().split('the update stayed up — keeping it').length > 2 || null, 30_000)+ newest = LATER+ }await daemon!.restart()- expect((await status())?.harnessd).toMatchObject({ masterVersion: NEXT, masterPid: daemon!.pid })+ expect((await status())?.harnessd).toMatchObject({ masterVersion: newest, masterPid: daemon!.pid })})it('this master returns to the released bundle, re-executing only when it answers the probe', async () => {@@ -355,7 +369,7 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\return current?.version === releasedVersion ? current : null}, 60_000, 250)expect(now.harnessd).toMatchObject({- masterVersion: releasedCanReexec ? releasedVersion : NEXT,+ masterVersion: releasedCanReexec ? releasedVersion : newest,masterPid: master,reexecs: releasedCanReexec ? 1 : 0,})diff --git a/cli/e2e/releaseRehearsal.e2e.ts b/cli/e2e/releaseRehearsal.e2e.tsindex 42dafd6be..2538d0a48 100644--- a/cli/e2e/releaseRehearsal.e2e.ts+++ b/cli/e2e/releaseRehearsal.e2e.ts@@ -46,6 +46,8 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published releaselet offered: 'from' | 'next' = 'from'let fromVersion = ''let fromHasMaster = false+ /** The master a core with no master handed the machine to, when the release had none. */+ let handedTo: number | null = nullconst bundles = new Map<'from' | 'next', { cli: Buffer; notify: Buffer; version: string }>()const cliDir = () => join(scratch, 'cli')const status = async (): Promise<Row | null> =>@@ -211,6 +213,7 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published releaseafterAll(async () => {// Whatever runs the port now, a successor the release spawned on its own included, goes with the test.+ if (handedTo && IsolatedDaemon.alive(handedTo)) { try { process.kill(handedTo, 'SIGTERM') } catch { /* gone */ } }for (const pid of listeners()) { try { process.kill(pid, 'SIGTERM') } catch { /* gone */ } }await daemon?.close()await backend?.close()@@ -257,6 +260,14 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published releaseoffered = 'next'await until(`the machine to run ${NEXT}`, async () => (await status())?.version === NEXT || null, 150_000, 250)client.close()+ if (!fromHasMaster) {+ // A release with no master: once it has gone, the core it started hands the machine to a master,+ // which runs the updater (core/updateHandoff.ts), seconds after the release's own handoff.+ handedTo = await until('the core to hand the machine to a master', async () => {+ const pid = (await status())?.harnessd?.masterPid+ return typeof pid === 'number' && IsolatedDaemon.alive(pid) ? pid : null+ }, 120_000, 500)+ }// The window reconnects, as the desktop app does.client = await connect(d)@@ -291,7 +302,7 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published releasefor (const file of [files.claude, files.codex]) expect(readFileSync(file, 'utf8')).toContain(join(cliDir(), 'notify.mjs'))// Every process accounted for: under the release's master, its core on the new bundle as its child;- // without one, the successor the release spawned, and nothing of the release left running.+ // without one, the core of the master the successor handed the machine to, and nothing of the release.const status1 = (await status())!// A release without a master hands over by spawning the successor and watching it for up to half a// minute more (its waitForReady) before it leaves: one core once it has.@@ -304,7 +315,8 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published releaseexpect(cores[0].ppid).toBe(d.pid)await until('the master to keep the update', () => d.log().slice(updateFrom).includes('the update stayed up — keeping it'), 60_000)} else {- expect(listeners()).toEqual([cores[0].pid])+ // The master the core handed the machine to serves it, its core its child.+ expect(cores[0].ppid).toBe(handedTo)}expect(existsSync(join(cliDir(), 'cli.js.prev')) || existsSync(join(cliDir(), 'update-pending.json'))).toBe(false)memory(`after the update from ${fromVersion}`)@@ -312,6 +324,7 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published release// The next `harness start` (a reboot, the desktop app) starts this checkout as harnessd: master and core.await d.stop()+ if (handedTo) process.kill(handedTo, 'SIGTERM')for (const pid of listeners()) process.kill(pid, 'SIGTERM')await until('the daemon to be down', () => listeners().length === 0 || null, 30_000, 250);(d.options as { noMaster?: boolean }).noMaster = falsediff --git a/cli/e2e/update.e2e.ts b/cli/e2e/update.e2e.tsindex e40777d66..d54a216ac 100644--- a/cli/e2e/update.e2e.ts+++ b/cli/e2e/update.e2e.ts@@ -1,6 +1,7 @@/**- * A self-update end to end, on real release bundles: the daemon finds a newer build in its manifest,- * stages it, and harnessd's master restarts the core on it. A build that stays up is kept. One that+ * A self-update end to end, on real release bundles: the master's updater, in a process of its own,+ * finds a newer build in its manifest and stages it, and harnessd's master has the core hand over and+ * restarts it on the new build. The core itself downloads nothing. A build that stays up is kept. One that* crashes is rolled back, remembered, and not tried again until a newer one is published — before,* every machine on it restarted once a minute until a fix shipped.*@@ -21,6 +22,7 @@ const FIRST = '41.0.1'const sha = (bytes: Buffer): string => createHash('sha256').update(bytes).digest('hex')const count = (text: string, part: string): number => text.split(part).length - 1const sleep = (ms: number) => new Promise((done) => setTimeout(done, ms))+const parentOf = (pid: number): number => Number(execFileSync('ps', ['-o', 'ppid=', '-p', String(pid)], { encoding: 'utf8' }).trim())describe('a self-update under harnessd', () => {let scratch = ''@@ -34,6 +36,12 @@ describe('a self-update under harnessd', () => {const cliDir = () => join(scratch, 'cli')const installed = () => readFileSync(join(cliDir(), 'cli.js'))const rejected = () => JSON.parse(readFileSync(join(cliDir(), 'update-rejected.json'), 'utf8')) as string[]+ /** The updater's process the master last started, while it runs. */+ const updaterPid = (): number | null => {+ const started = [...daemon!.log().matchAll(/\[harnessd\] service updater started \(pid (\d+)\)/g)].at(-1)+ const pid = started ? Number(started[1]) : null+ return pid && IsolatedDaemon.alive(pid) ? pid : null+ }const running = async (): Promise<string | null> => {const health = await fetch(`http://127.0.0.1:${daemon!.port}/api/health`).then((response) => response.json()).catch(() => null)return (health as { version?: string } | null)?.version ?? null@@ -128,6 +136,11 @@ describe('a self-update under harnessd', () => {expect(existsSync(join(cliDir(), 'update-pending.json'))).toBe(false)expect(existsSync(join(cliDir(), 'update-rejected.json'))).toBe(false)expect(daemon!.pid).toBe(master)+ // The updater is its own process, the master's child; the core asked to hand over, and ran no updater.+ const updater = updaterPid()+ expect(updater).not.toBeNull()+ expect(parentOf(updater!)).toBe(master)+ expect(daemon!.log()).toContain('[harnessd] the updater staged 41.0.2 — asking the core to hand over')})it('rolls back a build that crashes, and never stages it again', async () => {@@ -150,8 +163,13 @@ describe('a self-update under harnessd', () => {expect(daemon!.pid).toBe(master)})- it('moves on when a newer build is published', async () => {+ it('moves on when a newer build is published, after its updater was killed outright', async () => {showLog()+ // Killed: the master starts it again, and the core never noticed.+ const killed = updaterPid()!+ process.kill(killed, 'SIGKILL')+ await until('the master to start the updater again', () => { const now = updaterPid(); return now && now !== killed ? now : null }, 30_000, 250)+ expect(await running()).toBe('41.0.2')offered = '41.0.4'await until('the core to run 41.0.4', async () => (await running()) === '41.0.4', 90_000, 250)await until('the master to keep it', () => count(daemon!.log(), 'the update stayed up — keeping it') === 2, 30_000)diff --git a/cli/e2e/updateOnItsOwn.e2e.ts b/cli/e2e/updateOnItsOwn.e2e.tsindex 6eab3b76a..da5a87774 100644--- a/cli/e2e/updateOnItsOwn.e2e.ts+++ b/cli/e2e/updateOnItsOwn.e2e.ts@@ -1,52 +1,61 @@/**- * A self-update of a core that runs with no master, on real release bundles: what this build's core does- * when an older release's own handoff started it (it spawned `cli.js __run`, e2e/migration.e2e.ts), or- * `HARNESS_NO_MASTER=1` asked for it. It hands the machine to harnessd's master on the staged bundle,- * which judges the update as it judges every one: a build whose core crashes is rolled back, remembered- * and the master re-executed onto the build before; a good one is kept. A build whose master cannot- * start is refused before anything is let go, and this build goes on under a master of its own. Before,- * the core spawned a core like itself and judged it, and the successor had no master either. The same- * holds for an update staged while such a core is still starting, or is in safe mode, as the boot- * handoff hands it over (core/main.ts `bootHandoff`).+ * A core with no master, on real release bundles. The updater is the master's, in a process of its own+ * (services/updaterProcess.ts), and the core never downloads a build, so a core with no master gets no+ * update by itself:+ *+ * - one an older release's own handoff started (it spawns `cli.js __run`, judges it and leaves:+ * e2e/harness/olderReleaseHandoff.mjs) hands the machine to a master on its build once that release has+ * gone. The master's updater then updates it, and the master judges each build: one whose core crashes+ * is rolled back and remembered, a good one kept. Before, such a core ran its own updater and handed each+ * update to a core like itself, which again had no master;+ * - one `HARNESS_NO_MASTER=1` asked for runs without updates, as asked.** The bundles are built from this checkout at made-up versions and served from a local manifest, as in* e2e/update.e2e.ts, so nothing here reaches beyond the machine.*/import { execFileSync } from 'node:child_process'import { createHash } from 'node:crypto'-import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'import { createServer, type Server } from 'node:http'import { tmpdir } from 'node:os'import { join } from 'node:path'-import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'+import { afterAll, afterEach, beforeAll, describe, expect, it, onTestFailed } from 'vitest'import { LocalClient, type Frame } from './harness/client.js'import { CLI_ROOT, IsolatedDaemon, until } from './harness/daemon.js'import { atVersion } from './harness/release.js'const FIRST = '44.0.1'const sha = (bytes: Buffer): string => createHash('sha256').update(bytes).digest('hex')+const sleep = (ms: number) => new Promise((done) => setTimeout(done, ms))type Row = Record<string, any>-describe('a self-update of a core with no master', () => {+describe('a core with no master', () => {let scratch = ''let server: Server | undefinedlet daemon: IsolatedDaemon | undefined+ let master: number | null = nullconst releases = new Map<string, Buffer>()let notify = Buffer.alloc(0)let offered = FIRST+ let local = ''const cliDir = () => join(scratch, 'cli')const installed = () => readFileSync(join(cliDir(), 'cli.js'))const rejected = () => JSON.parse(readFileSync(join(cliDir(), 'update-rejected.json'), 'utf8')) as string[]const status = async (): Promise<Row | null> =>fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json() as Promise<Row>).catch(() => null)- /** What the master this test never started writes: the daemon's log file. */- const masterLog = (): string => { try { return readFileSync(join(daemon!.dataDir, 'harness.log'), 'utf8') } catch { return '' } }/** The pids answering on the daemon's port: whatever runs it now. */const listeners = (): number[] => {try {return execFileSync('lsof', ['-n', '-P', '-t', `-iTCP:${daemon!.port}`, '-sTCP:LISTEN']).toString().trim().split('\n').filter(Boolean).map(Number)} catch { return [] }}+ /** What the daemon has said: what this test's process printed, and the log file the master the core+ * started writes to (core/main.ts `startMasterHere`). */+ const logs = (): string => {+ let file = ''+ try { file = readFileSync(join(daemon!.dataDir, 'harness.log'), 'utf8') } catch { /* none yet */ }+ return `${daemon!.log()}\n${file}`+ }const parentOf = (pid: number): number => Number(execFileSync('ps', ['-o', 'ppid=', '-p', String(pid)], { encoding: 'utf8' }).trim())const rows = async (client: LocalClient): Promise<Row[]> =>(await client.request<{ agents: Row[] }>('agents_list', { includeStopped: true }, 30_000)).agents@@ -55,12 +64,40 @@ describe('a self-update of a core with no master', () => {client.send('message', { agentId, content })await ended}- /** The master the core handed the machine to, once its core on `version` answers. */+ /** The agent, on its conversation, taking a turn: proof it came through. */+ const goesOn = async (agent: Row, content: string) => {+ const client = await LocalClient.connect(daemon!)+ await until('the agent back on its conversation', async () => {+ const row = (await rows(client)).find((one) => one.id === agent.id)+ return row?.status === 'active' && row.sessionId === agent.sessionId ? row : null+ }, 60_000, 500)+ await turn(client, agent.id, content)+ client.close()+ }+ /** The master running the machine, once its core on `version` answers. */const underMaster = (version: string) => until(`the machine to run ${version} under a master`, async () => {const now = await status()return now?.version === version && now.harnessd?.masterPid && IsolatedDaemon.alive(now.harnessd.masterPid) ? now : null}, 120_000, 250)- let master: number | null = null+ const environment = () => ({+ ADAPTER_CLI_DIR: cliDir(),+ ADAPTER_UPDATE_DISABLE: 'false',+ ADAPTER_UPDATE_URL: `${local}/metadata.json`,+ ADAPTER_UPDATE_CHECK_MS: '1000',+ ADAPTER_UPDATE_SLOT_SEC: '-1',+ HARNESSD_UPDATE_PROBATION_MS: '3000',+ HARNESSD_INITIAL_BACKOFF_MS: '100',+ HARNESS_TUI_MANIFEST_URL: `${local}/tui/metadata.json`,+ ADAPTER_RUNTIME_METADATA_URL: `${local}/runtime/metadata.json`,+ ADAPTER_GRID_RUNTIME_METADATA_URL: `${local}/grid/metadata.json`,+ })+ const install = (version: string) => {+ rmSync(cliDir(), { recursive: true, force: true })+ mkdirSync(cliDir())+ writeFileSync(join(cliDir(), 'cli.js'), releases.get(version)!)+ writeFileSync(join(cliDir(), 'notify.mjs'), notify)+ writeFileSync(join(cliDir(), 'package.json'), '{"type":"module"}\n')+ }beforeAll(async () => {scratch = mkdtempSync(join(tmpdir(), 'harnessd-update-alone-'))@@ -79,11 +116,7 @@ describe('a self-update of a core with no master', () => {release(FIRST)// Starts a master, whose core dies on every start.release('44.0.2', ['__run'])- // Whose master cannot start at all: it does not answer its probe.- release('44.0.3', ['__harnessd', '__harnessd-probe'])- release('44.0.4')- release('44.0.5', ['__harnessd', '__harnessd-probe'])- release('44.0.6')+ release('44.0.3')server = createServer((request, response) => {const url = request.url ?? ''@@ -104,66 +137,36 @@ describe('a self-update of a core with no master', () => {response.end()})await new Promise<void>((done) => server!.listen(0, '127.0.0.1', done))- const local = `http://127.0.0.1:${(server.address() as { port: number }).port}`-- mkdirSync(cliDir())- writeFileSync(join(cliDir(), 'cli.js'), releases.get(FIRST)!)- writeFileSync(join(cliDir(), 'notify.mjs'), notify)- writeFileSync(join(cliDir(), 'package.json'), '{"type":"module"}\n')- daemon = await IsolatedDaemon.create({- scriptPath: join(cliDir(), 'cli.js'),- // A core on its own, as an older release's handoff, or HARNESS_NO_MASTER=1, starts it.- noMaster: true,- env: {- ADAPTER_CLI_DIR: cliDir(),- ADAPTER_UPDATE_DISABLE: 'false',- ADAPTER_UPDATE_URL: `${local}/metadata.json`,- ADAPTER_UPDATE_CHECK_MS: '1000',- ADAPTER_UPDATE_SLOT_SEC: '-1',- HARNESSD_UPDATE_PROBATION_MS: '3000',- HARNESSD_INITIAL_BACKOFF_MS: '100',- HARNESS_TUI_MANIFEST_URL: `${local}/tui/metadata.json`,- ADAPTER_RUNTIME_METADATA_URL: `${local}/runtime/metadata.json`,- ADAPTER_GRID_RUNTIME_METADATA_URL: `${local}/grid/metadata.json`,- },- })- await daemon.start()- expect((await status())?.version).toBe(FIRST)- expect((await status())?.harnessd).toBeFalsy()+ local = `http://127.0.0.1:${(server.address() as { port: number }).port}`}, 300_000)- afterAll(async () => {- // The master this core handed the machine to is no child of the test's: it goes first, with its core.+ afterEach(async () => {+ // A master the core handed the machine to is no child of the test's: it goes first, with its core.if (master && IsolatedDaemon.alive(master)) {process.kill(master, 'SIGTERM')await until('the master to stop', () => !IsolatedDaemon.alive(master) || null, 15_000, 100).catch(() => { try { process.kill(master!, 'SIGKILL') } catch { /* gone */ } })}+ master = nullfor (const pid of listeners()) { try { process.kill(pid, 'SIGTERM') } catch { /* gone */ } }await daemon?.close()+ daemon = undefined+ })+ afterAll(async () => {await new Promise<void>((done) => server ? server.close(() => done()) : done())if (scratch) rmSync(scratch, { recursive: true, force: true })})- /** The agent, on its conversation, taking a turn: proof it came through. */- const goesOn = async (agent: Row, content: string) => {- const client = await LocalClient.connect(daemon!)- await until('the agent back on its conversation', async () => {- const row = (await rows(client)).find((one) => one.id === agent.id)- return row?.status === 'active' && row.sessionId === agent.sessionId ? row : null- }, 60_000, 500)- await turn(client, agent.id, content)- client.close()- }- /** One daemon: the core this test started has left, and the one on the port is the master's child. */- const oneDaemon = async (under: number) => {- await until('the core this test started to have left', () => !IsolatedDaemon.alive(daemon!.pid) || null, 30_000, 250)- const cores = await until('one core on the port', () => { const now = listeners(); return now.length === 1 ? now : null }, 30_000, 250)- expect(parentOf(cores[0])).toBe(under)- }-- it('hands each update to a master, which judges it; refuses one whose master cannot start; agents go on', async () => {- const d = daemon!- onTestFailed(() => { console.log(`---- the core's log\n${d.log().split('\n').slice(-60).join('\n')}\n---- the daemon's log file\n${masterLog().split('\n').slice(-120).join('\n')}`) })+ it('one an older release started hands itself to a master once the release has gone, whose updater updates it', async () => {+ offered = FIRST+ install(FIRST)+ const mayLeave = join(scratch, 'release-may-leave')+ rmSync(mayLeave, { force: true })+ const d = await IsolatedDaemon.create({ scriptPath: join(cliDir(), 'cli.js'), noMaster: true, env: environment() })+ daemon = d+ ;(d.options as { launch?: string[] }).launch = [join(CLI_ROOT, 'e2e/harness/olderReleaseHandoff.mjs'), join(cliDir(), 'cli.js'), join(d.dataDir, 'adapter.pid'), mayLeave]+ onTestFailed(() => { console.log(`---- daemon log\n${logs().split('\n').slice(-150).join('\n')}`) })+ await d.start()+ expect((await status())?.harnessd).toBeFalsy()const client = await LocalClient.connect(d)const cwd = join(d.projectsDir, 'alone')mkdirSync(cwd, { recursive: true })@@ -176,77 +179,46 @@ describe('a self-update of a core with no master', () => {await turn(client, agent.id, `on ${FIRST}, on its own`)client.close()- // A build whose core crashes: the core on its own stages it and hands the machine to a master on it,- // which finds the update unjudged, puts its first core on probation, rolls it back when it crashes and- // re-executes onto the build before.- offered = '44.0.2'- await until('the core to hand 44.0.2 to a master', () => d.log().includes('handing 44.0.2 to a harnessd master, which judges it') || null, 90_000, 250)- await until('the master to roll it back', () => masterLog().includes('the updated core failed (code 3) — rolled back') || null, 90_000, 250)- master = (await underMaster(FIRST)).harnessd.masterPid- expect(master).not.toBe(d.pid)- expect(installed()).toEqual(releases.get(FIRST))- expect(rejected()).toEqual(['44.0.2'])- expect(existsSync(join(cliDir(), 'update-pending.json'))).toBe(false)- await oneDaemon(master!)- await goesOn(agent, `back on ${FIRST}, under the master`)+ // The release that started it is still judging it: it stays put meanwhile, and runs no updater.+ const before = await status()+ await sleep(3_000)+ expect((await status())?.pid).toBe(before?.pid)+ expect(d.log()).not.toContain('[update] self-update on')- // On its own again (an older release's handoff, HARNESS_NO_MASTER=1), and a build whose master cannot- // start: refused before anything is let go, and this build goes on under a master of its own.- process.kill(master!, 'SIGTERM')- await until('the master to stop', () => (!IsolatedDaemon.alive(master) && listeners().length === 0) || null, 30_000, 250)- await d.start()- expect((await status())?.harnessd).toBeFalsy()- const from = d.log().length- const masterFrom = masterLog().length- offered = '44.0.3'- await until('the core to refuse 44.0.3', () => d.log().slice(from).includes('44.0.3\'s master did not answer its probe') || null, 90_000, 250)+ // The release leaves: the core hands the machine to a master on this build.+ writeFileSync(mayLeave, '')+ await until('the core to hand over', () => d.log().includes(`handing ${FIRST} to a harnessd master, which runs the updater`) || null, 60_000, 250)master = (await underMaster(FIRST)).harnessd.masterPid- expect(rejected()).toEqual(['44.0.2', '44.0.3'])- expect(installed()).toEqual(releases.get(FIRST))- await oneDaemon(master!)- // Its master started on this build, with nothing to judge: 44.0.3 never ran.- expect(masterLog().slice(masterFrom)).not.toContain('on probation')- await goesOn(agent, `on ${FIRST}, under a master of its own`)+ const cores = await until('one core on the port', () => { const now = listeners(); return now.length === 1 ? now : null }, 60_000, 250)+ expect(parentOf(cores[0])).toBe(master)+ await goesOn(agent, `on ${FIRST}, under a master`)- // A good build: now the master's to apply, re-executing on it in its own process, and to keep.- offered = '44.0.4'- await underMaster('44.0.4')- await until('the master to keep it', () => masterLog().includes('the update stayed up — keeping it') || null, 60_000, 250)+ // The master's updater: a build whose core crashes is rolled back and remembered, a good one is kept.+ offered = '44.0.2'+ await until('the master to roll 44.0.2 back', () => logs().includes('the updated core failed (code 3) — rolled back') || null, 120_000, 250)+ await underMaster(FIRST)+ expect(rejected()).toEqual(['44.0.2'])+ expect(installed()).toEqual(releases.get(FIRST))+ offered = '44.0.3'+ await underMaster('44.0.3')+ await until('the master to keep it', () => logs().includes('the update stayed up — keeping it') || null, 60_000, 250)expect((await status())?.harnessd.masterPid).toBe(master)- expect(installed()).toEqual(releases.get('44.0.4'))- expect(rejected()).toEqual(['44.0.2', '44.0.3'])- await goesOn(agent, 'on 44.0.4, kept')-- // The boot handoff: a core on its own in safe mode (its start-up threw; only its updater runs) hands an- // update over without finishing start-up. A build whose master cannot start is rolled back first, and- // this build's master starts, out of safe mode: the safe mode was the core's, not the master's to keep.- const inSafeMode = async () => {- process.kill(master!, 'SIGTERM')- await until('the master to stop', () => (!IsolatedDaemon.alive(master) && listeners().length === 0) || null, 30_000, 250)- d.env.HARNESSD_SAFE_MODE = 'an e2e start-up that failed'- await d.start({ ready: 'none' })- await until('the core to be in safe mode', async () => (await status())?.safeMode === true || null, 60_000, 250)- delete d.env.HARNESSD_SAFE_MODE- }- await inSafeMode()- offered = '44.0.5'- await until('the boot handoff to refuse 44.0.5', () => d.log().includes('44.0.5\'s master did not answer its probe') || null, 90_000, 250)- master = (await underMaster('44.0.4')).harnessd.masterPid- expect(rejected()).toEqual(['44.0.2', '44.0.3', '44.0.5'])- expect(installed()).toEqual(releases.get('44.0.4'))- await oneDaemon(master!)- expect((await status())?.safeMode).toBeFalsy()- await goesOn(agent, 'on 44.0.4, out of safe mode under a master')+ expect(installed()).toEqual(releases.get('44.0.3'))+ await goesOn(agent, 'on 44.0.3, kept')+ }, 600_000)- // …and a good build, which the master judges on its first core and keeps.- await inSafeMode()- const keptBefore = masterLog().split('the update stayed up — keeping it').length- offered = '44.0.6'- await until('the boot handoff to hand 44.0.6 over', () => d.log().includes('44.0.6 staged during start-up — handing off to harnessd\'s master') || null, 90_000, 250)- master = (await underMaster('44.0.6')).harnessd.masterPid- await until('the master to keep it', () => masterLog().split('the update stayed up — keeping it').length > keptBefore || null, 60_000, 250)- expect(installed()).toEqual(releases.get('44.0.6'))- await oneDaemon(master!)- await goesOn(agent, 'on 44.0.6, kept')- }, 900_000)+ it('one HARNESS_NO_MASTER=1 asked for runs without updates', async () => {+ offered = '44.0.3'+ install(FIRST)+ const d = await IsolatedDaemon.create({ scriptPath: join(cliDir(), 'cli.js'), noMaster: true, env: { ...environment(), HARNESS_NO_MASTER: '1' } })+ daemon = d+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-80).join('\n')}`) })+ await d.start()+ // A check a second, for five: nothing downloaded, nothing staged, the same core.+ const pid = (await status())?.pid+ await sleep(5_000)+ expect(await status()).toMatchObject({ version: FIRST, pid })+ expect(installed()).toEqual(releases.get(FIRST))+ expect(d.log()).not.toMatch(/\[update\] (newer build available|staged)/)+ }, 300_000)})diff --git a/cli/src/architecture.spec.ts b/cli/src/architecture.spec.tsindex 4b008108f..53f1098c8 100644--- a/cli/src/architecture.spec.ts+++ b/cli/src/architecture.spec.ts@@ -152,8 +152,10 @@ const WALK_TIMEOUT_MS = 60_000* D2): the devices' link to their own process and its routes, as every service there has, 2,213 to 2,232.* Then 2,230 (step 9, D3): the Wi-Fi device's wiring went with them (core/wifi.ts, core/wifiAgents.ts),* 2,232 lines to 2,227, though its link to the devices' process came in.+ *+ * Down to 2,215 the same day, at 2,209 from 2,228: the updater left the core for a process the master runs.*/-const RUN_FOREGROUND_BUDGET = 2_230+const RUN_FOREGROUND_BUDGET = 2_215/** Lowered from 2,180 when the relay and its E2EE left the socket for the gateway (step 10, R1: 1,440).* The Wi-Fi device's relay came back to it in R2, beside the device service it answers for, over the* gateway's sessions (lib/autonomous-device/overGateway.ts): 1,460. Models' grid commands, grid name and@@ -305,7 +307,11 @@ const SERVICE_MAY_IMPORT: Record<string, string> = {// session create, kill and rename). Before tmux 3.7 the two meeting crashed the tmux server, and every agent// with it (windows.e2e.ts, 7 of 27 CI runs on Ubuntu's 3.4). It guards the core's own terminals, so it// cannot move to a service. The budget keeps the 101 lines of room it had.-const CORE_CLOSURE_BUDGET = 73_512+//+// Then at 72,134 in 333, from 73,411 in 338: the updater left the core for a process the master runs+// (services/updaterProcess.ts), and the core never downloads a build. The CLI's and hn's updaters+// (lib/selfUpdate.ts, tui/update.ts, tui/install.ts) went, with the spawn lock they staged under.+const CORE_CLOSURE_BUDGET = 72_230/** What is not the core's, by path: each goes to a service or its own process, in the plan's order. */const EDGE: RegExp[] = [@@ -323,6 +329,9 @@ const EDGE: RegExp[] = [/^dsh\/(catalog|install|update|updates|registry|wire|service|lock|builtins|viewer|viewerLedger|verdict|artifacts)\.ts$/,// Search's index; the readers of other engines' sessions (external.ts, externals/) are the core's, for adoption./^lib\/sessionSearch\/(?!external\.ts$|externals\/)/,+ // Downloading builds: the updater's, in a process the master runs (services/updaterProcess.ts). The core+ // never downloads a build.+ /^lib\/(selfUpdate|runtimeInstall)\.ts$/, /^tui\/(update|install)\.ts$/,]/**diff --git a/cli/src/core/main.ts b/cli/src/core/main.tsindex 64fb3e8f6..4eb8024aa 100644--- a/cli/src/core/main.ts+++ b/cli/src/core/main.ts@@ -12,7 +12,7 @@import { ensureBundledCoreHarnesses } from '../dsh/builtins.js'import { readFileSync, writeFileSync, openSync, existsSync, rmSync, statSync } from 'fs'import { join } from 'path'-import { execFile, spawn, spawnSync } from 'child_process'+import { execFile, spawn } from 'child_process'import { createServer, type Server } from 'http'import { homedir, hostname } from 'os'import { env } from '../config/env.js'@@ -28,10 +28,9 @@ import { engineSessionTitle } from '../lib/sessionTitle.js'import { machineNames } from '../lib/machineNames.js'import { installCodexHooks } from '../lib/hooks.js'import { DAEMON_LOG_FILE, PID_FILE, daemonPort, isAlive, readPid, LEGACY_LOG_FILE, MACHINE_NAME_FILE, tildify, computerId, thisDeviceLabel } from '../lib/daemonState.js'-import { clearSafeModeMarker, runBootHandoff, safeModeDisposition, safeModeStatusBody, SafeModeRequest, writeSafeModeMarker } from '../lib/daemonSafeMode.js'+import { clearSafeModeMarker, safeModeDisposition, safeModeStatusBody, SafeModeRequest, writeSafeModeMarker } from '../lib/daemonSafeMode.js'import { awakeTimeout } from '../lib/sleepAware.js'import { removePidFileIf, onError } from '../lib/daemonLaunch.js'-import { describeSpawnLockOwner, withSpawnLock } from '../lib/daemonSpawnLock.js'import { ensureTmuxOnPath } from '../lib/tmuxOnPath.js'import { AUTH_DIR, AuthSessionError, AuthSessionManager, clearAuthSession, ensureSignInEpoch, readAuthSession, signInOf, type AuthSession } from '../lib/authSession.js'import { ENGINES, enginePathOverride } from '../lib/engineBin.js'@@ -88,7 +87,8 @@ import { createQuestions } from './questions.js'import { createTurnActivity } from './turns/activity.js'import { createLastTurnReader } from './transcripts/lastTurn.js'import { createRecaps } from './turns/recaps.js'-import { createUpdateHandoff, probeStagedMaster, probeVerdict } from './updateHandoff.js'+import { createUpdateHandoff, handOverOnceReleased, probeStagedMaster, type TeardownStep } from './updateHandoff.js'+import { needsUpdaterBeside, startUpdaterBeside } from './updaterBeside.js'import { createHeartbeats } from './turns/heartbeats.js'import { createEventFunnel } from './turns/funnel.js'import { createAgyBackstop } from './turns/agyBackstop.js'@@ -139,7 +139,7 @@ import { daemonCommand } from '../lib/daemonCommand.js'import { KNOWN_SERVICES, servicesTheMasterRuns } from '../harnessd/services.js'import { createTeamsLink, teamsOutOfProcess } from './teamsLink.js'import { createDevicesLink } from './devicesLink.js'-import { CORE_EXIT_STOP, CORE_EXIT_UPDATE, PROBE_COMMAND, PROBE_TIMEOUT_MS } from '../harnessd/protocol.js'+import { CORE_EXIT_STOP, CORE_EXIT_UPDATE, PROBE_COMMAND } from '../harnessd/protocol.js'import { localSocketPath, refuseServedDataFolder, type LocalSocketServer } from '../lib/localSocket.js'import { saveDaemonPort } from '../lib/daemonEndpoint.js'import { publishHookRoute } from '../lib/hookRoutes.js'@@ -154,9 +154,8 @@ import { autonomousDeviceLocalRequest } from '../lib/autonomous-device/localApi.import { attachLocalWsServer, LOCAL_WS_PATH, LOCAL_WS_PROTOCOL_VERSION } from '../localWsServer.js'import { TERMINAL_BINARY_VERSION } from '../lib/terminalBinary.js'import { isLoopbackRequest, loopbackHosts } from '../lib/loopbackRequest.js'-import { startSelfUpdater, restore as restoreUpdate, DOWNLOAD_LIMITS, type Poller } from '../lib/selfUpdate.js'+import { isInstalledCopy } from '../lib/installedCopy.js'import { managedNodePath } from '../lib/nodeRuntime.js'-import { startTuiUpdater } from '../tui/update.js'import { type ActivityFrame } from '../lib/turnActivity.js'import { CursorTranscriptDiscovery } from '../engines/cursor/discovery.js'import { cursorDataDir } from '../engines/cursor/home.js'@@ -269,23 +268,23 @@ function startMasterHere(): void {* What a staged update does while the daemon is still starting up — and the little the boot needs to* know about itself to do it.*- * The self-updater is started in `runForeground`'s prologue, before anything that can throw or hang,- * because a daemon that cannot finish booting is a daemon that can never be fixed: there is no- * supervisor, and the desktop app only re-runs `harness start` on the same broken bytes, once a- * minute, for ever. Its `onStaged` therefore has to mean something LONG before `restartForUpdate`- * exists — hence the indirection: `applyStagedUpdate` is `bootHandoff` until the body has built- * everything `restartForUpdate` tears down, and is swapped for it at that one line.+ * The master's updater (services/updaterProcess.ts) can stage a build at any moment, and the master then+ * asks this core to hand over (`harnessd:update`), which `runForeground`'s prologue listens for before+ * anything that can throw or hang: a daemon that cannot finish booting must still leave for its fix. That+ * request therefore has to mean something LONG before `restartForUpdate` exists — hence the indirection:+ * `applyStagedUpdate` is `bootHandoff` until the body has built everything `restartForUpdate` tears down,+ * and is swapped for it at that one line.*//** This process's channel to a harnessd master, when one started it (see harnessd/coreLink.ts).- * Inert otherwise: a daemon run on its own claims its pid file and hands off updates itself. */+ * Inert otherwise: a daemon run on its own claims its pid file, and gets no updates. */const coreLink = connectToMaster()/** When the devices' process starts: once there is a device, a dial's port, a Wi-Fi device or a request for one* (core/devicesWake.ts). Asked of the master through the channel above. */const devicesWake = createDevicesWake({ want: (service) => coreLink.want(service), dataDir: env.ADAPTER_DATA_DIR, cableDisabled: env.CABLE_DISABLE, testDialPort: process.env.HARNESSD_TEST_DIAL_PORT })const daemonBoot: {- updater: Poller | null- tuiUpdater: Poller | null+ /** Stops the updater this core runs beside itself under a master too old to run it (core/updaterBeside.ts). */+ updaterBeside: (() => void) | null/** The hook server, once bound — the only thing a mid-boot handoff has to release. */hookServer: Server | null/** Its Unix-socket twin (lib/localSocket.ts), when one could be opened. Read by `/api/status`. */@@ -298,52 +297,24 @@ const daemonBoot: {applyStagedUpdate: (version: string) => void | Promise<void>/** Opens the request gate of a start-up that did not finish, so its clients are answered (safe mode). */openRequests: (() => void) | null-} = { updater: null, tuiUpdater: null, hookServer: null, localSocket: null, markNotReady: null, safeMode: null, handingOff: false, applyStagedUpdate: bootHandoff, openRequests: null }+} = { updaterBeside: null, hookServer: null, localSocket: null, markNotReady: null, safeMode: null, handingOff: false, applyStagedUpdate: bootHandoff, openRequests: null }/**- * Hand the machine to a newer build without finishing start-up.+ * Hand the machine to a newer build without finishing start-up: the master asked (its updater staged one),+ * and starts the new bundle the moment this exits, and judges it.** SYNCHRONOUS END TO END, and that is the whole safety argument: never awaiting means the half-built- * `runForeground` body cannot interleave between the port closing and the exit, so it can never- * reach the code that would bind the port the successor is about to take, and two daemons are- * impossible by construction. That is also why it does not wait for the master to claim the pid file the- * way `restartForUpdate` does — waiting would leave this process running alongside the new core, both- * reconciling tmux and writing the registry.- *- * Without a master it spawns one rather than merely exiting, because on a machine with no desktop app- * nothing else would ever start the successor, and even with one the next spawn window is up to ~70s- * away. The master judges the update it starts on (core/updateHandoff.ts); the core this once spawned- * instead ran it unjudged.+ * `runForeground` body cannot interleave between the port closing and the exit, so it can never reach the+ * code that would bind the port the successor is about to take. Only a master asks; a core without one+ * gets no updates.*/function bootHandoff(version: string): void {if (daemonBoot.handingOff) returndaemonBoot.handingOff = true- daemonBoot.tuiUpdater?.stop()- if (coreLink.supervised) {- // The master starts the new bundle the moment this exits, and rolls it back if it does not stay- // up; a successor spawned from here would be a daemon outside its supervision.- console.log(`[update] ${VERSION} → ${version} staged during start-up — handing back to harnessd`)- try { daemonBoot.hookServer?.close() } catch { /* already gone */ }- try { daemonBoot.localSocket?.closeSync() } catch { /* already gone */ }- process.exit(CORE_EXIT_UPDATE)- }- runBootHandoff(VERSION, version, {- // The hook port has no fallback: a successor that cannot bind it is a daemon that does not come up.- closeServer: () => {- try { (daemonBoot.hookServer as unknown as { closeAllConnections?: () => void } | null)?.closeAllConnections?.() } catch { /* already gone */ }- try { daemonBoot.hookServer?.close() } catch { /* already gone */ }- try { daemonBoot.localSocket?.closeSync() } catch { /* already gone */ }- },- // Synchronous, as all of this is: start-up has not finished, and has nothing to serve meanwhile.- probeMaster: () => {- const probe = spawnSync(managedNodePath(), [SCRIPT_PATH, PROBE_COMMAND], { encoding: 'utf8', timeout: PROBE_TIMEOUT_MS })- return probeVerdict(probe.error ?? (probe.status === 0 ? null : new Error(probe.signal ? `signal ${probe.signal}` : `exit ${probe.status}`)), probe.stdout ?? '')- },- rollBack: () => restoreUpdate(env.ADAPTER_CLI_DIR),- startMaster: startMasterHere,- exit: (code) => process.exit(code),- log: (message) => console.log(message),- })+ console.log(`[update] ${VERSION} → ${version} staged during start-up — handing back to harnessd`)+ try { daemonBoot.hookServer?.close() } catch { /* already gone */ }+ try { daemonBoot.localSocket?.closeSync() } catch { /* already gone */ }+ process.exit(CORE_EXIT_UPDATE)}/** Set by runForeground once the DSH companions exist; a frame projected before that carries none. */@@ -398,75 +369,49 @@ async function runForeground(session: AuthSession | null): Promise<void> {console.error('[fatal-guard] uncaughtException:', err instanceof Error ? (err.stack ?? err.message) : err)})- // ── THE UPDATER GOES FIRST. Everything below this point can throw, hang, or wait on a vendor file,- // a port, or tmux — and a daemon that never finishes starting is a daemon that can never be fixed:- // there is no supervisor, and the desktop app only re-runs `harness start` on the same broken bytes.- // Started here, a published fix lands on its own however badly the rest of the boot goes.+ // ── THE MASTER'S UPDATE IS HEARD FIRST. Everything below this point can throw, hang, or wait on a vendor+ // file, a port, or tmux — and a daemon that never finishes starting must still leave for its fix. The+ // updater is the master's, in a process of its own (services/updaterProcess.ts): the core never+ // downloads a build. When it stages one, the master asks this core to hand over, and the request is+ // listened for here, before anything that can fail.//- // `onStaged` is one indirection on purpose: the handoff's teardown does not exist yet and must not move- // (it tears down two dozen subsystems declared further down). Until it is ready, a staged update is+ // `applyStagedUpdate` is one indirection on purpose: the handoff's teardown does not exist yet and must not+ // move (it tears down two dozen subsystems declared further down). Until it is ready, a staged update is// applied by `bootHandoff`, which hands the machine over without finishing start-up.-- // Self-update ONLY manages the INSTALLED copy (`~/.harness/cli/cli.js`). A dev/repo run — `tsx`- // (`npm run dev`) OR `node dist/cli.js` from the checkout — must NEVER self-update: it would swap- // the published bundle into ~/.harness/cli and restart, hijacking the version you're developing.- // Match by inode so symlinks/realpath don't fool it; fall back to a path compare.- const installedCli = join(env.ADAPTER_CLI_DIR, 'cli.js')- let isInstalledCopy = SCRIPT_PATH === installedCli- try { isInstalledCopy = statSync(SCRIPT_PATH).ino === statSync(installedCli).ino } catch { /* keep path compare */ }- if (isInstalledCopy && !env.ADAPTER_UPDATE_DISABLE) {- daemonBoot.updater = startSelfUpdater({- currentVersion: VERSION,- url: env.ADAPTER_UPDATE_URL,- key: env.ADAPTER_UPDATE_KEY,- dir: env.ADAPTER_CLI_DIR,- intervalMs: env.ADAPTER_UPDATE_CHECK_MS,- slotSecond: env.ADAPTER_UPDATE_SLOT_SEC,- // The lock spans the byte swap AND the handoff it triggers, as one critical section: a- // `harness start` that lands between the two would otherwise stage over our .prev, and one- // that lands during the handoff would spawn a second daemon.- withLock: (fn) => withSpawnLock('handoff', fn, {- onWaiting: (owner) => console.log(`[update] waiting — the daemon is ${describeSpawnLockOwner(owner)}`),- }),- onStaged: (v) => daemonBoot.applyStagedUpdate(v),- limits: { idleMs: env.ADAPTER_UPDATE_IDLE_MS, deadlineMs: env.ADAPTER_UPDATE_DEADLINE_MS, floorBytesPerSecond: DOWNLOAD_LIMITS.floorBytesPerSecond },- stageWhileJudged: !coreLink.supervised || process.env.HARNESSD_JUDGES_SUPERSEDED === '1',+ coreLink.onUpdate((version) => { void daemonBoot.applyStagedUpdate(version) })+ // …or, under a master too old to run the updater, from the updater this core runs beside itself+ // (core/updaterBeside.ts), in a process of its own: the core downloads no build either way.+ if (needsUpdaterBeside(process.env, coreLink.supervised, isInstalledCopy(SCRIPT_PATH, env.ADAPTER_CLI_DIR), env.ADAPTER_UPDATE_DISABLE)) {+ daemonBoot.updaterBeside = startUpdaterBeside({+ spawn: () => {+ const updater = spawn(managedNodePath(), [SCRIPT_PATH, '__service', 'updater'], {+ env: { ...process.env, HARNESSD_SERVICE: 'updater', HARNESSD_UPDATER_BESIDE_CORE: '1' }, stdio: ['ignore', 'inherit', 'inherit', 'ipc'],+ })+ updater.on('error', (e) => console.error('[update] the updater could not start:', e instanceof Error ? e.message : e))+ return { onMessage: (listener) => { updater.on('message', listener) }, onExit: (listener) => { updater.on('exit', listener) }, kill: () => { updater.kill() } }+ },+ staged: (version) => { void daemonBoot.applyStagedUpdate(version) },+ log: (line) => console.log(line),})- const slotted = env.ADAPTER_UPDATE_SLOT_SEC >= 0 && 60_000 % env.ADAPTER_UPDATE_CHECK_MS === 0- console.log(`[update] self-update on · v${VERSION} · every ${Math.round(env.ADAPTER_UPDATE_CHECK_MS / 1000)}s`- + (slotted ? ` at :${String(env.ADAPTER_UPDATE_SLOT_SEC % 60).padStart(2, '0')}` : ''))- } else if (!env.ADAPTER_UPDATE_DISABLE) {- console.log(`[update] self-update off · running a dev/repo build (v${VERSION}), not the installed copy`)}-- // Keep the CLI's recovery updater armed first. hn is an independent, optional download.- daemonBoot.tuiUpdater = startTuiUpdater({- currentVersion: VERSION,- isInstalledCopy,- disabled: env.ADAPTER_UPDATE_DISABLE,- intervalMs: env.ADAPTER_UPDATE_CHECK_MS,- slotSecond: env.ADAPTER_UPDATE_SLOT_SEC,- })- // The update handoff (core/updateHandoff.ts): after the updaters, never above them (startupOrder.spec.ts),- // and ahead of the /api/status handler that reads whether it is under way and of shutdown(), which takes- // a successor it is judging down with us.+ // The update handoff (core/updateHandoff.ts): after the update is listened for, never above it+ // (startupOrder.spec.ts), and ahead of the /api/status handler that reads whether it is under way.const updateHandoff = createUpdateHandoff({version: VERSION, supervised: coreLink.supervised, exitForUpdate: () => process.exit(CORE_EXIT_UPDATE),- // A core on its own: the staged bundle's master must answer its probe, as a master asks before it- // re-executes on one; then a master takes the machine, and judges the update.+ // A core on its own that an older release started: its master must answer its probe, as a master asks+ // before it re-executes on a bundle; then the master takes the machine, and runs the updater.probeMaster: () => probeStagedMaster((done) => execFile(managedNodePath(), [SCRIPT_PATH, PROBE_COMMAND], { encoding: 'utf8' }, (error, stdout) => done(error, stdout))),- rollBack: () => restoreUpdate(env.ADAPTER_CLI_DIR),handOff: () => { startMasterHere(); process.exit(0) },log: (line) => console.log(line), error: (line) => console.error(line),})// Another daemon serves this data folder: leave before reading or writing anything of its — its- // registry, its viewers, its panes. Just after the updaters, never before them (startupOrder.spec.ts);- // their first check is a slot away, and a daemon that leaves here is gone long before it.+ // registry, its viewers, its panes. Just after the update is listened for, never before it+ // (startupOrder.spec.ts).await refuseServedDataFolder(localSocketPath(env.ADAPTER_DATA_DIR, env.PORT))// harnessd's master saw this core crash again and again: start nothing that could do it again. The- // updaters above keep running, so a published fix still lands (`enterSafeMode`).+ // master's updater runs on, and a published fix still lands (`enterSafeMode`).if (process.env.HARNESSD_SAFE_MODE) throw new SafeModeRequest(process.env.HARNESSD_SAFE_MODE)const savedApis = new ApiConnections(env.ADAPTER_DATA_DIR)@@ -503,7 +448,7 @@ async function runForeground(session: AuthSession | null): Promise<void> {// Missing tmux is a STATE, not a reason to refuse to start. The daemon already models a machine// without it — `tmuxBackend` is null whenever the config omits tmux, every caller tests it, and the// create/restart/resume paths answer `TMUX_UNAVAILABLE` — so it can still serve its status, the- // local socket, the backend link and its updater, and say what is missing. Refusing instead left a+ // local socket and the backend link, leave for an update, and say what is missing. Refusing instead left a// machine whose PATH lost tmux with a daemon that could not start and therefore could not be fixed.let tmuxUnavailable: string | null = nullif (tmuxPathPromise) {@@ -1664,8 +1609,8 @@ async function runForeground(session: AuthSession | null): Promise<void> {restarting: updateHandoff.restarting(),discoveryReady,discoveryError: discoveryError ?? (tmuxUnavailable ? `tmux unavailable: ${tmuxUnavailable}` : null),- // Present only when start-up failed and this daemon is holding the machine open for its- // updater. Clients key on `discoveryReady`; this says WHY, in one word, for a person reading it.+ // Present only when start-up failed and this daemon is holding the machine open until a fix+ // arrives. Clients key on `discoveryReady`; this says WHY, in one word, for a person reading it....(daemonBoot.safeMode ? { safeMode: true } : {}),// Agents whose history is being read right now, and how many wait their turn. Normally empty or// gone in a second; one that stays here names the store that is slow, which no other field does.@@ -2447,12 +2392,12 @@ async function runForeground(session: AuthSession | null): Promise<void> {console.log(`[cli] not signed in — serving this computer only · watching registered sessions for ${ENGINES.length} engines`)}- // ── self-update: a staged bundle restarts the daemon IMMEDIATELY (core/updateHandoff.ts, and- // handOffToMaster for a core run on its own). The handler stops being `bootHandoff` HERE, and not a- // line earlier: everything the teardown releases exists by now. A straight-line assignment, never a- // wait: if the body never reaches this line the handler stays `bootHandoff`, and the fix still lands.- daemonBoot.applyStagedUpdate = (v) => updateHandoff.restartForUpdate(v, [- ['the registry', () => registry.flush()], ['the updaters', () => { daemonBoot.updater?.stop(); daemonBoot.tuiUpdater?.stop() }],+ // ── self-update: a staged bundle restarts the daemon IMMEDIATELY (core/updateHandoff.ts). The handler+ // stops being `bootHandoff` HERE, and not a line earlier: everything the teardown releases exists by now.+ // A straight-line assignment, never a wait: if the body never reaches this line the handler stays+ // `bootHandoff`, and the fix still lands.+ const updateTeardown = (): TeardownStep[] => [+ ['the registry', () => registry.flush()], ['the updater', () => daemonBoot.updaterBeside?.()],['the reconciler', () => agentReconciler.stop()],['the timers', () => { clearInterval(logTrimTimer); clearInterval(runtimeReconcileTimer); clearInterval(paneTitleSyncTimer) }],['the question watchers', () => questionWatcher.stopAll()],@@ -2472,7 +2417,15 @@ async function runForeground(session: AuthSession | null): Promise<void> {['the viewers', () => ports.viewers?.stop()], ['the gateway', () => gateway.stop()],// A graceful close releases the backend's one-machine claim, given a moment before the reclaim.['the backend', () => backend.stop()], ['a grace', () => new Promise((r) => setTimeout(r, 1000))],- ])+ ]+ daemonBoot.applyStagedUpdate = (v) => updateHandoff.restartForUpdate(v, updateTeardown())+ // A core an older release's own handoff started (it spawned this `cli.js __run` and judged it) has no+ // master, and so no updater: it would run this build until its next start. Once that release has gone,+ // it hands the machine to a master on this build (core/updateHandoff.ts). Not a core HARNESS_NO_MASTER=1+ // asked for, and not a dev or repo run.+ if (!coreLink.supervised && process.env.ADAPTER_UPDATED_TO && process.env.HARNESS_NO_MASTER !== '1' && isInstalledCopy(SCRIPT_PATH, env.ADAPTER_CLI_DIR)) {+ handOverOnceReleased({ startedBy: process.ppid, parent: () => process.ppid, alive: isAlive, handOver: () => { void updateHandoff.handOver(updateTeardown()) } })+ }/** Stopping for good — removed from the account, or connected from elsewhere: tell harnessd's master,* which restarts any other exit (harnessd/protocol.ts). */@@ -2488,8 +2441,7 @@ async function runForeground(session: AuthSession | null): Promise<void> {// Release the serial port first. It is exclusive, and a daemon that exits still holding it makes// esptool fail in a way that reads exactly like dead hardware.void ports.devices?.stop()- daemonBoot.updater?.stop()- daemonBoot.tuiUpdater?.stop()+ daemonBoot.updaterBeside?.()agentReconciler.stop()clearInterval(logTrimTimer)clearInterval(runtimeReconcileTimer)@@ -2602,14 +2554,14 @@ async function runForeground(session: AuthSession | null): Promise<void> {}/**- * The daemon's start-up threw. STAY UP anyway, running nothing but the updater.+ * The daemon's start-up threw. STAY UP anyway, waiting for a fix.*- * Exiting here is what made one bad build unrecoverable: nothing supervises this process, the desktop- * app answers a dead port by running `harness start` again — the same bytes, about once a minute, for- * ever — and the updater that could have fixed it lives most of the way down a body that never- * finished. The updater is started in the prologue now (see `runForeground`), so by the time this- * runs it is already polling; all this has to do is keep the process alive long enough for a- * published fix to land, and tell everyone what state the machine is in.+ * Exiting here is what made one bad build unrecoverable: the desktop app answers a dead port by running+ * `harness start` again — the same bytes, about once a minute, for ever. The updater is the master's, in a+ * process of its own (services/updaterProcess.ts), and the request to hand over for its fix is listened for+ * in the prologue (see `runForeground`); all this has to do is keep the process alive long enough for a+ * published fix to land, and tell everyone what state the machine is in. A core with no master gets no fix+ * here: its safe mode runs out (`ADAPTER_SAFE_MODE_MS`) and a clean start tries again.** Three ways it earns its keep, in order: the bound control port answers `discoveryReady: false`, so* the app reads the machine as not-ready instead of dead and STOPS respawning; the pid file stays@@ -2627,8 +2579,8 @@ const enterSafeMode = (err: unknown): void => {}const detail = err instanceof Error ? (err.stack ?? err.message) : String(err)console.error('Failed to start adapter:', err)- console.error(`[safe-mode] staying up on v${VERSION} with the updater only — a published fix will be`- + ' applied on its own. Nothing else on this machine works until then.')+ console.error(`[safe-mode] staying up on v${VERSION} — a published fix will be applied on its own.`+ + ' Nothing else on this machine works until then.')writeSafeModeMarker(env.ADAPTER_DATA_DIR, { pid: process.pid, version: VERSION, at: Date.now(), error: detail })daemonBoot.safeMode = disposition.reasondaemonBoot.markNotReady?.(disposition.reason)@@ -2643,8 +2595,8 @@ const enterSafeMode = (err: unknown): void => {}process.on('SIGINT', () => leave('SIGINT — leaving safe mode', 0))process.on('SIGTERM', () => leave('SIGTERM — leaving safe mode', 0))- // Up, though not ready: the master must neither give up waiting for a bind nor take it for hung,- // or the updater that can fix this build would never get its chance. It hears why, and rolls back an+ // Up, though not ready: the master must neither give up waiting for a bind nor take it for hung, or the+ // fix its updater stages would find no core to hand over. It hears why, and rolls back an// update whose first core ends up here.coreLink.bound(daemonPort())coreLink.ready(disposition.reason)@@ -2668,7 +2620,7 @@ const enterSafeMode = (err: unknown): void => {})status.on('error', (e) => {console.error(`[safe-mode] could not serve status on ${port}: ${e instanceof Error ? e.message : e}`)- // A ref'd timer, unlike the updater's: something has to hold the event loop open.+ // A ref'd timer: something has to hold the event loop open.setInterval(() => console.log(`[safe-mode] still waiting for a fixed build · v${VERSION}`), 10 * 60_000)})status.listen(port, '127.0.0.1')diff --git a/cli/src/core/updateHandoff.spec.ts b/cli/src/core/updateHandoff.spec.tsindex 547a4aa2d..deed5c0cd 100644--- a/cli/src/core/updateHandoff.spec.ts+++ b/cli/src/core/updateHandoff.spec.ts@@ -1,6 +1,8 @@import { afterEach, describe, expect, it, vi } from 'vitest'import { PROBE_ANSWER } from '../harnessd/protocol.js'-import { TEARDOWN_DEADLINE_MS, createUpdateHandoff, probeStagedMaster, probeVerdict, type TeardownStep, type UpdateHandoffDeps } from './updateHandoff.js'+import {+ RELEASED_POLL_MS, TEARDOWN_DEADLINE_MS, createUpdateHandoff, handOverOnceReleased, probeStagedMaster, probeVerdict, type TeardownStep, type UpdateHandoffDeps,+} from './updateHandoff.js'describe('the update handoff', () => {afterEach(() => { vi.useRealTimers() })@@ -11,7 +13,6 @@ describe('the update handoff', () => {version: '1.0.0', supervised: true,exitForUpdate: () => calls.push('exit 75'),probeMaster: async () => { calls.push('probe the master'); return null },- rollBack: () => calls.push('roll back'),handOff: () => calls.push('start a master, exit'),log: (line) => calls.push(line),error: (line) => calls.push(line),@@ -68,30 +69,34 @@ describe('the update handoff', () => {expect(plain.calls.at(-1)).toBe('exit 75')})- it('without a master, gives the machine to a master on the staged build once its master answers, after the same teardown', async () => {+ it('without a master, hands this build to one once its master answers, after the same teardown, once', async () => {const { handoff, calls } = make({ supervised: false })- const done = handoff.restartForUpdate('2.0.0', [step(calls, 'the registry'), step(calls, 'the backend', () => { throw new Error('no') })])+ const done = handoff.handOver([step(calls, 'the registry'), step(calls, 'the backend', () => { throw new Error('no') })])// Asked before anything is let go: the core serves on meanwhile.expect(handoff.restarting()).toBe(false)- await handoff.restartForUpdate('2.0.0', [])- await done+ expect(await handoff.handOver([])).toBe(false)+ expect(await done).toBe(true)expect(handoff.restarting()).toBe(true)expect(calls).toEqual([- '[update] applying 1.0.0 → 2.0.0 — restarting daemon', 'probe the master', 'release the registry', 'release the backend',- '[update] the backend did not let go (no) — handing over all the same',- '[update] handing 2.0.0 to a harnessd master, which judges it — this core is leaving', 'start a master, exit',+ 'probe the master',+ '[update] handing 1.0.0 to a harnessd master, which runs the updater — this core is leaving',+ 'release the registry', 'release the backend', '[update] the backend did not let go (no) — handing over all the same',+ 'start a master, exit',])+ // Leaving, it hands nothing else over.+ await handoff.restartForUpdate('2.0.0', [])+ expect(calls.at(-1)).toBe('start a master, exit')})- it('without a master, rolls a build whose master does not answer back, and hands this build to a master of its own', async () => {- const { handoff, calls } = make({ supervised: false, probeMaster: async () => 'harnessd-probe failed: a bad build' })- await handoff.restartForUpdate('2.0.0', [step(calls, 'the registry')])- expect(calls).toEqual([- '[update] applying 1.0.0 → 2.0.0 — restarting daemon',- '[update] 2.0.0\'s master did not answer its probe (harnessd-probe failed: a bad build) — rolled back; this build goes on under a master of its own',- 'roll back', 'release the registry',- '[update] handing 1.0.0 to a harnessd master, which judges it — this core is leaving', 'start a master, exit',- ])+ it('without a master, carries on when its master does not answer, and may try again', async () => {+ let answers = ['harnessd-probe failed: a bad build', null]+ const { handoff, calls } = make({ supervised: false, probeMaster: async () => answers.shift() ?? null })+ expect(await handoff.handOver([step(calls, 'the registry')])).toBe(false)+ expect(handoff.restarting()).toBe(false)+ expect(calls).toEqual(['[update] this build\'s master did not answer its probe (harnessd-probe failed: a bad build) — carrying on without one'])+ expect(await handoff.handOver([step(calls, 'the registry')])).toBe(true)+ expect(calls.at(-1)).toBe('start a master, exit')+ answers = []})it('under harnessd, asks no probe: the master asks its own before it re-executes', async () => {@@ -101,6 +106,53 @@ describe('the update handoff', () => {})})+describe('handing over once the release that started this core has gone', () => {+ const make = (alive: boolean[], parents: number[]) => {+ let tick: (() => void) | null = null+ const calls: string[] = []+ const stop = handOverOnceReleased({+ startedBy: 77,+ parent: () => parents.shift() ?? 77,+ alive: () => alive.shift() ?? true,+ handOver: () => calls.push('hand over'),+ setTimer: (run, ms) => { calls.push(`every ${ms}`); tick = run; return 'timer' },+ clearTimer: (timer) => calls.push(`cleared ${String(timer)}`),+ })+ return { tick: () => tick!(), calls, stop }+ }++ it('waits while the release is there, and hands over once, when its process ends', () => {+ const m = make([true, true, false], [77, 77, 77])+ m.tick()+ m.tick()+ expect(m.calls).toEqual([`every ${RELEASED_POLL_MS}`])+ m.tick()+ expect(m.calls).toEqual([`every ${RELEASED_POLL_MS}`, 'cleared timer', 'hand over'])+ })++ it('hands over when this core is reparented away from it, and can be stopped before', () => {+ const m = make([], [1])+ m.tick()+ expect(m.calls.slice(-1)).toEqual(['hand over'])+ const stopped = make([], [])+ stopped.stop()+ expect(stopped.calls).toEqual([`every ${RELEASED_POLL_MS}`, 'cleared timer'])+ })++ it('polls on an interval that keeps no process alive by default', () => {+ vi.useFakeTimers()+ const handOver = vi.fn()+ let gone = false+ const stop = handOverOnceReleased({ startedBy: 77, parent: () => 77, alive: () => !gone, handOver })+ vi.advanceTimersByTime(RELEASED_POLL_MS * 3)+ expect(handOver).not.toHaveBeenCalled()+ gone = true+ vi.advanceTimersByTime(RELEASED_POLL_MS * 3)+ expect(handOver).toHaveBeenCalledOnce()+ stop()+ })+})+describe('the probe of a staged bundle\'s master', () => {/** A probe that ends as told, and the deadline it is given. */const run = (error: Error | null, stdout: string) => {diff --git a/cli/src/core/updateHandoff.ts b/cli/src/core/updateHandoff.tsindex b91815608..3b3c734d8 100644--- a/cli/src/core/updateHandoff.ts+++ b/cli/src/core/updateHandoff.ts@@ -1,20 +1,21 @@/**- * The core handing the machine to a newer build its updater has just staged (`selfUpdate.ts`- * `startSelfUpdater`'s `onStaged`), once start-up has finished: release what the next core needs (the- * fixed ports, the backend's one-machine claim, the watchers and timers), then go.+ * The core handing the machine to a newer build harnessd's updater has just staged (the master asks:+ * `harnessd:update`, harnessd/coreLink.ts), once start-up has finished: release what the next core needs+ * (the fixed ports, the backend's one-machine claim, the watchers and timers), then exit 75. The master+ * starts the new bundle the moment this core exits, and judges it.*- * Under harnessd the master starts the new bundle the moment this core exits 75, and judges it. A- * teardown step that threw used to end the handoff where it stood: the core stayed on the old build with- * its servers half closed, its updater stopped and the new bundle staged but never judged- * (e2e/updateHostile.e2e.ts, round 40). Now every step is tried, one that fails is said and passed, and- * the core exits 75 whatever happened; one that hangs is given up on after `TEARDOWN_DEADLINE_MS`.+ * A teardown step that threw used to end the handoff where it stood: the core stayed on the old build with+ * its servers half closed and the new bundle staged but never judged (e2e/updateHostile.e2e.ts, round 40).+ * Now every step is tried, one that fails is said and passed, and the core exits 75 whatever happened; one+ * that hangs is given up on after `TEARDOWN_DEADLINE_MS`.*- * Without a master (a core an older release's own handoff started, e2e/migration.e2e.ts, or- * `HARNESS_NO_MASTER=1`) the core gives the machine to one, which judges the build as every update: the- * staged bundle's master must answer its probe first, as before a re-execution (harnessd/reexec.ts), or the- * build before goes back, rejected; then the same teardown, and `handOff` starts `__harnessd` and exits.- * The master finds the pending note and puts its first core on probation (harnessd/supervisor.ts,- * `unjudgedUpdate`). The core once spawned a core like itself and judged it, which then had no master.+ * A core with no master gets no update: the updater is the master's (services/updaterProcess.ts), and the+ * core never downloads a build. One an older release's own handoff started (it spawned `cli.js __run`,+ * e2e/migration.e2e.ts) would run this build until its next start, so once that release has gone it hands+ * the machine to a master (`handOver`, `handOverOnceReleased`): its master answers its probe first, as a+ * master asks before it re-executes on a bundle (harnessd/reexec.ts), then the same teardown, then+ * `handOff` starts `__harnessd`, names it in the pid file and exits. One `HARNESS_NO_MASTER=1` asked for+ * runs without updates, as asked.** Staged means restart now. The restart once waited for the computer to go idle, and "idle" is a set of* latches (an open turn, a settling composer, an awaited submit, the control lock, a recap in flight):@@ -41,11 +42,9 @@ export interface UpdateHandoffDeps {supervised: boolean/** Exit for the update (`CORE_EXIT_UPDATE`), to the master that starts the new bundle. */exitForUpdate(): void- /** Without a master: whether the staged bundle's master answers its probe (`probeStagedMaster`); null when- * it does, why not otherwise. */+ /** Without a master: whether the master of the bundle on disk answers its probe (`probeStagedMaster`); null+ * when it does, why not otherwise. */probeMaster(): Promise<string | null>- /** Without a master: put the build before back (`selfUpdate.restore`), remembering this one as rejected. */- rollBack(): void/** Without a master: start harnessd's master on the bundle now on disk, name it in the pid file, exit. */handOff(): voidlog(line: string): void@@ -83,28 +82,36 @@ export function createUpdateHandoff(deps: UpdateHandoffDeps) {return {/** Between the teardown starting and this core leaving (`/api/status`, and a signal mid-handoff). */restarting: (): boolean => restarting,- /** Hand over to `newVersion`, releasing `teardown` first. Once: a second call while one is under way does nothing. */+ /** Hand over to `newVersion`, releasing `teardown` first, and exit for the update. Once: a second call+ * while one is under way does nothing. */async restartForUpdate(newVersion: string, teardown: readonly TeardownStep[]): Promise<void> {if (started) returnstarted = true+ restarting = truedeps.log(`[update] applying ${deps.version} → ${newVersion} — restarting daemon`)- // Asked before anything is let go: the core serves on while it answers.- const refused = deps.supervised ? null : await deps.probeMaster()+ await releaseAll(teardown)+ // Everything above is released, or said not to be; the master starts the new bundle as soon as this+ // exits and rolls back to the .prev bytes if it does not come up and stay up.+ deps.log(`[update] handing ${newVersion} to harnessd`)+ deps.exitForUpdate()+ },+ /** Without a master: give the machine to one on this build, once its master answers; otherwise carry on.+ * Resolves whether it handed over. */+ async handOver(teardown: readonly TeardownStep[]): Promise<boolean> {+ if (started) return false+ started = true+ // Asked before anything is let go: the core serves on while it answers, and on if it does not.+ const refused = await deps.probeMaster()if (refused !== null) {- deps.error(`[update] ${newVersion}'s master did not answer its probe (${refused}) — rolled back; this build goes on under a master of its own`)- deps.rollBack()+ deps.error(`[update] this build's master did not answer its probe (${refused}) — carrying on without one`)+ started = false+ return false}restarting = true+ deps.log(`[update] handing ${deps.version} to a harnessd master, which runs the updater — this core is leaving`)await releaseAll(teardown)- if (deps.supervised) {- // Everything above is released, or said not to be; the master starts the new bundle as soon as- // this exits and rolls back to the .prev bytes if it does not come up and stay up.- deps.log(`[update] handing ${newVersion} to harnessd`)- deps.exitForUpdate()- return- }- deps.log(`[update] handing ${refused === null ? newVersion : deps.version} to a harnessd master, which judges it — this core is leaving`)deps.handOff()+ return true},}}@@ -133,9 +140,35 @@ export function probeStagedMaster(run: ProbeRun, timeoutMs = PROBE_TIMEOUT_MS, d})}-/** What a master probe's end says: null when it answered as a master, why not otherwise. Shared by the- * boot handoff's synchronous probe (lib/daemonSafeMode.ts `runBootHandoff`). */+/** What a master probe's end says: null when it answered as a master, why not otherwise. */export function probeVerdict(error: Error | null | undefined, stdout: string): string | null {if (!error && stdout.includes(PROBE_ANSWER)) return nullreturn stdout.trim().split('\n').pop()?.trim() || error?.message || 'no answer'}++/** How often a core an older release started looks for it to have gone. */+export const RELEASED_POLL_MS = 2_000++/**+ * Hand the machine to a master once the older release that started this core has gone: it spawned this+ * core detached, judged it by its bind and its connection (up to a minute and a half) and leaves; a core+ * that left first would be read as the new build failing, and rolled back. Gone is its process ended, or+ * this core reparented away from it. Returns a stop.+ */+export function handOverOnceReleased(deps: {+ startedBy: number+ parent(): number+ alive(pid: number): boolean+ handOver(): void+ setTimer?: (run: () => void, ms: number) => unknown+ clearTimer?: (timer: unknown) => void+}): () => void {+ const setTimer = deps.setTimer ?? ((run, ms) => { const timer = setInterval(run, ms); timer.unref?.(); return timer })+ const clearTimer = deps.clearTimer ?? ((timer) => clearInterval(timer as ReturnType<typeof setInterval>))+ const timer = setTimer(() => {+ if (deps.parent() === deps.startedBy && deps.alive(deps.startedBy)) return+ clearTimer(timer)+ deps.handOver()+ }, RELEASED_POLL_MS)+ return () => clearTimer(timer)+}diff --git a/cli/src/core/updaterBeside.spec.ts b/cli/src/core/updaterBeside.spec.tsnew file mode 100644index 000000000..d14e1265d--- /dev/null+++ b/cli/src/core/updaterBeside.spec.ts@@ -0,0 +1,88 @@+import { afterEach, describe, expect, it, vi } from 'vitest'+import { UPDATER_RESTART_MS, needsUpdaterBeside, startUpdaterBeside, type UpdaterChild } from './updaterBeside.js'++class FakeChild implements UpdaterChild {+ private messages: Array<(message: unknown) => void> = []+ private exits: Array<() => void> = []+ killed = false+ onMessage(listener: (message: unknown) => void): void { this.messages.push(listener) }+ onExit(listener: () => void): void { this.exits.push(listener) }+ kill(): void { this.killed = true }+ say(message: unknown): void { for (const listener of this.messages) listener(message) }+ exit(): void { for (const listener of this.exits) listener() }+}++describe('the updater beside a core whose master runs none', () => {+ afterEach(() => { vi.useRealTimers() })++ it('is needed only under a master that does not say it runs the updater, for the installed copy with updates on', () => {+ expect(needsUpdaterBeside({}, true, true, false)).toBe(true)+ expect(needsUpdaterBeside({ HARNESSD_UPDATES: 'master' }, true, true, false)).toBe(false)+ expect(needsUpdaterBeside({}, false, true, false)).toBe(false)+ expect(needsUpdaterBeside({}, true, false, false)).toBe(false)+ expect(needsUpdaterBeside({}, true, true, true)).toBe(false)+ })++ const make = () => {+ const children: FakeChild[] = []+ const staged: string[] = []+ const lines: string[] = []+ let pending: (() => void) | null = null+ const stop = startUpdaterBeside({+ spawn: () => { const child = new FakeChild(); children.push(child); return child },+ staged: (version) => staged.push(version),+ log: (line) => lines.push(line),+ setTimer: (run, ms) => { lines.push(`in ${ms}`); pending = run; return 'timer' },+ clearTimer: () => { lines.push('cleared'); pending = null },+ })+ return { children, staged, lines, stop, fire: () => pending!() }+ }++ it('hands over once for what it stages, and ignores what else it says', () => {+ const m = make()+ expect(m.lines).toEqual(['[update] this core\'s master runs no updater — running it beside this core'])+ m.children[0].say({ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1 })+ m.children[0].say({ type: 'harnessd:staged', version: 3 })+ m.children[0].say(null)+ expect(m.staged).toEqual([])+ m.children[0].say({ type: 'harnessd:staged', version: '9.9.9' })+ m.children[0].say({ type: 'harnessd:staged', version: '9.9.10' })+ expect(m.staged).toEqual(['9.9.9'])+ expect(m.lines.at(-1)).toBe('[update] the updater staged 9.9.9 — handing over to this core\'s master')+ // It exits to be started again as the new build; this core is leaving instead.+ m.children[0].exit()+ expect(m.children).toHaveLength(1)+ })++ it('starts it again a while after it ends, and stops it, and any restart, on request', () => {+ const m = make()+ m.children[0].exit()+ expect(m.lines.slice(-2)).toEqual([`[update] the updater ended — starting it again in ${UPDATER_RESTART_MS / 1000} s`, `in ${UPDATER_RESTART_MS}`])+ m.fire()+ expect(m.children).toHaveLength(2)+ // A process it let go of says nothing that counts.+ m.children[0].exit()+ expect(m.children).toHaveLength(2)+ m.stop()+ expect(m.children[1].killed).toBe(true)+ m.children[1].exit()+ expect(m.children).toHaveLength(2)+ const waiting = make()+ waiting.children[0].exit()+ waiting.stop()+ expect(waiting.lines.at(-1)).toBe('cleared')+ })++ it('waits on a timer that keeps no process alive by default', () => {+ vi.useFakeTimers()+ const children: FakeChild[] = []+ const stop = startUpdaterBeside({ spawn: () => { const child = new FakeChild(); children.push(child); return child }, staged: () => {}, log: () => {} })+ children[0].exit()+ vi.advanceTimersByTime(UPDATER_RESTART_MS)+ expect(children).toHaveLength(2)+ children[1].exit()+ stop()+ vi.advanceTimersByTime(UPDATER_RESTART_MS)+ expect(children).toHaveLength(2)+ })+})diff --git a/cli/src/core/updaterBeside.ts b/cli/src/core/updaterBeside.tsnew file mode 100644index 000000000..e7dbec00f--- /dev/null+++ b/cli/src/core/updaterBeside.ts@@ -0,0 +1,74 @@+/**+ * The updater beside a core whose master is too old to run it: one released before the updater left the+ * core (services/updaterProcess.ts), that could not re-execute on this build (it predates re-execution, or+ * this build's master did not answer its probe). Such a master tells its core nothing about updates+ * (`HARNESSD_UPDATES`), and until something restarts it, a core that ran none would leave the machine on+ * this build for good, a broken one included. So this core starts the same updater, in a process of its+ * own (`cli.js __service updater`) and never in the core's: the core downloads no build. When it stages+ * one, the core hands over to its master (exit 75), as it did when the updater ran inside it, and that+ * master judges the new build as it always has. A new master re-executes on it and runs the updater itself.+ */++/** The updater's process, as the core watches it. */+export interface UpdaterChild {+ onMessage(listener: (message: unknown) => void): void+ onExit(listener: () => void): void+ kill(): void+}++/** How long after the updater's process ends it is started again: its checks are a minute apart anyway. */+export const UPDATER_RESTART_MS = 30_000++export interface UpdaterBesideDeps {+ /** Start `__service updater`, with what it must know of this core's master (`HARNESSD_JUDGES_SUPERSEDED`). */+ spawn(): UpdaterChild+ /** The updater staged `version`: hand over for it. */+ staged(version: string): void+ log(line: string): void+ setTimer?: (run: () => void, ms: number) => unknown+ clearTimer?: (timer: unknown) => void+}++/** Whether this core must run the updater beside itself: supervised by a master that does not say it runs one. */+export function needsUpdaterBeside(env: NodeJS.ProcessEnv, supervised: boolean, installedCopy: boolean, updatesOff: boolean): boolean {+ return supervised && env.HARNESSD_UPDATES !== 'master' && installedCopy && !updatesOff+}++const isStaged = (message: unknown): message is { type: 'harnessd:staged'; version: string } =>+ !!message && typeof message === 'object' && (message as { type?: unknown }).type === 'harnessd:staged'+ && typeof (message as { version?: unknown }).version === 'string'++/** Run the updater beside this core until it stages a build; returns a stop. */+export function startUpdaterBeside(deps: UpdaterBesideDeps): () => void {+ const setTimer = deps.setTimer ?? ((run, ms) => { const timer = setTimeout(run, ms); timer.unref?.(); return timer })+ const clearTimer = deps.clearTimer ?? ((timer) => clearTimeout(timer as ReturnType<typeof setTimeout>))+ let child: UpdaterChild | null = null+ let timer: unknown = null+ let done = false+ const start = (): void => {+ timer = null+ const started = deps.spawn()+ child = started+ started.onMessage((message) => {+ if (!isStaged(message) || done) return+ done = true+ deps.log(`[update] the updater staged ${message.version} — handing over to this core's master`)+ deps.staged(message.version)+ })+ started.onExit(() => {+ if (child !== started) return+ child = null+ if (done) return+ deps.log(`[update] the updater ended — starting it again in ${UPDATER_RESTART_MS / 1000} s`)+ timer = setTimer(start, UPDATER_RESTART_MS)+ })+ }+ deps.log('[update] this core\'s master runs no updater — running it beside this core')+ start()+ return () => {+ done = true+ if (timer !== null) clearTimer(timer)+ child?.kill()+ child = null+ }+}diff --git a/cli/src/harnessd/AGENTS.md b/cli/src/harnessd/AGENTS.mdindex a33808ad3..1abf98334 100644--- a/cli/src/harnessd/AGENTS.md+++ b/cli/src/harnessd/AGENTS.md@@ -38,3 +38,9 @@ It keeps the core and the services running, and nothing else.`<runtime>/node-…/libexec/harnessd/` — never `bin/`, which agent panes get on PATH — are made only forthe managed runtime's node, never a bare `harness` (the dial's `pgrep -x harness` means the app), andany failure runs the process as `node`. Children a process starts use `baseNode(process.execPath)`.+9. **Updates are the master's to run, never the core's.** The updater runs in a process the master starts+ (`UPDATER_HOST`, `../services/updaterProcess.ts`): it downloads, verifies, canaries and stages a build,+ then says so (`harnessd:staged`); the master asks the core to hand over (`harnessd:update`) and judges+ the new core on probation, as before. The master itself still makes no network call. Every core this+ master starts hears `HARNESSD_UPDATES=master`; a core under a master from before that runs the updater+ beside itself, in its own process (`../core/updaterBeside.ts`).diff --git a/cli/src/harnessd/coreLink.spec.ts b/cli/src/harnessd/coreLink.spec.tsindex ff3d661f0..738817187 100644--- a/cli/src/harnessd/coreLink.spec.ts+++ b/cli/src/harnessd/coreLink.spec.ts@@ -47,9 +47,13 @@ describe('connectToMaster', () => {link.startHeartbeat()const gone = vi.fn()link.onMasterGone(gone)+ const update = vi.fn()+ link.onUpdate(update)+ channel.say({ type: 'harnessd:update', version: '9.9.9' })channel.leave()vi.advanceTimersByTime(60_000)expect(gone).not.toHaveBeenCalled()+ expect(update).not.toHaveBeenCalled()expect(link.status()).toBeNull()link.close()}@@ -107,6 +111,27 @@ describe('connectToMaster', () => {expect(channel.sent).toEqual([{ type: 'harnessd:ready', safeMode: 'no tmux' }])})+ it('hears an update the master asks for, kept until something listens', async () => {+ const channel = new FakeChannel()+ const link = connectToMaster(channel, supervised)+ // Asked before start-up got as far as listening: not lost.+ channel.say({ type: 'harnessd:update', version: '9.9.8' })+ channel.say({ type: 'harnessd:update', version: '9.9.9' })+ const update = vi.fn()+ link.onUpdate(update)+ expect(update).not.toHaveBeenCalled()+ await Promise.resolve()+ expect(update.mock.calls).toEqual([['9.9.9']])+ channel.say({ type: 'harnessd:update', version: '9.9.10' })+ expect(update.mock.calls).toEqual([['9.9.9'], ['9.9.10']])+ expect(link.status()).toBeNull()+ // Nothing kept, nothing said to a listener that comes later.+ const later = vi.fn()+ link.onUpdate(later)+ await Promise.resolve()+ expect(later).not.toHaveBeenCalled()+ })+it('keeps what the master says about itself, and hears when the master goes', () => {const channel = new FakeChannel()const link = connectToMaster(channel, supervised)diff --git a/cli/src/harnessd/coreLink.ts b/cli/src/harnessd/coreLink.tsindex 897563f90..bc2f1b119 100644--- a/cli/src/harnessd/coreLink.ts+++ b/cli/src/harnessd/coreLink.ts@@ -74,6 +74,9 @@ export interface CoreLink {startHeartbeat(): void/** The master is gone. A core without one stops, so nothing is left holding the port. */onMasterGone(listener: () => void): void+ /** The master's updater staged a build: hand over for it (`harnessd:update`). Heard from the start, as+ * a staged update may arrive while start-up is still under way. */+ onUpdate(listener: (version: string) => void): void/** What the master last said about itself (restarts, the last exit), for `/api/status`. */status(): SupervisorStatus | nullclose(): void@@ -104,8 +107,17 @@ export function connectToMaster(if (!supervised) returntry { channel.send!(message) } catch { /* closing */ }}+ // The last update the master asked for, kept until something listens: the listener is set at start-up,+ // but a master can stage one before, and a request missed is an update never applied.+ let update: string | null = null+ let updateListener: ((version: string) => void) | null = nullif (supervised) {- channel.on('message', (message) => { if (isMasterMessage(message)) status = message.status })+ channel.on('message', (message) => {+ if (!isMasterMessage(message)) return+ if (message.type === 'harnessd:status') { status = message.status; return }+ if (updateListener) updateListener(message.version)+ else update = message.version+ })}// ⚠️ Listened for from the start and remembered, not from whenever the caller subscribes.// runForeground subscribes some 1,100 lines into start-up, long after this core has bound and begun@@ -144,6 +156,15 @@ export function connectToMaster(if (masterGone) queueMicrotask(listener)else goneListeners.push(listener)},+ onUpdate: (listener) => {+ if (!supervised) return+ updateListener = listener+ if (update !== null) {+ const version = update+ update = null+ queueMicrotask(() => listener(version))+ }+ },status: () => status,close: () => {if (heartbeat) clearInterval(heartbeat)diff --git a/cli/src/harnessd/master.spec.ts b/cli/src/harnessd/master.spec.tsindex c26d61c89..3771fe89e 100644--- a/cli/src/harnessd/master.spec.ts+++ b/cli/src/harnessd/master.spec.ts@@ -490,6 +490,45 @@ describe('runMaster', { timeout: 60_000 }, () => {expect(exits).toEqual([0])})+ it('runs the updater when told to, whatever HARNESSD_SERVICES names, and has the core hand over for what it stages', async () => {+ const pidFile = join(dir, 'adapter.pid')+ const seen = join(dir, 'seen')+ const script = join(dir, 'daemon.cjs')+ writeFileSync(script, `+ const { appendFileSync } = require('node:fs')+ const role = process.argv[2] === '__service' ? 'service:' + process.argv[3] : process.argv[2]+ const say = (what) => appendFileSync(${JSON.stringify(seen)}, JSON.stringify({ role, what, restarts: process.env.HARNESSD_RESTARTS }) + '\\n')+ say('started')+ if (role === '__run') {+ process.send({ type: 'harnessd:bound', protocol: 3, port: 1 }); process.send({ type: 'harnessd:ready' })+ process.on('message', (message) => { if (message.type === 'harnessd:update') { say('asked for ' + message.version); process.exit(75) } })+ }+ if (role === 'service:updater' && process.env.HARNESSD_RESTARTS === '0') setTimeout(() => process.send({ type: 'harnessd:staged', version: '9.9.9' }), 100)+ setInterval(() => process.send({ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1, loopDelayMs: 0 }), 50)+ process.on('SIGTERM', () => process.exit(0))+ `)+ const exits: number[] = []+ const signals = new Map<string, () => void>()+ const lines: string[] = []+ const log = vi.spyOn(console, 'log').mockImplementation((line: string) => { lines.push(line) })+ try {+ runMaster({+ nodePath: process.execPath, execArgv: [], scriptPath: script, pidFile, updater: true,+ restoreUpdate: () => {}, confirmUpdate: () => {},+ env: { ...process.env, HARNESSD_SERVICES: 'none', HARNESSD_UPDATE_PROBATION_MS: '50' },+ exit: (code) => exits.push(code), onSignal: (signal, listener) => signals.set(signal, listener),+ })+ const said = (): Array<Record<string, string>> => existsSync(seen) ? readFileSync(seen, 'utf8').trim().split('\n').map((line) => JSON.parse(line)) : []+ await until('the core to hand over and a new one to start', () => said().filter((line) => line.role === '__run' && line.what === 'started').length === 2)+ expect(said().filter((line) => line.role.startsWith('service:')).map((line) => line.role)).toEqual(['service:updater'])+ expect(said()).toContainEqual({ role: '__run', what: 'asked for 9.9.9', restarts: '0' })+ expect(lines.some((line) => line.endsWith('[harnessd] the updater staged 9.9.9 — asking the core to hand over'))).toBe(true)+ expect(lines.some((line) => line.endsWith('[harnessd] core exited (code 75) for an update — restarting'))).toBe(true)+ signals.get('SIGTERM')!()+ await until('the master to finish', () => exits.length > 0)+ } finally { log.mockRestore() }+ })+it('starts the services from the lean bundle when given one, and the core from the CLI entry', async () => {// The lean bundle cli.js carries (./leanBundle.ts): each service parses its own code, not the CLI's.const pidFile = join(dir, 'adapter.pid')diff --git a/cli/src/harnessd/master.ts b/cli/src/harnessd/master.tsindex 68c3c65d6..65879a574 100644--- a/cli/src/harnessd/master.ts+++ b/cli/src/harnessd/master.ts@@ -16,7 +16,7 @@ import type { MasterMessage } from './protocol.js'import {PROBE_ANSWER, RESUME_ENV, createReexec, decodeResume, fingerprint, readMarker, recoverFailedReexec, removeMarker, runProbe, writeMarker,} from './reexec.js'-import { SERVICE_HOSTS, ServiceSupervisor, serviceOptions, serviceProcessesEnv, serviceSpecs } from './services.js'+import { SERVICE_HOSTS, ServiceSupervisor, UPDATER_HOST, serviceOptions, serviceProcessesEnv, serviceSpecs } from './services.js'import {DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorDeps, type SupervisorOptions, type SupervisorStatus,} from './supervisor.js'@@ -67,6 +67,8 @@ export interface MasterConfig {/** The version of the update the bundle with this fingerprint is, when it is one no master kept or* rolled back (`selfUpdate.unjudgedUpdate`); null otherwise. Left out, never. */unjudgedUpdate?: (bundle: string | null) => string | null+ /** Run the updater (`UPDATER_HOST`): the installed copy, with updates on (masterProcess.ts). */+ updater?: boolean}export type Execve = (file: string, args: string[], env: NodeJS.ProcessEnv) => void@@ -295,7 +297,10 @@ export function runMaster(config: MasterConfig): Supervisor {folderFingerprint, exists: existsSync, sameBundle: () => bundle() === own, log,})const specs = serviceSpecs(env, SERVICE_HOSTS)- const services = new ServiceSupervisor(specs, {+ // The updater beside them, whatever HARNESSD_SERVICES says: the core neither routes to it nor runs it.+ const processes = config.updater ? [...specs, { name: 'updater', ...UPDATER_HOST }] : specs+ let supervisor: Supervisor | null = null+ const services = new ServiceSupervisor(processes, {spawnService: (spec, extra) => {const script = lean.scriptFor(spec.name)// One process for every service it hosts, each on its own link to the core (services/process.ts).@@ -311,6 +316,8 @@ export function runMaster(config: MasterConfig): Supervisor {setTimer: (run, ms) => setTimeout(run, ms),clearTimer: (timer) => clearTimeout(timer as ReturnType<typeof setTimeout>),log,+ // The updater staged a build: the core is asked to hand over for it, and the new core judged.+ staged: (version) => supervisor?.updateStaged(version),}, serviceOptions(env), { HARNESSD_SERVICE_TOKEN: token })const reexec = markerFile ? createReexec({own, current: bundle, nodePath: named('harnessd'), execArgv: config.execArgv, scriptPath: config.scriptPath, env, pid: process.pid,@@ -324,7 +331,7 @@ export function runMaster(config: MasterConfig): Supervisor {now: () => Date.now(),log,}) : null- const supervisor = new Supervisor({+ supervisor = new Supervisor({spawnCore: (extra) => coreHandle(spawn(named('harnessd-core'), [...execArgv, config.scriptPath, '__run'], {// Told which services this master runs, so it routes to exactly those and runs the rest itself.env: { ...env, ...extra, HARNESSD_SERVICE_TOKEN: token, ...serviceProcessesEnv(specs) },diff --git a/cli/src/harnessd/protocol.spec.ts b/cli/src/harnessd/protocol.spec.tsindex 4c4c654c5..b869318bd 100644--- a/cli/src/harnessd/protocol.spec.ts+++ b/cli/src/harnessd/protocol.spec.ts@@ -1,5 +1,5 @@import { describe, expect, it } from 'vitest'-import { HARNESSD_PROTOCOL, isCoreMessage, isMasterMessage } from './protocol.js'+import { HARNESSD_PROTOCOL, isCoreMessage, isMasterMessage, isUpdaterMessage } from './protocol.js'describe('the spawn channel protocol', () => {it.each([@@ -28,9 +28,23 @@ describe('the spawn channel protocol', () => {[{ type: 'harnessd:status', status: null }, false],[{ type: 'harnessd:status', status: 'running' }, false],[{ type: 'harnessd:bound', protocol: 1, port: 1 }, false],+ [{ type: 'harnessd:update', version: '1.2.3' }, true],+ [{ type: 'harnessd:update', version: '' }, false],+ [{ type: 'harnessd:update' }, false],[undefined, false],[7, false],])('a master message %j is %s', (message, valid) => {expect(isMasterMessage(message)).toBe(valid)})++ it.each([+ [{ type: 'harnessd:staged', version: '1.2.3' }, true],+ [{ type: 'harnessd:staged', version: '' }, false],+ [{ type: 'harnessd:staged', version: 3 }, false],+ [{ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1 }, false],+ [null, false],+ ['harnessd:staged', false],+ ])('an updater message %j is %s', (message, valid) => {+ expect(isUpdaterMessage(message)).toBe(valid)+ })})diff --git a/cli/src/harnessd/protocol.ts b/cli/src/harnessd/protocol.tsindex 3a2af1c22..87ea4e896 100644--- a/cli/src/harnessd/protocol.ts+++ b/cli/src/harnessd/protocol.ts@@ -2,8 +2,9 @@* What harnessd's master and its core say to each other over the spawn channel (Node IPC).** Small on purpose. The core says when it is bound, when it is ready and that it is alive; the master- * tells it its own status. Everything else the daemon does goes through the core's local API, never- * through here.+ * tells it its own status, and when an update is staged, to hand over. The updater, a process the master+ * runs (services/updaterProcess.ts), says when it has staged one. Everything else the daemon does goes+ * through the core's local API, never through here.** Messages are only ever added, and a side ignores what it does not know: during an update an older* master supervises a newer core, and after a rollback a newer master supervises an older one. Each@@ -12,8 +13,9 @@*/import type { SupervisorStatus } from './supervisor.js'-/** 2: `ready`, the event-loop delay on the heartbeat, and the exit-code contract below. 3: `want`. 4: `want`- * for the devices' process too, which runs only once there is a device (./services.ts `askedSince`). */+/** 2: `ready`, the event-loop delay on the heartbeat, and the exit-code contract below. 3: `want`, and the+ * master's updater: the master tells its core to hand over for a build its updater staged (`harnessd:update`).+ * 4: `want` for the devices' process too, which runs only once there is a device (./services.ts `askedSince`). */export const HARNESSD_PROTOCOL = 4/** The command a bundle's master answers its probe on, and what it answers. */@@ -59,6 +61,21 @@ export type CoreMessage =export type MasterMessage =| { type: 'harnessd:status'; status: SupervisorStatus }+ /** The updater staged `version`: hand over for it, as a core whose own updater staged it did (exit 75). */+ | { type: 'harnessd:update'; version: string }++/** What the updater says to the master: it staged `version` on disk. */+export type UpdaterMessage = { type: 'harnessd:staged'; version: string }++/** The exit code a service uses to be started again at once, on the bundle now on disk, without counting+ * as a crash: the updater, once it has staged a build, so that it next runs as that build. */+export const SERVICE_EXIT_RESTART = 75++export function isUpdaterMessage(value: unknown): value is UpdaterMessage {+ if (!value || typeof value !== 'object') return false+ const message = value as Record<string, unknown>+ return message.type === 'harnessd:staged' && typeof message.version === 'string' && message.version.length > 0+}export function isCoreMessage(value: unknown): value is CoreMessage {if (!value || typeof value !== 'object') return false@@ -81,5 +98,6 @@ export function isCoreMessage(value: unknown): value is CoreMessage {export function isMasterMessage(value: unknown): value is MasterMessage {if (!value || typeof value !== 'object') return falseconst message = value as Record<string, unknown>+ if (message.type === 'harnessd:update') return typeof message.version === 'string' && message.version.length > 0return message.type === 'harnessd:status' && !!message.status && typeof message.status === 'object'}diff --git a/cli/src/harnessd/services.spec.ts b/cli/src/harnessd/services.spec.tsindex 11657696e..2269c38b3 100644--- a/cli/src/harnessd/services.spec.ts+++ b/cli/src/harnessd/services.spec.ts@@ -1,6 +1,6 @@import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'import {- DEFAULT_SERVICE_OPTIONS, KNOWN_SERVICES, SERVICE_HOSTS, SERVICE_PROCESSES_ENV, ServiceSupervisor, serviceOptions, serviceProcessesEnv, serviceSpecs, servicesTheMasterRuns,+ DEFAULT_SERVICE_OPTIONS, KNOWN_SERVICES, SERVICE_HOSTS, SERVICE_PROCESSES_ENV, ServiceSupervisor, UPDATER_HOST, serviceOptions, serviceProcessesEnv, serviceSpecs, servicesTheMasterRuns,type ServiceSpec, type ServiceSupervisorOptions,} from './services.js'import type { CoreHandle } from './supervisor.js'@@ -40,8 +40,10 @@ describe('ServiceSupervisor', () => {let children: FakeService[]let lines: string[]const latest = (name = 'search') => children.filter((child) => child.spec.name === name).at(-1)!- const make = (specs: ServiceSpec[] = [search], overrides: Partial<ServiceSupervisorOptions> = {}, env: Record<string, string> = {}) =>+ const staged: string[] = []+ const make = (specs: ServiceSpec[] = [search], overrides: Partial<ServiceSupervisorOptions> = {}, env: Record<string, string> = {}, tell = true) =>new ServiceSupervisor(specs, {+ ...(tell ? { staged: (version: string) => { staged.push(version) } } : {}),spawnService: (spec, extra) => {const child = new FakeService(2000 + children.length, spec, extra)children.push(child)@@ -75,6 +77,34 @@ describe('ServiceSupervisor', () => {expect(supervisor.status()[0].state).toBe('running')})+ it('passes on what the updater staged, and starts it again at once when it asks: not a crash', () => {+ const updater: ServiceSpec = { name: 'updater', ...UPDATER_HOST }+ staged.length = 0+ const supervisor = make([updater], { parkCrashes: 2 })+ supervisor.start()+ latest('updater').beat()+ latest('updater').say({ type: 'harnessd:staged', version: '9.9.9' })+ expect(staged).toEqual(['9.9.9'])+ expect(lines).toContain('[harnessd] service updater staged 9.9.9')+ for (let i = 0; i < 3; i++) {+ latest('updater').exit(75)+ vi.advanceTimersByTime(0)+ }+ expect(children).toHaveLength(4)+ expect(supervisor.status()[0]).toMatchObject({ state: 'restarting', restarts: 3, lastExit: 'code 75', lastExitReason: 'restart' })+ expect(lines).toContain('[harnessd] service updater asked to start again (code 75) — restarting')+ // Killed for hanging, its 75 is no request.+ latest('updater').beat()+ vi.advanceTimersByTime(options.heartbeatTimeoutMs * 2)+ latest('updater').exit(75)+ expect(supervisor.status()[0].lastExitReason).toBe('hung')+ // A master that was not told how to pass a staging on only says it.+ const quiet = make([updater], {}, {}, false)+ quiet.start()+ latest('updater').say({ type: 'harnessd:staged', version: '9.9.10' })+ expect(staged).toEqual(['9.9.9'])+ })+it('restarts a crashed service with a backoff that doubles, caps, and is forgiven after a long run', () => {const supervisor = make([search], { parkCrashes: 99 })supervisor.start()diff --git a/cli/src/harnessd/services.ts b/cli/src/harnessd/services.tsindex 103ad9f9a..42065ae6f 100644--- a/cli/src/harnessd/services.ts+++ b/cli/src/harnessd/services.ts@@ -12,7 +12,7 @@* bind and no update to prove; it beats, or it is restarted. Everything that touches the operating* system is injected, so every decision here is tested without one.*/-import { heartbeatGraceMs, isCoreMessage } from './protocol.js'+import { SERVICE_EXIT_RESTART, heartbeatGraceMs, isCoreMessage, isUpdaterMessage } from './protocol.js'import type { CoreHandle } from './supervisor.js'export interface ServiceSpec {@@ -42,6 +42,8 @@ export interface ServiceSupervisorDeps {setTimer(run: () => void, ms: number): unknownclearTimer(timer: unknown): voidlog(line: string): void+ /** The updater staged `version` on disk (`harnessd:staged`): the core is to hand over for it. */+ staged?(version: string): void}export interface ServiceSupervisorOptions {@@ -77,7 +79,7 @@ export const DEFAULT_SERVICE_OPTIONS: ServiceSupervisorOptions = {/** `off`: an experiment's process that no one has asked for yet, and so was never started. */export type ServiceState = 'off' | 'starting' | 'running' | 'restarting' | 'parked' | 'stopping' | 'stopped'-export type ServiceExitReason = 'crashed' | 'hung' | 'memory' | 'stopped'+export type ServiceExitReason = 'crashed' | 'hung' | 'memory' | 'stopped' | 'restart'export interface ServiceStatus {name: string@@ -178,6 +180,11 @@ class Service {}private onMessage(child: CoreHandle, message: unknown): void {+ if (isUpdaterMessage(message)) {+ this.deps.log(`[harnessd] service ${this.spec.name} staged ${message.version}`)+ this.deps.staged?.(message.version)+ return+ }if (!isCoreMessage(message) || message.type !== 'harnessd:heartbeat') returnthis.watchHeartbeat(child)if (this.state === 'starting' || this.state === 'restarting') this.setState('running')@@ -209,6 +216,16 @@ class Service {this.stopped()return}+ if (code === SERVICE_EXIT_RESTART && !this.killReason) {+ // Asked for: the updater, once it has staged a build, so that it next runs as that build (a lean+ // bundle that no longer matches cli.js is not used, ./leanServices.ts). Not a crash, and not delayed.+ this.lastExitReason = 'restart'+ this.restarts+++ this.deps.log(`[harnessd] service ${this.spec.name} asked to start again (${exit}) — restarting`)+ this.setState('restarting')+ this.armTimer('restartTimer', () => { this.restartTimer = null; this.start() }, 0)+ return+ }const reason = this.killReason ?? 'crashed'this.lastExitReason = reasonthis.restarts++@@ -377,6 +394,16 @@ export const SERVICE_HOSTS: Readonly<Record<string, ServiceHostSpec>> = {/** Every service this build can run outside the core's process: what `HARNESSD_SERVICES` names. */export const KNOWN_SERVICES: readonly string[] = Object.values(SERVICE_HOSTS).flatMap((host) => host.services)+/**+ * The updater's process (services/updaterProcess.ts): checks for a newer build of the CLI and of hn, and+ * downloads, verifies, canaries and stages it, then tells the master, which has the core hand over. Its own+ * process and never the core's: the core never downloads a build, and a core that cannot start (safe mode)+ * or a host whose services keep crashing must not stop the fix from arriving. Not one of `SERVICE_HOSTS`:+ * the core neither routes to it nor runs it, and `HARNESSD_SERVICES` does not turn it off; the master runs+ * it when it runs the installed copy with updates on (masterProcess.ts). It holds one download at a time.+ */+export const UPDATER_HOST: ServiceHostSpec = { services: ['updater'], heapLimitMiB: 256, rssLimitMiB: 512 }+/** Service timings from the environment (for tests and support); anything unset or invalid keeps its default. */export function serviceOptions(env: NodeJS.ProcessEnv): ServiceSupervisorOptions {const read = (name: string, fallback: number, min: number): number => {diff --git a/cli/src/harnessd/supervisor.spec.ts b/cli/src/harnessd/supervisor.spec.tsindex f918cf657..1aeff287a 100644--- a/cli/src/harnessd/supervisor.spec.ts+++ b/cli/src/harnessd/supervisor.spec.ts@@ -1,7 +1,7 @@import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'import { CORE_EXIT_STOP, CORE_EXIT_UPDATE, HARNESSD_PROTOCOL, type MasterMessage } from './protocol.js'import {- DEFAULT_SUPERVISOR_OPTIONS, Supervisor, namesTheDisk, type CoreHandle, type ReexecOutcome, type ResumeState, type SupervisorDeps, type SupervisorOptions,+ DEFAULT_SUPERVISOR_OPTIONS, HAND_OVER_GRACE_MS, Supervisor, namesTheDisk, type CoreHandle, type ReexecOutcome, type ResumeState, type SupervisorDeps, type SupervisorOptions,type SupervisorStatus,} from './supervisor.js'@@ -95,7 +95,7 @@ describe('Supervisor', () => {supervisor.start()supervisor.start()expect(cores).toHaveLength(1)- expect(core().env).toEqual({ HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: '0', HARNESSD_WATCHDOG_MS: '6000', HARNESSD_JUDGES_SUPERSEDED: '1' })+ expect(core().env).toEqual({ HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: '0', HARNESSD_WATCHDOG_MS: '6000', HARNESSD_JUDGES_SUPERSEDED: '1', HARNESSD_UPDATES: 'master' })expect(supervisor.status()).toMatchObject({ state: 'starting', corePid: 1000 })core().bind()expect(calls).toEqual(['claim'])@@ -111,7 +111,7 @@ describe('Supervisor', () => {expect(lines.at(-1)).toBe('[harnessd] core ready (pid 1000)')// Every change is written for `harness status`, and told to the bound core.expect(statuses.map((status) => status.state)).toEqual(['starting', 'listening', 'running'])- expect(core().sent.map((message) => message.status.state)).toEqual(['listening', 'running'])+ expect(core().sent.map((message) => message.type === 'harnessd:status' ? message.status.state : message.type)).toEqual(['listening', 'running'])core().ready()expect(statuses).toHaveLength(3)})@@ -161,7 +161,7 @@ describe('Supervisor', () => {expect(cores).toHaveLength(1)vi.advanceTimersByTime(1)expect(cores).toHaveLength(2)- expect(core().env).toEqual({ HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: '1', HARNESSD_WATCHDOG_MS: '6000', HARNESSD_JUDGES_SUPERSEDED: '1', HARNESSD_LAST_EXIT: 'signal SIGKILL' })+ expect(core().env).toEqual({ HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: '1', HARNESSD_WATCHDOG_MS: '6000', HARNESSD_JUDGES_SUPERSEDED: '1', HARNESSD_UPDATES: 'master', HARNESSD_LAST_EXIT: 'signal SIGKILL' })expect(lines.at(-1)).toBe('[harnessd] core started (pid ?) · restart 1')core().bind()expect(lines.at(-1)).toBe(`[harnessd] core bound (pid ?, protocol ${HARNESSD_PROTOCOL})`)@@ -380,6 +380,94 @@ describe('Supervisor', () => {})})+ describe('an update the updater staged', () => {+ it('asks the running core to hand over, once, and judges its exit as the update\'s', () => {+ const supervisor = make()+ supervisor.start()+ core().up()+ supervisor.updateStaged('9.9.9')+ supervisor.updateStaged('9.9.9')+ expect(core().sent.filter((message) => message.type === 'harnessd:update')).toEqual([{ type: 'harnessd:update', version: '9.9.9' }])+ expect(lines.at(-1)).toBe('[harnessd] the updater staged 9.9.9 — asking the core to hand over')+ core().exit(CORE_EXIT_UPDATE)+ vi.advanceTimersByTime(0)+ expect(cores).toHaveLength(2)+ expect(lines).toContain('[harnessd] core exited (code 75) for an update — restarting')+ core().up()+ // The grace it had to hand over in ended with it.+ live(HAND_OVER_GRACE_MS)+ expect(core().kills).toEqual([])+ expect(calls).toContain('confirm')+ })++ it('stops a core that does not hand over within its grace, and counts that exit as the update\'s', () => {+ const supervisor = make()+ supervisor.start()+ core().up()+ supervisor.updateStaged('9.9.9')+ live(HAND_OVER_GRACE_MS - 1_000)+ expect(core().kills).toEqual([])+ live(1_000)+ expect(core().kills).toEqual(['SIGTERM'])+ expect(lines).toContain('[harnessd] core did not hand over for 9.9.9 within 45000 ms — stopping it')+ core().exit(0)+ vi.advanceTimersByTime(0)+ expect(supervisor.status()).toMatchObject({ lastExitReason: 'update', restarts: 1 })+ expect(cores).toHaveLength(2)+ core().up()+ live(options.updateProbationMs)+ expect(calls).toContain('confirm')+ })++ it('with no core running, starts the next one on the new bundle at once, on probation, with a clean slate', () => {+ const supervisor = make()+ supervisor.start()+ // Into safe mode: the crashes a fix would end.+ for (let i = 0; i < options.crashLoopCrashes; i++) { core().up(); crash() }+ expect(core().env.HARNESSD_SAFE_MODE).toBe('crash-loop')+ core().exit(1)+ expect(supervisor.status().state).toBe('restarting')+ supervisor.updateStaged('9.9.9')+ expect(lines.at(-1)).toBe('[harnessd] the updater staged 9.9.9 — the next core starts on it')+ vi.advanceTimersByTime(0)+ expect(core().env.HARNESSD_SAFE_MODE).toBeUndefined()+ core().up()+ core().exit(1)+ vi.advanceTimersByTime(0)+ // On probation: its crash rolls the update back.+ expect(calls).toContain('restore')+ })++ it('judges the first core when a build is staged before the master starts one', () => {+ const supervisor = make()+ supervisor.updateStaged('9.9.9')+ expect(cores).toHaveLength(0)+ supervisor.start()+ core().up()+ core().exit(1)+ expect(calls).toContain('restore')+ })++ it('leaves a core being started to start on the bundle on disk, and ignores a staging while stopping', () => {+ const deciding: Array<(outcome: ReexecOutcome) => void> = []+ const supervisor = make({}, { reexec: (_state, proceed) => { deciding.push(proceed) } })+ supervisor.start()+ expect(cores).toHaveLength(0)+ supervisor.updateStaged('9.9.9')+ expect(cores).toHaveLength(0)+ deciding[0]('kept')+ expect(cores).toHaveLength(1)+ core().up()+ core().exit(1)+ vi.advanceTimersByTime(0)+ expect(calls).toContain('restore')+ supervisor.stop('stopping')+ const before = lines.length+ supervisor.updateStaged('9.9.10')+ expect(lines.length).toBe(before)+ })+ })+it('stops the core and itself on request, and kills the core on a second request', () => {const supervisor = make()supervisor.start()diff --git a/cli/src/harnessd/supervisor.ts b/cli/src/harnessd/supervisor.tsindex e34e0c698..ae61b0c67 100644--- a/cli/src/harnessd/supervisor.ts+++ b/cli/src/harnessd/supervisor.ts@@ -179,7 +179,14 @@ export interface SupervisorStatus {reexecs: number}-type TimerName = 'bindTimer' | 'readyTimer' | 'heartbeatTimer' | 'killTimer' | 'restartTimer' | 'probationTimer'+type TimerName = 'bindTimer' | 'readyTimer' | 'heartbeatTimer' | 'killTimer' | 'restartTimer' | 'probationTimer' | 'handOverTimer'++/**+ * How long a core asked to hand over for an update (`harnessd:update`) has before the master stops it all+ * the same: its teardown gives up after 15 s (core/updateHandoff.ts) and a grace of 1 s follows, so this+ * is a core that did not hear, or did not act. Its exit then counts as the update's.+ */+export const HAND_OVER_GRACE_MS = 45_000const describeExit = (code: number | null, signal: NodeJS.Signals | null): string =>signal ? `signal ${signal}` : `code ${code}`@@ -227,6 +234,7 @@ export class Supervisor {private heartbeatTimer: unknown = nullprivate killTimer: unknown = nullprivate restartTimer: unknown = null+ private handOverTimer: unknown = nullprivate readonly masterVersion: string | nullprivate reexecs = 0private unproven = 0@@ -308,6 +316,37 @@ export class Supervisor {})}+ /**+ * The updater (services/updaterProcess.ts) staged `version` on disk. The core used to stage it itself+ * and exit 75; now the master asks it to (`harnessd:update`), and its exit is the update's, judged as+ * before. A core that does not leave within `HAND_OVER_GRACE_MS` is stopped, and that exit counts as the+ * update's too. With no core running (a restart's backoff, safe mode's), the next one starts on the new+ * bundle at once, on probation, with a clean slate: whatever was crashing may be what it fixes.+ */+ updateStaged(version: string): void {+ if (this.state === 'stopping' || this.state === 'stopped') return+ const core = this.core+ if (!core) {+ this.update = 'pending'+ this.crashes = []+ this.safeMode = null+ this.deps.log(`[harnessd] the updater staged ${version} — the next core starts on it`)+ // Waiting out a backoff: no longer. Otherwise the next core is already being started (a+ // re-execution deciding), and starts on the bundle on disk.+ if (this.restartTimer !== null) this.scheduleSpawn(0)+ return+ }+ if (this.handOverTimer !== null) return+ this.deps.log(`[harnessd] the updater staged ${version} — asking the core to hand over`)+ core.send({ type: 'harnessd:update', version })+ this.armTimer('handOverTimer', () => {+ this.handOverTimer = null+ this.deps.log(`[harnessd] core did not hand over for ${version} within ${HAND_OVER_GRACE_MS} ms — stopping it`)+ this.killReason = 'update'+ this.end('restart', 'SIGTERM')+ }, HAND_OVER_GRACE_MS)+ }+/** Stop the core and the master. A second call while stopping kills the core outright. */stop(reason: string): void {if (this.state === 'stopped') return@@ -346,9 +385,11 @@ export class Supervisor {// HARNESSD_JUDGES_SUPERSEDED: this master judges a build its core on probation stages on its own// (#807). A master from before rolls back on any exit during probation, the update's own included,// and its core must not stage then: see `stageWhileJudged` in lib/selfUpdate.ts.+ // HARNESSD_UPDATES: updates are this master's to run (services/updaterProcess.ts), not the core's. A core+ // under a master from before it starts the updater beside itself (core/updaterBeside.ts).const env: Record<string, string> = {HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: String(this.restarts), HARNESSD_WATCHDOG_MS: String(this.options.heartbeatTimeoutMs),- HARNESSD_JUDGES_SUPERSEDED: '1',+ HARNESSD_JUDGES_SUPERSEDED: '1', HARNESSD_UPDATES: 'master',}if (this.lastExit) env.HARNESSD_LAST_EXIT = this.lastExitif (this.safeMode) env.HARNESSD_SAFE_MODE = this.safeMode@@ -460,7 +501,7 @@ export class Supervisor {const exit = describeExit(code, signal)const reason = this.killReason ?? (code === CORE_EXIT_UPDATE ? 'update' : 'crashed')const wasSafe = this.safeMode ?? this.coreSafeMode- for (const timer of ['bindTimer', 'readyTimer', 'heartbeatTimer', 'killTimer', 'probationTimer'] as const) this.clearTimer(timer)+ for (const timer of ['bindTimer', 'readyTimer', 'heartbeatTimer', 'killTimer', 'probationTimer', 'handOverTimer'] as const) this.clearTimer(timer)this.core = nullthis.bound = falsethis.coreSafeMode = nulldiff --git a/cli/src/leanEntry.spec.ts b/cli/src/leanEntry.spec.tsindex 5b882152c..e29e91268 100644--- a/cli/src/leanEntry.spec.ts+++ b/cli/src/leanEntry.spec.ts@@ -16,9 +16,10 @@ import { PROBE_ANSWER } from './harnessd/reexec.js'import { KNOWN_SERVICES, SERVICE_HOSTS } from './harnessd/services.js'const CLI_ROOT = fileURLToPath(new URL('..', import.meta.url))-const ROLES = ['master', ...KNOWN_SERVICES]+// The updater, beside the services the core knows: the master runs it (harnessd/services.ts `UPDATER_HOST`).+const ROLES = ['master', ...KNOWN_SERVICES, 'updater']const RUNNER: Record<string, string> = { master: 'masterProcess' }-for (const name of KNOWN_SERVICES) RUNNER[name] = `${name}Process`+for (const name of [...KNOWN_SERVICES, 'updater']) RUNNER[name] = `${name}Process`describe('the lean bundle a release carries', () => {let scratch = ''@@ -58,10 +59,10 @@ describe('the lean bundle a release carries', () => {}})- it('brings no zod to the master, search or the edge host', () => {+ it('brings no zod to the master, search, the updater or the edge host', () => {expect(hasZod(new Set(files.keys())), 'the bundle still holds zod, for the services whose own code uses it').toBe(true)// The edge host's services share one process: zod in any of them would be in all of them.- for (const role of ['master', 'search', ...SERVICE_HOSTS.edge.services]) expect(hasZod(loads(role)), role).toBe(false)+ for (const role of ['master', 'search', 'updater', ...SERVICE_HOSTS.edge.services]) expect(hasZod(loads(role)), role).toBe(false)})it('passes the release script\'s check, which a lean bundle that breaks a process does not', async () => {diff --git a/cli/src/lib/daemonSafeMode.spec.ts b/cli/src/lib/daemonSafeMode.spec.tsindex d08f0f2fe..e45684233 100644--- a/cli/src/lib/daemonSafeMode.spec.ts+++ b/cli/src/lib/daemonSafeMode.spec.ts@@ -3,8 +3,8 @@ import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'import { tmpdir } from 'node:os'import { join } from 'node:path'import {- clearSafeModeMarker, readSafeModeMarker, runBootHandoff, safeModeDisposition,- safeModeFile, safeModeStatusBody, SafeModeRequest, writeSafeModeMarker, type BootHandoffDeps,+ clearSafeModeMarker, readSafeModeMarker, safeModeDisposition,+ safeModeFile, safeModeStatusBody, SafeModeRequest, writeSafeModeMarker,} from './daemonSafeMode.js'let dir = ''@@ -80,50 +80,3 @@ describe('the safe-mode marker', () => {expect(readFileSync(safeModeFile(dir), 'utf-8')).toBe('{ not json')})})--describe('runBootHandoff', () => {- function harness(over: Partial<BootHandoffDeps> = {}) {- const calls: string[] = []- let exited: number | null = null- const deps: BootHandoffDeps = {- closeServer: () => calls.push('close'),- probeMaster: () => { calls.push('probe'); return null },- rollBack: () => calls.push('roll back'),- startMaster: () => calls.push('start the master'),- exit: ((code: number) => { calls.push(`exit:${code}`); exited = code; return undefined as never }),- log: () => {},- ...over,- }- return { deps, calls, exited: () => exited }- }-- it('probes the new build\'s master, releases the port, then starts the master on the new build, and leaves', () => {- const h = harness()- runBootHandoff('0.3.5', '0.3.6', h.deps)- expect(h.calls).toEqual(['probe', 'close', 'start the master', 'exit:0'])- })-- it('rolls a build whose master does not answer back, and starts this build\'s master instead', () => {- const lines: string[] = []- const h = harness({ probeMaster: () => 'harnessd-probe failed: a bad build', log: (m) => lines.push(m) })- runBootHandoff('0.3.5', '0.3.6', h.deps)- expect(h.calls).toEqual(['roll back', 'close', 'start the master', 'exit:0'])- expect(lines[1]).toBe('[update] 0.3.6\'s master did not answer its probe (harnessd-probe failed: a bad build) — rolled back; 0.3.5 goes on under a master of its own')- })-- it('has already exited by the time it returns — the property that makes two daemons impossible', () => {- // Nothing is awaited, so the half-built boot cannot interleave between closing the port and the- // exit, and can never reach the code that would bind the port the successor is taking.- const h = harness()- runBootHandoff('0.3.5', '0.3.6', h.deps)- expect(h.exited()).toBe(0) // no await, no tick: true immediately after the call returns- })-- it('says what it hands over, and to whom', () => {- const lines: string[] = []- const h = harness({ log: (m) => lines.push(m) })- runBootHandoff('0.3.5', '0.3.6', h.deps)- expect(lines[0]).toContain('0.3.5 → 0.3.6')- expect(lines[0]).toContain('harnessd\'s master')- })-})diff --git a/cli/src/lib/daemonSafeMode.ts b/cli/src/lib/daemonSafeMode.tsindex 3962be854..85f090499 100644--- a/cli/src/lib/daemonSafeMode.ts+++ b/cli/src/lib/daemonSafeMode.ts@@ -111,50 +111,6 @@ export function clearSafeModeMarker(dataDir: string): void {try { rmSync(safeModeFile(dataDir), { force: true }) } catch { /* ignore */ }}-export interface BootHandoffDeps {- /** The bound control port, if start-up ever got that far. */- closeServer: () => void- /** Whether the staged bundle's master answers its probe, synchronously: null when it does, why not otherwise. */- probeMaster: () => string | null- /** Put the build before back (`selfUpdate.restore`), remembering this one as rejected. */- rollBack: () => void- /** Start harnessd's master on the bundle now on disk, named in the pid file: it starts the core, and- * judges the update. */- startMaster: () => void- exit: (code: number) => never- log: (message: string) => void-}--/**- * Hand the machine to a newer build without finishing start-up.- *- * SYNCHRONOUS END TO END, and that is the whole safety argument: never awaiting means the half-built- * boot cannot interleave between the port closing and the exit, so it can never reach the code that- * would bind the port the successor is about to take. Two daemons are impossible by construction.- * That is also why it does not supervise the child the way a normal update restart does — waiting- * would leave this process running alongside the new one for up to a minute, both reconciling tmux- * and writing the registry.- *- * It spawns rather than merely exiting because nothing supervises this core: on a machine with no- * desktop app nothing else would ever start the successor. What it spawns is harnessd's master, which- * judges the update its core starts on; the core it once spawned ran the update unjudged. The staged- * bundle's master is probed first, as a master asks before it re-executes on a new bundle, and- * synchronously like the rest: one that cannot start would leave nothing running and nothing to judge, so- * the build before goes back and its master is started instead.- */-export function runBootHandoff(from: string, to: string, deps: BootHandoffDeps): void {- deps.log(`[update] ${from} → ${to} staged during start-up — handing off to harnessd's master without finishing boot`)- const refused = deps.probeMaster()- if (refused !== null) {- deps.log(`[update] ${to}'s master did not answer its probe (${refused}) — rolled back; ${from} goes on under a master of its own`)- deps.rollBack()- }- deps.closeServer()- deps.startMaster()- deps.log('[update] boot handoff · this process is leaving')- deps.exit(0)-}-/** The marker, or null — including for one left behind by a process that is no longer running. */export function readSafeModeMarker(dataDir: string,diff --git a/cli/src/lib/installedCopy.spec.ts b/cli/src/lib/installedCopy.spec.tsnew file mode 100644index 000000000..3d5eca098--- /dev/null+++ b/cli/src/lib/installedCopy.spec.ts@@ -0,0 +1,27 @@+import { linkSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'+import { tmpdir } from 'node:os'+import { join } from 'node:path'+import { afterEach, beforeEach, describe, expect, it } from 'vitest'+import { isInstalledCopy } from './installedCopy.js'++describe('the installed copy', () => {+ let dir = ''+ beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'installed-copy-')) })+ afterEach(() => { rmSync(dir, { recursive: true, force: true }) })++ it('is cli.js in the CLI folder, by inode, under any name that reaches it', () => {+ writeFileSync(join(dir, 'cli.js'), 'cli')+ symlinkSync(join(dir, 'cli.js'), join(dir, 'link.js'))+ linkSync(join(dir, 'cli.js'), join(dir, 'hard.js'))+ writeFileSync(join(dir, 'other.js'), 'other')+ expect(isInstalledCopy(join(dir, 'cli.js'), dir)).toBe(true)+ expect(isInstalledCopy(join(dir, 'link.js'), dir)).toBe(true)+ expect(isInstalledCopy(join(dir, 'hard.js'), dir)).toBe(true)+ expect(isInstalledCopy(join(dir, 'other.js'), dir)).toBe(false)+ })++ it('is matched by path when it cannot be read', () => {+ expect(isInstalledCopy(join(dir, 'cli.js'), dir)).toBe(true)+ expect(isInstalledCopy(join(dir, 'src', 'cli.ts'), dir)).toBe(false)+ })+})diff --git a/cli/src/lib/installedCopy.ts b/cli/src/lib/installedCopy.tsnew file mode 100644index 000000000..5945f8823--- /dev/null+++ b/cli/src/lib/installedCopy.ts@@ -0,0 +1,13 @@+/**+ * Whether a script is the installed copy of the CLI (`~/.harness/cli/cli.js`), the one the updater+ * replaces. A dev or repo run (`tsx`, or `node dist/cli.js` from a checkout) never updates itself: it would+ * swap the published bundle into ~/.harness/cli and restart, hijacking the version being developed. Matched+ * by inode, so a symlink or a realpath does not fool it; by path when either cannot be read.+ */+import { statSync } from 'node:fs'+import { join } from 'node:path'++export function isInstalledCopy(scriptPath: string, cliDir: string): boolean {+ const installed = join(cliDir, 'cli.js')+ try { return statSync(scriptPath).ino === statSync(installed).ino } catch { return scriptPath === installed }+}diff --git a/cli/src/lib/selfUpdate.spec.ts b/cli/src/lib/selfUpdate.spec.tsindex 41d4503f2..218736d39 100644--- a/cli/src/lib/selfUpdate.spec.ts+++ b/cli/src/lib/selfUpdate.spec.ts@@ -206,8 +206,8 @@ describe('startSelfUpdater', () => {it('is already stopped when onStaged runs — a handler that defers loses the updater for good', async () => {// `done = true` and `stop()` happen BEFORE `onStaged`, so a handler that returns without handing- // the machine over leaves no timer and no way back. This is why the daemon's boot-time handler- // always takes over and exits rather than waiting for start-up to finish (`runBootHandoff`).+ // the machine over leaves no timer and no way back. This is why the updater's process, once it has+ // staged, exits to be started again as the new build (services/updaterProcess.ts).serveUpdate()const dir = tempDir()let staged = 0diff --git a/cli/src/masterProcess.spec.ts b/cli/src/masterProcess.spec.tsindex 65c6dd379..abfe207a1 100644--- a/cli/src/masterProcess.spec.ts+++ b/cli/src/masterProcess.spec.ts@@ -2,6 +2,7 @@ import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:import { tmpdir } from 'node:os'import { join } from 'node:path'import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'+import { env } from './config/env.js'import { PROBE_ANSWER } from './harnessd/reexec.js'import { LEAN_ENTRY, leanFingerprint, type LeanBundle } from './harnessd/leanBundle.js'@@ -167,6 +168,11 @@ describe('starting the master', () => {const config = runMaster.mock.calls[0][0] as Record<string, unknown> & { restoreUpdate(): void; confirmUpdate(): void }expect(config).toMatchObject({ scriptPath: '/cli/cli.js', serviceScriptPath: '/lean.mjs', bundleFingerprint: 'f', nodePath: process.execPath })expect(String(config.reexecMarkerFile)).toMatch(/harnessd-reexec\.json$/)+ // Not the installed copy: no updater. The installed one runs it.+ expect(config.updater).toBe(false)+ runMaster.mockClear()+ startMaster({ scriptPath: join(env.ADAPTER_CLI_DIR, 'cli.js') })+ expect((runMaster.mock.calls[0][0] as { updater: boolean }).updater).toBe(true)expect(LEAN_DIR).toMatch(/[\\/]lean$/)expect(LEAN_OFF_FILE).toMatch(/[\\/]lean-off$/)})diff --git a/cli/src/masterProcess.ts b/cli/src/masterProcess.tsindex e75b84ff0..71481cde9 100644--- a/cli/src/masterProcess.ts+++ b/cli/src/masterProcess.ts@@ -18,6 +18,7 @@ import { baseNode, namedNode } from './harnessd/processName.js'import { PROBE_ANSWER, PROBE_TIMEOUT_MS } from './harnessd/reexec.js'import { ensureUtf8Locale } from './lib/childLocale.js'import { DAEMON_LOG_FILE, HARNESSD_STATUS_FILE, PID_FILE } from './lib/daemonState.js'+import { isInstalledCopy } from './lib/installedCopy.js'import { leanFingerprint, readLeanBundle, releaseLeanClaim, writeLeanBundle, type LeanBundle } from './harnessd/leanBundle.js'import { ts } from './lib/log.js'import { confirm as confirmUpdate, restore as restoreUpdate, unjudgedUpdate } from './lib/selfUpdate.js'@@ -69,6 +70,8 @@ export function startMaster(start: MasterStart, exit: (code: number) => void = (version: VERSION,reexecMarkerFile: HARNESSD_REEXEC_FILE,unjudgedUpdate: (bundle) => unjudgedUpdate(env.ADAPTER_CLI_DIR, bundle),+ // The updater runs for the installed copy alone (lib/installedCopy.ts), beside the core, never in it.+ updater: !env.ADAPTER_UPDATE_DISABLE && isInstalledCopy(start.scriptPath, env.ADAPTER_CLI_DIR),// A master that ends cleanly gives up its claim on its lean bundle, so the next one to start can// clear the folder; one that dies leaves a claim whose pid is gone, which counts for nothing.exit: (code) => {diff --git a/cli/src/serviceProcess.spec.ts b/cli/src/serviceProcess.spec.tsindex 1a1181256..59efc8807 100644--- a/cli/src/serviceProcess.spec.ts+++ b/cli/src/serviceProcess.spec.ts@@ -1,7 +1,7 @@import { afterEach, describe, expect, it, vi } from 'vitest'import { SERVICE_RUNNERS, startServiceProcess, type ServiceProcessOptions } from './serviceProcess.js'import type { ServiceHost, ServiceHostOptions, ServiceProcess } from './services/process.js'-import { KNOWN_SERVICES } from './harnessd/services.js'+import { KNOWN_SERVICES, UPDATER_HOST } from './harnessd/services.js'const local = vi.hoisted(() => ({ socket: '/data/daemon-18473.sock' as string | null }))vi.mock('./lib/localSocket.js', async (real) => ({ ...await real<object>(), localSocketPath: () => local.socket }))@@ -13,8 +13,8 @@ describe('a service in its own process', () => {const title = process.titleafterEach(() => { process.title = title; local.socket = '/data/daemon-18473.sock'; vi.restoreAllMocks() })- it('can run every service the master knows, and loads each one\'s runner on its own', async () => {- expect([...SERVICE_RUNNERS.keys()].sort()).toEqual([...KNOWN_SERVICES].sort())+ it('can run every service the master knows, and the updater it runs beside them, and loads each one\'s runner on its own', async () => {+ expect([...SERVICE_RUNNERS.keys()].sort()).toEqual([...KNOWN_SERVICES, ...UPDATER_HOST.services].sort())for (const [name, load] of SERVICE_RUNNERS) expect(typeof await load(), name).toBe('function')// Every service's own code, imported: seconds on a loaded machine, not the default five.}, 60_000)diff --git a/cli/src/serviceProcess.ts b/cli/src/serviceProcess.tsindex caec86a87..732f7d784 100644--- a/cli/src/serviceProcess.ts+++ b/cli/src/serviceProcess.ts@@ -38,6 +38,8 @@ export const SERVICE_RUNNERS: ReadonlyMap<string, () => Promise<Runner>> = new M['sharing', async () => (await import('./services/sharingProcess.js')).runSharingService],['orchestrator', async () => (await import('./services/orchestratorProcess.js')).runOrchestratorService],['commandBar', async () => (await import('./services/commandBarProcess.js')).runCommandBarService],+ // Not a service the core knows: the master runs it beside them (harnessd/services.ts `UPDATER_HOST`).+ ['updater', async () => (await import('./services/updaterProcess.js')).runUpdaterService],['gateway', async () => (await import('./gateway/gatewayProcess.js')).runGatewayService],['models', async () => (await import('./services/modelsProcess.js')).runModelsService],['devices', async () => (await import('./services/devicesProcess.js')).runDevicesService],diff --git a/cli/src/services/updaterProcess.spec.ts b/cli/src/services/updaterProcess.spec.tsnew file mode 100644index 000000000..d934b345c--- /dev/null+++ b/cli/src/services/updaterProcess.spec.ts@@ -0,0 +1,111 @@+import { afterEach, describe, expect, it, vi } from 'vitest'+import { SERVICE_EXIT_RESTART, type UpdaterMessage } from '../harnessd/protocol.js'+import type { startSelfUpdater } from '../lib/selfUpdate.js'+import type { startTuiUpdater } from '../tui/update.js'+import { VERSION } from '../version.js'+import { processUpdaterDeps, runUpdaterService, type UpdaterServiceDeps } from './updaterProcess.js'++// A lock someone else holds a moment: what the updater says while it waits for it.+vi.mock('../lib/daemonSpawnLock.js', async (real) => ({+ ...await real<object>(),+ withSpawnLock: async (_purpose: string, fn: () => Promise<unknown>, opts: { onWaiting?: (owner: object) => void } = {}) => {+ opts.onWaiting?.({ pid: 4242, purpose: 'start', since: Date.now(), startMarker: 'x', token: 't' })+ return fn()+ },+}))++type CliOptions = Parameters<typeof startSelfUpdater>[0]+type HnOptions = Parameters<typeof startTuiUpdater>[0]++describe('the updater in its own process', () => {+ afterEach(() => { vi.restoreAllMocks() })++ const make = () => {+ const calls: string[] = []+ const told: UpdaterMessage[] = []+ let cli: CliOptions | null = null+ let hn: HnOptions | null = null+ let sent: (() => void) | null = null+ const deps: UpdaterServiceDeps = {+ tell: (message, done) => { told.push(message); sent = done },+ exit: (code) => { calls.push(`exit ${code}`) },+ startCli: (options) => { cli = options; return { stop: () => calls.push('stop cli') } },+ startHn: (options) => { hn = options; return { stop: () => calls.push('stop hn') } },+ log: (line) => calls.push(line),+ }+ return { deps, calls, told, cli: () => cli!, hn: () => hn!, sent: () => sent! }+ }++ it('checks for this build\'s successor, and hn\'s, on the installed copy, and stops both', () => {+ const m = make()+ const service = runUpdaterService(undefined, m.deps)+ expect(m.cli()).toMatchObject({ currentVersion: VERSION, stageWhileJudged: true })+ expect(m.hn()).toMatchObject({ currentVersion: VERSION, isInstalledCopy: true })+ expect(m.calls[0]).toMatch(new RegExp(`^\\[update\\] self-update on · v${VERSION.replace(/\./g, '\\.')} · every \\d+s`))+ service.stop()+ expect(m.calls.slice(-2)).toEqual(['stop cli', 'stop hn'])+ })++ it('beside a core, stages over a build on probation only under a master that judges that itself', () => {+ const saved = { beside: process.env.HARNESSD_UPDATER_BESIDE_CORE, judges: process.env.HARNESSD_JUDGES_SUPERSEDED }+ try {+ process.env.HARNESSD_UPDATER_BESIDE_CORE = '1'+ delete process.env.HARNESSD_JUDGES_SUPERSEDED+ const older = make()+ runUpdaterService(undefined, older.deps)+ expect(older.cli().stageWhileJudged).toBe(false)+ process.env.HARNESSD_JUDGES_SUPERSEDED = '1'+ const newer = make()+ runUpdaterService(undefined, newer.deps)+ expect(newer.cli().stageWhileJudged).toBe(true)+ } finally {+ for (const [name, value] of [['HARNESSD_UPDATER_BESIDE_CORE', saved.beside], ['HARNESSD_JUDGES_SUPERSEDED', saved.judges]] as const) {+ if (value === undefined) delete process.env[name]+ else process.env[name] = value+ }+ }+ })++ it('tells the master what it staged, and only once that has left asks to be started again as the new build', async () => {+ const m = make()+ runUpdaterService(undefined, m.deps)+ const staged = m.cli().onStaged('9.9.9') as Promise<void>+ expect(m.told).toEqual([{ type: 'harnessd:staged', version: '9.9.9' }])+ expect(m.calls).not.toContain(`exit ${SERVICE_EXIT_RESTART}`)+ m.sent()()+ await staged+ expect(m.calls).toContain(`exit ${SERVICE_EXIT_RESTART}`)+ })++ it('stages under the spawn lock, saying who holds it while it waits', async () => {+ const m = make()+ runUpdaterService(undefined, m.deps)+ expect(await m.cli().withLock!(async () => 'staged')).toBe('staged')+ expect(m.calls.at(-1)).toMatch(/^\[update\] waiting — the daemon is .*4242/)+ })++ it('reaches the master over the spawn channel, or tells no one when started by hand', () => {+ const deps = processUpdaterDeps()+ const exit = vi.spyOn(process, 'exit').mockImplementation((() => {}) as never)+ deps.exit(SERVICE_EXIT_RESTART)+ expect(exit).toHaveBeenCalledWith(SERVICE_EXIT_RESTART)+ const log = vi.spyOn(console, 'log').mockImplementation(() => {})+ deps.log('a line')+ expect(log).toHaveBeenCalledWith('a line')+ const original = process.send+ try {+ const sent = vi.fn()+ process.send = undefined+ deps.tell({ type: 'harnessd:staged', version: '9.9.9' }, sent)+ expect(sent).toHaveBeenCalledOnce()+ const send = vi.fn((_message: unknown, callback: () => void) => { callback(); return true })+ process.send = send as unknown as typeof process.send+ const left = vi.fn()+ deps.tell({ type: 'harnessd:staged', version: '9.9.9' }, left)+ expect(send.mock.calls[0][0]).toEqual({ type: 'harnessd:staged', version: '9.9.9' })+ expect(left).toHaveBeenCalledOnce()+ } finally { process.send = original }+ expect(deps.startCli).toBeTypeOf('function')+ expect(deps.startHn).toBeTypeOf('function')+ })+})diff --git a/cli/src/services/updaterProcess.ts b/cli/src/services/updaterProcess.tsnew file mode 100644index 000000000..b87243bc2--- /dev/null+++ b/cli/src/services/updaterProcess.ts@@ -0,0 +1,80 @@+/**+ * Harness's updater, in a process of its own that harnessd's master runs (`harness __service updater`,+ * harnessd/services.ts `UPDATER_HOST`), only for the installed copy with updates on (masterProcess.ts).+ *+ * It checks the release manifest for a newer CLI and a newer hn, downloads and verifies them, runs the new+ * CLI's canary and stages it (lib/selfUpdate.ts, tui/install.ts), then tells the master (`harnessd:staged`).+ * The master asks the core to hand over (`harnessd:update`) and judges the new build on probation, as when+ * the core staged it itself. That is what it was until now: the core's process downloaded builds, a 15 s+ * canary and minutes of downloads in a process that must never go down, and a core in safe mode ran on+ * only for its updater. Here a broken download or a hung check costs this process alone, and a core that+ * cannot start at all still gets its fix (docs/design/2026-10-06-core-boundary-next.md, "Updaters").+ *+ * Once it has staged a build it exits with `SERVICE_EXIT_RESTART`, and the master starts it again at once:+ * a staging is the updater's last act on the bundle it runs, and the next check is the new build's.+ */+import { env } from '../config/env.js'+import { SERVICE_EXIT_RESTART, type UpdaterMessage } from '../harnessd/protocol.js'+import { describeSpawnLockOwner, withSpawnLock } from '../lib/daemonSpawnLock.js'+import { DOWNLOAD_LIMITS, startSelfUpdater } from '../lib/selfUpdate.js'+import { startTuiUpdater } from '../tui/update.js'+import { VERSION } from '../version.js'+import type { ServiceProcess } from './process.js'++export interface UpdaterServiceDeps {+ /** Tell the master; `sent` once the message has left (an exit before it would lose it). */+ tell(message: UpdaterMessage, sent: () => void): void+ exit(code: number): void+ startCli: typeof startSelfUpdater+ startHn: typeof startTuiUpdater+ log(line: string): void+}++export const processUpdaterDeps = (): UpdaterServiceDeps => ({+ tell: (message, sent) => {+ // Under its master, the spawn channel; started by hand, there is no one to tell, and nothing to wait on.+ if (!process.send) { sent(); return }+ process.send(message, () => sent())+ },+ exit: (code) => process.exit(code),+ startCli: startSelfUpdater,+ startHn: startTuiUpdater,+ log: (line) => console.log(line),+})++export function runUpdaterService(_options?: unknown, deps: UpdaterServiceDeps = processUpdaterDeps()): ServiceProcess {+ const cli = deps.startCli({+ currentVersion: VERSION,+ url: env.ADAPTER_UPDATE_URL,+ key: env.ADAPTER_UPDATE_KEY,+ dir: env.ADAPTER_CLI_DIR,+ intervalMs: env.ADAPTER_UPDATE_CHECK_MS,+ slotSecond: env.ADAPTER_UPDATE_SLOT_SEC,+ // The lock spans the byte swap and the word to the master: a `harness start` or `harness update` that+ // lands between them would stage over the backup this one just made.+ withLock: (fn) => withSpawnLock('handoff', fn, {+ onWaiting: (owner) => deps.log(`[update] waiting — the daemon is ${describeSpawnLockOwner(owner)}`),+ }),+ onStaged: (version) => new Promise<void>((done) => {+ deps.tell({ type: 'harnessd:staged', version }, () => {+ done()+ deps.exit(SERVICE_EXIT_RESTART)+ })+ }),+ limits: { idleMs: env.ADAPTER_UPDATE_IDLE_MS, deadlineMs: env.ADAPTER_UPDATE_DEADLINE_MS, floorBytesPerSecond: DOWNLOAD_LIMITS.floorBytesPerSecond },+ // A master judges a build staged while the one before is on probation (#807). Beside a core, under a+ // master too old to run the updater (core/updaterBeside.ts), only one that says so: one from before+ // rolled the newer build back with the one it judged.+ stageWhileJudged: process.env.HARNESSD_UPDATER_BESIDE_CORE !== '1' || process.env.HARNESSD_JUDGES_SUPERSEDED === '1',+ })+ // hn is an independent, optional download: its failure never holds up the CLI's fix.+ const hn = deps.startHn({+ currentVersion: VERSION,+ isInstalledCopy: true,+ disabled: env.ADAPTER_UPDATE_DISABLE,+ intervalMs: env.ADAPTER_UPDATE_CHECK_MS,+ slotSecond: env.ADAPTER_UPDATE_SLOT_SEC,+ })+ deps.log(`[update] self-update on · v${VERSION} · every ${Math.round(env.ADAPTER_UPDATE_CHECK_MS / 1000)}s`)+ return { stop: () => { cli.stop(); hn.stop() } }+}diff --git a/cli/src/startupOrder.spec.ts b/cli/src/startupOrder.spec.tsindex a0180b4f2..b4b9d2254 100644--- a/cli/src/startupOrder.spec.ts+++ b/cli/src/startupOrder.spec.ts@@ -82,13 +82,10 @@ function callArguments(source: string, call: string): { text: string; at: number}/** Every call whose dependencies run DURING start-up, before `runForeground` has finished its body. */-const STARTUP_CALLS = ['await repairClaudeCwd({', 'await restoreAgents({', 'startSelfUpdater({', 'startTuiUpdater({']+const STARTUP_CALLS = ['await repairClaudeCwd({', 'await restoreAgents({']-/** What the prologue is allowed to do before the updater is running: nothing that can throw. */-const PROLOGUE_CALLS = new Set([- 'installTimestampedConsole', 'startSelfUpdater', 'statSync', 'join', 'String', 'Number', 'Date',- 'withSpawnLock', 'describeSpawnLockOwner',-])+/** What the prologue is allowed to do before the master's update is listened for: nothing that can throw. */+const PROLOGUE_CALLS = new Set(['installTimestampedConsole'])const KEYWORDS = new Set(['if', 'for', 'while', 'switch', 'catch', 'return', 'typeof', 'new', 'await', 'function'])describe('the core\'s start-up order (core/main.ts)', () => {@@ -101,16 +98,17 @@ describe('the core\'s start-up order (core/main.ts)', () => {})// ── The invariant the whole safe-boot design rests on ───────────────────────────────────────────- // A daemon that cannot finish starting can only be fixed by its own updater, and the updater can- // only run if start-up reached it. So it goes first, and everything that can throw or hang goes- // after. These three say so in the order they are worth reading.+ // A daemon that cannot finish starting is fixed by the build the master's updater stages, and leaves for+ // it only if start-up got as far as listening for the master's request. So that goes first, and+ // everything that can throw or hang goes after. These three say so in the order they are worth reading.- it('starts the updater before anything that can throw or hang', () => {+ it('listens for the master\'s update before anything that can throw or hang, and runs no updater itself', () => {const source = code(SOURCE)- const updater = source.indexOf('startSelfUpdater({')- expect(updater, 'core/main.ts still starts the self-updater').toBeGreaterThan(-1)+ const updater = source.indexOf('coreLink.onUpdate(')+ expect(updater, 'core/main.ts still listens for the master\'s update').toBeGreaterThan(-1)+ // The updater is the master's, in its own process (services/updaterProcess.ts): the core downloads no build.+ expect(source).not.toMatch(/startSelfUpdater\(|startTuiUpdater\(/)for (const risky of [- 'startTuiUpdater(', // optional hn download must never precede CLI recovery'requireTmuxAvailable(', // throws outright when tmux is missing'await startHookServer(', // EADDRINUSE on a fixed port with no fallback'installSessionHooks(', // 13 vendor settings files, any of which can be unreadable@@ -120,7 +118,7 @@ describe('the core\'s start-up order (core/main.ts)', () => {]) {const at = source.indexOf(risky)if (at < 0) continue // renamed or gone; the TDZ lint above still covers what remains- expect(at, `\`${risky}\` must come after startSelfUpdater({ — a crash there would strand the machine`)+ expect(at, `\`${risky}\` must come after coreLink.onUpdate( — a crash there would strand the machine`).toBeGreaterThan(updater)}})@@ -129,15 +127,15 @@ describe('the core\'s start-up order (core/main.ts)', () => {// Stronger than listing today's hazards: this is what catches the NEXT line somebody adds on top.const source = code(SOURCE)const from = source.indexOf('async function runForeground(')- const to = source.indexOf('startSelfUpdater({')+ const to = source.indexOf('coreLink.onUpdate(')const prologue = source.slice(source.indexOf('{', from), to)- expect(prologue.includes('\n await '), 'no await may precede the updater — a hang there is unrecoverable').toBe(false)- expect(/\n\s*throw /.test(prologue), 'no throw may precede the updater').toBe(false)+ expect(prologue.includes('\n await '), 'no await may precede the update — a hang there is unrecoverable').toBe(false)+ expect(/\n\s*throw /.test(prologue), 'no throw may precede the update').toBe(false)const called = [...prologue.replace(/\.\s*[A-Za-z_$][\w$]*\s*\(/g, ' ').matchAll(/([A-Za-z_$][\w$]*)\s*\(/g)].map(match => match[1]).filter(name => !KEYWORDS.has(name))expect([...new Set(called.filter(name => !PROLOGUE_CALLS.has(name)))].sort(),- 'only calls that cannot fail belong above the updater; move this below it, or add it to PROLOGUE_CALLS with a reason')+ 'only calls that cannot fail belong above the update; move this below it, or add it to PROLOGUE_CALLS with a reason').toEqual([])})@@ -148,13 +146,16 @@ describe('the core\'s start-up order (core/main.ts)', () => {const source = code(SOURCE)const flip = source.indexOf('daemonBoot.applyStagedUpdate = ')expect(flip, 'the handoff handler is still swapped in core/main.ts').toBeGreaterThan(-1)- const body = callArguments(source, 'updateHandoff.restartForUpdate(')- expect(body.at, 'the handler is the one it swaps in').toBeGreaterThan(flip)+ expect(callArguments(source, 'updateHandoff.restartForUpdate(').at, 'the handler is the one it swaps in').toBeGreaterThan(flip)+ // Its teardown, which a core with no master hands over with too: built just before the swap.+ const body = callArguments(source, 'const updateTeardown = (')+ const teardown = { text: source.slice(body.at, source.indexOf('\n ]\n', body.at)), at: body.at }+ expect(teardown.at).toBeLessThan(flip)const declaredAt = localsOf(source)- const late = referencedIn(body.text).filter(used => (declaredAt.get(used) ?? -1) > flip).sort()+ const late = referencedIn(teardown.text).filter(used => (declaredAt.get(used) ?? -1) > teardown.at).sort()expect(late, 'these are torn down by the update handoff but declared after it is armed').toEqual([])// What it tears down is all there is to it: the handoff module reaches nothing of runForeground's.- expect(referencedIn(body.text)).toEqual(expect.arrayContaining(['registry', 'hookServer', 'localWsServer', 'backend', 'watcher']))+ expect(referencedIn(teardown.text)).toEqual(expect.arrayContaining(['registry', 'hookServer', 'localWsServer', 'backend', 'watcher']))})it('the daemon arm survives its own start-up failure', () => {