Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

The updater leaves the core for a process the master runs, and the core never downloads a build (#925)

Dee08c80d76dd21
diff --git a/cli/AGENTS.md b/cli/AGENTS.md
index 0a76e4bc1..dc3272571 100644
--- a/cli/AGENTS.md
+++ b/cli/AGENTS.md
@@ -18,9 +18,12 @@ SERVICES src/services/ everything else: search, viewers, models, workspaces,
`src/core/main.ts` `runForeground()` is the composition root: it creates the modules and wires them
together. It is the core's own entry (`harness __run`), which the master starts; `src/cli.ts` is the CLI,
and calls in for `__run`. `harness start -f` runs the master in the foreground, as launchd and systemd do.
-A core with no master (`HARNESS_NO_MASTER=1`, or one an older release's handoff started) hands each update
-to a master that judges it (`src/core/updateHandoff.ts`). `src/backendSocket.ts` is the transport: it
-receives frames and dispatches them.
+The updater is the master's, in a process of its own (`src/services/updaterProcess.ts`): the core never
+downloads a build. When it stages one, the master has the core hand over (`harnessd:update`) and judges the
+new build. A core with no master gets no updates: one an older release's own handoff started hands itself
+to a master once that release has gone, and one `HARNESS_NO_MASTER=1` asked for runs as it is
+(`src/core/updateHandoff.ts`). `src/backendSocket.ts` is the transport: it receives frames and dispatches
+them.
`src/gateway/` is the relay: the backend link, the E2EE sessions and keys, and every rule about what a
remote client may send and how what it is sent is sealed. It runs in a process of its own
(`src/gateway/gatewayProcess.ts`; the core's side is `src/core/gatewayLink.ts`), or in the core's with
diff --git a/cli/e2e/harness/daemon.ts b/cli/e2e/harness/daemon.ts
index af07a87b4..8b21fb48d 100644
--- a/cli/e2e/harness/daemon.ts
+++ b/cli/e2e/harness/daemon.ts
@@ -57,6 +57,8 @@ export interface DaemonOptions {
noMaster?: boolean
/** Start as a supervisor does, `harness start -f`: the master in the foreground, the core its child. */
foreground?: boolean
+ /** Start with these arguments to node instead (a launcher the test provides), in the daemon's environment. */
+ launch?: string[]
/** Run this bundle (an installed `cli.js`) instead of the checkout's source. */
scriptPath?: string
/** Keep the daemon's data folder here instead of under the throwaway root (a test volume). The fake
@@ -308,7 +310,7 @@ export class IsolatedDaemon {
// A test's own bundle (an old release), the run's bundle (`E2E_BUNDLE`, see bundle.ts), or the sources.
const bundle = this.options.scriptPath ?? process.env.E2E_BUNDLE_PATH
const script = bundle ? [bundle] : ['--import', 'tsx', 'src/cli.ts']
- const child = spawn(process.execPath, [...heap, ...script, ...entry], {
+ const child = spawn(process.execPath, this.options.launch ?? [...heap, ...script, ...entry], {
cwd: CLI_ROOT, env: this.env, stdio: ['ignore', 'pipe', 'pipe'],
})
this.child = child
diff --git a/cli/e2e/harness/olderReleaseHandoff.mjs b/cli/e2e/harness/olderReleaseHandoff.mjs
new file mode 100644
index 000000000..49dc3cd45
--- /dev/null
+++ b/cli/e2e/harness/olderReleaseHandoff.mjs
@@ -0,0 +1,21 @@
+// What an older release's own update handoff does, for e2e/updateOnItsOwn.e2e.ts: it spawns the new build's
+// core (`cli.js __run`, detached, told ADAPTER_UPDATED_TO), waits for it to claim the pid file, judges it
+// a while longer (the release waits for its connection), and leaves. Releases before harnessd's master did
+// exactly this, and so do later ones run without a master; the core it leaves behind has no master.
+//
+// Usage: node olderReleaseHandoff.mjs <cli.js> <pid file> <file whose presence lets it leave>
+import { spawn } from 'node:child_process'
+import { existsSync, readFileSync } from 'node:fs'
+
+const [cli, pidFile, mayLeave] = process.argv.slice(2)
+const core = spawn(process.execPath, [cli, '__run'], {
+ detached: true, stdio: 'inherit', env: { ...process.env, ADAPTER_UPDATED_TO: process.env.HANDED_OVER_VERSION ?? '0.0.0' },
+})
+core.unref()
+const deadline = Date.now() + 120_000
+const judged = setInterval(() => {
+ let claimed = false
+ try { claimed = Number(readFileSync(pidFile, 'utf8').trim()) === core.pid } catch { /* not yet */ }
+ if (claimed && existsSync(mayLeave)) { clearInterval(judged); process.exit(0) }
+ if (Date.now() > deadline) process.exit(1)
+}, 200)
diff --git a/cli/e2e/migration.e2e.ts b/cli/e2e/migration.e2e.ts
index 563de4193..e96cccbbe 100644
--- a/cli/e2e/migration.e2e.ts
+++ b/cli/e2e/migration.e2e.ts
@@ -3,8 +3,9 @@
* does when a release ships. A daemon of the released build (`MIGRATION_FROM`, its bundled `cli.js`)
* runs with agents at work, finds this checkout's build in its update manifest, and hands over to it
* the way that release hands over: it spawns the new `cli.js __run` and exits. The new core comes up
- * on its own, with no master, keeps every agent and its conversation, and the next start brings up
- * harnessd's master over the same agents.
+ * on its own, with no master, keeps every agent and its conversation, and once that release has gone it
+ * hands the machine to harnessd's master, which runs the updater (core/updateHandoff.ts). Then the next
+ * start brings up the master over the same agents again.
*
* Skipped unless MIGRATION_FROM names a bundle, so CI does not build old releases. Before a release:
* build the last released tag's bundle (`node build-bundle.mjs` in a checkout of it) and run
@@ -103,8 +104,17 @@ describe.skipIf(!FROM)('upgrading a machine from a released build', () => {
expect((await health())?.version).toBe(fromVersion)
}, 300_000)
+ /** The master the handed-over core started, if it is running: no child of the test's. */
+ const handedTo = async (): Promise<number | null> => {
+ const status = await fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json() as Promise<Record<string, any>>).catch(() => null)
+ const pid = status?.harnessd?.masterPid
+ return typeof pid === 'number' && pid !== daemon!.pid && IsolatedDaemon.alive(pid) ? pid : null
+ }
+
afterAll(async () => {
- // Whatever runs the port now — the detached successor included — goes with the test.
+ // Whatever runs the port now — a master the handed-over core started included — goes with the test.
+ const master = daemon ? await handedTo() : null
+ if (master) { try { process.kill(master, 'SIGTERM') } catch { /* gone */ } }
for (const pid of listeners()) { try { process.kill(pid, 'SIGTERM') } catch { /* gone */ } }
await daemon?.close()
await new Promise<void>((done) => server ? server.close(() => done()) : done())
@@ -135,21 +145,26 @@ describe.skipIf(!FROM)('upgrading a machine from a released build', () => {
offered = 'next'
await until(`the machine to run ${NEXT}`, async () => (await health())?.version === NEXT || null, 120_000, 500)
client.close()
+
+ // Once the released build has gone, the core it left with no master hands the machine to one, which
+ // runs the updater: a second restart, seconds after the first, as any update's is. The agents go on.
+ const master = await until('the handed-over core to hand the machine to a master', () => handedTo(), 120_000, 500)
client = await LocalClient.connect(d)
for (const agent of agents) {
- await until(`${agent.id.slice(0, 8)} back on ${NEXT}`, async () => {
+ await until(`${agent.id.slice(0, 8)} back under the master the core started`, async () => {
const now = (await rows(client)).find((one) => one.id === agent.id)
return now?.status === 'active' && now.sessionId === agent.sessionId ? now : null
}, 60_000, 500)
- await turn(client, agent.id, `on ${NEXT}, handed over`)
+ await turn(client, agent.id, `on ${NEXT}, under the master the core started`)
}
client.close()
// The next `harness start` (a reboot, the desktop app) starts the new build as harnessd: master and core.
// The released build's own process may still be about, waiting on its successor; it goes first.
await d.stop()
+ process.kill(master, 'SIGTERM')
for (const pid of listeners()) process.kill(pid, 'SIGTERM')
- await until('the handed-over core to stop', () => listeners().length === 0 || null, 30_000, 250)
+ await until('the daemon to stop', () => (listeners().length === 0 && !IsolatedDaemon.alive(master)) || null, 30_000, 250)
;(d.options as { noMaster?: boolean }).noMaster = false
await d.start()
expect((await health())?.version).toBe(NEXT)
diff --git a/cli/e2e/reexec.e2e.ts b/cli/e2e/reexec.e2e.ts
index b5108c955..488a641c0 100644
--- a/cli/e2e/reexec.e2e.ts
+++ b/cli/e2e/reexec.e2e.ts
@@ -260,10 +260,13 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\
let scratch = ''
let server: Server | undefined
let daemon: IsolatedDaemon | undefined
- let offered: 'released' | 'next' = 'released'
+ let offered: 'released' | 'next' | 'later' = 'released'
let releasedVersion = ''
let releasedCanReexec = false
- const bundles = new Map<'released' | 'next', { cli: Buffer; notify: Buffer; version: string }>()
+ const bundles = new Map<'released' | 'next' | 'later', { cli: Buffer; notify: Buffer; version: string }>()
+ const LATER = '0.99.1'
+ /** The newest build this machine has kept: NEXT, or LATER once a released master's core updated to it. */
+ let newest = NEXT
const cliDir = () => join(scratch, 'cli')
const status = async (): Promise<Record<string, any> | null> =>
fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json()).catch(() => null)
@@ -273,6 +276,7 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\
const out = join(scratch, 'build')
execFileSync(process.execPath, ['build-bundle.mjs'], { cwd: CLI_ROOT, env: { ...process.env, ADAPTER_VERSION: NEXT, BUNDLE_OUT_DIR: out }, stdio: 'pipe' })
bundles.set('next', { cli: readFileSync(join(out, 'cli.js')), notify: readFileSync(join(out, 'notify.mjs')), version: NEXT })
+ bundles.set('later', { cli: Buffer.from(atVersion(readFileSync(join(out, 'cli.js'), 'utf8'), NEXT, LATER)), notify: readFileSync(join(out, 'notify.mjs')), version: LATER })
releasedVersion = execFileSync(process.execPath, [RELEASED!, 'version'], { encoding: 'utf8' }).trim()
bundles.set('released', { cli: readFileSync(RELEASED!), notify: readFileSync(join(dirname(RELEASED!), 'notify.mjs')), version: releasedVersion })
server = createServer((request, response) => {
@@ -339,8 +343,18 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\
expect(daemon!.log()).not.toContain('re-executing')
}
await until('the released master to keep the update', () => daemon!.log().includes('the update stayed up — keeping it'), 30_000)
+ if (!releasedCanReexec) {
+ // A master from before the updater left the core runs none, and could not become this build's: the
+ // core runs it beside itself (core/updaterBeside.ts), so the next build still arrives.
+ expect(daemon!.log()).toContain('[update] this core\'s master runs no updater — running it beside this core')
+ offered = 'later'
+ await until(`the released master to run ${LATER}`, async () => (await status())?.version === LATER || null, 90_000, 250)
+ expect((await status())?.harnessd.masterPid).toBe(master)
+ await until('the released master to keep it too', () => daemon!.log().split('the update stayed up — keeping it').length > 2 || null, 30_000)
+ newest = LATER
+ }
await daemon!.restart()
- expect((await status())?.harnessd).toMatchObject({ masterVersion: NEXT, masterPid: daemon!.pid })
+ expect((await status())?.harnessd).toMatchObject({ masterVersion: newest, masterPid: daemon!.pid })
})
it('this master returns to the released bundle, re-executing only when it answers the probe', async () => {
@@ -355,7 +369,7 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\
return current?.version === releasedVersion ? current : null
}, 60_000, 250)
expect(now.harnessd).toMatchObject({
- masterVersion: releasedCanReexec ? releasedVersion : NEXT,
+ masterVersion: releasedCanReexec ? releasedVersion : newest,
masterPid: master,
reexecs: releasedCanReexec ? 1 : 0,
})
diff --git a/cli/e2e/releaseRehearsal.e2e.ts b/cli/e2e/releaseRehearsal.e2e.ts
index 42dafd6be..2538d0a48 100644
--- a/cli/e2e/releaseRehearsal.e2e.ts
+++ b/cli/e2e/releaseRehearsal.e2e.ts
@@ -46,6 +46,8 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published release
let offered: 'from' | 'next' = 'from'
let fromVersion = ''
let fromHasMaster = false
+ /** The master a core with no master handed the machine to, when the release had none. */
+ let handedTo: number | null = null
const bundles = new Map<'from' | 'next', { cli: Buffer; notify: Buffer; version: string }>()
const cliDir = () => join(scratch, 'cli')
const status = async (): Promise<Row | null> =>
@@ -211,6 +213,7 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published release
afterAll(async () => {
// Whatever runs the port now, a successor the release spawned on its own included, goes with the test.
+ if (handedTo && IsolatedDaemon.alive(handedTo)) { try { process.kill(handedTo, 'SIGTERM') } catch { /* gone */ } }
for (const pid of listeners()) { try { process.kill(pid, 'SIGTERM') } catch { /* gone */ } }
await daemon?.close()
await backend?.close()
@@ -257,6 +260,14 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published release
offered = 'next'
await until(`the machine to run ${NEXT}`, async () => (await status())?.version === NEXT || null, 150_000, 250)
client.close()
+ if (!fromHasMaster) {
+ // A release with no master: once it has gone, the core it started hands the machine to a master,
+ // which runs the updater (core/updateHandoff.ts), seconds after the release's own handoff.
+ handedTo = await until('the core to hand the machine to a master', async () => {
+ const pid = (await status())?.harnessd?.masterPid
+ return typeof pid === 'number' && IsolatedDaemon.alive(pid) ? pid : null
+ }, 120_000, 500)
+ }
// The window reconnects, as the desktop app does.
client = await connect(d)
@@ -291,7 +302,7 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published release
for (const file of [files.claude, files.codex]) expect(readFileSync(file, 'utf8')).toContain(join(cliDir(), 'notify.mjs'))
// Every process accounted for: under the release's master, its core on the new bundle as its child;
- // without one, the successor the release spawned, and nothing of the release left running.
+ // without one, the core of the master the successor handed the machine to, and nothing of the release.
const status1 = (await status())!
// A release without a master hands over by spawning the successor and watching it for up to half a
// minute more (its waitForReady) before it leaves: one core once it has.
@@ -304,7 +315,8 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published release
expect(cores[0].ppid).toBe(d.pid)
await until('the master to keep the update', () => d.log().slice(updateFrom).includes('the update stayed up — keeping it'), 60_000)
} else {
- expect(listeners()).toEqual([cores[0].pid])
+ // The master the core handed the machine to serves it, its core its child.
+ expect(cores[0].ppid).toBe(handedTo)
}
expect(existsSync(join(cliDir(), 'cli.js.prev')) || existsSync(join(cliDir(), 'update-pending.json'))).toBe(false)
memory(`after the update from ${fromVersion}`)
@@ -312,6 +324,7 @@ describe.skipIf(!FROM)('a release rehearsal: the update from a published release
// The next `harness start` (a reboot, the desktop app) starts this checkout as harnessd: master and core.
await d.stop()
+ if (handedTo) process.kill(handedTo, 'SIGTERM')
for (const pid of listeners()) process.kill(pid, 'SIGTERM')
await until('the daemon to be down', () => listeners().length === 0 || null, 30_000, 250)
;(d.options as { noMaster?: boolean }).noMaster = false
diff --git a/cli/e2e/update.e2e.ts b/cli/e2e/update.e2e.ts
index e40777d66..d54a216ac 100644
--- a/cli/e2e/update.e2e.ts
+++ b/cli/e2e/update.e2e.ts
@@ -1,6 +1,7 @@
/**
- * A self-update end to end, on real release bundles: the daemon finds a newer build in its manifest,
- * stages it, and harnessd's master restarts the core on it. A build that stays up is kept. One that
+ * A self-update end to end, on real release bundles: the master's updater, in a process of its own,
+ * finds a newer build in its manifest and stages it, and harnessd's master has the core hand over and
+ * restarts it on the new build. The core itself downloads nothing. A build that stays up is kept. One that
* crashes is rolled back, remembered, and not tried again until a newer one is published — before,
* every machine on it restarted once a minute until a fix shipped.
*
@@ -21,6 +22,7 @@ const FIRST = '41.0.1'
const sha = (bytes: Buffer): string => createHash('sha256').update(bytes).digest('hex')
const count = (text: string, part: string): number => text.split(part).length - 1
const sleep = (ms: number) => new Promise((done) => setTimeout(done, ms))
+const parentOf = (pid: number): number => Number(execFileSync('ps', ['-o', 'ppid=', '-p', String(pid)], { encoding: 'utf8' }).trim())
describe('a self-update under harnessd', () => {
let scratch = ''
@@ -34,6 +36,12 @@ describe('a self-update under harnessd', () => {
const cliDir = () => join(scratch, 'cli')
const installed = () => readFileSync(join(cliDir(), 'cli.js'))
const rejected = () => JSON.parse(readFileSync(join(cliDir(), 'update-rejected.json'), 'utf8')) as string[]
+ /** The updater's process the master last started, while it runs. */
+ const updaterPid = (): number | null => {
+ const started = [...daemon!.log().matchAll(/\[harnessd\] service updater started \(pid (\d+)\)/g)].at(-1)
+ const pid = started ? Number(started[1]) : null
+ return pid && IsolatedDaemon.alive(pid) ? pid : null
+ }
const running = async (): Promise<string | null> => {
const health = await fetch(`http://127.0.0.1:${daemon!.port}/api/health`).then((response) => response.json()).catch(() => null)
return (health as { version?: string } | null)?.version ?? null
@@ -128,6 +136,11 @@ describe('a self-update under harnessd', () => {
expect(existsSync(join(cliDir(), 'update-pending.json'))).toBe(false)
expect(existsSync(join(cliDir(), 'update-rejected.json'))).toBe(false)
expect(daemon!.pid).toBe(master)
+ // The updater is its own process, the master's child; the core asked to hand over, and ran no updater.
+ const updater = updaterPid()
+ expect(updater).not.toBeNull()
+ expect(parentOf(updater!)).toBe(master)
+ expect(daemon!.log()).toContain('[harnessd] the updater staged 41.0.2 — asking the core to hand over')
})
it('rolls back a build that crashes, and never stages it again', async () => {
@@ -150,8 +163,13 @@ describe('a self-update under harnessd', () => {
expect(daemon!.pid).toBe(master)
})
- it('moves on when a newer build is published', async () => {
+ it('moves on when a newer build is published, after its updater was killed outright', async () => {
showLog()
+ // Killed: the master starts it again, and the core never noticed.
+ const killed = updaterPid()!
+ process.kill(killed, 'SIGKILL')
+ await until('the master to start the updater again', () => { const now = updaterPid(); return now && now !== killed ? now : null }, 30_000, 250)
+ expect(await running()).toBe('41.0.2')
offered = '41.0.4'
await until('the core to run 41.0.4', async () => (await running()) === '41.0.4', 90_000, 250)
await until('the master to keep it', () => count(daemon!.log(), 'the update stayed up — keeping it') === 2, 30_000)
diff --git a/cli/e2e/updateOnItsOwn.e2e.ts b/cli/e2e/updateOnItsOwn.e2e.ts
index 6eab3b76a..da5a87774 100644
--- a/cli/e2e/updateOnItsOwn.e2e.ts
+++ b/cli/e2e/updateOnItsOwn.e2e.ts
@@ -1,52 +1,61 @@
/**
- * A self-update of a core that runs with no master, on real release bundles: what this build's core does
- * when an older release's own handoff started it (it spawned `cli.js __run`, e2e/migration.e2e.ts), or
- * `HARNESS_NO_MASTER=1` asked for it. It hands the machine to harnessd's master on the staged bundle,
- * which judges the update as it judges every one: a build whose core crashes is rolled back, remembered
- * and the master re-executed onto the build before; a good one is kept. A build whose master cannot
- * start is refused before anything is let go, and this build goes on under a master of its own. Before,
- * the core spawned a core like itself and judged it, and the successor had no master either. The same
- * holds for an update staged while such a core is still starting, or is in safe mode, as the boot
- * handoff hands it over (core/main.ts `bootHandoff`).
+ * A core with no master, on real release bundles. The updater is the master's, in a process of its own
+ * (services/updaterProcess.ts), and the core never downloads a build, so a core with no master gets no
+ * update by itself:
+ *
+ * - one an older release's own handoff started (it spawns `cli.js __run`, judges it and leaves:
+ * e2e/harness/olderReleaseHandoff.mjs) hands the machine to a master on its build once that release has
+ * gone. The master's updater then updates it, and the master judges each build: one whose core crashes
+ * is rolled back and remembered, a good one kept. Before, such a core ran its own updater and handed each
+ * update to a core like itself, which again had no master;
+ * - one `HARNESS_NO_MASTER=1` asked for runs without updates, as asked.
*
* The bundles are built from this checkout at made-up versions and served from a local manifest, as in
* e2e/update.e2e.ts, so nothing here reaches beyond the machine.
*/
import { execFileSync } from 'node:child_process'
import { createHash } from 'node:crypto'
-import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { createServer, type Server } from 'node:http'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
-import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
+import { afterAll, afterEach, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
import { LocalClient, type Frame } from './harness/client.js'
import { CLI_ROOT, IsolatedDaemon, until } from './harness/daemon.js'
import { atVersion } from './harness/release.js'
const FIRST = '44.0.1'
const sha = (bytes: Buffer): string => createHash('sha256').update(bytes).digest('hex')
+const sleep = (ms: number) => new Promise((done) => setTimeout(done, ms))
type Row = Record<string, any>
-describe('a self-update of a core with no master', () => {
+describe('a core with no master', () => {
let scratch = ''
let server: Server | undefined
let daemon: IsolatedDaemon | undefined
+ let master: number | null = null
const releases = new Map<string, Buffer>()
let notify = Buffer.alloc(0)
let offered = FIRST
+ let local = ''
const cliDir = () => join(scratch, 'cli')
const installed = () => readFileSync(join(cliDir(), 'cli.js'))
const rejected = () => JSON.parse(readFileSync(join(cliDir(), 'update-rejected.json'), 'utf8')) as string[]
const status = async (): Promise<Row | null> =>
fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json() as Promise<Row>).catch(() => null)
- /** What the master this test never started writes: the daemon's log file. */
- const masterLog = (): string => { try { return readFileSync(join(daemon!.dataDir, 'harness.log'), 'utf8') } catch { return '' } }
/** The pids answering on the daemon's port: whatever runs it now. */
const listeners = (): number[] => {
try {
return execFileSync('lsof', ['-n', '-P', '-t', `-iTCP:${daemon!.port}`, '-sTCP:LISTEN']).toString().trim().split('\n').filter(Boolean).map(Number)
} catch { return [] }
}
+ /** What the daemon has said: what this test's process printed, and the log file the master the core
+ * started writes to (core/main.ts `startMasterHere`). */
+ const logs = (): string => {
+ let file = ''
+ try { file = readFileSync(join(daemon!.dataDir, 'harness.log'), 'utf8') } catch { /* none yet */ }
+ return `${daemon!.log()}\n${file}`
+ }
const parentOf = (pid: number): number => Number(execFileSync('ps', ['-o', 'ppid=', '-p', String(pid)], { encoding: 'utf8' }).trim())
const rows = async (client: LocalClient): Promise<Row[]> =>
(await client.request<{ agents: Row[] }>('agents_list', { includeStopped: true }, 30_000)).agents
@@ -55,12 +64,40 @@ describe('a self-update of a core with no master', () => {
client.send('message', { agentId, content })
await ended
}
- /** The master the core handed the machine to, once its core on `version` answers. */
+ /** The agent, on its conversation, taking a turn: proof it came through. */
+ const goesOn = async (agent: Row, content: string) => {
+ const client = await LocalClient.connect(daemon!)
+ await until('the agent back on its conversation', async () => {
+ const row = (await rows(client)).find((one) => one.id === agent.id)
+ return row?.status === 'active' && row.sessionId === agent.sessionId ? row : null
+ }, 60_000, 500)
+ await turn(client, agent.id, content)
+ client.close()
+ }
+ /** The master running the machine, once its core on `version` answers. */
const underMaster = (version: string) => until(`the machine to run ${version} under a master`, async () => {
const now = await status()
return now?.version === version && now.harnessd?.masterPid && IsolatedDaemon.alive(now.harnessd.masterPid) ? now : null
}, 120_000, 250)
- let master: number | null = null
+ const environment = () => ({
+ ADAPTER_CLI_DIR: cliDir(),
+ ADAPTER_UPDATE_DISABLE: 'false',
+ ADAPTER_UPDATE_URL: `${local}/metadata.json`,
+ ADAPTER_UPDATE_CHECK_MS: '1000',
+ ADAPTER_UPDATE_SLOT_SEC: '-1',
+ HARNESSD_UPDATE_PROBATION_MS: '3000',
+ HARNESSD_INITIAL_BACKOFF_MS: '100',
+ HARNESS_TUI_MANIFEST_URL: `${local}/tui/metadata.json`,
+ ADAPTER_RUNTIME_METADATA_URL: `${local}/runtime/metadata.json`,
+ ADAPTER_GRID_RUNTIME_METADATA_URL: `${local}/grid/metadata.json`,
+ })
+ const install = (version: string) => {
+ rmSync(cliDir(), { recursive: true, force: true })
+ mkdirSync(cliDir())
+ writeFileSync(join(cliDir(), 'cli.js'), releases.get(version)!)
+ writeFileSync(join(cliDir(), 'notify.mjs'), notify)
+ writeFileSync(join(cliDir(), 'package.json'), '{"type":"module"}\n')
+ }
beforeAll(async () => {
scratch = mkdtempSync(join(tmpdir(), 'harnessd-update-alone-'))
@@ -79,11 +116,7 @@ describe('a self-update of a core with no master', () => {
release(FIRST)
// Starts a master, whose core dies on every start.
release('44.0.2', ['__run'])
- // Whose master cannot start at all: it does not answer its probe.
- release('44.0.3', ['__harnessd', '__harnessd-probe'])
- release('44.0.4')
- release('44.0.5', ['__harnessd', '__harnessd-probe'])
- release('44.0.6')
+ release('44.0.3')
server = createServer((request, response) => {
const url = request.url ?? ''
@@ -104,66 +137,36 @@ describe('a self-update of a core with no master', () => {
response.end()
})
await new Promise<void>((done) => server!.listen(0, '127.0.0.1', done))
- const local = `http://127.0.0.1:${(server.address() as { port: number }).port}`
-
- mkdirSync(cliDir())
- writeFileSync(join(cliDir(), 'cli.js'), releases.get(FIRST)!)
- writeFileSync(join(cliDir(), 'notify.mjs'), notify)
- writeFileSync(join(cliDir(), 'package.json'), '{"type":"module"}\n')
- daemon = await IsolatedDaemon.create({
- scriptPath: join(cliDir(), 'cli.js'),
- // A core on its own, as an older release's handoff, or HARNESS_NO_MASTER=1, starts it.
- noMaster: true,
- env: {
- ADAPTER_CLI_DIR: cliDir(),
- ADAPTER_UPDATE_DISABLE: 'false',
- ADAPTER_UPDATE_URL: `${local}/metadata.json`,
- ADAPTER_UPDATE_CHECK_MS: '1000',
- ADAPTER_UPDATE_SLOT_SEC: '-1',
- HARNESSD_UPDATE_PROBATION_MS: '3000',
- HARNESSD_INITIAL_BACKOFF_MS: '100',
- HARNESS_TUI_MANIFEST_URL: `${local}/tui/metadata.json`,
- ADAPTER_RUNTIME_METADATA_URL: `${local}/runtime/metadata.json`,
- ADAPTER_GRID_RUNTIME_METADATA_URL: `${local}/grid/metadata.json`,
- },
- })
- await daemon.start()
- expect((await status())?.version).toBe(FIRST)
- expect((await status())?.harnessd).toBeFalsy()
+ local = `http://127.0.0.1:${(server.address() as { port: number }).port}`
}, 300_000)
- afterAll(async () => {
- // The master this core handed the machine to is no child of the test's: it goes first, with its core.
+ afterEach(async () => {
+ // A master the core handed the machine to is no child of the test's: it goes first, with its core.
if (master && IsolatedDaemon.alive(master)) {
process.kill(master, 'SIGTERM')
await until('the master to stop', () => !IsolatedDaemon.alive(master) || null, 15_000, 100).catch(() => { try { process.kill(master!, 'SIGKILL') } catch { /* gone */ } })
}
+ master = null
for (const pid of listeners()) { try { process.kill(pid, 'SIGTERM') } catch { /* gone */ } }
await daemon?.close()
+ daemon = undefined
+ })
+ afterAll(async () => {
await new Promise<void>((done) => server ? server.close(() => done()) : done())
if (scratch) rmSync(scratch, { recursive: true, force: true })
})
- /** The agent, on its conversation, taking a turn: proof it came through. */
- const goesOn = async (agent: Row, content: string) => {
- const client = await LocalClient.connect(daemon!)
- await until('the agent back on its conversation', async () => {
- const row = (await rows(client)).find((one) => one.id === agent.id)
- return row?.status === 'active' && row.sessionId === agent.sessionId ? row : null
- }, 60_000, 500)
- await turn(client, agent.id, content)
- client.close()
- }
- /** One daemon: the core this test started has left, and the one on the port is the master's child. */
- const oneDaemon = async (under: number) => {
- await until('the core this test started to have left', () => !IsolatedDaemon.alive(daemon!.pid) || null, 30_000, 250)
- const cores = await until('one core on the port', () => { const now = listeners(); return now.length === 1 ? now : null }, 30_000, 250)
- expect(parentOf(cores[0])).toBe(under)
- }
-
- it('hands each update to a master, which judges it; refuses one whose master cannot start; agents go on', async () => {
- const d = daemon!
- onTestFailed(() => { console.log(`---- the core's log\n${d.log().split('\n').slice(-60).join('\n')}\n---- the daemon's log file\n${masterLog().split('\n').slice(-120).join('\n')}`) })
+ it('one an older release started hands itself to a master once the release has gone, whose updater updates it', async () => {
+ offered = FIRST
+ install(FIRST)
+ const mayLeave = join(scratch, 'release-may-leave')
+ rmSync(mayLeave, { force: true })
+ const d = await IsolatedDaemon.create({ scriptPath: join(cliDir(), 'cli.js'), noMaster: true, env: environment() })
+ daemon = d
+ ;(d.options as { launch?: string[] }).launch = [join(CLI_ROOT, 'e2e/harness/olderReleaseHandoff.mjs'), join(cliDir(), 'cli.js'), join(d.dataDir, 'adapter.pid'), mayLeave]
+ onTestFailed(() => { console.log(`---- daemon log\n${logs().split('\n').slice(-150).join('\n')}`) })
+ await d.start()
+ expect((await status())?.harnessd).toBeFalsy()
const client = await LocalClient.connect(d)
const cwd = join(d.projectsDir, 'alone')
mkdirSync(cwd, { recursive: true })
@@ -176,77 +179,46 @@ describe('a self-update of a core with no master', () => {
await turn(client, agent.id, `on ${FIRST}, on its own`)
client.close()
- // A build whose core crashes: the core on its own stages it and hands the machine to a master on it,
- // which finds the update unjudged, puts its first core on probation, rolls it back when it crashes and
- // re-executes onto the build before.
- offered = '44.0.2'
- await until('the core to hand 44.0.2 to a master', () => d.log().includes('handing 44.0.2 to a harnessd master, which judges it') || null, 90_000, 250)
- await until('the master to roll it back', () => masterLog().includes('the updated core failed (code 3) — rolled back') || null, 90_000, 250)
- master = (await underMaster(FIRST)).harnessd.masterPid
- expect(master).not.toBe(d.pid)
- expect(installed()).toEqual(releases.get(FIRST))
- expect(rejected()).toEqual(['44.0.2'])
- expect(existsSync(join(cliDir(), 'update-pending.json'))).toBe(false)
- await oneDaemon(master!)
- await goesOn(agent, `back on ${FIRST}, under the master`)
+ // The release that started it is still judging it: it stays put meanwhile, and runs no updater.
+ const before = await status()
+ await sleep(3_000)
+ expect((await status())?.pid).toBe(before?.pid)
+ expect(d.log()).not.toContain('[update] self-update on')
- // On its own again (an older release's handoff, HARNESS_NO_MASTER=1), and a build whose master cannot
- // start: refused before anything is let go, and this build goes on under a master of its own.
- process.kill(master!, 'SIGTERM')
- await until('the master to stop', () => (!IsolatedDaemon.alive(master) && listeners().length === 0) || null, 30_000, 250)
- await d.start()
- expect((await status())?.harnessd).toBeFalsy()
- const from = d.log().length
- const masterFrom = masterLog().length
- offered = '44.0.3'
- await until('the core to refuse 44.0.3', () => d.log().slice(from).includes('44.0.3\'s master did not answer its probe') || null, 90_000, 250)
+ // The release leaves: the core hands the machine to a master on this build.
+ writeFileSync(mayLeave, '')
+ await until('the core to hand over', () => d.log().includes(`handing ${FIRST} to a harnessd master, which runs the updater`) || null, 60_000, 250)
master = (await underMaster(FIRST)).harnessd.masterPid
- expect(rejected()).toEqual(['44.0.2', '44.0.3'])
- expect(installed()).toEqual(releases.get(FIRST))
- await oneDaemon(master!)
- // Its master started on this build, with nothing to judge: 44.0.3 never ran.
- expect(masterLog().slice(masterFrom)).not.toContain('on probation')
- await goesOn(agent, `on ${FIRST}, under a master of its own`)
+ const cores = await until('one core on the port', () => { const now = listeners(); return now.length === 1 ? now : null }, 60_000, 250)
+ expect(parentOf(cores[0])).toBe(master)
+ await goesOn(agent, `on ${FIRST}, under a master`)
- // A good build: now the master's to apply, re-executing on it in its own process, and to keep.
- offered = '44.0.4'
- await underMaster('44.0.4')
- await until('the master to keep it', () => masterLog().includes('the update stayed up — keeping it') || null, 60_000, 250)
+ // The master's updater: a build whose core crashes is rolled back and remembered, a good one is kept.
+ offered = '44.0.2'
+ await until('the master to roll 44.0.2 back', () => logs().includes('the updated core failed (code 3) — rolled back') || null, 120_000, 250)
+ await underMaster(FIRST)
+ expect(rejected()).toEqual(['44.0.2'])
+ expect(installed()).toEqual(releases.get(FIRST))
+ offered = '44.0.3'
+ await underMaster('44.0.3')
+ await until('the master to keep it', () => logs().includes('the update stayed up — keeping it') || null, 60_000, 250)
expect((await status())?.harnessd.masterPid).toBe(master)
- expect(installed()).toEqual(releases.get('44.0.4'))
- expect(rejected()).toEqual(['44.0.2', '44.0.3'])
- await goesOn(agent, 'on 44.0.4, kept')
-
- // The boot handoff: a core on its own in safe mode (its start-up threw; only its updater runs) hands an
- // update over without finishing start-up. A build whose master cannot start is rolled back first, and
- // this build's master starts, out of safe mode: the safe mode was the core's, not the master's to keep.
- const inSafeMode = async () => {
- process.kill(master!, 'SIGTERM')
- await until('the master to stop', () => (!IsolatedDaemon.alive(master) && listeners().length === 0) || null, 30_000, 250)
- d.env.HARNESSD_SAFE_MODE = 'an e2e start-up that failed'
- await d.start({ ready: 'none' })
- await until('the core to be in safe mode', async () => (await status())?.safeMode === true || null, 60_000, 250)
- delete d.env.HARNESSD_SAFE_MODE
- }
- await inSafeMode()
- offered = '44.0.5'
- await until('the boot handoff to refuse 44.0.5', () => d.log().includes('44.0.5\'s master did not answer its probe') || null, 90_000, 250)
- master = (await underMaster('44.0.4')).harnessd.masterPid
- expect(rejected()).toEqual(['44.0.2', '44.0.3', '44.0.5'])
- expect(installed()).toEqual(releases.get('44.0.4'))
- await oneDaemon(master!)
- expect((await status())?.safeMode).toBeFalsy()
- await goesOn(agent, 'on 44.0.4, out of safe mode under a master')
+ expect(installed()).toEqual(releases.get('44.0.3'))
+ await goesOn(agent, 'on 44.0.3, kept')
+ }, 600_000)
- // …and a good build, which the master judges on its first core and keeps.
- await inSafeMode()
- const keptBefore = masterLog().split('the update stayed up — keeping it').length
- offered = '44.0.6'
- await until('the boot handoff to hand 44.0.6 over', () => d.log().includes('44.0.6 staged during start-up — handing off to harnessd\'s master') || null, 90_000, 250)
- master = (await underMaster('44.0.6')).harnessd.masterPid
- await until('the master to keep it', () => masterLog().split('the update stayed up — keeping it').length > keptBefore || null, 60_000, 250)
- expect(installed()).toEqual(releases.get('44.0.6'))
- await oneDaemon(master!)
- await goesOn(agent, 'on 44.0.6, kept')
- }, 900_000)
+ it('one HARNESS_NO_MASTER=1 asked for runs without updates', async () => {
+ offered = '44.0.3'
+ install(FIRST)
+ const d = await IsolatedDaemon.create({ scriptPath: join(cliDir(), 'cli.js'), noMaster: true, env: { ...environment(), HARNESS_NO_MASTER: '1' } })
+ daemon = d
+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-80).join('\n')}`) })
+ await d.start()
+ // A check a second, for five: nothing downloaded, nothing staged, the same core.
+ const pid = (await status())?.pid
+ await sleep(5_000)
+ expect(await status()).toMatchObject({ version: FIRST, pid })
+ expect(installed()).toEqual(releases.get(FIRST))
+ expect(d.log()).not.toMatch(/\[update\] (newer build available|staged)/)
+ }, 300_000)
})
diff --git a/cli/src/architecture.spec.ts b/cli/src/architecture.spec.ts
index 4b008108f..53f1098c8 100644
--- a/cli/src/architecture.spec.ts
+++ b/cli/src/architecture.spec.ts
@@ -152,8 +152,10 @@ const WALK_TIMEOUT_MS = 60_000
* D2): the devices' link to their own process and its routes, as every service there has, 2,213 to 2,232.
* Then 2,230 (step 9, D3): the Wi-Fi device's wiring went with them (core/wifi.ts, core/wifiAgents.ts),
* 2,232 lines to 2,227, though its link to the devices' process came in.
+ *
+ * Down to 2,215 the same day, at 2,209 from 2,228: the updater left the core for a process the master runs.
*/
-const RUN_FOREGROUND_BUDGET = 2_230
+const RUN_FOREGROUND_BUDGET = 2_215
/** Lowered from 2,180 when the relay and its E2EE left the socket for the gateway (step 10, R1: 1,440).
* The Wi-Fi device's relay came back to it in R2, beside the device service it answers for, over the
* gateway's sessions (lib/autonomous-device/overGateway.ts): 1,460. Models' grid commands, grid name and
@@ -305,7 +307,11 @@ const SERVICE_MAY_IMPORT: Record<string, string> = {
// session create, kill and rename). Before tmux 3.7 the two meeting crashed the tmux server, and every agent
// with it (windows.e2e.ts, 7 of 27 CI runs on Ubuntu's 3.4). It guards the core's own terminals, so it
// cannot move to a service. The budget keeps the 101 lines of room it had.
-const CORE_CLOSURE_BUDGET = 73_512
+//
+// Then at 72,134 in 333, from 73,411 in 338: the updater left the core for a process the master runs
+// (services/updaterProcess.ts), and the core never downloads a build. The CLI's and hn's updaters
+// (lib/selfUpdate.ts, tui/update.ts, tui/install.ts) went, with the spawn lock they staged under.
+const CORE_CLOSURE_BUDGET = 72_230
/** What is not the core's, by path: each goes to a service or its own process, in the plan's order. */
const EDGE: RegExp[] = [
@@ -323,6 +329,9 @@ const EDGE: RegExp[] = [
/^dsh\/(catalog|install|update|updates|registry|wire|service|lock|builtins|viewer|viewerLedger|verdict|artifacts)\.ts$/,
// Search's index; the readers of other engines' sessions (external.ts, externals/) are the core's, for adoption.
/^lib\/sessionSearch\/(?!external\.ts$|externals\/)/,
+ // Downloading builds: the updater's, in a process the master runs (services/updaterProcess.ts). The core
+ // never downloads a build.
+ /^lib\/(selfUpdate|runtimeInstall)\.ts$/, /^tui\/(update|install)\.ts$/,
]
/**
diff --git a/cli/src/core/main.ts b/cli/src/core/main.ts
index 64fb3e8f6..4eb8024aa 100644
--- a/cli/src/core/main.ts
+++ b/cli/src/core/main.ts
@@ -12,7 +12,7 @@
import { ensureBundledCoreHarnesses } from '../dsh/builtins.js'
import { readFileSync, writeFileSync, openSync, existsSync, rmSync, statSync } from 'fs'
import { join } from 'path'
-import { execFile, spawn, spawnSync } from 'child_process'
+import { execFile, spawn } from 'child_process'
import { createServer, type Server } from 'http'
import { homedir, hostname } from 'os'
import { env } from '../config/env.js'
@@ -28,10 +28,9 @@ import { engineSessionTitle } from '../lib/sessionTitle.js'
import { machineNames } from '../lib/machineNames.js'
import { installCodexHooks } from '../lib/hooks.js'
import { DAEMON_LOG_FILE, PID_FILE, daemonPort, isAlive, readPid, LEGACY_LOG_FILE, MACHINE_NAME_FILE, tildify, computerId, thisDeviceLabel } from '../lib/daemonState.js'
-import { clearSafeModeMarker, runBootHandoff, safeModeDisposition, safeModeStatusBody, SafeModeRequest, writeSafeModeMarker } from '../lib/daemonSafeMode.js'
+import { clearSafeModeMarker, safeModeDisposition, safeModeStatusBody, SafeModeRequest, writeSafeModeMarker } from '../lib/daemonSafeMode.js'
import { awakeTimeout } from '../lib/sleepAware.js'
import { removePidFileIf, onError } from '../lib/daemonLaunch.js'
-import { describeSpawnLockOwner, withSpawnLock } from '../lib/daemonSpawnLock.js'
import { ensureTmuxOnPath } from '../lib/tmuxOnPath.js'
import { AUTH_DIR, AuthSessionError, AuthSessionManager, clearAuthSession, ensureSignInEpoch, readAuthSession, signInOf, type AuthSession } from '../lib/authSession.js'
import { ENGINES, enginePathOverride } from '../lib/engineBin.js'
@@ -88,7 +87,8 @@ import { createQuestions } from './questions.js'
import { createTurnActivity } from './turns/activity.js'
import { createLastTurnReader } from './transcripts/lastTurn.js'
import { createRecaps } from './turns/recaps.js'
-import { createUpdateHandoff, probeStagedMaster, probeVerdict } from './updateHandoff.js'
+import { createUpdateHandoff, handOverOnceReleased, probeStagedMaster, type TeardownStep } from './updateHandoff.js'
+import { needsUpdaterBeside, startUpdaterBeside } from './updaterBeside.js'
import { createHeartbeats } from './turns/heartbeats.js'
import { createEventFunnel } from './turns/funnel.js'
import { createAgyBackstop } from './turns/agyBackstop.js'
@@ -139,7 +139,7 @@ import { daemonCommand } from '../lib/daemonCommand.js'
import { KNOWN_SERVICES, servicesTheMasterRuns } from '../harnessd/services.js'
import { createTeamsLink, teamsOutOfProcess } from './teamsLink.js'
import { createDevicesLink } from './devicesLink.js'
-import { CORE_EXIT_STOP, CORE_EXIT_UPDATE, PROBE_COMMAND, PROBE_TIMEOUT_MS } from '../harnessd/protocol.js'
+import { CORE_EXIT_STOP, CORE_EXIT_UPDATE, PROBE_COMMAND } from '../harnessd/protocol.js'
import { localSocketPath, refuseServedDataFolder, type LocalSocketServer } from '../lib/localSocket.js'
import { saveDaemonPort } from '../lib/daemonEndpoint.js'
import { publishHookRoute } from '../lib/hookRoutes.js'
@@ -154,9 +154,8 @@ import { autonomousDeviceLocalRequest } from '../lib/autonomous-device/localApi.
import { attachLocalWsServer, LOCAL_WS_PATH, LOCAL_WS_PROTOCOL_VERSION } from '../localWsServer.js'
import { TERMINAL_BINARY_VERSION } from '../lib/terminalBinary.js'
import { isLoopbackRequest, loopbackHosts } from '../lib/loopbackRequest.js'
-import { startSelfUpdater, restore as restoreUpdate, DOWNLOAD_LIMITS, type Poller } from '../lib/selfUpdate.js'
+import { isInstalledCopy } from '../lib/installedCopy.js'
import { managedNodePath } from '../lib/nodeRuntime.js'
-import { startTuiUpdater } from '../tui/update.js'
import { type ActivityFrame } from '../lib/turnActivity.js'
import { CursorTranscriptDiscovery } from '../engines/cursor/discovery.js'
import { cursorDataDir } from '../engines/cursor/home.js'
@@ -269,23 +268,23 @@ function startMasterHere(): void {
* What a staged update does while the daemon is still starting up — and the little the boot needs to
* know about itself to do it.
*
- * The self-updater is started in `runForeground`'s prologue, before anything that can throw or hang,
- * because a daemon that cannot finish booting is a daemon that can never be fixed: there is no
- * supervisor, and the desktop app only re-runs `harness start` on the same broken bytes, once a
- * minute, for ever. Its `onStaged` therefore has to mean something LONG before `restartForUpdate`
- * exists — hence the indirection: `applyStagedUpdate` is `bootHandoff` until the body has built
- * everything `restartForUpdate` tears down, and is swapped for it at that one line.
+ * The master's updater (services/updaterProcess.ts) can stage a build at any moment, and the master then
+ * asks this core to hand over (`harnessd:update`), which `runForeground`'s prologue listens for before
+ * anything that can throw or hang: a daemon that cannot finish booting must still leave for its fix. That
+ * request therefore has to mean something LONG before `restartForUpdate` exists — hence the indirection:
+ * `applyStagedUpdate` is `bootHandoff` until the body has built everything `restartForUpdate` tears down,
+ * and is swapped for it at that one line.
*/
/** This process's channel to a harnessd master, when one started it (see harnessd/coreLink.ts).
- * Inert otherwise: a daemon run on its own claims its pid file and hands off updates itself. */
+ * Inert otherwise: a daemon run on its own claims its pid file, and gets no updates. */
const coreLink = connectToMaster()
/** When the devices' process starts: once there is a device, a dial's port, a Wi-Fi device or a request for one
* (core/devicesWake.ts). Asked of the master through the channel above. */
const devicesWake = createDevicesWake({ want: (service) => coreLink.want(service), dataDir: env.ADAPTER_DATA_DIR, cableDisabled: env.CABLE_DISABLE, testDialPort: process.env.HARNESSD_TEST_DIAL_PORT })
const daemonBoot: {
- updater: Poller | null
- tuiUpdater: Poller | null
+ /** Stops the updater this core runs beside itself under a master too old to run it (core/updaterBeside.ts). */
+ updaterBeside: (() => void) | null
/** The hook server, once bound — the only thing a mid-boot handoff has to release. */
hookServer: Server | null
/** Its Unix-socket twin (lib/localSocket.ts), when one could be opened. Read by `/api/status`. */
@@ -298,52 +297,24 @@ const daemonBoot: {
applyStagedUpdate: (version: string) => void | Promise<void>
/** Opens the request gate of a start-up that did not finish, so its clients are answered (safe mode). */
openRequests: (() => void) | null
-} = { updater: null, tuiUpdater: null, hookServer: null, localSocket: null, markNotReady: null, safeMode: null, handingOff: false, applyStagedUpdate: bootHandoff, openRequests: null }
+} = { updaterBeside: null, hookServer: null, localSocket: null, markNotReady: null, safeMode: null, handingOff: false, applyStagedUpdate: bootHandoff, openRequests: null }
/**
- * Hand the machine to a newer build without finishing start-up.
+ * Hand the machine to a newer build without finishing start-up: the master asked (its updater staged one),
+ * and starts the new bundle the moment this exits, and judges it.
*
* SYNCHRONOUS END TO END, and that is the whole safety argument: never awaiting means the half-built
- * `runForeground` body cannot interleave between the port closing and the exit, so it can never
- * reach the code that would bind the port the successor is about to take, and two daemons are
- * impossible by construction. That is also why it does not wait for the master to claim the pid file the
- * way `restartForUpdate` does — waiting would leave this process running alongside the new core, both
- * reconciling tmux and writing the registry.
- *
- * Without a master it spawns one rather than merely exiting, because on a machine with no desktop app
- * nothing else would ever start the successor, and even with one the next spawn window is up to ~70s
- * away. The master judges the update it starts on (core/updateHandoff.ts); the core this once spawned
- * instead ran it unjudged.
+ * `runForeground` body cannot interleave between the port closing and the exit, so it can never reach the
+ * code that would bind the port the successor is about to take. Only a master asks; a core without one
+ * gets no updates.
*/
function bootHandoff(version: string): void {
if (daemonBoot.handingOff) return
daemonBoot.handingOff = true
- daemonBoot.tuiUpdater?.stop()
- if (coreLink.supervised) {
- // The master starts the new bundle the moment this exits, and rolls it back if it does not stay
- // up; a successor spawned from here would be a daemon outside its supervision.
- console.log(`[update] ${VERSION} → ${version} staged during start-up — handing back to harnessd`)
- try { daemonBoot.hookServer?.close() } catch { /* already gone */ }
- try { daemonBoot.localSocket?.closeSync() } catch { /* already gone */ }
- process.exit(CORE_EXIT_UPDATE)
- }
- runBootHandoff(VERSION, version, {
- // The hook port has no fallback: a successor that cannot bind it is a daemon that does not come up.
- closeServer: () => {
- try { (daemonBoot.hookServer as unknown as { closeAllConnections?: () => void } | null)?.closeAllConnections?.() } catch { /* already gone */ }
- try { daemonBoot.hookServer?.close() } catch { /* already gone */ }
- try { daemonBoot.localSocket?.closeSync() } catch { /* already gone */ }
- },
- // Synchronous, as all of this is: start-up has not finished, and has nothing to serve meanwhile.
- probeMaster: () => {
- const probe = spawnSync(managedNodePath(), [SCRIPT_PATH, PROBE_COMMAND], { encoding: 'utf8', timeout: PROBE_TIMEOUT_MS })
- return probeVerdict(probe.error ?? (probe.status === 0 ? null : new Error(probe.signal ? `signal ${probe.signal}` : `exit ${probe.status}`)), probe.stdout ?? '')
- },
- rollBack: () => restoreUpdate(env.ADAPTER_CLI_DIR),
- startMaster: startMasterHere,
- exit: (code) => process.exit(code),
- log: (message) => console.log(message),
- })
+ console.log(`[update] ${VERSION} → ${version} staged during start-up — handing back to harnessd`)
+ try { daemonBoot.hookServer?.close() } catch { /* already gone */ }
+ try { daemonBoot.localSocket?.closeSync() } catch { /* already gone */ }
+ process.exit(CORE_EXIT_UPDATE)
}
/** Set by runForeground once the DSH companions exist; a frame projected before that carries none. */
@@ -398,75 +369,49 @@ async function runForeground(session: AuthSession | null): Promise<void> {
console.error('[fatal-guard] uncaughtException:', err instanceof Error ? (err.stack ?? err.message) : err)
})
- // ── THE UPDATER GOES FIRST. Everything below this point can throw, hang, or wait on a vendor file,
- // a port, or tmux — and a daemon that never finishes starting is a daemon that can never be fixed:
- // there is no supervisor, and the desktop app only re-runs `harness start` on the same broken bytes.
- // Started here, a published fix lands on its own however badly the rest of the boot goes.
+ // ── THE MASTER'S UPDATE IS HEARD FIRST. Everything below this point can throw, hang, or wait on a vendor
+ // file, a port, or tmux — and a daemon that never finishes starting must still leave for its fix. The
+ // updater is the master's, in a process of its own (services/updaterProcess.ts): the core never
+ // downloads a build. When it stages one, the master asks this core to hand over, and the request is
+ // listened for here, before anything that can fail.
//
- // `onStaged` is one indirection on purpose: the handoff's teardown does not exist yet and must not move
- // (it tears down two dozen subsystems declared further down). Until it is ready, a staged update is
+ // `applyStagedUpdate` is one indirection on purpose: the handoff's teardown does not exist yet and must not
+ // move (it tears down two dozen subsystems declared further down). Until it is ready, a staged update is
// applied by `bootHandoff`, which hands the machine over without finishing start-up.
-
- // Self-update ONLY manages the INSTALLED copy (`~/.harness/cli/cli.js`). A dev/repo run — `tsx`
- // (`npm run dev`) OR `node dist/cli.js` from the checkout — must NEVER self-update: it would swap
- // the published bundle into ~/.harness/cli and restart, hijacking the version you're developing.
- // Match by inode so symlinks/realpath don't fool it; fall back to a path compare.
- const installedCli = join(env.ADAPTER_CLI_DIR, 'cli.js')
- let isInstalledCopy = SCRIPT_PATH === installedCli
- try { isInstalledCopy = statSync(SCRIPT_PATH).ino === statSync(installedCli).ino } catch { /* keep path compare */ }
- if (isInstalledCopy && !env.ADAPTER_UPDATE_DISABLE) {
- daemonBoot.updater = startSelfUpdater({
- currentVersion: VERSION,
- url: env.ADAPTER_UPDATE_URL,
- key: env.ADAPTER_UPDATE_KEY,
- dir: env.ADAPTER_CLI_DIR,
- intervalMs: env.ADAPTER_UPDATE_CHECK_MS,
- slotSecond: env.ADAPTER_UPDATE_SLOT_SEC,
- // The lock spans the byte swap AND the handoff it triggers, as one critical section: a
- // `harness start` that lands between the two would otherwise stage over our .prev, and one
- // that lands during the handoff would spawn a second daemon.
- withLock: (fn) => withSpawnLock('handoff', fn, {
- onWaiting: (owner) => console.log(`[update] waiting — the daemon is ${describeSpawnLockOwner(owner)}`),
- }),
- onStaged: (v) => daemonBoot.applyStagedUpdate(v),
- limits: { idleMs: env.ADAPTER_UPDATE_IDLE_MS, deadlineMs: env.ADAPTER_UPDATE_DEADLINE_MS, floorBytesPerSecond: DOWNLOAD_LIMITS.floorBytesPerSecond },
- stageWhileJudged: !coreLink.supervised || process.env.HARNESSD_JUDGES_SUPERSEDED === '1',
+ coreLink.onUpdate((version) => { void daemonBoot.applyStagedUpdate(version) })
+ // …or, under a master too old to run the updater, from the updater this core runs beside itself
+ // (core/updaterBeside.ts), in a process of its own: the core downloads no build either way.
+ if (needsUpdaterBeside(process.env, coreLink.supervised, isInstalledCopy(SCRIPT_PATH, env.ADAPTER_CLI_DIR), env.ADAPTER_UPDATE_DISABLE)) {
+ daemonBoot.updaterBeside = startUpdaterBeside({
+ spawn: () => {
+ const updater = spawn(managedNodePath(), [SCRIPT_PATH, '__service', 'updater'], {
+ env: { ...process.env, HARNESSD_SERVICE: 'updater', HARNESSD_UPDATER_BESIDE_CORE: '1' }, stdio: ['ignore', 'inherit', 'inherit', 'ipc'],
+ })
+ updater.on('error', (e) => console.error('[update] the updater could not start:', e instanceof Error ? e.message : e))
+ return { onMessage: (listener) => { updater.on('message', listener) }, onExit: (listener) => { updater.on('exit', listener) }, kill: () => { updater.kill() } }
+ },
+ staged: (version) => { void daemonBoot.applyStagedUpdate(version) },
+ log: (line) => console.log(line),
})
- const slotted = env.ADAPTER_UPDATE_SLOT_SEC >= 0 && 60_000 % env.ADAPTER_UPDATE_CHECK_MS === 0
- console.log(`[update] self-update on · v${VERSION} · every ${Math.round(env.ADAPTER_UPDATE_CHECK_MS / 1000)}s`
- + (slotted ? ` at :${String(env.ADAPTER_UPDATE_SLOT_SEC % 60).padStart(2, '0')}` : ''))
- } else if (!env.ADAPTER_UPDATE_DISABLE) {
- console.log(`[update] self-update off · running a dev/repo build (v${VERSION}), not the installed copy`)
}
-
- // Keep the CLI's recovery updater armed first. hn is an independent, optional download.
- daemonBoot.tuiUpdater = startTuiUpdater({
- currentVersion: VERSION,
- isInstalledCopy,
- disabled: env.ADAPTER_UPDATE_DISABLE,
- intervalMs: env.ADAPTER_UPDATE_CHECK_MS,
- slotSecond: env.ADAPTER_UPDATE_SLOT_SEC,
- })
- // The update handoff (core/updateHandoff.ts): after the updaters, never above them (startupOrder.spec.ts),
- // and ahead of the /api/status handler that reads whether it is under way and of shutdown(), which takes
- // a successor it is judging down with us.
+ // The update handoff (core/updateHandoff.ts): after the update is listened for, never above it
+ // (startupOrder.spec.ts), and ahead of the /api/status handler that reads whether it is under way.
const updateHandoff = createUpdateHandoff({
version: VERSION, supervised: coreLink.supervised, exitForUpdate: () => process.exit(CORE_EXIT_UPDATE),
- // A core on its own: the staged bundle's master must answer its probe, as a master asks before it
- // re-executes on one; then a master takes the machine, and judges the update.
+ // A core on its own that an older release started: its master must answer its probe, as a master asks
+ // before it re-executes on a bundle; then the master takes the machine, and runs the updater.
probeMaster: () => probeStagedMaster((done) => execFile(managedNodePath(), [SCRIPT_PATH, PROBE_COMMAND], { encoding: 'utf8' }, (error, stdout) => done(error, stdout))),
- rollBack: () => restoreUpdate(env.ADAPTER_CLI_DIR),
handOff: () => { startMasterHere(); process.exit(0) },
log: (line) => console.log(line), error: (line) => console.error(line),
})
// Another daemon serves this data folder: leave before reading or writing anything of its — its
- // registry, its viewers, its panes. Just after the updaters, never before them (startupOrder.spec.ts);
- // their first check is a slot away, and a daemon that leaves here is gone long before it.
+ // registry, its viewers, its panes. Just after the update is listened for, never before it
+ // (startupOrder.spec.ts).
await refuseServedDataFolder(localSocketPath(env.ADAPTER_DATA_DIR, env.PORT))
// harnessd's master saw this core crash again and again: start nothing that could do it again. The
- // updaters above keep running, so a published fix still lands (`enterSafeMode`).
+ // master's updater runs on, and a published fix still lands (`enterSafeMode`).
if (process.env.HARNESSD_SAFE_MODE) throw new SafeModeRequest(process.env.HARNESSD_SAFE_MODE)
const savedApis = new ApiConnections(env.ADAPTER_DATA_DIR)
@@ -503,7 +448,7 @@ async function runForeground(session: AuthSession | null): Promise<void> {
// Missing tmux is a STATE, not a reason to refuse to start. The daemon already models a machine
// without it — `tmuxBackend` is null whenever the config omits tmux, every caller tests it, and the
// create/restart/resume paths answer `TMUX_UNAVAILABLE` — so it can still serve its status, the
- // local socket, the backend link and its updater, and say what is missing. Refusing instead left a
+ // local socket and the backend link, leave for an update, and say what is missing. Refusing instead left a
// machine whose PATH lost tmux with a daemon that could not start and therefore could not be fixed.
let tmuxUnavailable: string | null = null
if (tmuxPathPromise) {
@@ -1664,8 +1609,8 @@ async function runForeground(session: AuthSession | null): Promise<void> {
restarting: updateHandoff.restarting(),
discoveryReady,
discoveryError: discoveryError ?? (tmuxUnavailable ? `tmux unavailable: ${tmuxUnavailable}` : null),
- // Present only when start-up failed and this daemon is holding the machine open for its
- // updater. Clients key on `discoveryReady`; this says WHY, in one word, for a person reading it.
+ // Present only when start-up failed and this daemon is holding the machine open until a fix
+ // arrives. Clients key on `discoveryReady`; this says WHY, in one word, for a person reading it.
...(daemonBoot.safeMode ? { safeMode: true } : {}),
// Agents whose history is being read right now, and how many wait their turn. Normally empty or
// gone in a second; one that stays here names the store that is slow, which no other field does.
@@ -2447,12 +2392,12 @@ async function runForeground(session: AuthSession | null): Promise<void> {
console.log(`[cli] not signed in — serving this computer only · watching registered sessions for ${ENGINES.length} engines`)
}
- // ── self-update: a staged bundle restarts the daemon IMMEDIATELY (core/updateHandoff.ts, and
- // handOffToMaster for a core run on its own). The handler stops being `bootHandoff` HERE, and not a
- // line earlier: everything the teardown releases exists by now. A straight-line assignment, never a
- // wait: if the body never reaches this line the handler stays `bootHandoff`, and the fix still lands.
- daemonBoot.applyStagedUpdate = (v) => updateHandoff.restartForUpdate(v, [
- ['the registry', () => registry.flush()], ['the updaters', () => { daemonBoot.updater?.stop(); daemonBoot.tuiUpdater?.stop() }],
+ // ── self-update: a staged bundle restarts the daemon IMMEDIATELY (core/updateHandoff.ts). The handler
+ // stops being `bootHandoff` HERE, and not a line earlier: everything the teardown releases exists by now.
+ // A straight-line assignment, never a wait: if the body never reaches this line the handler stays
+ // `bootHandoff`, and the fix still lands.
+ const updateTeardown = (): TeardownStep[] => [
+ ['the registry', () => registry.flush()], ['the updater', () => daemonBoot.updaterBeside?.()],
['the reconciler', () => agentReconciler.stop()],
['the timers', () => { clearInterval(logTrimTimer); clearInterval(runtimeReconcileTimer); clearInterval(paneTitleSyncTimer) }],
['the question watchers', () => questionWatcher.stopAll()],
@@ -2472,7 +2417,15 @@ async function runForeground(session: AuthSession | null): Promise<void> {
['the viewers', () => ports.viewers?.stop()], ['the gateway', () => gateway.stop()],
// A graceful close releases the backend's one-machine claim, given a moment before the reclaim.
['the backend', () => backend.stop()], ['a grace', () => new Promise((r) => setTimeout(r, 1000))],
- ])
+ ]
+ daemonBoot.applyStagedUpdate = (v) => updateHandoff.restartForUpdate(v, updateTeardown())
+ // A core an older release's own handoff started (it spawned this `cli.js __run` and judged it) has no
+ // master, and so no updater: it would run this build until its next start. Once that release has gone,
+ // it hands the machine to a master on this build (core/updateHandoff.ts). Not a core HARNESS_NO_MASTER=1
+ // asked for, and not a dev or repo run.
+ if (!coreLink.supervised && process.env.ADAPTER_UPDATED_TO && process.env.HARNESS_NO_MASTER !== '1' && isInstalledCopy(SCRIPT_PATH, env.ADAPTER_CLI_DIR)) {
+ handOverOnceReleased({ startedBy: process.ppid, parent: () => process.ppid, alive: isAlive, handOver: () => { void updateHandoff.handOver(updateTeardown()) } })
+ }
/** Stopping for good — removed from the account, or connected from elsewhere: tell harnessd's master,
* which restarts any other exit (harnessd/protocol.ts). */
@@ -2488,8 +2441,7 @@ async function runForeground(session: AuthSession | null): Promise<void> {
// Release the serial port first. It is exclusive, and a daemon that exits still holding it makes
// esptool fail in a way that reads exactly like dead hardware.
void ports.devices?.stop()
- daemonBoot.updater?.stop()
- daemonBoot.tuiUpdater?.stop()
+ daemonBoot.updaterBeside?.()
agentReconciler.stop()
clearInterval(logTrimTimer)
clearInterval(runtimeReconcileTimer)
@@ -2602,14 +2554,14 @@ async function runForeground(session: AuthSession | null): Promise<void> {
}
/**
- * The daemon's start-up threw. STAY UP anyway, running nothing but the updater.
+ * The daemon's start-up threw. STAY UP anyway, waiting for a fix.
*
- * Exiting here is what made one bad build unrecoverable: nothing supervises this process, the desktop
- * app answers a dead port by running `harness start` again — the same bytes, about once a minute, for
- * ever — and the updater that could have fixed it lives most of the way down a body that never
- * finished. The updater is started in the prologue now (see `runForeground`), so by the time this
- * runs it is already polling; all this has to do is keep the process alive long enough for a
- * published fix to land, and tell everyone what state the machine is in.
+ * Exiting here is what made one bad build unrecoverable: the desktop app answers a dead port by running
+ * `harness start` again — the same bytes, about once a minute, for ever. The updater is the master's, in a
+ * process of its own (services/updaterProcess.ts), and the request to hand over for its fix is listened for
+ * in the prologue (see `runForeground`); all this has to do is keep the process alive long enough for a
+ * published fix to land, and tell everyone what state the machine is in. A core with no master gets no fix
+ * here: its safe mode runs out (`ADAPTER_SAFE_MODE_MS`) and a clean start tries again.
*
* Three ways it earns its keep, in order: the bound control port answers `discoveryReady: false`, so
* the app reads the machine as not-ready instead of dead and STOPS respawning; the pid file stays
@@ -2627,8 +2579,8 @@ const enterSafeMode = (err: unknown): void => {
}
const detail = err instanceof Error ? (err.stack ?? err.message) : String(err)
console.error('Failed to start adapter:', err)
- console.error(`[safe-mode] staying up on v${VERSION} with the updater only — a published fix will be`
- + ' applied on its own. Nothing else on this machine works until then.')
+ console.error(`[safe-mode] staying up on v${VERSION} — a published fix will be applied on its own.`
+ + ' Nothing else on this machine works until then.')
writeSafeModeMarker(env.ADAPTER_DATA_DIR, { pid: process.pid, version: VERSION, at: Date.now(), error: detail })
daemonBoot.safeMode = disposition.reason
daemonBoot.markNotReady?.(disposition.reason)
@@ -2643,8 +2595,8 @@ const enterSafeMode = (err: unknown): void => {
}
process.on('SIGINT', () => leave('SIGINT — leaving safe mode', 0))
process.on('SIGTERM', () => leave('SIGTERM — leaving safe mode', 0))
- // Up, though not ready: the master must neither give up waiting for a bind nor take it for hung,
- // or the updater that can fix this build would never get its chance. It hears why, and rolls back an
+ // Up, though not ready: the master must neither give up waiting for a bind nor take it for hung, or the
+ // fix its updater stages would find no core to hand over. It hears why, and rolls back an
// update whose first core ends up here.
coreLink.bound(daemonPort())
coreLink.ready(disposition.reason)
@@ -2668,7 +2620,7 @@ const enterSafeMode = (err: unknown): void => {
})
status.on('error', (e) => {
console.error(`[safe-mode] could not serve status on ${port}: ${e instanceof Error ? e.message : e}`)
- // A ref'd timer, unlike the updater's: something has to hold the event loop open.
+ // A ref'd timer: something has to hold the event loop open.
setInterval(() => console.log(`[safe-mode] still waiting for a fixed build · v${VERSION}`), 10 * 60_000)
})
status.listen(port, '127.0.0.1')
diff --git a/cli/src/core/updateHandoff.spec.ts b/cli/src/core/updateHandoff.spec.ts
index 547a4aa2d..deed5c0cd 100644
--- a/cli/src/core/updateHandoff.spec.ts
+++ b/cli/src/core/updateHandoff.spec.ts
@@ -1,6 +1,8 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { PROBE_ANSWER } from '../harnessd/protocol.js'
-import { TEARDOWN_DEADLINE_MS, createUpdateHandoff, probeStagedMaster, probeVerdict, type TeardownStep, type UpdateHandoffDeps } from './updateHandoff.js'
+import {
+ RELEASED_POLL_MS, TEARDOWN_DEADLINE_MS, createUpdateHandoff, handOverOnceReleased, probeStagedMaster, probeVerdict, type TeardownStep, type UpdateHandoffDeps,
+} from './updateHandoff.js'
describe('the update handoff', () => {
afterEach(() => { vi.useRealTimers() })
@@ -11,7 +13,6 @@ describe('the update handoff', () => {
version: '1.0.0', supervised: true,
exitForUpdate: () => calls.push('exit 75'),
probeMaster: async () => { calls.push('probe the master'); return null },
- rollBack: () => calls.push('roll back'),
handOff: () => calls.push('start a master, exit'),
log: (line) => calls.push(line),
error: (line) => calls.push(line),
@@ -68,30 +69,34 @@ describe('the update handoff', () => {
expect(plain.calls.at(-1)).toBe('exit 75')
})
- it('without a master, gives the machine to a master on the staged build once its master answers, after the same teardown', async () => {
+ it('without a master, hands this build to one once its master answers, after the same teardown, once', async () => {
const { handoff, calls } = make({ supervised: false })
- const done = handoff.restartForUpdate('2.0.0', [step(calls, 'the registry'), step(calls, 'the backend', () => { throw new Error('no') })])
+ const done = handoff.handOver([step(calls, 'the registry'), step(calls, 'the backend', () => { throw new Error('no') })])
// Asked before anything is let go: the core serves on meanwhile.
expect(handoff.restarting()).toBe(false)
- await handoff.restartForUpdate('2.0.0', [])
- await done
+ expect(await handoff.handOver([])).toBe(false)
+ expect(await done).toBe(true)
expect(handoff.restarting()).toBe(true)
expect(calls).toEqual([
- '[update] applying 1.0.0 → 2.0.0 — restarting daemon', 'probe the master', 'release the registry', 'release the backend',
- '[update] the backend did not let go (no) — handing over all the same',
- '[update] handing 2.0.0 to a harnessd master, which judges it — this core is leaving', 'start a master, exit',
+ 'probe the master',
+ '[update] handing 1.0.0 to a harnessd master, which runs the updater — this core is leaving',
+ 'release the registry', 'release the backend', '[update] the backend did not let go (no) — handing over all the same',
+ 'start a master, exit',
])
+ // Leaving, it hands nothing else over.
+ await handoff.restartForUpdate('2.0.0', [])
+ expect(calls.at(-1)).toBe('start a master, exit')
})
- it('without a master, rolls a build whose master does not answer back, and hands this build to a master of its own', async () => {
- const { handoff, calls } = make({ supervised: false, probeMaster: async () => 'harnessd-probe failed: a bad build' })
- await handoff.restartForUpdate('2.0.0', [step(calls, 'the registry')])
- expect(calls).toEqual([
- '[update] applying 1.0.0 → 2.0.0 — restarting daemon',
- '[update] 2.0.0\'s master did not answer its probe (harnessd-probe failed: a bad build) — rolled back; this build goes on under a master of its own',
- 'roll back', 'release the registry',
- '[update] handing 1.0.0 to a harnessd master, which judges it — this core is leaving', 'start a master, exit',
- ])
+ it('without a master, carries on when its master does not answer, and may try again', async () => {
+ let answers = ['harnessd-probe failed: a bad build', null]
+ const { handoff, calls } = make({ supervised: false, probeMaster: async () => answers.shift() ?? null })
+ expect(await handoff.handOver([step(calls, 'the registry')])).toBe(false)
+ expect(handoff.restarting()).toBe(false)
+ expect(calls).toEqual(['[update] this build\'s master did not answer its probe (harnessd-probe failed: a bad build) — carrying on without one'])
+ expect(await handoff.handOver([step(calls, 'the registry')])).toBe(true)
+ expect(calls.at(-1)).toBe('start a master, exit')
+ answers = []
})
it('under harnessd, asks no probe: the master asks its own before it re-executes', async () => {
@@ -101,6 +106,53 @@ describe('the update handoff', () => {
})
})
+describe('handing over once the release that started this core has gone', () => {
+ const make = (alive: boolean[], parents: number[]) => {
+ let tick: (() => void) | null = null
+ const calls: string[] = []
+ const stop = handOverOnceReleased({
+ startedBy: 77,
+ parent: () => parents.shift() ?? 77,
+ alive: () => alive.shift() ?? true,
+ handOver: () => calls.push('hand over'),
+ setTimer: (run, ms) => { calls.push(`every ${ms}`); tick = run; return 'timer' },
+ clearTimer: (timer) => calls.push(`cleared ${String(timer)}`),
+ })
+ return { tick: () => tick!(), calls, stop }
+ }
+
+ it('waits while the release is there, and hands over once, when its process ends', () => {
+ const m = make([true, true, false], [77, 77, 77])
+ m.tick()
+ m.tick()
+ expect(m.calls).toEqual([`every ${RELEASED_POLL_MS}`])
+ m.tick()
+ expect(m.calls).toEqual([`every ${RELEASED_POLL_MS}`, 'cleared timer', 'hand over'])
+ })
+
+ it('hands over when this core is reparented away from it, and can be stopped before', () => {
+ const m = make([], [1])
+ m.tick()
+ expect(m.calls.slice(-1)).toEqual(['hand over'])
+ const stopped = make([], [])
+ stopped.stop()
+ expect(stopped.calls).toEqual([`every ${RELEASED_POLL_MS}`, 'cleared timer'])
+ })
+
+ it('polls on an interval that keeps no process alive by default', () => {
+ vi.useFakeTimers()
+ const handOver = vi.fn()
+ let gone = false
+ const stop = handOverOnceReleased({ startedBy: 77, parent: () => 77, alive: () => !gone, handOver })
+ vi.advanceTimersByTime(RELEASED_POLL_MS * 3)
+ expect(handOver).not.toHaveBeenCalled()
+ gone = true
+ vi.advanceTimersByTime(RELEASED_POLL_MS * 3)
+ expect(handOver).toHaveBeenCalledOnce()
+ stop()
+ })
+})
+
describe('the probe of a staged bundle\'s master', () => {
/** A probe that ends as told, and the deadline it is given. */
const run = (error: Error | null, stdout: string) => {
diff --git a/cli/src/core/updateHandoff.ts b/cli/src/core/updateHandoff.ts
index b91815608..3b3c734d8 100644
--- a/cli/src/core/updateHandoff.ts
+++ b/cli/src/core/updateHandoff.ts
@@ -1,20 +1,21 @@
/**
- * The core handing the machine to a newer build its updater has just staged (`selfUpdate.ts`
- * `startSelfUpdater`'s `onStaged`), once start-up has finished: release what the next core needs (the
- * fixed ports, the backend's one-machine claim, the watchers and timers), then go.
+ * The core handing the machine to a newer build harnessd's updater has just staged (the master asks:
+ * `harnessd:update`, harnessd/coreLink.ts), once start-up has finished: release what the next core needs
+ * (the fixed ports, the backend's one-machine claim, the watchers and timers), then exit 75. The master
+ * starts the new bundle the moment this core exits, and judges it.
*
- * Under harnessd the master starts the new bundle the moment this core exits 75, and judges it. A
- * teardown step that threw used to end the handoff where it stood: the core stayed on the old build with
- * its servers half closed, its updater stopped and the new bundle staged but never judged
- * (e2e/updateHostile.e2e.ts, round 40). Now every step is tried, one that fails is said and passed, and
- * the core exits 75 whatever happened; one that hangs is given up on after `TEARDOWN_DEADLINE_MS`.
+ * A teardown step that threw used to end the handoff where it stood: the core stayed on the old build with
+ * its servers half closed and the new bundle staged but never judged (e2e/updateHostile.e2e.ts, round 40).
+ * Now every step is tried, one that fails is said and passed, and the core exits 75 whatever happened; one
+ * that hangs is given up on after `TEARDOWN_DEADLINE_MS`.
*
- * Without a master (a core an older release's own handoff started, e2e/migration.e2e.ts, or
- * `HARNESS_NO_MASTER=1`) the core gives the machine to one, which judges the build as every update: the
- * staged bundle's master must answer its probe first, as before a re-execution (harnessd/reexec.ts), or the
- * build before goes back, rejected; then the same teardown, and `handOff` starts `__harnessd` and exits.
- * The master finds the pending note and puts its first core on probation (harnessd/supervisor.ts,
- * `unjudgedUpdate`). The core once spawned a core like itself and judged it, which then had no master.
+ * A core with no master gets no update: the updater is the master's (services/updaterProcess.ts), and the
+ * core never downloads a build. One an older release's own handoff started (it spawned `cli.js __run`,
+ * e2e/migration.e2e.ts) would run this build until its next start, so once that release has gone it hands
+ * the machine to a master (`handOver`, `handOverOnceReleased`): its master answers its probe first, as a
+ * master asks before it re-executes on a bundle (harnessd/reexec.ts), then the same teardown, then
+ * `handOff` starts `__harnessd`, names it in the pid file and exits. One `HARNESS_NO_MASTER=1` asked for
+ * runs without updates, as asked.
*
* Staged means restart now. The restart once waited for the computer to go idle, and "idle" is a set of
* latches (an open turn, a settling composer, an awaited submit, the control lock, a recap in flight):
@@ -41,11 +42,9 @@ export interface UpdateHandoffDeps {
supervised: boolean
/** Exit for the update (`CORE_EXIT_UPDATE`), to the master that starts the new bundle. */
exitForUpdate(): void
- /** Without a master: whether the staged bundle's master answers its probe (`probeStagedMaster`); null when
- * it does, why not otherwise. */
+ /** Without a master: whether the master of the bundle on disk answers its probe (`probeStagedMaster`); null
+ * when it does, why not otherwise. */
probeMaster(): Promise<string | null>
- /** Without a master: put the build before back (`selfUpdate.restore`), remembering this one as rejected. */
- rollBack(): void
/** Without a master: start harnessd's master on the bundle now on disk, name it in the pid file, exit. */
handOff(): void
log(line: string): void
@@ -83,28 +82,36 @@ export function createUpdateHandoff(deps: UpdateHandoffDeps) {
return {
/** Between the teardown starting and this core leaving (`/api/status`, and a signal mid-handoff). */
restarting: (): boolean => restarting,
- /** Hand over to `newVersion`, releasing `teardown` first. Once: a second call while one is under way does nothing. */
+ /** Hand over to `newVersion`, releasing `teardown` first, and exit for the update. Once: a second call
+ * while one is under way does nothing. */
async restartForUpdate(newVersion: string, teardown: readonly TeardownStep[]): Promise<void> {
if (started) return
started = true
+ restarting = true
deps.log(`[update] applying ${deps.version} → ${newVersion} — restarting daemon`)
- // Asked before anything is let go: the core serves on while it answers.
- const refused = deps.supervised ? null : await deps.probeMaster()
+ await releaseAll(teardown)
+ // Everything above is released, or said not to be; the master starts the new bundle as soon as this
+ // exits and rolls back to the .prev bytes if it does not come up and stay up.
+ deps.log(`[update] handing ${newVersion} to harnessd`)
+ deps.exitForUpdate()
+ },
+ /** Without a master: give the machine to one on this build, once its master answers; otherwise carry on.
+ * Resolves whether it handed over. */
+ async handOver(teardown: readonly TeardownStep[]): Promise<boolean> {
+ if (started) return false
+ started = true
+ // Asked before anything is let go: the core serves on while it answers, and on if it does not.
+ const refused = await deps.probeMaster()
if (refused !== null) {
- deps.error(`[update] ${newVersion}'s master did not answer its probe (${refused}) — rolled back; this build goes on under a master of its own`)
- deps.rollBack()
+ deps.error(`[update] this build's master did not answer its probe (${refused}) — carrying on without one`)
+ started = false
+ return false
}
restarting = true
+ deps.log(`[update] handing ${deps.version} to a harnessd master, which runs the updater — this core is leaving`)
await releaseAll(teardown)
- if (deps.supervised) {
- // Everything above is released, or said not to be; the master starts the new bundle as soon as
- // this exits and rolls back to the .prev bytes if it does not come up and stay up.
- deps.log(`[update] handing ${newVersion} to harnessd`)
- deps.exitForUpdate()
- return
- }
- deps.log(`[update] handing ${refused === null ? newVersion : deps.version} to a harnessd master, which judges it — this core is leaving`)
deps.handOff()
+ return true
},
}
}
@@ -133,9 +140,35 @@ export function probeStagedMaster(run: ProbeRun, timeoutMs = PROBE_TIMEOUT_MS, d
})
}
-/** What a master probe's end says: null when it answered as a master, why not otherwise. Shared by the
- * boot handoff's synchronous probe (lib/daemonSafeMode.ts `runBootHandoff`). */
+/** What a master probe's end says: null when it answered as a master, why not otherwise. */
export function probeVerdict(error: Error | null | undefined, stdout: string): string | null {
if (!error && stdout.includes(PROBE_ANSWER)) return null
return stdout.trim().split('\n').pop()?.trim() || error?.message || 'no answer'
}
+
+/** How often a core an older release started looks for it to have gone. */
+export const RELEASED_POLL_MS = 2_000
+
+/**
+ * Hand the machine to a master once the older release that started this core has gone: it spawned this
+ * core detached, judged it by its bind and its connection (up to a minute and a half) and leaves; a core
+ * that left first would be read as the new build failing, and rolled back. Gone is its process ended, or
+ * this core reparented away from it. Returns a stop.
+ */
+export function handOverOnceReleased(deps: {
+ startedBy: number
+ parent(): number
+ alive(pid: number): boolean
+ handOver(): void
+ setTimer?: (run: () => void, ms: number) => unknown
+ clearTimer?: (timer: unknown) => void
+}): () => void {
+ const setTimer = deps.setTimer ?? ((run, ms) => { const timer = setInterval(run, ms); timer.unref?.(); return timer })
+ const clearTimer = deps.clearTimer ?? ((timer) => clearInterval(timer as ReturnType<typeof setInterval>))
+ const timer = setTimer(() => {
+ if (deps.parent() === deps.startedBy && deps.alive(deps.startedBy)) return
+ clearTimer(timer)
+ deps.handOver()
+ }, RELEASED_POLL_MS)
+ return () => clearTimer(timer)
+}
diff --git a/cli/src/core/updaterBeside.spec.ts b/cli/src/core/updaterBeside.spec.ts
new file mode 100644
index 000000000..d14e1265d
--- /dev/null
+++ b/cli/src/core/updaterBeside.spec.ts
@@ -0,0 +1,88 @@
+import { afterEach, describe, expect, it, vi } from 'vitest'
+import { UPDATER_RESTART_MS, needsUpdaterBeside, startUpdaterBeside, type UpdaterChild } from './updaterBeside.js'
+
+class FakeChild implements UpdaterChild {
+ private messages: Array<(message: unknown) => void> = []
+ private exits: Array<() => void> = []
+ killed = false
+ onMessage(listener: (message: unknown) => void): void { this.messages.push(listener) }
+ onExit(listener: () => void): void { this.exits.push(listener) }
+ kill(): void { this.killed = true }
+ say(message: unknown): void { for (const listener of this.messages) listener(message) }
+ exit(): void { for (const listener of this.exits) listener() }
+}
+
+describe('the updater beside a core whose master runs none', () => {
+ afterEach(() => { vi.useRealTimers() })
+
+ it('is needed only under a master that does not say it runs the updater, for the installed copy with updates on', () => {
+ expect(needsUpdaterBeside({}, true, true, false)).toBe(true)
+ expect(needsUpdaterBeside({ HARNESSD_UPDATES: 'master' }, true, true, false)).toBe(false)
+ expect(needsUpdaterBeside({}, false, true, false)).toBe(false)
+ expect(needsUpdaterBeside({}, true, false, false)).toBe(false)
+ expect(needsUpdaterBeside({}, true, true, true)).toBe(false)
+ })
+
+ const make = () => {
+ const children: FakeChild[] = []
+ const staged: string[] = []
+ const lines: string[] = []
+ let pending: (() => void) | null = null
+ const stop = startUpdaterBeside({
+ spawn: () => { const child = new FakeChild(); children.push(child); return child },
+ staged: (version) => staged.push(version),
+ log: (line) => lines.push(line),
+ setTimer: (run, ms) => { lines.push(`in ${ms}`); pending = run; return 'timer' },
+ clearTimer: () => { lines.push('cleared'); pending = null },
+ })
+ return { children, staged, lines, stop, fire: () => pending!() }
+ }
+
+ it('hands over once for what it stages, and ignores what else it says', () => {
+ const m = make()
+ expect(m.lines).toEqual(['[update] this core\'s master runs no updater — running it beside this core'])
+ m.children[0].say({ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1 })
+ m.children[0].say({ type: 'harnessd:staged', version: 3 })
+ m.children[0].say(null)
+ expect(m.staged).toEqual([])
+ m.children[0].say({ type: 'harnessd:staged', version: '9.9.9' })
+ m.children[0].say({ type: 'harnessd:staged', version: '9.9.10' })
+ expect(m.staged).toEqual(['9.9.9'])
+ expect(m.lines.at(-1)).toBe('[update] the updater staged 9.9.9 — handing over to this core\'s master')
+ // It exits to be started again as the new build; this core is leaving instead.
+ m.children[0].exit()
+ expect(m.children).toHaveLength(1)
+ })
+
+ it('starts it again a while after it ends, and stops it, and any restart, on request', () => {
+ const m = make()
+ m.children[0].exit()
+ expect(m.lines.slice(-2)).toEqual([`[update] the updater ended — starting it again in ${UPDATER_RESTART_MS / 1000} s`, `in ${UPDATER_RESTART_MS}`])
+ m.fire()
+ expect(m.children).toHaveLength(2)
+ // A process it let go of says nothing that counts.
+ m.children[0].exit()
+ expect(m.children).toHaveLength(2)
+ m.stop()
+ expect(m.children[1].killed).toBe(true)
+ m.children[1].exit()
+ expect(m.children).toHaveLength(2)
+ const waiting = make()
+ waiting.children[0].exit()
+ waiting.stop()
+ expect(waiting.lines.at(-1)).toBe('cleared')
+ })
+
+ it('waits on a timer that keeps no process alive by default', () => {
+ vi.useFakeTimers()
+ const children: FakeChild[] = []
+ const stop = startUpdaterBeside({ spawn: () => { const child = new FakeChild(); children.push(child); return child }, staged: () => {}, log: () => {} })
+ children[0].exit()
+ vi.advanceTimersByTime(UPDATER_RESTART_MS)
+ expect(children).toHaveLength(2)
+ children[1].exit()
+ stop()
+ vi.advanceTimersByTime(UPDATER_RESTART_MS)
+ expect(children).toHaveLength(2)
+ })
+})
diff --git a/cli/src/core/updaterBeside.ts b/cli/src/core/updaterBeside.ts
new file mode 100644
index 000000000..e7dbec00f
--- /dev/null
+++ b/cli/src/core/updaterBeside.ts
@@ -0,0 +1,74 @@
+/**
+ * The updater beside a core whose master is too old to run it: one released before the updater left the
+ * core (services/updaterProcess.ts), that could not re-execute on this build (it predates re-execution, or
+ * this build's master did not answer its probe). Such a master tells its core nothing about updates
+ * (`HARNESSD_UPDATES`), and until something restarts it, a core that ran none would leave the machine on
+ * this build for good, a broken one included. So this core starts the same updater, in a process of its
+ * own (`cli.js __service updater`) and never in the core's: the core downloads no build. When it stages
+ * one, the core hands over to its master (exit 75), as it did when the updater ran inside it, and that
+ * master judges the new build as it always has. A new master re-executes on it and runs the updater itself.
+ */
+
+/** The updater's process, as the core watches it. */
+export interface UpdaterChild {
+ onMessage(listener: (message: unknown) => void): void
+ onExit(listener: () => void): void
+ kill(): void
+}
+
+/** How long after the updater's process ends it is started again: its checks are a minute apart anyway. */
+export const UPDATER_RESTART_MS = 30_000
+
+export interface UpdaterBesideDeps {
+ /** Start `__service updater`, with what it must know of this core's master (`HARNESSD_JUDGES_SUPERSEDED`). */
+ spawn(): UpdaterChild
+ /** The updater staged `version`: hand over for it. */
+ staged(version: string): void
+ log(line: string): void
+ setTimer?: (run: () => void, ms: number) => unknown
+ clearTimer?: (timer: unknown) => void
+}
+
+/** Whether this core must run the updater beside itself: supervised by a master that does not say it runs one. */
+export function needsUpdaterBeside(env: NodeJS.ProcessEnv, supervised: boolean, installedCopy: boolean, updatesOff: boolean): boolean {
+ return supervised && env.HARNESSD_UPDATES !== 'master' && installedCopy && !updatesOff
+}
+
+const isStaged = (message: unknown): message is { type: 'harnessd:staged'; version: string } =>
+ !!message && typeof message === 'object' && (message as { type?: unknown }).type === 'harnessd:staged'
+ && typeof (message as { version?: unknown }).version === 'string'
+
+/** Run the updater beside this core until it stages a build; returns a stop. */
+export function startUpdaterBeside(deps: UpdaterBesideDeps): () => void {
+ const setTimer = deps.setTimer ?? ((run, ms) => { const timer = setTimeout(run, ms); timer.unref?.(); return timer })
+ const clearTimer = deps.clearTimer ?? ((timer) => clearTimeout(timer as ReturnType<typeof setTimeout>))
+ let child: UpdaterChild | null = null
+ let timer: unknown = null
+ let done = false
+ const start = (): void => {
+ timer = null
+ const started = deps.spawn()
+ child = started
+ started.onMessage((message) => {
+ if (!isStaged(message) || done) return
+ done = true
+ deps.log(`[update] the updater staged ${message.version} — handing over to this core's master`)
+ deps.staged(message.version)
+ })
+ started.onExit(() => {
+ if (child !== started) return
+ child = null
+ if (done) return
+ deps.log(`[update] the updater ended — starting it again in ${UPDATER_RESTART_MS / 1000} s`)
+ timer = setTimer(start, UPDATER_RESTART_MS)
+ })
+ }
+ deps.log('[update] this core\'s master runs no updater — running it beside this core')
+ start()
+ return () => {
+ done = true
+ if (timer !== null) clearTimer(timer)
+ child?.kill()
+ child = null
+ }
+}
diff --git a/cli/src/harnessd/AGENTS.md b/cli/src/harnessd/AGENTS.md
index a33808ad3..1abf98334 100644
--- a/cli/src/harnessd/AGENTS.md
+++ b/cli/src/harnessd/AGENTS.md
@@ -38,3 +38,9 @@ It keeps the core and the services running, and nothing else.
`<runtime>/node-…/libexec/harnessd/` — never `bin/`, which agent panes get on PATH — are made only for
the managed runtime's node, never a bare `harness` (the dial's `pgrep -x harness` means the app), and
any failure runs the process as `node`. Children a process starts use `baseNode(process.execPath)`.
+9. **Updates are the master's to run, never the core's.** The updater runs in a process the master starts
+ (`UPDATER_HOST`, `../services/updaterProcess.ts`): it downloads, verifies, canaries and stages a build,
+ then says so (`harnessd:staged`); the master asks the core to hand over (`harnessd:update`) and judges
+ the new core on probation, as before. The master itself still makes no network call. Every core this
+ master starts hears `HARNESSD_UPDATES=master`; a core under a master from before that runs the updater
+ beside itself, in its own process (`../core/updaterBeside.ts`).
diff --git a/cli/src/harnessd/coreLink.spec.ts b/cli/src/harnessd/coreLink.spec.ts
index ff3d661f0..738817187 100644
--- a/cli/src/harnessd/coreLink.spec.ts
+++ b/cli/src/harnessd/coreLink.spec.ts
@@ -47,9 +47,13 @@ describe('connectToMaster', () => {
link.startHeartbeat()
const gone = vi.fn()
link.onMasterGone(gone)
+ const update = vi.fn()
+ link.onUpdate(update)
+ channel.say({ type: 'harnessd:update', version: '9.9.9' })
channel.leave()
vi.advanceTimersByTime(60_000)
expect(gone).not.toHaveBeenCalled()
+ expect(update).not.toHaveBeenCalled()
expect(link.status()).toBeNull()
link.close()
}
@@ -107,6 +111,27 @@ describe('connectToMaster', () => {
expect(channel.sent).toEqual([{ type: 'harnessd:ready', safeMode: 'no tmux' }])
})
+ it('hears an update the master asks for, kept until something listens', async () => {
+ const channel = new FakeChannel()
+ const link = connectToMaster(channel, supervised)
+ // Asked before start-up got as far as listening: not lost.
+ channel.say({ type: 'harnessd:update', version: '9.9.8' })
+ channel.say({ type: 'harnessd:update', version: '9.9.9' })
+ const update = vi.fn()
+ link.onUpdate(update)
+ expect(update).not.toHaveBeenCalled()
+ await Promise.resolve()
+ expect(update.mock.calls).toEqual([['9.9.9']])
+ channel.say({ type: 'harnessd:update', version: '9.9.10' })
+ expect(update.mock.calls).toEqual([['9.9.9'], ['9.9.10']])
+ expect(link.status()).toBeNull()
+ // Nothing kept, nothing said to a listener that comes later.
+ const later = vi.fn()
+ link.onUpdate(later)
+ await Promise.resolve()
+ expect(later).not.toHaveBeenCalled()
+ })
+
it('keeps what the master says about itself, and hears when the master goes', () => {
const channel = new FakeChannel()
const link = connectToMaster(channel, supervised)
diff --git a/cli/src/harnessd/coreLink.ts b/cli/src/harnessd/coreLink.ts
index 897563f90..bc2f1b119 100644
--- a/cli/src/harnessd/coreLink.ts
+++ b/cli/src/harnessd/coreLink.ts
@@ -74,6 +74,9 @@ export interface CoreLink {
startHeartbeat(): void
/** The master is gone. A core without one stops, so nothing is left holding the port. */
onMasterGone(listener: () => void): void
+ /** The master's updater staged a build: hand over for it (`harnessd:update`). Heard from the start, as
+ * a staged update may arrive while start-up is still under way. */
+ onUpdate(listener: (version: string) => void): void
/** What the master last said about itself (restarts, the last exit), for `/api/status`. */
status(): SupervisorStatus | null
close(): void
@@ -104,8 +107,17 @@ export function connectToMaster(
if (!supervised) return
try { channel.send!(message) } catch { /* closing */ }
}
+ // The last update the master asked for, kept until something listens: the listener is set at start-up,
+ // but a master can stage one before, and a request missed is an update never applied.
+ let update: string | null = null
+ let updateListener: ((version: string) => void) | null = null
if (supervised) {
- channel.on('message', (message) => { if (isMasterMessage(message)) status = message.status })
+ channel.on('message', (message) => {
+ if (!isMasterMessage(message)) return
+ if (message.type === 'harnessd:status') { status = message.status; return }
+ if (updateListener) updateListener(message.version)
+ else update = message.version
+ })
}
// ⚠️ Listened for from the start and remembered, not from whenever the caller subscribes.
// runForeground subscribes some 1,100 lines into start-up, long after this core has bound and begun
@@ -144,6 +156,15 @@ export function connectToMaster(
if (masterGone) queueMicrotask(listener)
else goneListeners.push(listener)
},
+ onUpdate: (listener) => {
+ if (!supervised) return
+ updateListener = listener
+ if (update !== null) {
+ const version = update
+ update = null
+ queueMicrotask(() => listener(version))
+ }
+ },
status: () => status,
close: () => {
if (heartbeat) clearInterval(heartbeat)
diff --git a/cli/src/harnessd/master.spec.ts b/cli/src/harnessd/master.spec.ts
index c26d61c89..3771fe89e 100644
--- a/cli/src/harnessd/master.spec.ts
+++ b/cli/src/harnessd/master.spec.ts
@@ -490,6 +490,45 @@ describe('runMaster', { timeout: 60_000 }, () => {
expect(exits).toEqual([0])
})
+ it('runs the updater when told to, whatever HARNESSD_SERVICES names, and has the core hand over for what it stages', async () => {
+ const pidFile = join(dir, 'adapter.pid')
+ const seen = join(dir, 'seen')
+ const script = join(dir, 'daemon.cjs')
+ writeFileSync(script, `
+ const { appendFileSync } = require('node:fs')
+ const role = process.argv[2] === '__service' ? 'service:' + process.argv[3] : process.argv[2]
+ const say = (what) => appendFileSync(${JSON.stringify(seen)}, JSON.stringify({ role, what, restarts: process.env.HARNESSD_RESTARTS }) + '\\n')
+ say('started')
+ if (role === '__run') {
+ process.send({ type: 'harnessd:bound', protocol: 3, port: 1 }); process.send({ type: 'harnessd:ready' })
+ process.on('message', (message) => { if (message.type === 'harnessd:update') { say('asked for ' + message.version); process.exit(75) } })
+ }
+ if (role === 'service:updater' && process.env.HARNESSD_RESTARTS === '0') setTimeout(() => process.send({ type: 'harnessd:staged', version: '9.9.9' }), 100)
+ setInterval(() => process.send({ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1, loopDelayMs: 0 }), 50)
+ process.on('SIGTERM', () => process.exit(0))
+ `)
+ const exits: number[] = []
+ const signals = new Map<string, () => void>()
+ const lines: string[] = []
+ const log = vi.spyOn(console, 'log').mockImplementation((line: string) => { lines.push(line) })
+ try {
+ runMaster({
+ nodePath: process.execPath, execArgv: [], scriptPath: script, pidFile, updater: true,
+ restoreUpdate: () => {}, confirmUpdate: () => {},
+ env: { ...process.env, HARNESSD_SERVICES: 'none', HARNESSD_UPDATE_PROBATION_MS: '50' },
+ exit: (code) => exits.push(code), onSignal: (signal, listener) => signals.set(signal, listener),
+ })
+ const said = (): Array<Record<string, string>> => existsSync(seen) ? readFileSync(seen, 'utf8').trim().split('\n').map((line) => JSON.parse(line)) : []
+ await until('the core to hand over and a new one to start', () => said().filter((line) => line.role === '__run' && line.what === 'started').length === 2)
+ expect(said().filter((line) => line.role.startsWith('service:')).map((line) => line.role)).toEqual(['service:updater'])
+ expect(said()).toContainEqual({ role: '__run', what: 'asked for 9.9.9', restarts: '0' })
+ expect(lines.some((line) => line.endsWith('[harnessd] the updater staged 9.9.9 — asking the core to hand over'))).toBe(true)
+ expect(lines.some((line) => line.endsWith('[harnessd] core exited (code 75) for an update — restarting'))).toBe(true)
+ signals.get('SIGTERM')!()
+ await until('the master to finish', () => exits.length > 0)
+ } finally { log.mockRestore() }
+ })
+
it('starts the services from the lean bundle when given one, and the core from the CLI entry', async () => {
// The lean bundle cli.js carries (./leanBundle.ts): each service parses its own code, not the CLI's.
const pidFile = join(dir, 'adapter.pid')
diff --git a/cli/src/harnessd/master.ts b/cli/src/harnessd/master.ts
index 68c3c65d6..65879a574 100644
--- a/cli/src/harnessd/master.ts
+++ b/cli/src/harnessd/master.ts
@@ -16,7 +16,7 @@ import type { MasterMessage } from './protocol.js'
import {
PROBE_ANSWER, RESUME_ENV, createReexec, decodeResume, fingerprint, readMarker, recoverFailedReexec, removeMarker, runProbe, writeMarker,
} from './reexec.js'
-import { SERVICE_HOSTS, ServiceSupervisor, serviceOptions, serviceProcessesEnv, serviceSpecs } from './services.js'
+import { SERVICE_HOSTS, ServiceSupervisor, UPDATER_HOST, serviceOptions, serviceProcessesEnv, serviceSpecs } from './services.js'
import {
DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorDeps, type SupervisorOptions, type SupervisorStatus,
} from './supervisor.js'
@@ -67,6 +67,8 @@ export interface MasterConfig {
/** The version of the update the bundle with this fingerprint is, when it is one no master kept or
* rolled back (`selfUpdate.unjudgedUpdate`); null otherwise. Left out, never. */
unjudgedUpdate?: (bundle: string | null) => string | null
+ /** Run the updater (`UPDATER_HOST`): the installed copy, with updates on (masterProcess.ts). */
+ updater?: boolean
}
export type Execve = (file: string, args: string[], env: NodeJS.ProcessEnv) => void
@@ -295,7 +297,10 @@ export function runMaster(config: MasterConfig): Supervisor {
folderFingerprint, exists: existsSync, sameBundle: () => bundle() === own, log,
})
const specs = serviceSpecs(env, SERVICE_HOSTS)
- const services = new ServiceSupervisor(specs, {
+ // The updater beside them, whatever HARNESSD_SERVICES says: the core neither routes to it nor runs it.
+ const processes = config.updater ? [...specs, { name: 'updater', ...UPDATER_HOST }] : specs
+ let supervisor: Supervisor | null = null
+ const services = new ServiceSupervisor(processes, {
spawnService: (spec, extra) => {
const script = lean.scriptFor(spec.name)
// One process for every service it hosts, each on its own link to the core (services/process.ts).
@@ -311,6 +316,8 @@ export function runMaster(config: MasterConfig): Supervisor {
setTimer: (run, ms) => setTimeout(run, ms),
clearTimer: (timer) => clearTimeout(timer as ReturnType<typeof setTimeout>),
log,
+ // The updater staged a build: the core is asked to hand over for it, and the new core judged.
+ staged: (version) => supervisor?.updateStaged(version),
}, serviceOptions(env), { HARNESSD_SERVICE_TOKEN: token })
const reexec = markerFile ? createReexec({
own, current: bundle, nodePath: named('harnessd'), execArgv: config.execArgv, scriptPath: config.scriptPath, env, pid: process.pid,
@@ -324,7 +331,7 @@ export function runMaster(config: MasterConfig): Supervisor {
now: () => Date.now(),
log,
}) : null
- const supervisor = new Supervisor({
+ supervisor = new Supervisor({
spawnCore: (extra) => coreHandle(spawn(named('harnessd-core'), [...execArgv, config.scriptPath, '__run'], {
// Told which services this master runs, so it routes to exactly those and runs the rest itself.
env: { ...env, ...extra, HARNESSD_SERVICE_TOKEN: token, ...serviceProcessesEnv(specs) },
diff --git a/cli/src/harnessd/protocol.spec.ts b/cli/src/harnessd/protocol.spec.ts
index 4c4c654c5..b869318bd 100644
--- a/cli/src/harnessd/protocol.spec.ts
+++ b/cli/src/harnessd/protocol.spec.ts
@@ -1,5 +1,5 @@
import { describe, expect, it } from 'vitest'
-import { HARNESSD_PROTOCOL, isCoreMessage, isMasterMessage } from './protocol.js'
+import { HARNESSD_PROTOCOL, isCoreMessage, isMasterMessage, isUpdaterMessage } from './protocol.js'
describe('the spawn channel protocol', () => {
it.each([
@@ -28,9 +28,23 @@ describe('the spawn channel protocol', () => {
[{ type: 'harnessd:status', status: null }, false],
[{ type: 'harnessd:status', status: 'running' }, false],
[{ type: 'harnessd:bound', protocol: 1, port: 1 }, false],
+ [{ type: 'harnessd:update', version: '1.2.3' }, true],
+ [{ type: 'harnessd:update', version: '' }, false],
+ [{ type: 'harnessd:update' }, false],
[undefined, false],
[7, false],
])('a master message %j is %s', (message, valid) => {
expect(isMasterMessage(message)).toBe(valid)
})
+
+ it.each([
+ [{ type: 'harnessd:staged', version: '1.2.3' }, true],
+ [{ type: 'harnessd:staged', version: '' }, false],
+ [{ type: 'harnessd:staged', version: 3 }, false],
+ [{ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1 }, false],
+ [null, false],
+ ['harnessd:staged', false],
+ ])('an updater message %j is %s', (message, valid) => {
+ expect(isUpdaterMessage(message)).toBe(valid)
+ })
})
diff --git a/cli/src/harnessd/protocol.ts b/cli/src/harnessd/protocol.ts
index 3a2af1c22..87ea4e896 100644
--- a/cli/src/harnessd/protocol.ts
+++ b/cli/src/harnessd/protocol.ts
@@ -2,8 +2,9 @@
* What harnessd's master and its core say to each other over the spawn channel (Node IPC).
*
* Small on purpose. The core says when it is bound, when it is ready and that it is alive; the master
- * tells it its own status. Everything else the daemon does goes through the core's local API, never
- * through here.
+ * tells it its own status, and when an update is staged, to hand over. The updater, a process the master
+ * runs (services/updaterProcess.ts), says when it has staged one. Everything else the daemon does goes
+ * through the core's local API, never through here.
*
* Messages are only ever added, and a side ignores what it does not know: during an update an older
* master supervises a newer core, and after a rollback a newer master supervises an older one. Each
@@ -12,8 +13,9 @@
*/
import type { SupervisorStatus } from './supervisor.js'
-/** 2: `ready`, the event-loop delay on the heartbeat, and the exit-code contract below. 3: `want`. 4: `want`
- * for the devices' process too, which runs only once there is a device (./services.ts `askedSince`). */
+/** 2: `ready`, the event-loop delay on the heartbeat, and the exit-code contract below. 3: `want`, and the
+ * master's updater: the master tells its core to hand over for a build its updater staged (`harnessd:update`).
+ * 4: `want` for the devices' process too, which runs only once there is a device (./services.ts `askedSince`). */
export const HARNESSD_PROTOCOL = 4
/** The command a bundle's master answers its probe on, and what it answers. */
@@ -59,6 +61,21 @@ export type CoreMessage =
export type MasterMessage =
| { type: 'harnessd:status'; status: SupervisorStatus }
+ /** The updater staged `version`: hand over for it, as a core whose own updater staged it did (exit 75). */
+ | { type: 'harnessd:update'; version: string }
+
+/** What the updater says to the master: it staged `version` on disk. */
+export type UpdaterMessage = { type: 'harnessd:staged'; version: string }
+
+/** The exit code a service uses to be started again at once, on the bundle now on disk, without counting
+ * as a crash: the updater, once it has staged a build, so that it next runs as that build. */
+export const SERVICE_EXIT_RESTART = 75
+
+export function isUpdaterMessage(value: unknown): value is UpdaterMessage {
+ if (!value || typeof value !== 'object') return false
+ const message = value as Record<string, unknown>
+ return message.type === 'harnessd:staged' && typeof message.version === 'string' && message.version.length > 0
+}
export function isCoreMessage(value: unknown): value is CoreMessage {
if (!value || typeof value !== 'object') return false
@@ -81,5 +98,6 @@ export function isCoreMessage(value: unknown): value is CoreMessage {
export function isMasterMessage(value: unknown): value is MasterMessage {
if (!value || typeof value !== 'object') return false
const message = value as Record<string, unknown>
+ if (message.type === 'harnessd:update') return typeof message.version === 'string' && message.version.length > 0
return message.type === 'harnessd:status' && !!message.status && typeof message.status === 'object'
}
diff --git a/cli/src/harnessd/services.spec.ts b/cli/src/harnessd/services.spec.ts
index 11657696e..2269c38b3 100644
--- a/cli/src/harnessd/services.spec.ts
+++ b/cli/src/harnessd/services.spec.ts
@@ -1,6 +1,6 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
- DEFAULT_SERVICE_OPTIONS, KNOWN_SERVICES, SERVICE_HOSTS, SERVICE_PROCESSES_ENV, ServiceSupervisor, serviceOptions, serviceProcessesEnv, serviceSpecs, servicesTheMasterRuns,
+ DEFAULT_SERVICE_OPTIONS, KNOWN_SERVICES, SERVICE_HOSTS, SERVICE_PROCESSES_ENV, ServiceSupervisor, UPDATER_HOST, serviceOptions, serviceProcessesEnv, serviceSpecs, servicesTheMasterRuns,
type ServiceSpec, type ServiceSupervisorOptions,
} from './services.js'
import type { CoreHandle } from './supervisor.js'
@@ -40,8 +40,10 @@ describe('ServiceSupervisor', () => {
let children: FakeService[]
let lines: string[]
const latest = (name = 'search') => children.filter((child) => child.spec.name === name).at(-1)!
- const make = (specs: ServiceSpec[] = [search], overrides: Partial<ServiceSupervisorOptions> = {}, env: Record<string, string> = {}) =>
+ const staged: string[] = []
+ const make = (specs: ServiceSpec[] = [search], overrides: Partial<ServiceSupervisorOptions> = {}, env: Record<string, string> = {}, tell = true) =>
new ServiceSupervisor(specs, {
+ ...(tell ? { staged: (version: string) => { staged.push(version) } } : {}),
spawnService: (spec, extra) => {
const child = new FakeService(2000 + children.length, spec, extra)
children.push(child)
@@ -75,6 +77,34 @@ describe('ServiceSupervisor', () => {
expect(supervisor.status()[0].state).toBe('running')
})
+ it('passes on what the updater staged, and starts it again at once when it asks: not a crash', () => {
+ const updater: ServiceSpec = { name: 'updater', ...UPDATER_HOST }
+ staged.length = 0
+ const supervisor = make([updater], { parkCrashes: 2 })
+ supervisor.start()
+ latest('updater').beat()
+ latest('updater').say({ type: 'harnessd:staged', version: '9.9.9' })
+ expect(staged).toEqual(['9.9.9'])
+ expect(lines).toContain('[harnessd] service updater staged 9.9.9')
+ for (let i = 0; i < 3; i++) {
+ latest('updater').exit(75)
+ vi.advanceTimersByTime(0)
+ }
+ expect(children).toHaveLength(4)
+ expect(supervisor.status()[0]).toMatchObject({ state: 'restarting', restarts: 3, lastExit: 'code 75', lastExitReason: 'restart' })
+ expect(lines).toContain('[harnessd] service updater asked to start again (code 75) — restarting')
+ // Killed for hanging, its 75 is no request.
+ latest('updater').beat()
+ vi.advanceTimersByTime(options.heartbeatTimeoutMs * 2)
+ latest('updater').exit(75)
+ expect(supervisor.status()[0].lastExitReason).toBe('hung')
+ // A master that was not told how to pass a staging on only says it.
+ const quiet = make([updater], {}, {}, false)
+ quiet.start()
+ latest('updater').say({ type: 'harnessd:staged', version: '9.9.10' })
+ expect(staged).toEqual(['9.9.9'])
+ })
+
it('restarts a crashed service with a backoff that doubles, caps, and is forgiven after a long run', () => {
const supervisor = make([search], { parkCrashes: 99 })
supervisor.start()
diff --git a/cli/src/harnessd/services.ts b/cli/src/harnessd/services.ts
index 103ad9f9a..42065ae6f 100644
--- a/cli/src/harnessd/services.ts
+++ b/cli/src/harnessd/services.ts
@@ -12,7 +12,7 @@
* bind and no update to prove; it beats, or it is restarted. Everything that touches the operating
* system is injected, so every decision here is tested without one.
*/
-import { heartbeatGraceMs, isCoreMessage } from './protocol.js'
+import { SERVICE_EXIT_RESTART, heartbeatGraceMs, isCoreMessage, isUpdaterMessage } from './protocol.js'
import type { CoreHandle } from './supervisor.js'
export interface ServiceSpec {
@@ -42,6 +42,8 @@ export interface ServiceSupervisorDeps {
setTimer(run: () => void, ms: number): unknown
clearTimer(timer: unknown): void
log(line: string): void
+ /** The updater staged `version` on disk (`harnessd:staged`): the core is to hand over for it. */
+ staged?(version: string): void
}
export interface ServiceSupervisorOptions {
@@ -77,7 +79,7 @@ export const DEFAULT_SERVICE_OPTIONS: ServiceSupervisorOptions = {
/** `off`: an experiment's process that no one has asked for yet, and so was never started. */
export type ServiceState = 'off' | 'starting' | 'running' | 'restarting' | 'parked' | 'stopping' | 'stopped'
-export type ServiceExitReason = 'crashed' | 'hung' | 'memory' | 'stopped'
+export type ServiceExitReason = 'crashed' | 'hung' | 'memory' | 'stopped' | 'restart'
export interface ServiceStatus {
name: string
@@ -178,6 +180,11 @@ class Service {
}
private onMessage(child: CoreHandle, message: unknown): void {
+ if (isUpdaterMessage(message)) {
+ this.deps.log(`[harnessd] service ${this.spec.name} staged ${message.version}`)
+ this.deps.staged?.(message.version)
+ return
+ }
if (!isCoreMessage(message) || message.type !== 'harnessd:heartbeat') return
this.watchHeartbeat(child)
if (this.state === 'starting' || this.state === 'restarting') this.setState('running')
@@ -209,6 +216,16 @@ class Service {
this.stopped()
return
}
+ if (code === SERVICE_EXIT_RESTART && !this.killReason) {
+ // Asked for: the updater, once it has staged a build, so that it next runs as that build (a lean
+ // bundle that no longer matches cli.js is not used, ./leanServices.ts). Not a crash, and not delayed.
+ this.lastExitReason = 'restart'
+ this.restarts++
+ this.deps.log(`[harnessd] service ${this.spec.name} asked to start again (${exit}) — restarting`)
+ this.setState('restarting')
+ this.armTimer('restartTimer', () => { this.restartTimer = null; this.start() }, 0)
+ return
+ }
const reason = this.killReason ?? 'crashed'
this.lastExitReason = reason
this.restarts++
@@ -377,6 +394,16 @@ export const SERVICE_HOSTS: Readonly<Record<string, ServiceHostSpec>> = {
/** Every service this build can run outside the core's process: what `HARNESSD_SERVICES` names. */
export const KNOWN_SERVICES: readonly string[] = Object.values(SERVICE_HOSTS).flatMap((host) => host.services)
+/**
+ * The updater's process (services/updaterProcess.ts): checks for a newer build of the CLI and of hn, and
+ * downloads, verifies, canaries and stages it, then tells the master, which has the core hand over. Its own
+ * process and never the core's: the core never downloads a build, and a core that cannot start (safe mode)
+ * or a host whose services keep crashing must not stop the fix from arriving. Not one of `SERVICE_HOSTS`:
+ * the core neither routes to it nor runs it, and `HARNESSD_SERVICES` does not turn it off; the master runs
+ * it when it runs the installed copy with updates on (masterProcess.ts). It holds one download at a time.
+ */
+export const UPDATER_HOST: ServiceHostSpec = { services: ['updater'], heapLimitMiB: 256, rssLimitMiB: 512 }
+
/** Service timings from the environment (for tests and support); anything unset or invalid keeps its default. */
export function serviceOptions(env: NodeJS.ProcessEnv): ServiceSupervisorOptions {
const read = (name: string, fallback: number, min: number): number => {
diff --git a/cli/src/harnessd/supervisor.spec.ts b/cli/src/harnessd/supervisor.spec.ts
index f918cf657..1aeff287a 100644
--- a/cli/src/harnessd/supervisor.spec.ts
+++ b/cli/src/harnessd/supervisor.spec.ts
@@ -1,7 +1,7 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { CORE_EXIT_STOP, CORE_EXIT_UPDATE, HARNESSD_PROTOCOL, type MasterMessage } from './protocol.js'
import {
- DEFAULT_SUPERVISOR_OPTIONS, Supervisor, namesTheDisk, type CoreHandle, type ReexecOutcome, type ResumeState, type SupervisorDeps, type SupervisorOptions,
+ DEFAULT_SUPERVISOR_OPTIONS, HAND_OVER_GRACE_MS, Supervisor, namesTheDisk, type CoreHandle, type ReexecOutcome, type ResumeState, type SupervisorDeps, type SupervisorOptions,
type SupervisorStatus,
} from './supervisor.js'
@@ -95,7 +95,7 @@ describe('Supervisor', () => {
supervisor.start()
supervisor.start()
expect(cores).toHaveLength(1)
- expect(core().env).toEqual({ HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: '0', HARNESSD_WATCHDOG_MS: '6000', HARNESSD_JUDGES_SUPERSEDED: '1' })
+ expect(core().env).toEqual({ HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: '0', HARNESSD_WATCHDOG_MS: '6000', HARNESSD_JUDGES_SUPERSEDED: '1', HARNESSD_UPDATES: 'master' })
expect(supervisor.status()).toMatchObject({ state: 'starting', corePid: 1000 })
core().bind()
expect(calls).toEqual(['claim'])
@@ -111,7 +111,7 @@ describe('Supervisor', () => {
expect(lines.at(-1)).toBe('[harnessd] core ready (pid 1000)')
// Every change is written for `harness status`, and told to the bound core.
expect(statuses.map((status) => status.state)).toEqual(['starting', 'listening', 'running'])
- expect(core().sent.map((message) => message.status.state)).toEqual(['listening', 'running'])
+ expect(core().sent.map((message) => message.type === 'harnessd:status' ? message.status.state : message.type)).toEqual(['listening', 'running'])
core().ready()
expect(statuses).toHaveLength(3)
})
@@ -161,7 +161,7 @@ describe('Supervisor', () => {
expect(cores).toHaveLength(1)
vi.advanceTimersByTime(1)
expect(cores).toHaveLength(2)
- expect(core().env).toEqual({ HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: '1', HARNESSD_WATCHDOG_MS: '6000', HARNESSD_JUDGES_SUPERSEDED: '1', HARNESSD_LAST_EXIT: 'signal SIGKILL' })
+ expect(core().env).toEqual({ HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: '1', HARNESSD_WATCHDOG_MS: '6000', HARNESSD_JUDGES_SUPERSEDED: '1', HARNESSD_UPDATES: 'master', HARNESSD_LAST_EXIT: 'signal SIGKILL' })
expect(lines.at(-1)).toBe('[harnessd] core started (pid ?) · restart 1')
core().bind()
expect(lines.at(-1)).toBe(`[harnessd] core bound (pid ?, protocol ${HARNESSD_PROTOCOL})`)
@@ -380,6 +380,94 @@ describe('Supervisor', () => {
})
})
+ describe('an update the updater staged', () => {
+ it('asks the running core to hand over, once, and judges its exit as the update\'s', () => {
+ const supervisor = make()
+ supervisor.start()
+ core().up()
+ supervisor.updateStaged('9.9.9')
+ supervisor.updateStaged('9.9.9')
+ expect(core().sent.filter((message) => message.type === 'harnessd:update')).toEqual([{ type: 'harnessd:update', version: '9.9.9' }])
+ expect(lines.at(-1)).toBe('[harnessd] the updater staged 9.9.9 — asking the core to hand over')
+ core().exit(CORE_EXIT_UPDATE)
+ vi.advanceTimersByTime(0)
+ expect(cores).toHaveLength(2)
+ expect(lines).toContain('[harnessd] core exited (code 75) for an update — restarting')
+ core().up()
+ // The grace it had to hand over in ended with it.
+ live(HAND_OVER_GRACE_MS)
+ expect(core().kills).toEqual([])
+ expect(calls).toContain('confirm')
+ })
+
+ it('stops a core that does not hand over within its grace, and counts that exit as the update\'s', () => {
+ const supervisor = make()
+ supervisor.start()
+ core().up()
+ supervisor.updateStaged('9.9.9')
+ live(HAND_OVER_GRACE_MS - 1_000)
+ expect(core().kills).toEqual([])
+ live(1_000)
+ expect(core().kills).toEqual(['SIGTERM'])
+ expect(lines).toContain('[harnessd] core did not hand over for 9.9.9 within 45000 ms — stopping it')
+ core().exit(0)
+ vi.advanceTimersByTime(0)
+ expect(supervisor.status()).toMatchObject({ lastExitReason: 'update', restarts: 1 })
+ expect(cores).toHaveLength(2)
+ core().up()
+ live(options.updateProbationMs)
+ expect(calls).toContain('confirm')
+ })
+
+ it('with no core running, starts the next one on the new bundle at once, on probation, with a clean slate', () => {
+ const supervisor = make()
+ supervisor.start()
+ // Into safe mode: the crashes a fix would end.
+ for (let i = 0; i < options.crashLoopCrashes; i++) { core().up(); crash() }
+ expect(core().env.HARNESSD_SAFE_MODE).toBe('crash-loop')
+ core().exit(1)
+ expect(supervisor.status().state).toBe('restarting')
+ supervisor.updateStaged('9.9.9')
+ expect(lines.at(-1)).toBe('[harnessd] the updater staged 9.9.9 — the next core starts on it')
+ vi.advanceTimersByTime(0)
+ expect(core().env.HARNESSD_SAFE_MODE).toBeUndefined()
+ core().up()
+ core().exit(1)
+ vi.advanceTimersByTime(0)
+ // On probation: its crash rolls the update back.
+ expect(calls).toContain('restore')
+ })
+
+ it('judges the first core when a build is staged before the master starts one', () => {
+ const supervisor = make()
+ supervisor.updateStaged('9.9.9')
+ expect(cores).toHaveLength(0)
+ supervisor.start()
+ core().up()
+ core().exit(1)
+ expect(calls).toContain('restore')
+ })
+
+ it('leaves a core being started to start on the bundle on disk, and ignores a staging while stopping', () => {
+ const deciding: Array<(outcome: ReexecOutcome) => void> = []
+ const supervisor = make({}, { reexec: (_state, proceed) => { deciding.push(proceed) } })
+ supervisor.start()
+ expect(cores).toHaveLength(0)
+ supervisor.updateStaged('9.9.9')
+ expect(cores).toHaveLength(0)
+ deciding[0]('kept')
+ expect(cores).toHaveLength(1)
+ core().up()
+ core().exit(1)
+ vi.advanceTimersByTime(0)
+ expect(calls).toContain('restore')
+ supervisor.stop('stopping')
+ const before = lines.length
+ supervisor.updateStaged('9.9.10')
+ expect(lines.length).toBe(before)
+ })
+ })
+
it('stops the core and itself on request, and kills the core on a second request', () => {
const supervisor = make()
supervisor.start()
diff --git a/cli/src/harnessd/supervisor.ts b/cli/src/harnessd/supervisor.ts
index e34e0c698..ae61b0c67 100644
--- a/cli/src/harnessd/supervisor.ts
+++ b/cli/src/harnessd/supervisor.ts
@@ -179,7 +179,14 @@ export interface SupervisorStatus {
reexecs: number
}
-type TimerName = 'bindTimer' | 'readyTimer' | 'heartbeatTimer' | 'killTimer' | 'restartTimer' | 'probationTimer'
+type TimerName = 'bindTimer' | 'readyTimer' | 'heartbeatTimer' | 'killTimer' | 'restartTimer' | 'probationTimer' | 'handOverTimer'
+
+/**
+ * How long a core asked to hand over for an update (`harnessd:update`) has before the master stops it all
+ * the same: its teardown gives up after 15 s (core/updateHandoff.ts) and a grace of 1 s follows, so this
+ * is a core that did not hear, or did not act. Its exit then counts as the update's.
+ */
+export const HAND_OVER_GRACE_MS = 45_000
const describeExit = (code: number | null, signal: NodeJS.Signals | null): string =>
signal ? `signal ${signal}` : `code ${code}`
@@ -227,6 +234,7 @@ export class Supervisor {
private heartbeatTimer: unknown = null
private killTimer: unknown = null
private restartTimer: unknown = null
+ private handOverTimer: unknown = null
private readonly masterVersion: string | null
private reexecs = 0
private unproven = 0
@@ -308,6 +316,37 @@ export class Supervisor {
})
}
+ /**
+ * The updater (services/updaterProcess.ts) staged `version` on disk. The core used to stage it itself
+ * and exit 75; now the master asks it to (`harnessd:update`), and its exit is the update's, judged as
+ * before. A core that does not leave within `HAND_OVER_GRACE_MS` is stopped, and that exit counts as the
+ * update's too. With no core running (a restart's backoff, safe mode's), the next one starts on the new
+ * bundle at once, on probation, with a clean slate: whatever was crashing may be what it fixes.
+ */
+ updateStaged(version: string): void {
+ if (this.state === 'stopping' || this.state === 'stopped') return
+ const core = this.core
+ if (!core) {
+ this.update = 'pending'
+ this.crashes = []
+ this.safeMode = null
+ this.deps.log(`[harnessd] the updater staged ${version} — the next core starts on it`)
+ // Waiting out a backoff: no longer. Otherwise the next core is already being started (a
+ // re-execution deciding), and starts on the bundle on disk.
+ if (this.restartTimer !== null) this.scheduleSpawn(0)
+ return
+ }
+ if (this.handOverTimer !== null) return
+ this.deps.log(`[harnessd] the updater staged ${version} — asking the core to hand over`)
+ core.send({ type: 'harnessd:update', version })
+ this.armTimer('handOverTimer', () => {
+ this.handOverTimer = null
+ this.deps.log(`[harnessd] core did not hand over for ${version} within ${HAND_OVER_GRACE_MS} ms — stopping it`)
+ this.killReason = 'update'
+ this.end('restart', 'SIGTERM')
+ }, HAND_OVER_GRACE_MS)
+ }
+
/** Stop the core and the master. A second call while stopping kills the core outright. */
stop(reason: string): void {
if (this.state === 'stopped') return
@@ -346,9 +385,11 @@ export class Supervisor {
// HARNESSD_JUDGES_SUPERSEDED: this master judges a build its core on probation stages on its own
// (#807). A master from before rolls back on any exit during probation, the update's own included,
// and its core must not stage then: see `stageWhileJudged` in lib/selfUpdate.ts.
+ // HARNESSD_UPDATES: updates are this master's to run (services/updaterProcess.ts), not the core's. A core
+ // under a master from before it starts the updater beside itself (core/updaterBeside.ts).
const env: Record<string, string> = {
HARNESSD_SUPERVISED: '1', HARNESSD_RESTARTS: String(this.restarts), HARNESSD_WATCHDOG_MS: String(this.options.heartbeatTimeoutMs),
- HARNESSD_JUDGES_SUPERSEDED: '1',
+ HARNESSD_JUDGES_SUPERSEDED: '1', HARNESSD_UPDATES: 'master',
}
if (this.lastExit) env.HARNESSD_LAST_EXIT = this.lastExit
if (this.safeMode) env.HARNESSD_SAFE_MODE = this.safeMode
@@ -460,7 +501,7 @@ export class Supervisor {
const exit = describeExit(code, signal)
const reason = this.killReason ?? (code === CORE_EXIT_UPDATE ? 'update' : 'crashed')
const wasSafe = this.safeMode ?? this.coreSafeMode
- for (const timer of ['bindTimer', 'readyTimer', 'heartbeatTimer', 'killTimer', 'probationTimer'] as const) this.clearTimer(timer)
+ for (const timer of ['bindTimer', 'readyTimer', 'heartbeatTimer', 'killTimer', 'probationTimer', 'handOverTimer'] as const) this.clearTimer(timer)
this.core = null
this.bound = false
this.coreSafeMode = null
diff --git a/cli/src/leanEntry.spec.ts b/cli/src/leanEntry.spec.ts
index 5b882152c..e29e91268 100644
--- a/cli/src/leanEntry.spec.ts
+++ b/cli/src/leanEntry.spec.ts
@@ -16,9 +16,10 @@ import { PROBE_ANSWER } from './harnessd/reexec.js'
import { KNOWN_SERVICES, SERVICE_HOSTS } from './harnessd/services.js'
const CLI_ROOT = fileURLToPath(new URL('..', import.meta.url))
-const ROLES = ['master', ...KNOWN_SERVICES]
+// The updater, beside the services the core knows: the master runs it (harnessd/services.ts `UPDATER_HOST`).
+const ROLES = ['master', ...KNOWN_SERVICES, 'updater']
const RUNNER: Record<string, string> = { master: 'masterProcess' }
-for (const name of KNOWN_SERVICES) RUNNER[name] = `${name}Process`
+for (const name of [...KNOWN_SERVICES, 'updater']) RUNNER[name] = `${name}Process`
describe('the lean bundle a release carries', () => {
let scratch = ''
@@ -58,10 +59,10 @@ describe('the lean bundle a release carries', () => {
}
})
- it('brings no zod to the master, search or the edge host', () => {
+ it('brings no zod to the master, search, the updater or the edge host', () => {
expect(hasZod(new Set(files.keys())), 'the bundle still holds zod, for the services whose own code uses it').toBe(true)
// The edge host's services share one process: zod in any of them would be in all of them.
- for (const role of ['master', 'search', ...SERVICE_HOSTS.edge.services]) expect(hasZod(loads(role)), role).toBe(false)
+ for (const role of ['master', 'search', 'updater', ...SERVICE_HOSTS.edge.services]) expect(hasZod(loads(role)), role).toBe(false)
})
it('passes the release script\'s check, which a lean bundle that breaks a process does not', async () => {
diff --git a/cli/src/lib/daemonSafeMode.spec.ts b/cli/src/lib/daemonSafeMode.spec.ts
index d08f0f2fe..e45684233 100644
--- a/cli/src/lib/daemonSafeMode.spec.ts
+++ b/cli/src/lib/daemonSafeMode.spec.ts
@@ -3,8 +3,8 @@ import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import {
- clearSafeModeMarker, readSafeModeMarker, runBootHandoff, safeModeDisposition,
- safeModeFile, safeModeStatusBody, SafeModeRequest, writeSafeModeMarker, type BootHandoffDeps,
+ clearSafeModeMarker, readSafeModeMarker, safeModeDisposition,
+ safeModeFile, safeModeStatusBody, SafeModeRequest, writeSafeModeMarker,
} from './daemonSafeMode.js'
let dir = ''
@@ -80,50 +80,3 @@ describe('the safe-mode marker', () => {
expect(readFileSync(safeModeFile(dir), 'utf-8')).toBe('{ not json')
})
})
-
-describe('runBootHandoff', () => {
- function harness(over: Partial<BootHandoffDeps> = {}) {
- const calls: string[] = []
- let exited: number | null = null
- const deps: BootHandoffDeps = {
- closeServer: () => calls.push('close'),
- probeMaster: () => { calls.push('probe'); return null },
- rollBack: () => calls.push('roll back'),
- startMaster: () => calls.push('start the master'),
- exit: ((code: number) => { calls.push(`exit:${code}`); exited = code; return undefined as never }),
- log: () => {},
- ...over,
- }
- return { deps, calls, exited: () => exited }
- }
-
- it('probes the new build\'s master, releases the port, then starts the master on the new build, and leaves', () => {
- const h = harness()
- runBootHandoff('0.3.5', '0.3.6', h.deps)
- expect(h.calls).toEqual(['probe', 'close', 'start the master', 'exit:0'])
- })
-
- it('rolls a build whose master does not answer back, and starts this build\'s master instead', () => {
- const lines: string[] = []
- const h = harness({ probeMaster: () => 'harnessd-probe failed: a bad build', log: (m) => lines.push(m) })
- runBootHandoff('0.3.5', '0.3.6', h.deps)
- expect(h.calls).toEqual(['roll back', 'close', 'start the master', 'exit:0'])
- expect(lines[1]).toBe('[update] 0.3.6\'s master did not answer its probe (harnessd-probe failed: a bad build) — rolled back; 0.3.5 goes on under a master of its own')
- })
-
- it('has already exited by the time it returns — the property that makes two daemons impossible', () => {
- // Nothing is awaited, so the half-built boot cannot interleave between closing the port and the
- // exit, and can never reach the code that would bind the port the successor is taking.
- const h = harness()
- runBootHandoff('0.3.5', '0.3.6', h.deps)
- expect(h.exited()).toBe(0) // no await, no tick: true immediately after the call returns
- })
-
- it('says what it hands over, and to whom', () => {
- const lines: string[] = []
- const h = harness({ log: (m) => lines.push(m) })
- runBootHandoff('0.3.5', '0.3.6', h.deps)
- expect(lines[0]).toContain('0.3.5 → 0.3.6')
- expect(lines[0]).toContain('harnessd\'s master')
- })
-})
diff --git a/cli/src/lib/daemonSafeMode.ts b/cli/src/lib/daemonSafeMode.ts
index 3962be854..85f090499 100644
--- a/cli/src/lib/daemonSafeMode.ts
+++ b/cli/src/lib/daemonSafeMode.ts
@@ -111,50 +111,6 @@ export function clearSafeModeMarker(dataDir: string): void {
try { rmSync(safeModeFile(dataDir), { force: true }) } catch { /* ignore */ }
}
-export interface BootHandoffDeps {
- /** The bound control port, if start-up ever got that far. */
- closeServer: () => void
- /** Whether the staged bundle's master answers its probe, synchronously: null when it does, why not otherwise. */
- probeMaster: () => string | null
- /** Put the build before back (`selfUpdate.restore`), remembering this one as rejected. */
- rollBack: () => void
- /** Start harnessd's master on the bundle now on disk, named in the pid file: it starts the core, and
- * judges the update. */
- startMaster: () => void
- exit: (code: number) => never
- log: (message: string) => void
-}
-
-/**
- * Hand the machine to a newer build without finishing start-up.
- *
- * SYNCHRONOUS END TO END, and that is the whole safety argument: never awaiting means the half-built
- * boot cannot interleave between the port closing and the exit, so it can never reach the code that
- * would bind the port the successor is about to take. Two daemons are impossible by construction.
- * That is also why it does not supervise the child the way a normal update restart does — waiting
- * would leave this process running alongside the new one for up to a minute, both reconciling tmux
- * and writing the registry.
- *
- * It spawns rather than merely exiting because nothing supervises this core: on a machine with no
- * desktop app nothing else would ever start the successor. What it spawns is harnessd's master, which
- * judges the update its core starts on; the core it once spawned ran the update unjudged. The staged
- * bundle's master is probed first, as a master asks before it re-executes on a new bundle, and
- * synchronously like the rest: one that cannot start would leave nothing running and nothing to judge, so
- * the build before goes back and its master is started instead.
- */
-export function runBootHandoff(from: string, to: string, deps: BootHandoffDeps): void {
- deps.log(`[update] ${from} → ${to} staged during start-up — handing off to harnessd's master without finishing boot`)
- const refused = deps.probeMaster()
- if (refused !== null) {
- deps.log(`[update] ${to}'s master did not answer its probe (${refused}) — rolled back; ${from} goes on under a master of its own`)
- deps.rollBack()
- }
- deps.closeServer()
- deps.startMaster()
- deps.log('[update] boot handoff · this process is leaving')
- deps.exit(0)
-}
-
/** The marker, or null — including for one left behind by a process that is no longer running. */
export function readSafeModeMarker(
dataDir: string,
diff --git a/cli/src/lib/installedCopy.spec.ts b/cli/src/lib/installedCopy.spec.ts
new file mode 100644
index 000000000..3d5eca098
--- /dev/null
+++ b/cli/src/lib/installedCopy.spec.ts
@@ -0,0 +1,27 @@
+import { linkSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { afterEach, beforeEach, describe, expect, it } from 'vitest'
+import { isInstalledCopy } from './installedCopy.js'
+
+describe('the installed copy', () => {
+ let dir = ''
+ beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'installed-copy-')) })
+ afterEach(() => { rmSync(dir, { recursive: true, force: true }) })
+
+ it('is cli.js in the CLI folder, by inode, under any name that reaches it', () => {
+ writeFileSync(join(dir, 'cli.js'), 'cli')
+ symlinkSync(join(dir, 'cli.js'), join(dir, 'link.js'))
+ linkSync(join(dir, 'cli.js'), join(dir, 'hard.js'))
+ writeFileSync(join(dir, 'other.js'), 'other')
+ expect(isInstalledCopy(join(dir, 'cli.js'), dir)).toBe(true)
+ expect(isInstalledCopy(join(dir, 'link.js'), dir)).toBe(true)
+ expect(isInstalledCopy(join(dir, 'hard.js'), dir)).toBe(true)
+ expect(isInstalledCopy(join(dir, 'other.js'), dir)).toBe(false)
+ })
+
+ it('is matched by path when it cannot be read', () => {
+ expect(isInstalledCopy(join(dir, 'cli.js'), dir)).toBe(true)
+ expect(isInstalledCopy(join(dir, 'src', 'cli.ts'), dir)).toBe(false)
+ })
+})
diff --git a/cli/src/lib/installedCopy.ts b/cli/src/lib/installedCopy.ts
new file mode 100644
index 000000000..5945f8823
--- /dev/null
+++ b/cli/src/lib/installedCopy.ts
@@ -0,0 +1,13 @@
+/**
+ * Whether a script is the installed copy of the CLI (`~/.harness/cli/cli.js`), the one the updater
+ * replaces. A dev or repo run (`tsx`, or `node dist/cli.js` from a checkout) never updates itself: it would
+ * swap the published bundle into ~/.harness/cli and restart, hijacking the version being developed. Matched
+ * by inode, so a symlink or a realpath does not fool it; by path when either cannot be read.
+ */
+import { statSync } from 'node:fs'
+import { join } from 'node:path'
+
+export function isInstalledCopy(scriptPath: string, cliDir: string): boolean {
+ const installed = join(cliDir, 'cli.js')
+ try { return statSync(scriptPath).ino === statSync(installed).ino } catch { return scriptPath === installed }
+}
diff --git a/cli/src/lib/selfUpdate.spec.ts b/cli/src/lib/selfUpdate.spec.ts
index 41d4503f2..218736d39 100644
--- a/cli/src/lib/selfUpdate.spec.ts
+++ b/cli/src/lib/selfUpdate.spec.ts
@@ -206,8 +206,8 @@ describe('startSelfUpdater', () => {
it('is already stopped when onStaged runs — a handler that defers loses the updater for good', async () => {
// `done = true` and `stop()` happen BEFORE `onStaged`, so a handler that returns without handing
- // the machine over leaves no timer and no way back. This is why the daemon's boot-time handler
- // always takes over and exits rather than waiting for start-up to finish (`runBootHandoff`).
+ // the machine over leaves no timer and no way back. This is why the updater's process, once it has
+ // staged, exits to be started again as the new build (services/updaterProcess.ts).
serveUpdate()
const dir = tempDir()
let staged = 0
diff --git a/cli/src/masterProcess.spec.ts b/cli/src/masterProcess.spec.ts
index 65c6dd379..abfe207a1 100644
--- a/cli/src/masterProcess.spec.ts
+++ b/cli/src/masterProcess.spec.ts
@@ -2,6 +2,7 @@ import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { env } from './config/env.js'
import { PROBE_ANSWER } from './harnessd/reexec.js'
import { LEAN_ENTRY, leanFingerprint, type LeanBundle } from './harnessd/leanBundle.js'
@@ -167,6 +168,11 @@ describe('starting the master', () => {
const config = runMaster.mock.calls[0][0] as Record<string, unknown> & { restoreUpdate(): void; confirmUpdate(): void }
expect(config).toMatchObject({ scriptPath: '/cli/cli.js', serviceScriptPath: '/lean.mjs', bundleFingerprint: 'f', nodePath: process.execPath })
expect(String(config.reexecMarkerFile)).toMatch(/harnessd-reexec\.json$/)
+ // Not the installed copy: no updater. The installed one runs it.
+ expect(config.updater).toBe(false)
+ runMaster.mockClear()
+ startMaster({ scriptPath: join(env.ADAPTER_CLI_DIR, 'cli.js') })
+ expect((runMaster.mock.calls[0][0] as { updater: boolean }).updater).toBe(true)
expect(LEAN_DIR).toMatch(/[\\/]lean$/)
expect(LEAN_OFF_FILE).toMatch(/[\\/]lean-off$/)
})
diff --git a/cli/src/masterProcess.ts b/cli/src/masterProcess.ts
index e75b84ff0..71481cde9 100644
--- a/cli/src/masterProcess.ts
+++ b/cli/src/masterProcess.ts
@@ -18,6 +18,7 @@ import { baseNode, namedNode } from './harnessd/processName.js'
import { PROBE_ANSWER, PROBE_TIMEOUT_MS } from './harnessd/reexec.js'
import { ensureUtf8Locale } from './lib/childLocale.js'
import { DAEMON_LOG_FILE, HARNESSD_STATUS_FILE, PID_FILE } from './lib/daemonState.js'
+import { isInstalledCopy } from './lib/installedCopy.js'
import { leanFingerprint, readLeanBundle, releaseLeanClaim, writeLeanBundle, type LeanBundle } from './harnessd/leanBundle.js'
import { ts } from './lib/log.js'
import { confirm as confirmUpdate, restore as restoreUpdate, unjudgedUpdate } from './lib/selfUpdate.js'
@@ -69,6 +70,8 @@ export function startMaster(start: MasterStart, exit: (code: number) => void = (
version: VERSION,
reexecMarkerFile: HARNESSD_REEXEC_FILE,
unjudgedUpdate: (bundle) => unjudgedUpdate(env.ADAPTER_CLI_DIR, bundle),
+ // The updater runs for the installed copy alone (lib/installedCopy.ts), beside the core, never in it.
+ updater: !env.ADAPTER_UPDATE_DISABLE && isInstalledCopy(start.scriptPath, env.ADAPTER_CLI_DIR),
// A master that ends cleanly gives up its claim on its lean bundle, so the next one to start can
// clear the folder; one that dies leaves a claim whose pid is gone, which counts for nothing.
exit: (code) => {
diff --git a/cli/src/serviceProcess.spec.ts b/cli/src/serviceProcess.spec.ts
index 1a1181256..59efc8807 100644
--- a/cli/src/serviceProcess.spec.ts
+++ b/cli/src/serviceProcess.spec.ts
@@ -1,7 +1,7 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { SERVICE_RUNNERS, startServiceProcess, type ServiceProcessOptions } from './serviceProcess.js'
import type { ServiceHost, ServiceHostOptions, ServiceProcess } from './services/process.js'
-import { KNOWN_SERVICES } from './harnessd/services.js'
+import { KNOWN_SERVICES, UPDATER_HOST } from './harnessd/services.js'
const local = vi.hoisted(() => ({ socket: '/data/daemon-18473.sock' as string | null }))
vi.mock('./lib/localSocket.js', async (real) => ({ ...await real<object>(), localSocketPath: () => local.socket }))
@@ -13,8 +13,8 @@ describe('a service in its own process', () => {
const title = process.title
afterEach(() => { process.title = title; local.socket = '/data/daemon-18473.sock'; vi.restoreAllMocks() })
- it('can run every service the master knows, and loads each one\'s runner on its own', async () => {
- expect([...SERVICE_RUNNERS.keys()].sort()).toEqual([...KNOWN_SERVICES].sort())
+ it('can run every service the master knows, and the updater it runs beside them, and loads each one\'s runner on its own', async () => {
+ expect([...SERVICE_RUNNERS.keys()].sort()).toEqual([...KNOWN_SERVICES, ...UPDATER_HOST.services].sort())
for (const [name, load] of SERVICE_RUNNERS) expect(typeof await load(), name).toBe('function')
// Every service's own code, imported: seconds on a loaded machine, not the default five.
}, 60_000)
diff --git a/cli/src/serviceProcess.ts b/cli/src/serviceProcess.ts
index caec86a87..732f7d784 100644
--- a/cli/src/serviceProcess.ts
+++ b/cli/src/serviceProcess.ts
@@ -38,6 +38,8 @@ export const SERVICE_RUNNERS: ReadonlyMap<string, () => Promise<Runner>> = new M
['sharing', async () => (await import('./services/sharingProcess.js')).runSharingService],
['orchestrator', async () => (await import('./services/orchestratorProcess.js')).runOrchestratorService],
['commandBar', async () => (await import('./services/commandBarProcess.js')).runCommandBarService],
+ // Not a service the core knows: the master runs it beside them (harnessd/services.ts `UPDATER_HOST`).
+ ['updater', async () => (await import('./services/updaterProcess.js')).runUpdaterService],
['gateway', async () => (await import('./gateway/gatewayProcess.js')).runGatewayService],
['models', async () => (await import('./services/modelsProcess.js')).runModelsService],
['devices', async () => (await import('./services/devicesProcess.js')).runDevicesService],
diff --git a/cli/src/services/updaterProcess.spec.ts b/cli/src/services/updaterProcess.spec.ts
new file mode 100644
index 000000000..d934b345c
--- /dev/null
+++ b/cli/src/services/updaterProcess.spec.ts
@@ -0,0 +1,111 @@
+import { afterEach, describe, expect, it, vi } from 'vitest'
+import { SERVICE_EXIT_RESTART, type UpdaterMessage } from '../harnessd/protocol.js'
+import type { startSelfUpdater } from '../lib/selfUpdate.js'
+import type { startTuiUpdater } from '../tui/update.js'
+import { VERSION } from '../version.js'
+import { processUpdaterDeps, runUpdaterService, type UpdaterServiceDeps } from './updaterProcess.js'
+
+// A lock someone else holds a moment: what the updater says while it waits for it.
+vi.mock('../lib/daemonSpawnLock.js', async (real) => ({
+ ...await real<object>(),
+ withSpawnLock: async (_purpose: string, fn: () => Promise<unknown>, opts: { onWaiting?: (owner: object) => void } = {}) => {
+ opts.onWaiting?.({ pid: 4242, purpose: 'start', since: Date.now(), startMarker: 'x', token: 't' })
+ return fn()
+ },
+}))
+
+type CliOptions = Parameters<typeof startSelfUpdater>[0]
+type HnOptions = Parameters<typeof startTuiUpdater>[0]
+
+describe('the updater in its own process', () => {
+ afterEach(() => { vi.restoreAllMocks() })
+
+ const make = () => {
+ const calls: string[] = []
+ const told: UpdaterMessage[] = []
+ let cli: CliOptions | null = null
+ let hn: HnOptions | null = null
+ let sent: (() => void) | null = null
+ const deps: UpdaterServiceDeps = {
+ tell: (message, done) => { told.push(message); sent = done },
+ exit: (code) => { calls.push(`exit ${code}`) },
+ startCli: (options) => { cli = options; return { stop: () => calls.push('stop cli') } },
+ startHn: (options) => { hn = options; return { stop: () => calls.push('stop hn') } },
+ log: (line) => calls.push(line),
+ }
+ return { deps, calls, told, cli: () => cli!, hn: () => hn!, sent: () => sent! }
+ }
+
+ it('checks for this build\'s successor, and hn\'s, on the installed copy, and stops both', () => {
+ const m = make()
+ const service = runUpdaterService(undefined, m.deps)
+ expect(m.cli()).toMatchObject({ currentVersion: VERSION, stageWhileJudged: true })
+ expect(m.hn()).toMatchObject({ currentVersion: VERSION, isInstalledCopy: true })
+ expect(m.calls[0]).toMatch(new RegExp(`^\\[update\\] self-update on · v${VERSION.replace(/\./g, '\\.')} · every \\d+s`))
+ service.stop()
+ expect(m.calls.slice(-2)).toEqual(['stop cli', 'stop hn'])
+ })
+
+ it('beside a core, stages over a build on probation only under a master that judges that itself', () => {
+ const saved = { beside: process.env.HARNESSD_UPDATER_BESIDE_CORE, judges: process.env.HARNESSD_JUDGES_SUPERSEDED }
+ try {
+ process.env.HARNESSD_UPDATER_BESIDE_CORE = '1'
+ delete process.env.HARNESSD_JUDGES_SUPERSEDED
+ const older = make()
+ runUpdaterService(undefined, older.deps)
+ expect(older.cli().stageWhileJudged).toBe(false)
+ process.env.HARNESSD_JUDGES_SUPERSEDED = '1'
+ const newer = make()
+ runUpdaterService(undefined, newer.deps)
+ expect(newer.cli().stageWhileJudged).toBe(true)
+ } finally {
+ for (const [name, value] of [['HARNESSD_UPDATER_BESIDE_CORE', saved.beside], ['HARNESSD_JUDGES_SUPERSEDED', saved.judges]] as const) {
+ if (value === undefined) delete process.env[name]
+ else process.env[name] = value
+ }
+ }
+ })
+
+ it('tells the master what it staged, and only once that has left asks to be started again as the new build', async () => {
+ const m = make()
+ runUpdaterService(undefined, m.deps)
+ const staged = m.cli().onStaged('9.9.9') as Promise<void>
+ expect(m.told).toEqual([{ type: 'harnessd:staged', version: '9.9.9' }])
+ expect(m.calls).not.toContain(`exit ${SERVICE_EXIT_RESTART}`)
+ m.sent()()
+ await staged
+ expect(m.calls).toContain(`exit ${SERVICE_EXIT_RESTART}`)
+ })
+
+ it('stages under the spawn lock, saying who holds it while it waits', async () => {
+ const m = make()
+ runUpdaterService(undefined, m.deps)
+ expect(await m.cli().withLock!(async () => 'staged')).toBe('staged')
+ expect(m.calls.at(-1)).toMatch(/^\[update\] waiting — the daemon is .*4242/)
+ })
+
+ it('reaches the master over the spawn channel, or tells no one when started by hand', () => {
+ const deps = processUpdaterDeps()
+ const exit = vi.spyOn(process, 'exit').mockImplementation((() => {}) as never)
+ deps.exit(SERVICE_EXIT_RESTART)
+ expect(exit).toHaveBeenCalledWith(SERVICE_EXIT_RESTART)
+ const log = vi.spyOn(console, 'log').mockImplementation(() => {})
+ deps.log('a line')
+ expect(log).toHaveBeenCalledWith('a line')
+ const original = process.send
+ try {
+ const sent = vi.fn()
+ process.send = undefined
+ deps.tell({ type: 'harnessd:staged', version: '9.9.9' }, sent)
+ expect(sent).toHaveBeenCalledOnce()
+ const send = vi.fn((_message: unknown, callback: () => void) => { callback(); return true })
+ process.send = send as unknown as typeof process.send
+ const left = vi.fn()
+ deps.tell({ type: 'harnessd:staged', version: '9.9.9' }, left)
+ expect(send.mock.calls[0][0]).toEqual({ type: 'harnessd:staged', version: '9.9.9' })
+ expect(left).toHaveBeenCalledOnce()
+ } finally { process.send = original }
+ expect(deps.startCli).toBeTypeOf('function')
+ expect(deps.startHn).toBeTypeOf('function')
+ })
+})
diff --git a/cli/src/services/updaterProcess.ts b/cli/src/services/updaterProcess.ts
new file mode 100644
index 000000000..b87243bc2
--- /dev/null
+++ b/cli/src/services/updaterProcess.ts
@@ -0,0 +1,80 @@
+/**
+ * Harness's updater, in a process of its own that harnessd's master runs (`harness __service updater`,
+ * harnessd/services.ts `UPDATER_HOST`), only for the installed copy with updates on (masterProcess.ts).
+ *
+ * It checks the release manifest for a newer CLI and a newer hn, downloads and verifies them, runs the new
+ * CLI's canary and stages it (lib/selfUpdate.ts, tui/install.ts), then tells the master (`harnessd:staged`).
+ * The master asks the core to hand over (`harnessd:update`) and judges the new build on probation, as when
+ * the core staged it itself. That is what it was until now: the core's process downloaded builds, a 15 s
+ * canary and minutes of downloads in a process that must never go down, and a core in safe mode ran on
+ * only for its updater. Here a broken download or a hung check costs this process alone, and a core that
+ * cannot start at all still gets its fix (docs/design/2026-10-06-core-boundary-next.md, "Updaters").
+ *
+ * Once it has staged a build it exits with `SERVICE_EXIT_RESTART`, and the master starts it again at once:
+ * a staging is the updater's last act on the bundle it runs, and the next check is the new build's.
+ */
+import { env } from '../config/env.js'
+import { SERVICE_EXIT_RESTART, type UpdaterMessage } from '../harnessd/protocol.js'
+import { describeSpawnLockOwner, withSpawnLock } from '../lib/daemonSpawnLock.js'
+import { DOWNLOAD_LIMITS, startSelfUpdater } from '../lib/selfUpdate.js'
+import { startTuiUpdater } from '../tui/update.js'
+import { VERSION } from '../version.js'
+import type { ServiceProcess } from './process.js'
+
+export interface UpdaterServiceDeps {
+ /** Tell the master; `sent` once the message has left (an exit before it would lose it). */
+ tell(message: UpdaterMessage, sent: () => void): void
+ exit(code: number): void
+ startCli: typeof startSelfUpdater
+ startHn: typeof startTuiUpdater
+ log(line: string): void
+}
+
+export const processUpdaterDeps = (): UpdaterServiceDeps => ({
+ tell: (message, sent) => {
+ // Under its master, the spawn channel; started by hand, there is no one to tell, and nothing to wait on.
+ if (!process.send) { sent(); return }
+ process.send(message, () => sent())
+ },
+ exit: (code) => process.exit(code),
+ startCli: startSelfUpdater,
+ startHn: startTuiUpdater,
+ log: (line) => console.log(line),
+})
+
+export function runUpdaterService(_options?: unknown, deps: UpdaterServiceDeps = processUpdaterDeps()): ServiceProcess {
+ const cli = deps.startCli({
+ currentVersion: VERSION,
+ url: env.ADAPTER_UPDATE_URL,
+ key: env.ADAPTER_UPDATE_KEY,
+ dir: env.ADAPTER_CLI_DIR,
+ intervalMs: env.ADAPTER_UPDATE_CHECK_MS,
+ slotSecond: env.ADAPTER_UPDATE_SLOT_SEC,
+ // The lock spans the byte swap and the word to the master: a `harness start` or `harness update` that
+ // lands between them would stage over the backup this one just made.
+ withLock: (fn) => withSpawnLock('handoff', fn, {
+ onWaiting: (owner) => deps.log(`[update] waiting — the daemon is ${describeSpawnLockOwner(owner)}`),
+ }),
+ onStaged: (version) => new Promise<void>((done) => {
+ deps.tell({ type: 'harnessd:staged', version }, () => {
+ done()
+ deps.exit(SERVICE_EXIT_RESTART)
+ })
+ }),
+ limits: { idleMs: env.ADAPTER_UPDATE_IDLE_MS, deadlineMs: env.ADAPTER_UPDATE_DEADLINE_MS, floorBytesPerSecond: DOWNLOAD_LIMITS.floorBytesPerSecond },
+ // A master judges a build staged while the one before is on probation (#807). Beside a core, under a
+ // master too old to run the updater (core/updaterBeside.ts), only one that says so: one from before
+ // rolled the newer build back with the one it judged.
+ stageWhileJudged: process.env.HARNESSD_UPDATER_BESIDE_CORE !== '1' || process.env.HARNESSD_JUDGES_SUPERSEDED === '1',
+ })
+ // hn is an independent, optional download: its failure never holds up the CLI's fix.
+ const hn = deps.startHn({
+ currentVersion: VERSION,
+ isInstalledCopy: true,
+ disabled: env.ADAPTER_UPDATE_DISABLE,
+ intervalMs: env.ADAPTER_UPDATE_CHECK_MS,
+ slotSecond: env.ADAPTER_UPDATE_SLOT_SEC,
+ })
+ deps.log(`[update] self-update on · v${VERSION} · every ${Math.round(env.ADAPTER_UPDATE_CHECK_MS / 1000)}s`)
+ return { stop: () => { cli.stop(); hn.stop() } }
+}
diff --git a/cli/src/startupOrder.spec.ts b/cli/src/startupOrder.spec.ts
index a0180b4f2..b4b9d2254 100644
--- a/cli/src/startupOrder.spec.ts
+++ b/cli/src/startupOrder.spec.ts
@@ -82,13 +82,10 @@ function callArguments(source: string, call: string): { text: string; at: number
}
/** Every call whose dependencies run DURING start-up, before `runForeground` has finished its body. */
-const STARTUP_CALLS = ['await repairClaudeCwd({', 'await restoreAgents({', 'startSelfUpdater({', 'startTuiUpdater({']
+const STARTUP_CALLS = ['await repairClaudeCwd({', 'await restoreAgents({']
-/** What the prologue is allowed to do before the updater is running: nothing that can throw. */
-const PROLOGUE_CALLS = new Set([
- 'installTimestampedConsole', 'startSelfUpdater', 'statSync', 'join', 'String', 'Number', 'Date',
- 'withSpawnLock', 'describeSpawnLockOwner',
-])
+/** What the prologue is allowed to do before the master's update is listened for: nothing that can throw. */
+const PROLOGUE_CALLS = new Set(['installTimestampedConsole'])
const KEYWORDS = new Set(['if', 'for', 'while', 'switch', 'catch', 'return', 'typeof', 'new', 'await', 'function'])
describe('the core\'s start-up order (core/main.ts)', () => {
@@ -101,16 +98,17 @@ describe('the core\'s start-up order (core/main.ts)', () => {
})
// ── The invariant the whole safe-boot design rests on ───────────────────────────────────────────
- // A daemon that cannot finish starting can only be fixed by its own updater, and the updater can
- // only run if start-up reached it. So it goes first, and everything that can throw or hang goes
- // after. These three say so in the order they are worth reading.
+ // A daemon that cannot finish starting is fixed by the build the master's updater stages, and leaves for
+ // it only if start-up got as far as listening for the master's request. So that goes first, and
+ // everything that can throw or hang goes after. These three say so in the order they are worth reading.
- it('starts the updater before anything that can throw or hang', () => {
+ it('listens for the master\'s update before anything that can throw or hang, and runs no updater itself', () => {
const source = code(SOURCE)
- const updater = source.indexOf('startSelfUpdater({')
- expect(updater, 'core/main.ts still starts the self-updater').toBeGreaterThan(-1)
+ const updater = source.indexOf('coreLink.onUpdate(')
+ expect(updater, 'core/main.ts still listens for the master\'s update').toBeGreaterThan(-1)
+ // The updater is the master's, in its own process (services/updaterProcess.ts): the core downloads no build.
+ expect(source).not.toMatch(/startSelfUpdater\(|startTuiUpdater\(/)
for (const risky of [
- 'startTuiUpdater(', // optional hn download must never precede CLI recovery
'requireTmuxAvailable(', // throws outright when tmux is missing
'await startHookServer(', // EADDRINUSE on a fixed port with no fallback
'installSessionHooks(', // 13 vendor settings files, any of which can be unreadable
@@ -120,7 +118,7 @@ describe('the core\'s start-up order (core/main.ts)', () => {
]) {
const at = source.indexOf(risky)
if (at < 0) continue // renamed or gone; the TDZ lint above still covers what remains
- expect(at, `\`${risky}\` must come after startSelfUpdater({ — a crash there would strand the machine`)
+ expect(at, `\`${risky}\` must come after coreLink.onUpdate( — a crash there would strand the machine`)
.toBeGreaterThan(updater)
}
})
@@ -129,15 +127,15 @@ describe('the core\'s start-up order (core/main.ts)', () => {
// Stronger than listing today's hazards: this is what catches the NEXT line somebody adds on top.
const source = code(SOURCE)
const from = source.indexOf('async function runForeground(')
- const to = source.indexOf('startSelfUpdater({')
+ const to = source.indexOf('coreLink.onUpdate(')
const prologue = source.slice(source.indexOf('{', from), to)
- expect(prologue.includes('\n await '), 'no await may precede the updater — a hang there is unrecoverable').toBe(false)
- expect(/\n\s*throw /.test(prologue), 'no throw may precede the updater').toBe(false)
+ expect(prologue.includes('\n await '), 'no await may precede the update — a hang there is unrecoverable').toBe(false)
+ expect(/\n\s*throw /.test(prologue), 'no throw may precede the update').toBe(false)
const called = [...prologue.replace(/\.\s*[A-Za-z_$][\w$]*\s*\(/g, ' ').matchAll(/([A-Za-z_$][\w$]*)\s*\(/g)]
.map(match => match[1])
.filter(name => !KEYWORDS.has(name))
expect([...new Set(called.filter(name => !PROLOGUE_CALLS.has(name)))].sort(),
- 'only calls that cannot fail belong above the updater; move this below it, or add it to PROLOGUE_CALLS with a reason')
+ 'only calls that cannot fail belong above the update; move this below it, or add it to PROLOGUE_CALLS with a reason')
.toEqual([])
})
@@ -148,13 +146,16 @@ describe('the core\'s start-up order (core/main.ts)', () => {
const source = code(SOURCE)
const flip = source.indexOf('daemonBoot.applyStagedUpdate = ')
expect(flip, 'the handoff handler is still swapped in core/main.ts').toBeGreaterThan(-1)
- const body = callArguments(source, 'updateHandoff.restartForUpdate(')
- expect(body.at, 'the handler is the one it swaps in').toBeGreaterThan(flip)
+ expect(callArguments(source, 'updateHandoff.restartForUpdate(').at, 'the handler is the one it swaps in').toBeGreaterThan(flip)
+ // Its teardown, which a core with no master hands over with too: built just before the swap.
+ const body = callArguments(source, 'const updateTeardown = (')
+ const teardown = { text: source.slice(body.at, source.indexOf('\n ]\n', body.at)), at: body.at }
+ expect(teardown.at).toBeLessThan(flip)
const declaredAt = localsOf(source)
- const late = referencedIn(body.text).filter(used => (declaredAt.get(used) ?? -1) > flip).sort()
+ const late = referencedIn(teardown.text).filter(used => (declaredAt.get(used) ?? -1) > teardown.at).sort()
expect(late, 'these are torn down by the update handoff but declared after it is armed').toEqual([])
// What it tears down is all there is to it: the handoff module reaches nothing of runForeground's.
- expect(referencedIn(body.text)).toEqual(expect.arrayContaining(['registry', 'hookServer', 'localWsServer', 'backend', 'watcher']))
+ expect(referencedIn(teardown.text)).toEqual(expect.arrayContaining(['registry', 'hookServer', 'localWsServer', 'backend', 'watcher']))
})
it('the daemon arm survives its own start-up failure', () => {