Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

harnessd: the master re-executes itself on update, keeping its pid (#791)

Dee28cc10266cde
diff --git a/cli/e2e/reexec.e2e.ts b/cli/e2e/reexec.e2e.ts
new file mode 100644
index 000000000..13edb5bf7
--- /dev/null
+++ b/cli/e2e/reexec.e2e.ts
@@ -0,0 +1,345 @@
+/**
+ * The master re-executing itself on an update, on real release bundles (harnessd/reexec.ts): the core
+ * stages a newer build and exits 75, and the master, before it starts a core on it, replaces itself
+ * with that build in its own process. The pid stays, the core comes back on the new build under the
+ * new master, and the agent at work in tmux carries on.
+ *
+ * A build whose master does not answer its probe is rolled back and remembered, like one whose core
+ * fails. One whose master dies after it re-executed leaves its marker, and the next start (the
+ * desktop app's `harness start`, or launchd's restart) rolls it back and moves onto the build before.
+ * Nothing is left orphaned at any step: every core is the master's child.
+ *
+ * The bundles are built from this checkout at made-up versions and served from a local manifest, as in
+ * e2e/update.e2e.ts, so nothing here reaches beyond the machine.
+ */
+import { execFileSync } from 'node:child_process'
+import { createHash } from 'node:crypto'
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
+import { createServer, type Server } from 'node:http'
+import { tmpdir } from 'node:os'
+import { dirname, join } from 'node:path'
+import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
+import { LocalClient, type Frame } from './harness/client.js'
+import { CLI_ROOT, IsolatedDaemon, until } from './harness/daemon.js'
+
+const FIRST = '42.0.1'
+const sha = (bytes: Buffer): string => createHash('sha256').update(bytes).digest('hex')
+const count = (text: string, part: string): number => text.split(part).length - 1
+const sleep = (ms: number) => new Promise((done) => setTimeout(done, ms))
+
+describe('the master re-executing itself on an update', () => {
+ let scratch = ''
+ let server: Server | undefined
+ let daemon: IsolatedDaemon | undefined
+ let master: number | null = null
+ let agent: Record<string, any> = {}
+ const releases = new Map<string, Buffer>()
+ let notify = Buffer.alloc(0)
+ let offered = FIRST
+ const cliDir = () => join(scratch, 'cli')
+ const installed = () => readFileSync(join(cliDir(), 'cli.js'))
+ const rejected = () => JSON.parse(readFileSync(join(cliDir(), 'update-rejected.json'), 'utf8')) as string[]
+ const marker = () => join(daemon!.dataDir, 'harnessd-reexec.json')
+ const status = async (): Promise<Record<string, any> | null> =>
+ fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json()).catch(() => null)
+ /** The build the core runs, and the one the master runs, as the daemon reports them. */
+ const builds = async (): Promise<{ core: string | null; master: string | null; masterPid: number | null; reexecs: number | null }> => {
+ const now = await status()
+ return { core: now?.version ?? null, master: now?.harnessd?.masterVersion ?? null, masterPid: now?.harnessd?.masterPid ?? null, reexecs: now?.harnessd?.reexecs ?? null }
+ }
+ const on = (version: string) => until(`the core and the master to run ${version}`, async () => {
+ const now = await builds()
+ return now.core === version && now.master === version ? now : null
+ }, 90_000, 250)
+
+ const rows = async (client: LocalClient) =>
+ (await client.request<{ agents: Array<Record<string, any>> }>('agents_list', { includeStopped: true }, 30_000)).agents
+ /** The agent is back, the same conversation, and takes a turn: the core under the new master serves it. */
+ const agentCarriesOn = async () => {
+ const client = await LocalClient.connect(daemon!)
+ try {
+ await until('the agent to be back', async () => {
+ const row = (await rows(client)).find((candidate) => candidate.id === agent.id)
+ return row?.status === 'active' && row.sessionId === agent.sessionId ? row : null
+ }, 60_000, 500)
+ const ended = client.next((frame: Frame) => frame.type === 'turn_ended' && frame.agentId === agent.id, 45_000, 'a turn')
+ client.send('message', { agentId: agent.id, content: 'still there?' })
+ await ended
+ } finally { client.close() }
+ }
+
+ /**
+ * Every process running a build of this machine, from the process table. Each core must be the
+ * master's child: one left behind by a master that replaced itself would be an orphan holding the
+ * port, re-parented to launchd or init.
+ */
+ const daemonProcesses = (): Array<{ pid: number; ppid: number; command: string }> =>
+ execFileSync('ps', ['-axo', 'pid=,ppid=,command='], { encoding: 'utf8' }).split('\n')
+ .map((line) => /^\s*(\d+)\s+(\d+)\s+(.*)$/.exec(line))
+ .filter((match): match is RegExpExecArray => !!match && match[3].includes(join(cliDir(), 'cli.js')))
+ .map((match) => ({ pid: Number(match[1]), ppid: Number(match[2]), command: match[3] }))
+ const nothingOrphaned = async (masterPid: number) => {
+ // A probe is over within the update; give one a moment to end.
+ await until('only the master\'s own children', () => daemonProcesses().every((process) => process.ppid === masterPid && process.command.endsWith(' __run')), 10_000, 200)
+ expect(daemonProcesses().filter((process) => process.command.endsWith(' __run'))).toHaveLength(1)
+ }
+
+ beforeAll(async () => {
+ scratch = mkdtempSync(join(tmpdir(), 'harnessd-reexec-'))
+ const out = join(scratch, 'build')
+ execFileSync(process.execPath, ['build-bundle.mjs'], {
+ cwd: CLI_ROOT, env: { ...process.env, ADAPTER_VERSION: FIRST, BUNDLE_OUT_DIR: out }, stdio: 'pipe',
+ })
+ const first = readFileSync(join(out, 'cli.js'), 'utf8')
+ notify = readFileSync(join(out, 'notify.mjs'))
+ expect(count(first, FIRST)).toBeGreaterThanOrEqual(2)
+ const release = (version: string, inject = ''): void => {
+ // The version is baked in at build time; the others are the same bytes with it swapped, and a
+ // fault put in at the top where a test needs one.
+ const source = first.replaceAll(FIRST, version).replace('\n', `\n${inject}\n`)
+ const file = join(out, `cli-${version}.js`)
+ writeFileSync(file, source)
+ expect(execFileSync(process.execPath, [file, 'version'], { encoding: 'utf8' }).trim()).toBe(version)
+ releases.set(version, Buffer.from(source))
+ }
+ release(FIRST)
+ release('42.0.2')
+ // Passes the updater's canary (`cli.js version`), and its master does not answer its probe.
+ release('42.0.3', 'if(process.argv[2]==="__harnessd-probe")process.exit(3);')
+ release('42.0.4')
+ // Answers its probe, and dies as soon as a master re-executes on it.
+ release('42.0.5', 'if(process.argv[2]==="__harnessd"&&process.env.HARNESSD_RESUME)process.exit(4);')
+
+ server = createServer((request, response) => {
+ const url = request.url ?? ''
+ const origin = `http://127.0.0.1:${(server!.address() as { port: number }).port}`
+ if (url === '/metadata.json') {
+ const cli = releases.get(offered)!
+ response.end(JSON.stringify({ cli: {
+ version: offered,
+ cli: { url: `${origin}/cli-${offered}.js`, sha256: sha(cli), size: cli.length },
+ notify: { url: `${origin}/notify.mjs`, sha256: sha(notify), size: notify.length },
+ } }))
+ return
+ }
+ const asked = /^\/cli-(.+)\.js$/.exec(url)?.[1]
+ if (asked && releases.has(asked)) { response.end(releases.get(asked)); return }
+ if (url === '/notify.mjs') { response.end(notify); return }
+ response.statusCode = 404
+ response.end()
+ })
+ await new Promise<void>((done) => server!.listen(0, '127.0.0.1', done))
+ const local = `http://127.0.0.1:${(server.address() as { port: number }).port}`
+
+ mkdirSync(cliDir())
+ writeFileSync(join(cliDir(), 'cli.js'), releases.get(FIRST)!)
+ writeFileSync(join(cliDir(), 'notify.mjs'), notify)
+ writeFileSync(join(cliDir(), 'package.json'), '{"type":"module"}\n')
+ daemon = await IsolatedDaemon.create({
+ scriptPath: join(cliDir(), 'cli.js'),
+ env: {
+ ADAPTER_CLI_DIR: cliDir(),
+ ADAPTER_UPDATE_DISABLE: 'false',
+ ADAPTER_UPDATE_URL: `${local}/metadata.json`,
+ ADAPTER_UPDATE_CHECK_MS: '1000',
+ ADAPTER_UPDATE_SLOT_SEC: '-1',
+ HARNESSD_UPDATE_PROBATION_MS: '3000',
+ HARNESSD_INITIAL_BACKOFF_MS: '100',
+ HARNESS_TUI_MANIFEST_URL: `${local}/tui/metadata.json`,
+ ADAPTER_RUNTIME_METADATA_URL: `${local}/runtime/metadata.json`,
+ ADAPTER_GRID_RUNTIME_METADATA_URL: `${local}/grid/metadata.json`,
+ },
+ })
+ await daemon.start()
+ master = daemon.pid
+ expect(await builds()).toMatchObject({ core: FIRST, master: FIRST, masterPid: master, reexecs: 0 })
+
+ // An agent at work, whose pane lives in tmux through every swap below.
+ const client = await LocalClient.connect(daemon)
+ const cwd = join(daemon.projectsDir, 'reexec')
+ mkdirSync(cwd, { recursive: true })
+ const created = await client.request('agent_create', { engine: 'claude', cwd, bypassPermission: true }, 90_000)
+ agent = await until('the agent to bind its conversation', async () => {
+ const row = (await rows(client)).find((candidate) => candidate.id === created.agent.id)
+ return row?.sessionId && row.status === 'active' ? row : null
+ }, 60_000, 500)
+ client.close()
+ })
+
+ afterAll(async () => {
+ await daemon?.close()
+ await new Promise<void>((done) => server ? server.close(() => done()) : done())
+ if (scratch) rmSync(scratch, { recursive: true, force: true })
+ })
+
+ const showLog = () => onTestFailed(() => { console.log(`---- daemon log\n${daemon?.log().split('\n').slice(-150).join('\n')}`) })
+
+ it('replaces the master with the new build in its own process: same pid, and the core and the agent carry on', async () => {
+ showLog()
+ offered = '42.0.2'
+ const now = await on('42.0.2')
+ expect(now).toMatchObject({ masterPid: master, reexecs: 1 })
+ expect(daemon!.pid).toBe(master)
+ expect(IsolatedDaemon.alive(master)).toBe(true)
+ expect(daemon!.log()).toContain(`[harnessd] master re-executed (pid ${master}) · now v42.0.2`)
+ await until('the master to keep the update', () => daemon!.log().includes('the update stayed up — keeping it'), 30_000)
+ expect(installed()).toEqual(releases.get('42.0.2'))
+ expect(existsSync(marker())).toBe(false)
+ await agentCarriesOn()
+ await nothingOrphaned(master!)
+ })
+
+ it('rolls back a build whose master does not answer its probe, and remembers it; the master stays as it was', async () => {
+ showLog()
+ const from = daemon!.log().length
+ offered = '42.0.3'
+ await until('the rollback', () => daemon!.log().slice(from).includes('the new bundle\'s master would not start — rolled back to the previous bundle'), 90_000)
+ expect(daemon!.log().slice(from)).toContain('did not answer its probe')
+ expect(await on('42.0.2')).toMatchObject({ masterPid: master, reexecs: 1 })
+ expect(installed()).toEqual(releases.get('42.0.2'))
+ expect(rejected()).toEqual(['42.0.3'])
+ expect(existsSync(join(cliDir(), 'update-pending.json'))).toBe(false)
+ // A check every second: a few more of them, and the rejected build is not staged again.
+ await until('the core to say it waits for a newer build', () => daemon!.log().slice(from).includes('42.0.3 was rolled back on this machine'), 30_000)
+ const cores = daemon!.coresStarted()
+ await sleep(4_000)
+ expect(daemon!.coresStarted()).toBe(cores)
+ await agentCarriesOn()
+ await nothingOrphaned(master!)
+ })
+
+ it('moves on when a newer build is published', async () => {
+ showLog()
+ offered = '42.0.4'
+ expect(await on('42.0.4')).toMatchObject({ masterPid: master, reexecs: 2 })
+ await until('the master to keep the update', () => count(daemon!.log(), 'the update stayed up — keeping it') === 2, 30_000)
+ expect(rejected()).toEqual(['42.0.3'])
+ await agentCarriesOn()
+ await nothingOrphaned(master!)
+ })
+
+ it('rolls back, at the next start, a build whose master died after re-executing on it', async () => {
+ showLog()
+ offered = '42.0.5'
+ await until('the master that re-executed on 42.0.5 to die', () => !IsolatedDaemon.alive(master), 90_000, 200)
+ expect(readFileSync(marker(), 'utf8')).toContain(`"pid":${master}`)
+ expect(installed()).toEqual(releases.get('42.0.5'))
+ // The core it replaced had exited for the update; nothing was left running in its name.
+ expect(daemonProcesses()).toEqual([])
+
+ // What the desktop app's `harness start`, or launchd restarting the master, would do.
+ const from = daemon!.log().length
+ await daemon!.start()
+ const restarted = daemon!.pid!
+ expect(restarted).not.toBe(master)
+ expect(daemon!.log().slice(from)).toContain(`the master that re-executed on the new bundle (pid ${master}) never brought a core up — rolled back to the previous bundle`)
+ expect(await on('42.0.4')).toMatchObject({ masterPid: restarted, reexecs: 1 })
+ expect(installed()).toEqual(releases.get('42.0.4'))
+ expect(rejected()).toEqual(['42.0.3', '42.0.5'])
+ await until('the marker to be gone', () => !existsSync(marker()), 30_000)
+ await agentCarriesOn()
+ await nothingOrphaned(restarted)
+ })
+})
+
+/**
+ * A released master and this one, each over the other's core (protocol.ts only grows). Run when
+ * MIGRATION_FROM names a released bundle, as e2e/migration.e2e.ts is: one from before re-execution.
+ *
+ * An older master cannot re-execute: after an update it supervises the newer core until it is
+ * restarted, as every machine will once this ships. A newer master that finds the older bundle back on
+ * disk does not re-execute on it either: the older bundle does not know the probe, and is never run as
+ * a master to find out. Its core runs under the newer master.
+ */
+const RELEASED = process.env.MIGRATION_FROM
+describe.skipIf(!RELEASED)('a released master and this one, each over the other\'s core', () => {
+ const NEXT = '0.99.0'
+ let scratch = ''
+ let server: Server | undefined
+ let daemon: IsolatedDaemon | undefined
+ let offered: 'released' | 'next' = 'released'
+ let releasedVersion = ''
+ const bundles = new Map<'released' | 'next', { cli: Buffer; notify: Buffer; version: string }>()
+ const cliDir = () => join(scratch, 'cli')
+ const status = async (): Promise<Record<string, any> | null> =>
+ fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json()).catch(() => null)
+
+ beforeAll(async () => {
+ scratch = mkdtempSync(join(tmpdir(), 'harnessd-reexec-compat-'))
+ const out = join(scratch, 'build')
+ execFileSync(process.execPath, ['build-bundle.mjs'], { cwd: CLI_ROOT, env: { ...process.env, ADAPTER_VERSION: NEXT, BUNDLE_OUT_DIR: out }, stdio: 'pipe' })
+ bundles.set('next', { cli: readFileSync(join(out, 'cli.js')), notify: readFileSync(join(out, 'notify.mjs')), version: NEXT })
+ releasedVersion = execFileSync(process.execPath, [RELEASED!, 'version'], { encoding: 'utf8' }).trim()
+ bundles.set('released', { cli: readFileSync(RELEASED!), notify: readFileSync(join(dirname(RELEASED!), 'notify.mjs')), version: releasedVersion })
+ server = createServer((request, response) => {
+ const origin = `http://127.0.0.1:${(server!.address() as { port: number }).port}`
+ const bundle = bundles.get(offered)!
+ if (request.url === '/metadata.json') {
+ response.end(JSON.stringify({ cli: {
+ version: bundle.version,
+ cli: { url: `${origin}/cli.js`, sha256: sha(bundle.cli), size: bundle.cli.length },
+ notify: { url: `${origin}/notify.mjs`, sha256: sha(bundle.notify), size: bundle.notify.length },
+ } }))
+ return
+ }
+ if (request.url === '/cli.js') { response.end(bundle.cli); return }
+ if (request.url === '/notify.mjs') { response.end(bundle.notify); return }
+ response.statusCode = 404
+ response.end()
+ })
+ await new Promise<void>((done) => server!.listen(0, '127.0.0.1', done))
+ const local = `http://127.0.0.1:${(server.address() as { port: number }).port}`
+ mkdirSync(cliDir())
+ writeFileSync(join(cliDir(), 'cli.js'), bundles.get('released')!.cli)
+ writeFileSync(join(cliDir(), 'notify.mjs'), bundles.get('released')!.notify)
+ writeFileSync(join(cliDir(), 'package.json'), '{"type":"module"}\n')
+ daemon = await IsolatedDaemon.create({
+ scriptPath: join(cliDir(), 'cli.js'),
+ env: {
+ ADAPTER_CLI_DIR: cliDir(), ADAPTER_UPDATE_DISABLE: 'false', ADAPTER_UPDATE_URL: `${local}/metadata.json`,
+ ADAPTER_UPDATE_CHECK_MS: '1000', ADAPTER_UPDATE_SLOT_SEC: '-1', HARNESSD_UPDATE_PROBATION_MS: '3000', HARNESSD_INITIAL_BACKOFF_MS: '100',
+ HARNESS_TUI_MANIFEST_URL: `${local}/tui/metadata.json`, ADAPTER_RUNTIME_METADATA_URL: `${local}/runtime/metadata.json`,
+ ADAPTER_GRID_RUNTIME_METADATA_URL: `${local}/grid/metadata.json`,
+ },
+ })
+ await daemon.start()
+ })
+
+ afterAll(async () => {
+ await daemon?.close()
+ await new Promise<void>((done) => server ? server.close(() => done()) : done())
+ if (scratch) rmSync(scratch, { recursive: true, force: true })
+ })
+
+ it('an older master supervises the newer core an update brings, until it restarts; then this master runs', async () => {
+ onTestFailed(() => { console.log(`---- daemon log\n${daemon?.log().split('\n').slice(-120).join('\n')}`) })
+ const master = daemon!.pid
+ expect((await status())?.version).toBe(releasedVersion)
+ offered = 'next'
+ const now = await until('the newer core under the released master', async () => {
+ const current = await status()
+ return current?.version === NEXT ? current : null
+ }, 90_000, 250)
+ expect(now.harnessd.masterPid).toBe(master)
+ expect(now.harnessd.masterVersion).toBeUndefined()
+ expect(daemon!.log()).not.toContain('re-executing')
+ await until('the released master to keep the update', () => daemon!.log().includes('the update stayed up — keeping it'), 30_000)
+ await daemon!.restart()
+ expect((await status())?.harnessd).toMatchObject({ masterVersion: NEXT, masterPid: daemon!.pid })
+ })
+
+ it('this master runs the older bundle\'s core when it finds it back on disk, without re-executing on it', async () => {
+ onTestFailed(() => { console.log(`---- daemon log\n${daemon?.log().split('\n').slice(-120).join('\n')}`) })
+ offered = 'released' // not newer: the older core's updater leaves it alone
+ const master = daemon!.pid
+ const from = daemon!.log().length
+ writeFileSync(join(cliDir(), 'cli.js'), bundles.get('released')!.cli)
+ process.kill((await status())!.corePid as number, 'SIGKILL')
+ const now = await until('the older core under this master', async () => {
+ const current = await status()
+ return current?.version === releasedVersion ? current : null
+ }, 60_000, 250)
+ expect(now.harnessd).toMatchObject({ masterVersion: NEXT, masterPid: master, reexecs: 0 })
+ expect(daemon!.log().slice(from)).toContain('did not answer its probe (Unknown command: __harnessd-probe) — keeping this master')
+ })
+})
diff --git a/cli/src/cli.ts b/cli/src/cli.ts
index 8b9f66391..2ece36e23 100644
--- a/cli/src/cli.ts
+++ b/cli/src/cli.ts
@@ -177,7 +177,7 @@ import { STORE_REQUESTS, startStore } from './services/store.js'
import { startViewers } from './services/viewers.js'
import { MODELS_REQUESTS, startModels } from './services/models.js'
import { startWorkspaces } from './services/workspaces.js'
-import { describeMasterStatus, readStatusFile, runMaster } from './harnessd/master.js'
+import { describeMasterStatus, probeMaster, readStatusFile, runMaster } from './harnessd/master.js'
import { CORE_EXIT_STOP, CORE_EXIT_UPDATE } from './harnessd/protocol.js'
import { isLocalSocketName, localSocketPath, refuseServedDataFolder, type LocalSocketServer } from './lib/localSocket.js'
import { legacyDaemonStatus, localDaemonStatus, saveDaemonPort } from './lib/daemonEndpoint.js'
@@ -242,6 +242,8 @@ import {
// Daemon stdout/stderr. Capped at LOG_MAX_BYTES — see prepareLogFile/trimLogFile in lib/log.ts.
const LOG_FILE = DAEMON_LOG_FILE
+/** Where a master re-executing on a new bundle leaves word of it, until that master's core is up. */
+const HARNESSD_REEXEC_FILE = join(env.ADAPTER_DATA_DIR, 'harnessd-reexec.json')
// Pre-rename name. Adopted (renamed, keeping the inode) the first time a daemon opens the log, so a
// machine that updates mid-run keeps its history instead of stranding it in a file nobody tails.
// The log has had three names; this slot holds the OLDEST. The middle one (`machine.log`) is adopted
@@ -5628,8 +5630,13 @@ switch (cmd) {
logFile: LOG_FILE,
restoreUpdate: () => restoreUpdate(env.ADAPTER_CLI_DIR),
confirmUpdate: () => confirmUpdate(env.ADAPTER_CLI_DIR),
+ version: VERSION,
+ reexecMarkerFile: HARNESSD_REEXEC_FILE,
})
break
+ case '__harnessd-probe': // internal: a master about to re-execute on this bundle asks it first (harnessd/reexec.ts)
+ process.exitCode = probeMaster({ env: process.env, execArgv: process.execArgv, version: VERSION })
+ break
case '__service': { // internal: a service harnessd's master runs in its own process (services/process.ts)
const socketPath = localSocketPath(env.ADAPTER_DATA_DIR, env.PORT)
const runService = rest[0] === 'search' ? runSearchService : rest[0] === 'viewers' ? runViewersService
diff --git a/cli/src/harnessd/AGENTS.md b/cli/src/harnessd/AGENTS.md
index 896b8b179..41a63ec79 100644
--- a/cli/src/harnessd/AGENTS.md
+++ b/cli/src/harnessd/AGENTS.md
@@ -20,3 +20,6 @@ It keeps the core and the services running, and nothing else.
Under systemd, never stop the unit with a stop job (`systemctl stop`, `restart`, `disable --now`):
tmux built with systemd support makes every pane PartOf the unit that started its server, so a stop
job ends every agent. `stop()` signals the master instead. The desktop app does not use it yet.
+6. **The master replaces itself only in the gap before a core starts** (`reexec.ts`): no core running,
+ services stopped and reaped first, in place so the pid stays. Never exec with a child running: it
+ would lose its channel, and become an orphan or a zombie no one reaps.
diff --git a/cli/src/harnessd/master.spec.ts b/cli/src/harnessd/master.spec.ts
index 49e406c11..87a37e5e9 100644
--- a/cli/src/harnessd/master.spec.ts
+++ b/cli/src/harnessd/master.spec.ts
@@ -5,10 +5,11 @@ import { join } from 'node:path'
import { spawn, type ChildProcess } from 'node:child_process'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
- coreExecArgv, coreHandle, describeMasterStatus, heapLimitInArgv, masterDefaults, onProcessSignal, processExit,
+ coreExecArgv, coreHandle, describeMasterStatus, heapLimitInArgv, masterDefaults, onProcessSignal, probeMaster, processExecve, processExit,
readStatusFile, runMaster, supervisorOptions, trimLogEvery, writeStatusFile, type MasterStatusFile,
} from './master.js'
-import { DEFAULT_SUPERVISOR_OPTIONS } from './supervisor.js'
+import { PROBE_ANSWER, RESUME_ENV, decodeResume, encodeResume, fingerprint, readMarker, writeMarker } from './reexec.js'
+import { DEFAULT_SUPERVISOR_OPTIONS, type ResumeState } from './supervisor.js'
describe('supervisorOptions', () => {
it('reads valid overrides and keeps the defaults for anything unset or invalid', () => {
@@ -60,7 +61,7 @@ describe('the status file', () => {
afterEach(() => rmSync(dir, { recursive: true, force: true }))
const status = (over: Partial<MasterStatusFile> = {}): MasterStatusFile => ({
state: 'running', corePid: 7, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null,
- protocol: 2, since: 1, masterPid: 42, ...over,
+ protocol: 2, since: 1, masterVersion: null, reexecs: 0, masterPid: 42, ...over,
})
it('is written whole and read back only for the master that wrote it', () => {
@@ -137,10 +138,46 @@ describe('trimLogEvery', () => {
describe('masterDefaults', () => {
it('takes what the config gives, and this process for the rest', () => {
const base = { nodePath: 'n', execArgv: [], scriptPath: 's', pidFile: 'p', restoreUpdate: () => {}, confirmUpdate: () => {} }
- expect(masterDefaults(base)).toEqual({ env: process.env, exit: processExit, onSignal: onProcessSignal })
+ expect(masterDefaults(base)).toEqual({ env: process.env, exit: processExit, onSignal: onProcessSignal, execve: expect.any(Function) })
const exit = () => {}
const onSignal = () => {}
- expect(masterDefaults({ ...base, env: { A: '1' }, exit, onSignal })).toEqual({ env: { A: '1' }, exit, onSignal })
+ const execve = () => {}
+ expect(masterDefaults({ ...base, env: { A: '1' }, exit, onSignal, execve })).toEqual({ env: { A: '1' }, exit, onSignal, execve })
+ expect(masterDefaults({ ...base, execve: null }).execve).toBeNull()
+ })
+
+ it('replaces the process through process.execve where this Node has it, and has no way to where it has not', () => {
+ expect(processExecve({})).toBeNull()
+ const calls: unknown[][] = []
+ const proc = { execve(this: unknown, ...args: unknown[]) { calls.push([this === proc, ...args]) } }
+ processExecve(proc)!('/node', ['/node', 'cli.js'], { A: '1' })
+ expect(calls).toEqual([[true, '/node', ['/node', 'cli.js'], { A: '1' }]])
+ // The Node these tests run on has it (22.15 and 23.11 on).
+ expect(processExecve()).toEqual(expect.any(Function))
+ })
+})
+
+describe('the probe a re-executing master asks a bundle', () => {
+ it('answers when the master builds, from the state it would be handed or from none', () => {
+ const said: string[] = []
+ expect(probeMaster({ env: {}, execArgv: [], version: '9.9.9' }, (line) => said.push(line))).toBe(0)
+ const resume: ResumeState = { restarts: 1, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 1, unproven: 1 }
+ expect(probeMaster({ env: { [RESUME_ENV]: encodeResume(resume) }, execArgv: [], version: '9.9.9' }, (line) => said.push(line))).toBe(0)
+ expect(said).toEqual([`${PROBE_ANSWER} · protocol 2 · v9.9.9`, `${PROBE_ANSWER} · protocol 2 · v9.9.9`])
+ })
+
+ it('refuses a state it cannot read, and says so on stdout by default', () => {
+ const log = vi.spyOn(console, 'log').mockImplementation(() => {})
+ try {
+ expect(probeMaster({ env: { [RESUME_ENV]: '{"restarts":"many"}' }, execArgv: [], version: '9.9.9' })).toBe(1)
+ expect(log).toHaveBeenCalledWith('harnessd-probe failed: the state it would be handed is not one this master can read')
+ } finally { log.mockRestore() }
+ // A failure that is not an Error is reported all the same: here, the answer itself could not be said.
+ const said: string[] = []
+ let first = true
+ const say = (line: string) => { if (first) { first = false; throw 'stdout closed' } said.push(line) }
+ expect(probeMaster({ env: { HARNESSD_SERVICES: 'search' }, execArgv: [], version: '' }, say)).toBe(1)
+ expect(said).toEqual(['harnessd-probe failed: stdout closed'])
})
})
@@ -170,7 +207,7 @@ describe('coreHandle', () => {
const messages: unknown[] = []
handle.onMessage((message) => messages.push(message))
child.emit('message', { type: 'x' })
- handle.send({ type: 'harnessd:status', status: { state: 'running', corePid: 1, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null, protocol: 2, since: 0 } })
+ handle.send({ type: 'harnessd:status', status: { state: 'running', corePid: 1, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null, protocol: 2, since: 0, masterVersion: null, reexecs: 0 } })
handle.kill('SIGTERM')
expect(handle.pid).toBe(4242)
expect(messages).toEqual([{ type: 'x' }])
@@ -386,4 +423,126 @@ describe('runMaster', () => {
expect(await exited).toBe(0)
expect(output).toContain('[harnessd] SIGTERM — stopping')
})
+
+ describe('re-executing on the bundle on disk', () => {
+ const DEAD_PID = 2_000_000_000
+ /** One script plays the bundle: as a core it binds, is ready and beats; as a probe it answers, or not. */
+ const bundle = (core: string, probe = `console.log(${JSON.stringify(PROBE_ANSWER)} + ' · test'); process.exit(0)`) => {
+ const file = join(dir, 'bundle.cjs')
+ writeFileSync(file, `
+ const fs = require('node:fs')
+ if (process.argv[2] === '__harnessd-probe') { ${probe} }
+ else {
+ process.send({ type: 'harnessd:bound', protocol: 2, port: 1 })
+ process.send({ type: 'harnessd:ready' })
+ setInterval(() => process.send({ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1, loopDelayMs: 0 }), 50)
+ process.on('SIGTERM', () => process.exit(0))
+ ${core}
+ }
+ `)
+ return file
+ }
+ const start = (scriptPath: string, over: Record<string, unknown> = {}) => {
+ const execs: Array<{ file: string; args: string[]; env: NodeJS.ProcessEnv }> = []
+ const exits: number[] = []
+ const calls: string[] = []
+ const signals = new Map<string, () => void>()
+ const supervisor = runMaster({
+ nodePath: process.execPath, execArgv: [], scriptPath, pidFile: join(dir, 'adapter.pid'), statusFile: join(dir, 'harnessd-status.json'),
+ restoreUpdate: () => calls.push('restore'), confirmUpdate: () => calls.push('confirm'),
+ env: { ...process.env, HARNESSD_INITIAL_BACKOFF_MS: '10', HARNESSD_UPDATE_PROBATION_MS: '100' },
+ exit: (code) => exits.push(code), onSignal: (signal, listener) => signals.set(signal, listener),
+ version: '1.0.0', reexecMarkerFile: join(dir, 'harnessd-reexec.json'),
+ execve: (file: string, args: string[], env: NodeJS.ProcessEnv) => { execs.push({ file, args, env }) },
+ ...over,
+ })
+ const stop = async () => { signals.get('SIGTERM')!(); await until('the master to finish', () => exits.length > 0) }
+ return { supervisor, execs, exits, calls, stop }
+ }
+ const marker = () => join(dir, 'harnessd-reexec.json')
+
+ it('replaces itself with the bundle its core staged before it exited for the update, leaving its marker and handing on its state', async () => {
+ // A newer bundle written over this one, then the update exit: what the core's updater does.
+ const script = bundle(`if (process.env.HARNESSD_RESTARTS === '0') setTimeout(() => { fs.appendFileSync(__filename, '\\n// a newer build\\n'); process.exit(75) }, 100)`)
+ const own = fingerprint(script)
+ const master = start(script)
+ await until('the master to re-execute', () => master.execs.length > 0)
+ expect(master.execs[0].file).toBe(process.execPath)
+ expect(master.execs[0].args).toEqual([process.execPath, script, '__harnessd'])
+ expect(decodeResume(master.execs[0].env[RESUME_ENV])).toEqual({ restarts: 1, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 1, unproven: 1 })
+ expect(readMarker(marker())).toEqual({ pid: process.pid, from: own, to: fingerprint(script), at: expect.any(Number) })
+ expect(master.supervisor.status()).toMatchObject({ state: 'restarting', corePid: null, masterVersion: '1.0.0' })
+ await master.stop()
+ expect(master.exits).toEqual([0])
+ })
+
+ it('rolls an update back when the new bundle\'s master does not answer its probe, and keeps its own code', async () => {
+ const script = bundle(`if (process.env.HARNESSD_RESTARTS === '0') setTimeout(() => { fs.appendFileSync(__filename, '\\n// a newer build\\n'); process.exit(75) }, 100)`,
+ `console.error('Unknown command: ' + process.argv[2]); process.exit(1)`)
+ const master = start(script)
+ await until('the rollback', () => master.calls.includes('restore'))
+ await until('a core on the bundle restored', () => master.supervisor.status().state === 'running')
+ expect(master.execs).toEqual([])
+ expect(readMarker(marker())).toBeNull()
+ await master.stop()
+ })
+
+ it('carries on as itself, without its marker, when the exec fails', async () => {
+ const script = bundle(`if (process.env.HARNESSD_RESTARTS === '0') setTimeout(() => { fs.appendFileSync(__filename, '\\n// a newer build\\n'); process.exit(75) }, 100)`)
+ const master = start(script, { execve: () => { throw new Error('E2BIG') } })
+ await until('a core on the new bundle, under this master', () => master.supervisor.status().state === 'running' && master.supervisor.status().restarts === 1)
+ expect(readMarker(marker())).toBeNull()
+ await master.stop()
+ })
+
+ it('abandons a probe still running when it is stopped, and replaces nothing', async () => {
+ const script = bundle(`if (process.env.HARNESSD_RESTARTS === '0') setTimeout(() => { fs.appendFileSync(__filename, '\\n// a newer build\\n'); process.exit(75) }, 100)`,
+ `fs.writeFileSync(${JSON.stringify(join(dir, 'probing'))}, String(process.pid)); setInterval(() => {}, 1000)`)
+ const master = start(script)
+ await until('the probe to be running', () => existsSync(join(dir, 'probing')))
+ const probe = Number(readFileSync(join(dir, 'probing'), 'utf8'))
+ await master.stop()
+ await until('the probe to be gone', () => { try { process.kill(probe, 0); return false } catch { return true } })
+ expect(master.execs).toEqual([])
+ })
+
+ it('carries on as the master it re-executed from, without handing that state to its children', async () => {
+ const seen = join(dir, 'seen')
+ const script = bundle(`fs.appendFileSync(${JSON.stringify(seen)}, JSON.stringify({ restarts: process.env.HARNESSD_RESTARTS, resume: process.env[${JSON.stringify(RESUME_ENV)}] ?? null }) + '\\n')`)
+ const resume: ResumeState = { restarts: 3, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 1, unproven: 1 }
+ const lines: string[] = []
+ const log = vi.spyOn(console, 'log').mockImplementation((line: string) => { lines.push(line) })
+ try {
+ const master = start(script, { env: { ...process.env, HARNESSD_UPDATE_PROBATION_MS: '100', [RESUME_ENV]: encodeResume(resume) } })
+ await until('the update to be kept', () => master.calls.includes('confirm'))
+ expect(JSON.parse(readFileSync(seen, 'utf8').trim().split('\n')[0])).toEqual({ restarts: '3', resume: null })
+ expect(readStatusFile(join(dir, 'harnessd-status.json'), process.pid)).toMatchObject({ masterVersion: '1.0.0', reexecs: 1, restarts: 3 })
+ expect(lines.some((line) => line.endsWith(`[harnessd] master re-executed (pid ${process.pid}) · now v1.0.0`))).toBe(true)
+ await master.stop()
+ // Without a version it says so without one.
+ const plain = start(script, { version: undefined, env: { ...process.env, [RESUME_ENV]: encodeResume(resume) } })
+ await until('its core', () => plain.supervisor.status().state === 'running')
+ expect(lines.some((line) => line.endsWith(`[harnessd] master re-executed (pid ${process.pid})`))).toBe(true)
+ await plain.stop()
+ } finally { log.mockRestore() }
+ })
+
+ it('rolls back a re-execution that never brought a core up, then moves onto the bundle restored', async () => {
+ const script = bundle('')
+ writeMarker(marker(), { pid: DEAD_PID, from: 'the one before', to: fingerprint(script)!, at: 1 })
+ const master = start(script, { restoreUpdate: () => { writeFileSync(script, readFileSync(script, 'utf8') + '\n// the build before\n') } })
+ await until('the master to re-execute on the bundle restored', () => master.execs.length > 0)
+ expect(readMarker(marker())).toMatchObject({ pid: process.pid, to: fingerprint(script) })
+ await master.stop()
+ })
+
+ it('removes its marker once its core is up, and leaves alone what it does not judge', async () => {
+ const script = bundle('')
+ writeMarker(marker(), { pid: process.pid, from: 'the one before', to: fingerprint(script)!, at: 1 })
+ const master = start(script)
+ await until('its marker to go', () => !existsSync(marker()))
+ expect(master.calls).toEqual([])
+ await master.stop()
+ })
+ })
})
diff --git a/cli/src/harnessd/master.ts b/cli/src/harnessd/master.ts
index f737eb2bc..769959763 100644
--- a/cli/src/harnessd/master.ts
+++ b/cli/src/harnessd/master.ts
@@ -1,7 +1,7 @@
/**
* `harness __harnessd`: the master process `harness start` launches (see ./supervisor.ts for what it
* does). This file is only its wiring to the operating system: the core child, the pid file, the status
- * file, the log's size, signals.
+ * file, the log's size, signals, and re-executing on a new bundle (./reexec.ts).
*/
import { spawn, type ChildProcess } from 'node:child_process'
import { randomBytes } from 'node:crypto'
@@ -10,8 +10,13 @@ import { performance } from 'node:perf_hooks'
import { trimLogFile } from '../lib/log.js'
import { platformFromEnv, type PlatformName } from './platform.js'
import type { MasterMessage } from './protocol.js'
+import {
+ PROBE_ANSWER, RESUME_ENV, createReexec, decodeResume, fingerprint, readMarker, recoverFailedReexec, removeMarker, runProbe, writeMarker,
+} from './reexec.js'
import { KNOWN_SERVICES, ServiceSupervisor, serviceOptions, serviceSpecs } from './services.js'
-import { DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorOptions, type SupervisorStatus } from './supervisor.js'
+import {
+ DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorDeps, type SupervisorOptions, type SupervisorStatus,
+} from './supervisor.js'
/** How often the master keeps the daemon's log under its cap. */
export const LOG_TRIM_INTERVAL_MS = 60_000
@@ -34,15 +39,32 @@ export interface MasterConfig {
exit?: (code: number) => void
/** Where SIGTERM, SIGINT and SIGHUP are listened for; defaults to this process. */
onSignal?: (signal: NodeJS.Signals, listener: () => void) => void
+ /** This bundle's version, for the status (`harness status`, `/api/status`). */
+ version?: string
+ /** Where a master about to re-execute on the bundle on disk leaves its marker (./reexec.ts). Left
+ * out, the master never re-executes, as before. */
+ reexecMarkerFile?: string
+ /** Replace this process (`process.execve`); defaults to this Node's, null where it has none. */
+ execve?: Execve | null
}
+export type Execve = (file: string, args: string[], env: NodeJS.ProcessEnv) => void
+
/** The defaults `runMaster` acts on this process with. */
export const processExit = (code: number): void => { process.exit(code) }
export const onProcessSignal = (signal: NodeJS.Signals, listener: () => void): void => { process.on(signal, listener) }
+/** `process.execve` (Node 22.15 and 23.11 on, and not yet in the typings this builds with); null without it. */
+export function processExecve(proc: object = process): Execve | null {
+ const execve = (proc as { execve?: unknown }).execve
+ return typeof execve === 'function' ? (file, args, env) => { execve.call(proc, file, args, env) } : null
+}
/** What a config leaves out, taken from this process. Pure: choosing a default does not act on one. */
-export function masterDefaults(config: MasterConfig): Required<Pick<MasterConfig, 'env' | 'exit' | 'onSignal'>> {
- return { env: config.env ?? process.env, exit: config.exit ?? processExit, onSignal: config.onSignal ?? onProcessSignal }
+export function masterDefaults(config: MasterConfig): Required<Pick<MasterConfig, 'env' | 'exit' | 'onSignal'>> & { execve: Execve | null } {
+ return {
+ env: config.env ?? process.env, exit: config.exit ?? processExit, onSignal: config.onSignal ?? onProcessSignal,
+ execve: config.execve === undefined ? processExecve() : config.execve,
+ }
}
/** The heap limit `--max-old-space-size` sets in these flags, MiB; null when none does. */
@@ -165,9 +187,37 @@ export function coreHandle(child: ChildProcess): CoreHandle {
}
}
+/**
+ * `harness __harnessd-probe`: what a master about to re-execute on this bundle asks it first
+ * (./reexec.ts). Everything a master does before it starts its children, with the state it would be
+ * handed, and nothing after: the whole bundle loads, the state reads, a master is built. Exit 0 and
+ * the answer, or 1 and why.
+ */
+export function probeMaster(config: { env: NodeJS.ProcessEnv; execArgv: string[]; version: string }, say: (line: string) => void = console.log): number {
+ try {
+ const handed = config.env[RESUME_ENV]
+ const resume = decodeResume(handed)
+ if (handed !== undefined && !resume) throw new Error('the state it would be handed is not one this master can read')
+ serviceSpecs(config.env, KNOWN_SERVICES)
+ const inert = { wallClock: () => Date.now() } as unknown as SupervisorDeps
+ const status = new Supervisor(inert, supervisorOptions(config.env, config.execArgv), { version: config.version, resume }).status()
+ say(`${PROBE_ANSWER} · protocol ${status.protocol} · v${status.masterVersion}`)
+ return 0
+ } catch (error) {
+ say(`harnessd-probe failed: ${error instanceof Error ? error.message : String(error)}`)
+ return 1
+ }
+}
+
export function runMaster(config: MasterConfig): Supervisor {
- const { env, exit, onSignal } = masterDefaults(config)
+ const { env: given, exit, onSignal, execve } = masterDefaults(config)
process.title = 'harnessd'
+ // What the master this process was a moment ago handed on, if it was one (./reexec.ts). Taken out of
+ // the environment every child is given: a core or service has no use for it, and a later re-exec
+ // writes its own.
+ const resume = decodeResume(given[RESUME_ENV])
+ const env: NodeJS.ProcessEnv = { ...given }
+ delete env[RESUME_ENV]
const readPid = (): number | null => {
try { return Number.parseInt(readFileSync(config.pidFile, 'utf8').trim(), 10) || null } catch { return null }
}
@@ -180,6 +230,22 @@ export function runMaster(config: MasterConfig): Supervisor {
// Set by the launchd agent or systemd unit `harness service install` writes (./platform.ts).
const platform = platformFromEnv(env)
if (platform) log(`[harnessd] run by ${platform}, which starts this master again if it dies`)
+ if (resume) log(`[harnessd] master re-executed (pid ${process.pid})${config.version ? ` · now v${config.version}` : ''}`)
+ // The bundle this master's code came from, read once, as it starts; the bundle on disk is read
+ // again before every core starts.
+ const own = fingerprint(config.scriptPath)
+ const bundle = () => fingerprint(config.scriptPath)
+ const markerFile = config.reexecMarkerFile
+ const alive = (pid: number): boolean => { try { process.kill(pid, 0); return true } catch { return false } }
+ if (markerFile) {
+ // A re-execution before this start that never brought a core up: back to the previous bundle,
+ // which the supervisor then re-executes this master on before it starts anything.
+ recoverFailedReexec({
+ marker: readMarker(markerFile), pid: process.pid, alive, current: bundle,
+ restoreUpdate: config.restoreUpdate, removeMarker: () => removeMarker(markerFile), log,
+ })
+ }
+ let stopping = false
// A new one every boot, given to the core and to each service: how the core knows a service
// connection is one this master started, and no other local process.
const token = randomBytes(24).toString('hex')
@@ -194,6 +260,18 @@ export function runMaster(config: MasterConfig): Supervisor {
clearTimer: (timer) => clearTimeout(timer as ReturnType<typeof setTimeout>),
log,
}, serviceOptions(env), { HARNESSD_SERVICE_TOKEN: token })
+ const reexec = markerFile ? createReexec({
+ own, current: bundle, nodePath: config.nodePath, execArgv: config.execArgv, scriptPath: config.scriptPath, env, pid: process.pid,
+ execve,
+ probe: (args, probeEnv) => runProbe(config.nodePath, args, probeEnv),
+ stopChildren: (done) => services.stop(done),
+ startChildren: () => services.start(),
+ writeMarker: (marker) => writeMarker(markerFile, marker),
+ removeMarker: () => removeMarker(markerFile),
+ stopping: () => stopping,
+ now: () => Date.now(),
+ log,
+ }) : null
const supervisor = new Supervisor({
spawnCore: (extra) => coreHandle(spawn(config.nodePath, [...execArgv, config.scriptPath, '__run'], {
env: { ...env, ...extra, HARNESSD_SERVICE_TOKEN: token },
@@ -210,13 +288,29 @@ export function runMaster(config: MasterConfig): Supervisor {
confirmUpdate: config.confirmUpdate,
log,
// The services go with the master, after the core: none is left holding the core's socket.
- exit: (code) => services.stop(() => {
- stopTrimming()
- exit(code)
- }),
- }, options)
+ exit: (code) => {
+ stopping = true
+ reexec?.cancel()
+ services.stop(() => {
+ stopTrimming()
+ exit(code)
+ })
+ },
+ ...(reexec && markerFile ? {
+ reexec: reexec.reexec,
+ // The re-executed master has brought a core up: its marker has served.
+ coreUp: () => removeMarker(markerFile),
+ } : {}),
+ }, options, { version: config.version, resume })
// Services stop beside the core, inside the same grace `harness stop` gives the whole daemon.
- for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP'] as const) onSignal(signal, () => { services.stop(() => {}); supervisor.stop(signal) })
+ for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP'] as const) {
+ onSignal(signal, () => {
+ stopping = true
+ reexec?.cancel()
+ services.stop(() => {})
+ supervisor.stop(signal)
+ })
+ }
supervisor.start()
// Not waiting on the core: a service reaches it through its socket, and retries until it answers.
services.start()
diff --git a/cli/src/harnessd/reexec.spec.ts b/cli/src/harnessd/reexec.spec.ts
new file mode 100644
index 000000000..aaffa5e21
--- /dev/null
+++ b/cli/src/harnessd/reexec.spec.ts
@@ -0,0 +1,284 @@
+import { EventEmitter } from 'node:events'
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import type { ChildProcess } from 'node:child_process'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import {
+ PROBE_ANSWER, PROBE_COMMAND, PROBE_TIMEOUT_MS, REEXEC_LIMIT, RESUME_ENV, createReexec, decodeResume, encodeResume, fingerprint,
+ readMarker, recoverFailedReexec, removeMarker, runProbe, writeMarker, type ProbeResult, type ReexecDeps,
+} from './reexec.js'
+import type { ReexecOutcome, ResumeState } from './supervisor.js'
+
+const state = (over: Partial<ResumeState> = {}): ResumeState => ({
+ restarts: 2, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 0, unproven: 0, ...over,
+})
+
+describe('which bundle', () => {
+ let dir: string
+ beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'harnessd-reexec-')) })
+ afterEach(() => rmSync(dir, { recursive: true, force: true }))
+
+ it('is told by the sha256 of its bytes, or null when it cannot be read', () => {
+ const file = join(dir, 'cli.js')
+ writeFileSync(file, 'one')
+ const one = fingerprint(file)
+ expect(one).toMatch(/^[0-9a-f]{64}$/)
+ writeFileSync(file, 'two')
+ expect(fingerprint(file)).not.toBe(one)
+ expect(fingerprint(join(dir, 'none.js'))).toBeNull()
+ expect(fingerprint('x', () => Buffer.from('one'))).toBe(one)
+ })
+})
+
+describe('the state a master hands on', () => {
+ it('reads back what was written', () => {
+ expect(decodeResume(encodeResume(state()))).toEqual(state())
+ expect(decodeResume(encodeResume(state({ lastExit: null, lastExitReason: null, update: null, claimed: false })))).toEqual(state({ lastExit: null, lastExitReason: null, update: null, claimed: false }))
+ })
+
+ it('is none when absent, and refused when it is not one this code can read', () => {
+ expect(decodeResume(undefined)).toBeNull()
+ expect(decodeResume('{not json')).toBeNull()
+ for (const wrong of [
+ { restarts: -1 }, { restarts: 1.5 }, { lastExit: 3 }, { lastExitReason: 'bored' }, { update: 'probation' }, { claimed: 'yes' },
+ { reexecs: -1 }, { reexecs: '1' }, { unproven: 0.5 }, { unproven: null },
+ ]) expect(decodeResume(JSON.stringify({ ...state(), ...wrong })), JSON.stringify(wrong)).toBeNull()
+ })
+})
+
+describe('the marker', () => {
+ let dir: string
+ beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'harnessd-marker-')) })
+ afterEach(() => rmSync(dir, { recursive: true, force: true }))
+
+ it('is written whole, read back, and removed', () => {
+ const file = join(dir, 'harnessd-reexec.json')
+ expect(readMarker(file)).toBeNull()
+ writeMarker(file, { pid: 42, from: 'a', to: 'b', at: 1 })
+ expect(readMarker(file)).toEqual({ pid: 42, from: 'a', to: 'b', at: 1 })
+ writeMarker(file, { pid: 42, from: null, to: 'b', at: 1 })
+ expect(readMarker(file)).toEqual({ pid: 42, from: null, to: 'b', at: 1 })
+ removeMarker(file)
+ removeMarker(file)
+ expect(readMarker(file)).toBeNull()
+ })
+
+ it('is no marker when it is not one', () => {
+ const file = join(dir, 'harnessd-reexec.json')
+ for (const wrong of [{ pid: 'x', to: 'b', at: 1, from: null }, { pid: 1, to: 2, at: 1, from: null }, { pid: 1, to: 'b', at: 'now', from: null }, { pid: 1, to: 'b', at: 1, from: 3 }]) {
+ writeFileSync(file, JSON.stringify(wrong))
+ expect(readMarker(file), JSON.stringify(wrong)).toBeNull()
+ }
+ })
+
+ it('is removed even when something odd sits where it was, without throwing', () => {
+ const odd = join(dir, 'harnessd-reexec.json')
+ mkdirSync(odd)
+ expect(() => removeMarker(odd)).not.toThrow()
+ })
+})
+
+describe('a re-execution that never came up', () => {
+ const recover = (over: Partial<Parameters<typeof recoverFailedReexec>[0]> = {}) => {
+ const calls: string[] = []
+ const result = recoverFailedReexec({
+ marker: { pid: 7, from: 'old', to: 'new', at: 1 }, pid: 99, alive: () => false, current: () => 'new',
+ restoreUpdate: () => calls.push('restore'), removeMarker: () => calls.push('remove'), log: (line) => calls.push(line), ...over,
+ })
+ return { result, calls }
+ }
+
+ it('is rolled back when the process that re-executed is gone and its bundle is still on disk', () => {
+ expect(recover()).toEqual({ result: true, calls: ['remove', 'restore', '[harnessd] the master that re-executed on the new bundle (pid 7) never brought a core up — rolled back to the previous bundle'] })
+ })
+
+ it('is let be when there is none, when it is this process\'s own, when its process lives, or when the bundle moved on', () => {
+ expect(recover({ marker: null })).toEqual({ result: false, calls: [] })
+ expect(recover({ pid: 7 })).toEqual({ result: false, calls: [] })
+ expect(recover({ alive: () => true })).toEqual({ result: false, calls: [] })
+ expect(recover({ current: () => 'newer' })).toEqual({ result: false, calls: ['remove'] })
+ })
+})
+
+describe('the probe', () => {
+ let dir: string
+ beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'harnessd-probe-')) })
+ afterEach(() => rmSync(dir, { recursive: true, force: true }))
+ const script = (body: string) => {
+ const file = join(dir, 'bundle.cjs')
+ writeFileSync(file, body)
+ return file
+ }
+
+ it('passes a bundle whose master answers, and says why one that does not fails', async () => {
+ const ok = await runProbe(process.execPath, [script(`console.log('loading'); console.log(${JSON.stringify(PROBE_ANSWER)} + ' · protocol 2')`), PROBE_COMMAND], process.env).result
+ expect(ok).toEqual({ ok: true, detail: `${PROBE_ANSWER} · protocol 2` })
+ // A bundle from before probes: its usage on stdout, the reason on stderr, which is what is said.
+ expect(await runProbe(process.execPath, [script(`console.error('Unknown command: ' + process.argv[2]); console.log('Usage: harness …\\n harness --help'); process.exit(1)`), PROBE_COMMAND], process.env).result)
+ .toEqual({ ok: false, detail: `Unknown command: ${PROBE_COMMAND}` })
+ // An answer on stderr is no answer.
+ expect(await runProbe(process.execPath, [script(`console.error(${JSON.stringify(PROBE_ANSWER)})`), PROBE_COMMAND], process.env).result)
+ .toEqual({ ok: false, detail: PROBE_ANSWER })
+ expect(await runProbe(process.execPath, [script('console.log("something else")'), PROBE_COMMAND], process.env).result)
+ .toEqual({ ok: false, detail: 'something else' })
+ expect(await runProbe(process.execPath, [script('process.exit(4)')], process.env).result).toEqual({ ok: false, detail: 'exit 4' })
+ expect(await runProbe(process.execPath, [script('process.kill(process.pid, "SIGKILL")')], process.env).result).toEqual({ ok: false, detail: 'signal SIGKILL' })
+ expect(await runProbe(join(dir, 'no-node'), [], process.env).result).toEqual({ ok: false, detail: expect.stringContaining('ENOENT') })
+ })
+
+ it('gives up on one that does not answer in time, and can be abandoned', async () => {
+ const hung = script('setInterval(() => {}, 1000)')
+ expect(await runProbe(process.execPath, [hung], process.env, 300).result).toEqual({ ok: false, detail: 'no answer within 300 ms' })
+ const running = runProbe(process.execPath, [hung], process.env)
+ running.cancel()
+ expect(await running.result).toEqual({ ok: false, detail: 'signal SIGKILL' })
+ expect(PROBE_TIMEOUT_MS).toBeGreaterThanOrEqual(10_000)
+ })
+
+ it('survives a child that has no output to read and cannot be killed', async () => {
+ const child = Object.assign(new EventEmitter(), { stdout: null, stderr: null, kill: () => { throw new Error('ESRCH') } }) as unknown as ChildProcess
+ vi.useFakeTimers()
+ try {
+ const running = runProbe('node', [], {}, 100, () => child)
+ running.cancel()
+ vi.advanceTimersByTime(100)
+ expect(await running.result).toEqual({ ok: false, detail: 'no answer within 100 ms' })
+ child.emit('close', 0, null)
+ } finally { vi.useRealTimers() }
+ })
+})
+
+describe('re-executing', () => {
+ let calls: string[]
+ let probes: Array<{ args: string[]; env: NodeJS.ProcessEnv; settle: (result: ProbeResult) => void; cancelled: boolean }>
+ let outcomes: ReexecOutcome[]
+ let stopping: boolean
+ let onDisk: string | null
+ let execs: Array<{ file: string; args: string[]; env: NodeJS.ProcessEnv }>
+ let stopped: Array<() => void>
+
+ const make = (over: Partial<ReexecDeps> = {}) => createReexec({
+ own: 'old', current: () => onDisk, nodePath: '/opt/node', execArgv: ['--flag'], scriptPath: '/opt/cli.js',
+ env: { KEEP: '1' }, pid: 4242,
+ execve: (file, args, env) => { execs.push({ file, args, env }) },
+ probe: (args, env) => {
+ let settle!: (result: ProbeResult) => void
+ const entry = { args, env, settle: (result: ProbeResult) => settle(result), cancelled: false }
+ probes.push(entry)
+ return { result: new Promise<ProbeResult>((resolve) => { settle = resolve }), cancel: () => { entry.cancelled = true } }
+ },
+ stopChildren: (done) => { calls.push('stop children'); stopped.push(done) },
+ startChildren: () => calls.push('start children'),
+ writeMarker: (marker) => calls.push(`marker ${JSON.stringify(marker)}`),
+ removeMarker: () => calls.push('remove marker'),
+ stopping: () => stopping,
+ now: () => 1_000,
+ log: (line) => calls.push(line),
+ ...over,
+ })
+ const settle = async (result: ProbeResult) => { probes.at(-1)!.settle(result); await new Promise((done) => setImmediate(done)) }
+
+ beforeEach(() => {
+ calls = []
+ probes = []
+ outcomes = []
+ stopping = false
+ onDisk = 'new'
+ execs = []
+ stopped = []
+ })
+
+ it('does nothing when the bundle on disk is this master\'s own, or cannot be read', () => {
+ const { reexec } = make()
+ onDisk = 'old'
+ reexec(state(), (outcome) => outcomes.push(outcome))
+ onDisk = null
+ reexec(state(), (outcome) => outcomes.push(outcome))
+ expect(outcomes).toEqual(['same', 'same'])
+ expect(calls).toEqual([])
+ })
+
+ it('keeps this master where this Node cannot re-execute, and says so once', () => {
+ const { reexec } = make({ execve: null })
+ reexec(state(), (outcome) => outcomes.push(outcome))
+ reexec(state(), (outcome) => outcomes.push(outcome))
+ expect(outcomes).toEqual(['kept', 'kept'])
+ expect(calls).toEqual(['[harnessd] the bundle on disk is newer than this master, but this Node cannot re-execute a process: the master keeps its code until it restarts'])
+ })
+
+ it('keeps this master after re-executing again and again without a core coming up', () => {
+ const { reexec } = make()
+ reexec(state({ unproven: REEXEC_LIMIT }), (outcome) => outcomes.push(outcome))
+ expect(outcomes).toEqual(['kept'])
+ expect(calls).toEqual([`[harnessd] this master re-executed ${REEXEC_LIMIT} times without a core coming up — keeping its code; something keeps rewriting the bundle`])
+ })
+
+ it('stops its children, probes the bundle, leaves its marker and replaces itself, handing its state on', async () => {
+ const { reexec } = make()
+ reexec(state({ reexecs: 2, unproven: 1 }), (outcome) => outcomes.push(outcome))
+ expect(calls).toEqual(['[harnessd] the bundle on disk is not this master\'s code — re-executing on it', 'stop children'])
+ expect(probes).toEqual([])
+ stopped[0]()
+ expect(probes[0].args).toEqual(['--flag', '/opt/cli.js', PROBE_COMMAND])
+ expect(decodeResume(probes[0].env[RESUME_ENV])).toEqual(state({ reexecs: 3, unproven: 2 }))
+ expect(probes[0].env.KEEP).toBe('1')
+ await settle({ ok: true, detail: PROBE_ANSWER })
+ expect(calls.at(-1)).toBe(`marker ${JSON.stringify({ pid: 4242, from: 'old', to: 'new', at: 1_000 })}`)
+ expect(execs).toEqual([{ file: '/opt/node', args: ['/opt/node', '--flag', '/opt/cli.js', '__harnessd'], env: probes[0].env }])
+ expect(outcomes).toEqual([])
+ })
+
+ it('carries on as itself when the probe fails or the exec does, with its children back', async () => {
+ const refused = make()
+ refused.reexec(state(), (outcome) => outcomes.push(outcome))
+ stopped[0]()
+ await settle({ ok: false, detail: 'Unknown command: __harnessd-probe' })
+ expect(outcomes).toEqual(['refused'])
+ expect(calls.slice(-2)).toEqual(['[harnessd] the new bundle\'s master did not answer its probe (Unknown command: __harnessd-probe) — keeping this master', 'start children'])
+ expect(execs).toEqual([])
+
+ calls = []
+ const failing = make({ execve: () => { throw new Error('E2BIG') } })
+ failing.reexec(state(), (outcome) => outcomes.push(outcome))
+ stopped[1]()
+ await settle({ ok: true, detail: PROBE_ANSWER })
+ expect(outcomes).toEqual(['refused', 'kept'])
+ expect(calls.slice(-3)).toEqual(['remove marker', '[harnessd] could not re-execute this master (E2BIG) — keeping it', 'start children'])
+
+ const odd = make({ execve: () => { throw 'odd' } })
+ odd.reexec(state(), (outcome) => outcomes.push(outcome))
+ stopped[2]()
+ await settle({ ok: true, detail: PROBE_ANSWER })
+ expect(calls).toContain('[harnessd] could not re-execute this master (odd) — keeping it')
+ })
+
+ it('does nothing more once the master is stopping, at any step, and abandons its probe', async () => {
+ const first = make()
+ first.reexec(state(), (outcome) => outcomes.push(outcome))
+ stopping = true
+ stopped[0]()
+ expect(probes).toEqual([])
+
+ stopping = false
+ const second = make()
+ second.reexec(state(), (outcome) => outcomes.push(outcome))
+ stopped[1]()
+ stopping = true
+ second.cancel()
+ expect(probes[0].cancelled).toBe(true)
+ await settle({ ok: true, detail: PROBE_ANSWER })
+ expect(execs).toEqual([])
+
+ stopping = false
+ const third = make()
+ third.reexec(state(), (outcome) => outcomes.push(outcome))
+ stopped[2]()
+ stopping = true
+ await settle({ ok: false, detail: 'no' })
+ expect(outcomes).toEqual([])
+ expect(calls).not.toContain('start children')
+ third.cancel()
+ make().cancel()
+ })
+})
diff --git a/cli/src/harnessd/reexec.ts b/cli/src/harnessd/reexec.ts
new file mode 100644
index 000000000..70d34ce71
--- /dev/null
+++ b/cli/src/harnessd/reexec.ts
@@ -0,0 +1,251 @@
+/**
+ * The master replacing itself with the bundle on disk (docs/design/2026-10-03-harnessd.md, "Updates").
+ *
+ * An update replaced the core but not the master: the core staged the new bundle and exited 75, the
+ * master started a core on it, and went on running its own old code until something restarted it, and
+ * so did every master fix. Now, in the gap before a core starts, a master whose code is not the bundle
+ * on disk re-executes itself on it (`process.execve`).
+ *
+ * In place, keeping its pid, rather than handing over to a fresh master: launchd and systemd follow the
+ * pid they started (a master that exits is, to them, a job that ended, and its successor runs
+ * unsupervised), and so do the pid file, `harness stop` and the desktop app's owner check. Only in the
+ * gap: with no core running and the services stopped and reaped first, nothing loses its channel to
+ * the master and nothing is orphaned, and the agents live in tmux, out of reach of all of it.
+ *
+ * What exec gives up is the old master as a fallback, so two things stand in for it:
+ * - a probe: the bundle's own master is asked first (`__harnessd-probe`, `probeMaster` in ./master.ts)
+ * to load and to build itself from the state it would be handed. A bundle from before this answers
+ * "Unknown command", so it is never started as a probe, which could have started a second daemon;
+ * one that fails the probe fails the update, which is rolled back and remembered;
+ * - a marker (`harnessd-reexec.json`): written before the exec, removed once the new master's first
+ * core is up. A master that starts and finds one left by a process that is gone knows that re-exec
+ * never came up, and puts the previous bundle back (`recoverFailedReexec`). Under launchd or systemd,
+ * which restart a master that died, that happens by itself; otherwise at the next `harness start`.
+ *
+ * Everything that touches the operating system is injected (`ReexecDeps`).
+ */
+import { spawn, type ChildProcess } from 'node:child_process'
+import { createHash } from 'node:crypto'
+import { readFileSync, renameSync, rmSync, writeFileSync } from 'node:fs'
+import type { ExitReason, ReexecOutcome, ResumeState } from './supervisor.js'
+
+/** The environment variable that hands a master's state to the one it re-executes as. */
+export const RESUME_ENV = 'HARNESSD_RESUME'
+/** The command a bundle's master answers its probe on, and what it answers. */
+export const PROBE_COMMAND = '__harnessd-probe'
+export const PROBE_ANSWER = 'harnessd-probe ok'
+/** How long a probe may take: the updater gives its canary as long. */
+export const PROBE_TIMEOUT_MS = 15_000
+/**
+ * This many re-executions in a row without a core coming up, and the master keeps its code: each new
+ * master found yet another bundle before it could start one, and following them would be a loop. An
+ * update that is rolled back takes two (onto the bundle, and back), each brought up a core or tried.
+ */
+export const REEXEC_LIMIT = 3
+
+/** A file's sha256: which bundle a master runs, or which is on disk; null when it cannot be read. */
+export function fingerprint(path: string, read: (path: string) => Buffer = readFileSync): string | null {
+ try { return createHash('sha256').update(read(path)).digest('hex') } catch { return null }
+}
+
+const EXIT_REASONS: readonly ExitReason[] = ['crashed', 'hung', 'did-not-bind', 'not-ready', 'memory', 'update', 'stopped']
+
+export function encodeResume(state: ResumeState): string {
+ return JSON.stringify(state)
+}
+
+/** The state a master was handed, or null when there is none or it is not one this code can read. */
+export function decodeResume(text: string | undefined): ResumeState | null {
+ if (text === undefined) return null
+ try {
+ const value = JSON.parse(text) as Partial<ResumeState>
+ const ok = Number.isInteger(value.restarts) && value.restarts! >= 0
+ && (value.lastExit === null || typeof value.lastExit === 'string')
+ && (value.lastExitReason === null || EXIT_REASONS.includes(value.lastExitReason as ExitReason))
+ && (value.update === null || value.update === 'pending')
+ && typeof value.claimed === 'boolean'
+ && Number.isInteger(value.reexecs) && value.reexecs! >= 0
+ && Number.isInteger(value.unproven) && value.unproven! >= 0
+ return ok ? value as ResumeState : null
+ } catch {
+ return null
+ }
+}
+
+/** What a master writes before it re-executes: who, from which bundle to which, and when. */
+export interface ReexecMarker {
+ pid: number
+ from: string | null
+ to: string
+ at: number
+}
+
+export function writeMarker(file: string, marker: ReexecMarker): void {
+ const temp = `${file}.${marker.pid}.tmp`
+ writeFileSync(temp, `${JSON.stringify(marker)}\n`)
+ renameSync(temp, file)
+}
+
+export function readMarker(file: string): ReexecMarker | null {
+ try {
+ const marker = JSON.parse(readFileSync(file, 'utf8')) as Partial<ReexecMarker>
+ return Number.isInteger(marker.pid) && typeof marker.to === 'string' && typeof marker.at === 'number'
+ && (marker.from === null || typeof marker.from === 'string')
+ ? marker as ReexecMarker
+ : null
+ } catch {
+ return null
+ }
+}
+
+export function removeMarker(file: string): void {
+ try { rmSync(file, { force: true }) } catch { /* nothing more to do */ }
+}
+
+/**
+ * At a master's start: did a re-execution before it fail? A marker left by a process that is gone,
+ * with the bundle it moved to still on disk, means the master that process became never brought a core
+ * up. The previous bundle is put back (`restoreUpdate` also remembers the version, so the updater does
+ * not stage it again) and true says so: this master runs that bundle's code and must move off it before
+ * it starts anything. A marker that is this process's own is a re-execution in progress, and one whose
+ * process lives on is not this master's to judge.
+ */
+export function recoverFailedReexec(deps: {
+ marker: ReexecMarker | null
+ pid: number
+ alive(pid: number): boolean
+ /** The bundle on disk now. */
+ current(): string | null
+ restoreUpdate(): void
+ removeMarker(): void
+ log(line: string): void
+}): boolean {
+ const { marker } = deps
+ if (!marker || marker.pid === deps.pid || deps.alive(marker.pid)) return false
+ deps.removeMarker()
+ if (deps.current() !== marker.to) return false
+ deps.restoreUpdate()
+ deps.log(`[harnessd] the master that re-executed on the new bundle (pid ${marker.pid}) never brought a core up — rolled back to the previous bundle`)
+ return true
+}
+
+export interface ProbeResult { ok: boolean; detail: string }
+
+/** Run a bundle's master probe: `node <flags> <bundle> __harnessd-probe`, which must answer in time. */
+export function runProbe(
+ nodePath: string, args: string[], env: NodeJS.ProcessEnv, timeoutMs = PROBE_TIMEOUT_MS,
+ start: (file: string, args: string[], env: NodeJS.ProcessEnv) => ChildProcess = (file, argv, environment) => spawn(file, argv, { env: environment, stdio: ['ignore', 'pipe', 'pipe'] }),
+): { result: Promise<ProbeResult>; cancel(): void } {
+ const child = start(nodePath, args, env)
+ // Apart: a bundle from before probes answers with its usage on stdout and the reason on stderr.
+ let out = ''
+ let err = ''
+ child.stdout?.on('data', (chunk: Buffer) => { out += chunk.toString('utf8') })
+ child.stderr?.on('data', (chunk: Buffer) => { err += chunk.toString('utf8') })
+ const lastLine = (text: string): string => { const lines = text.trim().split('\n'); return lines[lines.length - 1].trim() }
+ const result = new Promise<ProbeResult>((resolve) => {
+ let settled = false
+ const done = (ok: boolean, detail: string) => {
+ if (settled) return
+ settled = true
+ clearTimeout(timer)
+ resolve({ ok, detail })
+ }
+ const timer = setTimeout(() => {
+ try { child.kill('SIGKILL') } catch { /* gone */ }
+ done(false, `no answer within ${timeoutMs} ms`)
+ }, timeoutMs)
+ child.on('error', (error) => done(false, error.message))
+ child.on('close', (code, signal) => {
+ done(code === 0 && out.includes(PROBE_ANSWER), lastLine(err) || lastLine(out) || (signal ? `signal ${signal}` : `exit ${code}`))
+ })
+ })
+ return { result, cancel: () => { try { child.kill('SIGKILL') } catch { /* gone */ } } }
+}
+
+export interface ReexecDeps {
+ /** The bundle this master was started from: its code. */
+ own: string | null
+ /** The bundle on disk now. */
+ current(): string | null
+ /** What a re-executed master runs: this node, these flags, this bundle. */
+ nodePath: string
+ execArgv: string[]
+ scriptPath: string
+ /** The master's own environment; the resume state is added to it. */
+ env: NodeJS.ProcessEnv
+ pid: number
+ /** `process.execve`, where this Node has it (22.15 and 23.11 on). */
+ execve: ((file: string, args: string[], env: NodeJS.ProcessEnv) => void) | null
+ probe(args: string[], env: NodeJS.ProcessEnv): { result: Promise<ProbeResult>; cancel(): void }
+ /** Stop the services and wait for them to be gone, so none is left a zombie or an orphan. */
+ stopChildren(done: () => void): void
+ /** Start them again, when this master carries on as itself. */
+ startChildren(): void
+ writeMarker(marker: ReexecMarker): void
+ removeMarker(): void
+ /** The master is stopping: nothing is replaced or restarted then. */
+ stopping(): boolean
+ /** The wall clock, ms. */
+ now(): number
+ log(line: string): void
+}
+
+/** The supervisor's `reexec`, and a way to abandon a probe still running when the master stops. */
+export function createReexec(deps: ReexecDeps): {
+ reexec(state: ResumeState, proceed: (outcome: ReexecOutcome) => void): void
+ cancel(): void
+} {
+ let warned = false
+ let probe: { cancel(): void } | null = null
+ return {
+ reexec: (state, proceed) => {
+ const next = deps.current()
+ if (next === null || next === deps.own) { proceed('same'); return }
+ if (!deps.execve) {
+ if (!warned) deps.log('[harnessd] the bundle on disk is newer than this master, but this Node cannot re-execute a process: the master keeps its code until it restarts')
+ warned = true
+ proceed('kept')
+ return
+ }
+ if (state.unproven >= REEXEC_LIMIT) {
+ deps.log(`[harnessd] this master re-executed ${state.unproven} times without a core coming up — keeping its code; something keeps rewriting the bundle`)
+ proceed('kept')
+ return
+ }
+ const now = deps.now()
+ const env = { ...deps.env, [RESUME_ENV]: encodeResume({ ...state, reexecs: state.reexecs + 1, unproven: state.unproven + 1 }) }
+ const args = [...deps.execArgv, deps.scriptPath]
+ // Called only once the master is known not to be stopping: its children come back, and its core.
+ const carryOn = (outcome: ReexecOutcome) => {
+ deps.startChildren()
+ proceed(outcome)
+ }
+ deps.log('[harnessd] the bundle on disk is not this master\'s code — re-executing on it')
+ deps.stopChildren(() => {
+ if (deps.stopping()) return
+ const running = deps.probe([...args, PROBE_COMMAND], env)
+ probe = running
+ void running.result.then(({ ok, detail }) => {
+ probe = null
+ if (deps.stopping()) return
+ if (!ok) {
+ deps.log(`[harnessd] the new bundle's master did not answer its probe (${detail}) — keeping this master`)
+ carryOn('refused')
+ return
+ }
+ deps.writeMarker({ pid: deps.pid, from: deps.own, to: next, at: now })
+ try {
+ // Never returns: from here this process is the new bundle's master, with the same pid.
+ deps.execve!(deps.nodePath, [deps.nodePath, ...args, '__harnessd'], env)
+ } catch (error) {
+ deps.removeMarker()
+ deps.log(`[harnessd] could not re-execute this master (${error instanceof Error ? error.message : String(error)}) — keeping it`)
+ carryOn('kept')
+ }
+ })
+ })
+ },
+ cancel: () => { probe?.cancel(); probe = null },
+ }
+}
diff --git a/cli/src/harnessd/supervisor.spec.ts b/cli/src/harnessd/supervisor.spec.ts
index a34e36ca5..13de403af 100644
--- a/cli/src/harnessd/supervisor.spec.ts
+++ b/cli/src/harnessd/supervisor.spec.ts
@@ -1,6 +1,9 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { CORE_EXIT_STOP, CORE_EXIT_UPDATE, HARNESSD_PROTOCOL, type MasterMessage } from './protocol.js'
-import { DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorOptions, type SupervisorStatus } from './supervisor.js'
+import {
+ DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type ReexecOutcome, type ResumeState, type SupervisorDeps, type SupervisorOptions,
+ type SupervisorStatus,
+} from './supervisor.js'
const MIB = 1024 * 1024
@@ -49,7 +52,7 @@ describe('Supervisor', () => {
let pidOf: (index: number) => number | undefined
const core = () => cores[cores.length - 1]
- const make = (overrides: Partial<SupervisorOptions> = {}) => new Supervisor({
+ const make = (overrides: Partial<SupervisorOptions> = {}, more: Partial<SupervisorDeps> = {}, identity: ConstructorParameters<typeof Supervisor>[2] = {}) => new Supervisor({
spawnCore: (env) => {
const next = new FakeCore(pidOf(cores.length), env)
cores.push(next)
@@ -66,7 +69,8 @@ describe('Supervisor', () => {
writeStatus: (status) => statuses.push(status),
log: (line) => lines.push(line),
exit: (code) => exited.push(code),
- }, { ...options, ...overrides })
+ ...more,
+ }, { ...options, ...overrides }, identity)
/** Let time pass for a running core, beating as a healthy one does. */
const live = (ms: number) => { for (let left = ms; left > 0; left -= 1_000) { vi.advanceTimersByTime(Math.min(1_000, left)); core().beat() } }
@@ -87,7 +91,7 @@ describe('Supervisor', () => {
it('starts one core, claims the pid file once it is bound, and runs it once it is ready', () => {
const supervisor = make()
expect(new Supervisor({ wallClock: () => 7 } as never).status())
- .toEqual({ state: 'idle', corePid: null, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null, protocol: HARNESSD_PROTOCOL, since: 7 })
+ .toEqual({ state: 'idle', corePid: null, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null, protocol: HARNESSD_PROTOCOL, since: 7, masterVersion: null, reexecs: 0 })
supervisor.start()
supervisor.start()
expect(cores).toHaveLength(1)
@@ -102,7 +106,7 @@ describe('Supervisor', () => {
core().ready()
expect(supervisor.status()).toEqual({
state: 'running', corePid: 1000, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null,
- protocol: HARNESSD_PROTOCOL, since: Date.now(),
+ protocol: HARNESSD_PROTOCOL, since: Date.now(), masterVersion: null, reexecs: 0,
})
expect(lines.at(-1)).toBe('[harnessd] core ready (pid 1000)')
// Every change is written for `harness status`, and told to the bound core.
@@ -490,4 +494,118 @@ describe('Supervisor', () => {
expect(calls).toEqual(['claim', 'release'])
})
})
+
+ describe('re-executing on the bundle on disk', () => {
+ /** A `reexec` that records what it was asked, and answers when the test says. */
+ const reexecs = () => {
+ const asked: Array<{ state: ResumeState; proceed: (outcome: ReexecOutcome) => void }> = []
+ return { asked, deps: { reexec: (state: ResumeState, proceed: (outcome: ReexecOutcome) => void) => { asked.push({ state, proceed }) }, coreUp: () => calls.push('up') } }
+ }
+
+ it('asks before every core it starts, and starts it when the master stays as it is', () => {
+ const { asked, deps } = reexecs()
+ const supervisor = make({}, deps)
+ supervisor.start()
+ expect(cores).toHaveLength(0)
+ expect(asked[0].state).toEqual({ restarts: 0, lastExit: null, lastExitReason: null, update: null, claimed: false, reexecs: 0, unproven: 0 })
+ asked[0].proceed('same')
+ expect(cores).toHaveLength(1)
+ core().up()
+ expect(calls).toEqual(['claim', 'up'])
+ crash()
+ expect(asked).toHaveLength(2)
+ asked[1].proceed('kept')
+ expect(cores).toHaveLength(2)
+ expect(supervisor.status().state).toBe('restarting')
+ })
+
+ it('hands an update on probation, and the pid file, to the master it becomes; and starts no core itself', () => {
+ const { asked, deps } = reexecs()
+ const supervisor = make({}, deps)
+ supervisor.start()
+ asked[0].proceed('same')
+ core().up()
+ core().exit(CORE_EXIT_UPDATE)
+ vi.advanceTimersByTime(0)
+ expect(asked[1].state).toEqual({ restarts: 1, lastExit: `code ${CORE_EXIT_UPDATE}`, lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 0, unproven: 0 })
+ // The process is replaced from here: nothing more of this master runs.
+ expect(cores).toHaveLength(1)
+ expect(supervisor.status()).toMatchObject({ state: 'restarting', corePid: null })
+ })
+
+ it('rolls an update back when the new bundle\'s master would not start, and starts the core on the bundle restored', () => {
+ const { asked, deps } = reexecs()
+ const supervisor = make({}, deps)
+ supervisor.start()
+ asked[0].proceed('same')
+ core().up()
+ core().exit(CORE_EXIT_UPDATE)
+ vi.advanceTimersByTime(0)
+ asked[1].proceed('refused')
+ expect(calls).toEqual(['claim', 'up', 'restore'])
+ expect(lines.at(-1)).toBe('[harnessd] the new bundle\'s master would not start — rolled back to the previous bundle; restarting')
+ expect(asked[2].state.update).toBeNull()
+ asked[2].proceed('same')
+ core().up()
+ vi.runOnlyPendingTimers()
+ expect(calls).toEqual(['claim', 'up', 'restore', 'up'])
+ // A bundle replaced by hand has no update to fail: its core starts under this master all the same.
+ crash()
+ asked[3].proceed('refused')
+ expect(cores).toHaveLength(3)
+ })
+
+ it('rolls a failed update back and asks again, so the master moves back to the bundle restored', () => {
+ const { asked, deps } = reexecs()
+ const supervisor = make({}, deps)
+ supervisor.start()
+ asked[0].proceed('same')
+ core().up()
+ core().exit(CORE_EXIT_UPDATE)
+ vi.advanceTimersByTime(0)
+ asked[1].proceed('same')
+ core().exit(3)
+ expect(calls).toEqual(['claim', 'up', 'restore'])
+ vi.advanceTimersByTime(0)
+ expect(asked[2].state).toMatchObject({ update: null, restarts: 2, lastExit: 'code 3', lastExitReason: 'crashed' })
+ expect(supervisor.status().restarts).toBe(2)
+ })
+
+ it('starts nothing when it was stopped while deciding', () => {
+ const { asked, deps } = reexecs()
+ const supervisor = make({}, deps)
+ supervisor.start()
+ supervisor.stop('SIGTERM')
+ expect(exited).toEqual([0])
+ asked[0].proceed('same')
+ asked[0].proceed('refused')
+ expect(cores).toHaveLength(0)
+ })
+
+ it('carries on as the master it was, once re-executed: its counts, its claim, its update on probation', () => {
+ const { asked, deps } = reexecs()
+ const resume: ResumeState = { restarts: 4, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 2, unproven: 1 }
+ const supervisor = make({}, deps, { version: '9.9.9', resume })
+ expect(supervisor.status()).toMatchObject({ restarts: 4, lastExit: 'code 75', lastExitReason: 'update', masterVersion: '9.9.9', reexecs: 2 })
+ supervisor.start()
+ asked[0].proceed('same')
+ expect(core().env.HARNESSD_RESTARTS).toBe('4')
+ core().up()
+ // Its pid file was claimed by the master it was: the same process.
+ expect(calls).toEqual(['up'])
+ vi.advanceTimersByTime(options.updateProbationMs)
+ expect(calls).toEqual(['up', 'confirm'])
+ crash()
+ // A core came up: the re-executions so far are proven, and no longer count towards a loop.
+ expect(asked[1].state).toMatchObject({ reexecs: 2, unproven: 0 })
+ })
+
+ it('counts a protocol 1 core as up once it is bound', () => {
+ const { asked, deps } = reexecs()
+ make({}, deps).start()
+ asked[0].proceed('same')
+ core().bind(1)
+ expect(calls).toEqual(['claim', 'up'])
+ })
+ })
})
diff --git a/cli/src/harnessd/supervisor.ts b/cli/src/harnessd/supervisor.ts
index 4817d63c3..966eca7c2 100644
--- a/cli/src/harnessd/supervisor.ts
+++ b/cli/src/harnessd/supervisor.ts
@@ -49,6 +49,46 @@ export interface SupervisorDeps {
writeStatus(status: SupervisorStatus): void
log(line: string): void
exit(code: number): void
+ /**
+ * Replace this master with the bundle on disk before a core is started on it, carrying `state`
+ * (./reexec.ts): asked in the gap before every core start, when no core is running, so the swap
+ * orphans nothing. `proceed` is called when it did not replace itself, and why. Left out, never.
+ */
+ reexec?(state: ResumeState, proceed: (outcome: ReexecOutcome) => void): void
+ /** A core is up: ready, or bound for a protocol 1 core. A re-executed master has proved itself. */
+ coreUp?(): void
+}
+
+/**
+ * What a master hands the one that replaces it in its own process (./reexec.ts). The crash times and
+ * the backoff are left behind: they are on the old process's monotonic clock, which a new one restarts.
+ */
+export interface ResumeState {
+ restarts: number
+ lastExit: string | null
+ lastExitReason: ExitReason | null
+ /** `pending`: the core about to start runs an update, on probation once it is ready. */
+ update: 'pending' | null
+ /** The pid file is this process's already. */
+ claimed: boolean
+ /** How many times this process has re-executed. */
+ reexecs: number
+ /** Of those, how many since a core last came up: a master that keeps re-executing without one is
+ * in a loop, and stops (./reexec.ts `REEXEC_LIMIT`). */
+ unproven: number
+}
+
+/**
+ * Why a master did not replace itself: the bundle on disk is its own code (`same`); it could not or
+ * would not, for a reason that is not the bundle's (`kept`); or the bundle's master did not answer its
+ * probe (`refused`), which fails an update to it.
+ */
+export type ReexecOutcome = 'same' | 'kept' | 'refused'
+
+/** Who this master is: its code's version, and what the master it replaced handed on. */
+export interface SupervisorIdentity {
+ version?: string
+ resume?: ResumeState | null
}
export interface SupervisorOptions {
@@ -112,6 +152,10 @@ export interface SupervisorStatus {
protocol: number
/** When the state last changed, wall clock ms. */
since: number
+ /** The version of the code this master runs, when it knows it. */
+ masterVersion: string | null
+ /** How many times this master's process has replaced itself with a newer or restored bundle. */
+ reexecs: number
}
type TimerName = 'bindTimer' | 'readyTimer' | 'heartbeatTimer' | 'killTimer' | 'restartTimer' | 'probationTimer'
@@ -151,10 +195,30 @@ export class Supervisor {
private heartbeatTimer: unknown = null
private killTimer: unknown = null
private restartTimer: unknown = null
+ private readonly masterVersion: string | null
+ private reexecs = 0
+ private unproven = 0
- constructor(private readonly deps: SupervisorDeps, private readonly options: SupervisorOptions = DEFAULT_SUPERVISOR_OPTIONS) {
+ constructor(
+ private readonly deps: SupervisorDeps,
+ private readonly options: SupervisorOptions = DEFAULT_SUPERVISOR_OPTIONS,
+ identity: SupervisorIdentity = {},
+ ) {
this.backoff = options.initialBackoffMs
this.since = deps.wallClock()
+ this.masterVersion = identity.version ?? null
+ const resume = identity.resume
+ if (resume) {
+ // The master this process was a moment ago, carried on: its core exited for an update (or was
+ // rolled back), and the one about to start is judged as it would have judged it.
+ this.restarts = resume.restarts
+ this.lastExit = resume.lastExit
+ this.lastExitReason = resume.lastExitReason
+ this.update = resume.update
+ this.claimed = resume.claimed
+ this.reexecs = resume.reexecs
+ this.unproven = resume.unproven
+ }
}
status(): SupervisorStatus {
@@ -167,12 +231,45 @@ export class Supervisor {
safeMode: this.coreSafeMode ?? this.safeMode,
protocol: HARNESSD_PROTOCOL,
since: this.since,
+ masterVersion: this.masterVersion,
+ reexecs: this.reexecs,
}
}
start(): void {
if (this.state !== 'idle') return
- this.spawn()
+ this.next()
+ }
+
+ /** What this master hands on if it replaces itself now. */
+ private resumeState(): ResumeState {
+ return {
+ restarts: this.restarts, lastExit: this.lastExit, lastExitReason: this.lastExitReason,
+ update: this.update === 'pending' ? 'pending' : null, claimed: this.claimed, reexecs: this.reexecs, unproven: this.unproven,
+ }
+ }
+
+ /**
+ * Start the next core, in the one moment no core is running: first, if the bundle on disk is not
+ * this master's code (an update the core staged, a rollback that put the previous one back, a bundle
+ * replaced by hand), the master replaces itself with it and the new master starts the core. A master
+ * left running old code supervised every core after it with that code until something restarted it.
+ */
+ private next(): void {
+ if (!this.deps.reexec) { this.spawn(); return }
+ this.deps.reexec(this.resumeState(), (outcome) => {
+ // Stopped while it was deciding: the stop has finished this master already.
+ if (this.state === 'stopping' || this.state === 'stopped') return
+ if (outcome === 'refused' && this.update === 'pending') {
+ // A bundle whose master cannot start is an update that failed, as one whose core cannot is.
+ this.update = null
+ this.deps.restoreUpdate()
+ this.deps.log('[harnessd] the new bundle\'s master would not start — rolled back to the previous bundle; restarting')
+ this.next()
+ return
+ }
+ this.spawn()
+ })
}
/** Stop the core and the master. A second call while stopping kills the core outright. */
@@ -241,7 +338,7 @@ export class Supervisor {
this.armTimer('readyTimer', () => this.kill(core, 'not-ready', `bound but not ready within ${this.options.readyTimeoutMs} ms`), this.options.readyTimeoutMs)
}
if (this.state !== 'stopping') this.setState(readiness ? 'listening' : 'running')
- if (!readiness) this.beginProbation()
+ if (!readiness) this.up()
return
}
case 'harnessd:ready':
@@ -256,7 +353,7 @@ export class Supervisor {
this.coreSafeMode = message.safeMode ?? null
this.deps.log(`[harnessd] core ready (pid ${core.pid ?? '?'})${message.safeMode === undefined ? '' : ` · in safe mode: ${message.safeMode}`}`)
this.setState('running')
- this.beginProbation()
+ this.up()
return
case 'harnessd:heartbeat': {
if (!this.bound) return
@@ -273,6 +370,13 @@ export class Supervisor {
}
}
+ /** A core is up: its update goes on probation, and a re-executed master has proved itself. */
+ private up(): void {
+ this.beginProbation()
+ this.unproven = 0
+ this.deps.coreUp?.()
+ }
+
private restartForMemory(why: string): void {
this.deps.log(`[harnessd] core ${why} — restarting it`)
this.killReason = 'memory'
@@ -358,7 +462,7 @@ export class Supervisor {
private scheduleSpawn(delay: number): void {
this.setState('restarting')
- this.armTimer('restartTimer', () => { this.restartTimer = null; this.spawn() }, delay)
+ this.armTimer('restartTimer', () => { this.restartTimer = null; this.next() }, delay)
}
/** Kill a core that broke a promise — to bind, to be ready, to beat — and remember which. */
diff --git a/docs/design/2026-10-03-harnessd.md b/docs/design/2026-10-03-harnessd.md
index a5d05f148..ae207e4b4 100644
--- a/docs/design/2026-10-03-harnessd.md
+++ b/docs/design/2026-10-03-harnessd.md
@@ -325,11 +325,13 @@ the parts still to build say so.
with a deadline the master enforces; `harness stop` waits for the master instead of killing it
after a fixed three seconds. A fast stop leaves agents running in tmux; that is the only kind.
- **Updates.** The core's updater stages a bundle and exits 75; the master restarts the core on it,
- on probation from `ready`, and rolls back if it fails. Next: when the master's own code changed,
- it re-executes itself in the gap between the old core's exit and the new core's start, as
- Syncthing's monitor does — re-executing with a core running would cost that core its channel.
- Bundles move to versioned directories behind one pointer, so a swap is one rename, and a version
- that was rolled back is not offered again.
+ on probation from `ready`, and rolls back if it fails. When the master's own code changed, it
+ re-executes itself in the gap between the old core's exit and the new core's start, as Syncthing's
+ monitor does — re-executing with a core running would cost that core its channel. Built
+ (`harnessd/reexec.ts`): in place, keeping its pid, so launchd, systemd, the pid file and the app's
+ owner check never see the master change; a probe of the bundle's own master first; a marker that
+ rolls back a master that died after re-executing. Next: bundles move to versioned directories
+ behind one pointer, so a swap is one rename. A version that was rolled back is not offered again.
- **Something supervises the supervisor.** On macOS a launchd user agent (registered through
`SMAppService`, which asks the user once) and on Linux a systemd user unit run the master in the
foreground and restart it; the desktop app then watches instead of respawning, so there is one
@@ -562,6 +564,31 @@ Done so far:
`HARNESSD_TEST_PLATFORM`) see start and stop go through them; another data folder's definition is
ignored.
+- **The master re-executes itself on the bundle on disk** (`cli/src/harnessd/reexec.ts`, at 100%).
+ Before, an update replaced the core and left the master on its old code until something restarted
+ it. Now, in the gap before any core starts (an update's exit 75, a rollback, a bundle replaced by
+ hand), a master whose code is not the bundle on disk replaces itself with it (`process.execve`).
+ - In place, not by handing over to a fresh master: launchd and systemd follow the pid they started,
+ and to them a master that exits is a job that ended; the pid file, `harness stop` and the desktop
+ app's owner check follow it too. Only in the gap, with the services stopped and reaped first: no
+ core loses its channel and nothing is orphaned. The agents live in tmux, out of reach of it all.
+ - The state handed on (`HARNESSD_RESUME`, kept from every child): restarts, the last exit, the update
+ on probation, the pid file's claim, and re-executions since a core last came up (after three, the
+ master keeps its code). The status and `/api/status` gain the master's version and that count.
+ - Exec gives up the old master as a fallback, so: a probe (`harness __harnessd-probe`) asks the
+ bundle's own master first, and one that fails it fails the update, rolled back and remembered in
+ `update-rejected.json` like a failing core; a bundle from before this answers "Unknown command"
+ and is never started as a probe. And a marker (`harnessd-reexec.json`), removed once the new
+ master's first core is up: a master that starts and finds one left by a process that is gone rolls
+ back and moves onto the bundle before. Without `process.execve` (before Node 22.15 and 23.11) the
+ master keeps its code, as before.
+ - `e2e/reexec.e2e.ts`, on real bundles: the pid stays through every swap while the master's version
+ follows the bundle; the core and an agent at work carry on; a build whose master fails its probe is
+ rolled back and not staged again; one whose master dies after re-executing is rolled back at the
+ next start; every core is the master's child throughout. With a released bundle
+ (`MIGRATION_FROM`): an older master supervises the newer core an update brings until it restarts,
+ and this master runs an older bundle's core without re-executing on it.
+
Next, in order:
1. **Removals:** requests kept only to refuse retired features, once no shipped app sends them.
@@ -574,8 +601,8 @@ Next, in order:
to in-process for one release: devices first, then relay (with the self-hosting settings),
models, workspaces, search, viewers, teams, the light host, Companion and Memory.
5. **Platform supervision by default** (built as an opt-in above; the desktop app watching instead of
- respawning, registered through `SMAppService` on macOS) and the master re-executing itself on
- update.
+ respawning, registered through `SMAppService` on macOS). The other half of this step, the master
+ re-executing itself on update, is built (above).
6. **State directory move,** last, once nothing else is changing.
### The core boundary, step by step