The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
harnessd: the master re-executes itself on update, keeping its pid (#791)
Dee28cc10266cde
diff --git a/cli/e2e/reexec.e2e.ts b/cli/e2e/reexec.e2e.tsnew file mode 100644index 000000000..13edb5bf7--- /dev/null+++ b/cli/e2e/reexec.e2e.ts@@ -0,0 +1,345 @@+/**+ * The master re-executing itself on an update, on real release bundles (harnessd/reexec.ts): the core+ * stages a newer build and exits 75, and the master, before it starts a core on it, replaces itself+ * with that build in its own process. The pid stays, the core comes back on the new build under the+ * new master, and the agent at work in tmux carries on.+ *+ * A build whose master does not answer its probe is rolled back and remembered, like one whose core+ * fails. One whose master dies after it re-executed leaves its marker, and the next start (the+ * desktop app's `harness start`, or launchd's restart) rolls it back and moves onto the build before.+ * Nothing is left orphaned at any step: every core is the master's child.+ *+ * The bundles are built from this checkout at made-up versions and served from a local manifest, as in+ * e2e/update.e2e.ts, so nothing here reaches beyond the machine.+ */+import { execFileSync } from 'node:child_process'+import { createHash } from 'node:crypto'+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'+import { createServer, type Server } from 'node:http'+import { tmpdir } from 'node:os'+import { dirname, join } from 'node:path'+import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'+import { LocalClient, type Frame } from './harness/client.js'+import { CLI_ROOT, IsolatedDaemon, until } from './harness/daemon.js'++const FIRST = '42.0.1'+const sha = (bytes: Buffer): string => createHash('sha256').update(bytes).digest('hex')+const count = (text: string, part: string): number => text.split(part).length - 1+const sleep = (ms: number) => new Promise((done) => setTimeout(done, ms))++describe('the master re-executing itself on an update', () => {+ let scratch = ''+ let server: Server | undefined+ let daemon: IsolatedDaemon | undefined+ let master: number | null = null+ let agent: Record<string, any> = {}+ const releases = new Map<string, Buffer>()+ let notify = Buffer.alloc(0)+ let offered = FIRST+ const cliDir = () => join(scratch, 'cli')+ const installed = () => readFileSync(join(cliDir(), 'cli.js'))+ const rejected = () => JSON.parse(readFileSync(join(cliDir(), 'update-rejected.json'), 'utf8')) as string[]+ const marker = () => join(daemon!.dataDir, 'harnessd-reexec.json')+ const status = async (): Promise<Record<string, any> | null> =>+ fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json()).catch(() => null)+ /** The build the core runs, and the one the master runs, as the daemon reports them. */+ const builds = async (): Promise<{ core: string | null; master: string | null; masterPid: number | null; reexecs: number | null }> => {+ const now = await status()+ return { core: now?.version ?? null, master: now?.harnessd?.masterVersion ?? null, masterPid: now?.harnessd?.masterPid ?? null, reexecs: now?.harnessd?.reexecs ?? null }+ }+ const on = (version: string) => until(`the core and the master to run ${version}`, async () => {+ const now = await builds()+ return now.core === version && now.master === version ? now : null+ }, 90_000, 250)++ const rows = async (client: LocalClient) =>+ (await client.request<{ agents: Array<Record<string, any>> }>('agents_list', { includeStopped: true }, 30_000)).agents+ /** The agent is back, the same conversation, and takes a turn: the core under the new master serves it. */+ const agentCarriesOn = async () => {+ const client = await LocalClient.connect(daemon!)+ try {+ await until('the agent to be back', async () => {+ const row = (await rows(client)).find((candidate) => candidate.id === agent.id)+ return row?.status === 'active' && row.sessionId === agent.sessionId ? row : null+ }, 60_000, 500)+ const ended = client.next((frame: Frame) => frame.type === 'turn_ended' && frame.agentId === agent.id, 45_000, 'a turn')+ client.send('message', { agentId: agent.id, content: 'still there?' })+ await ended+ } finally { client.close() }+ }++ /**+ * Every process running a build of this machine, from the process table. Each core must be the+ * master's child: one left behind by a master that replaced itself would be an orphan holding the+ * port, re-parented to launchd or init.+ */+ const daemonProcesses = (): Array<{ pid: number; ppid: number; command: string }> =>+ execFileSync('ps', ['-axo', 'pid=,ppid=,command='], { encoding: 'utf8' }).split('\n')+ .map((line) => /^\s*(\d+)\s+(\d+)\s+(.*)$/.exec(line))+ .filter((match): match is RegExpExecArray => !!match && match[3].includes(join(cliDir(), 'cli.js')))+ .map((match) => ({ pid: Number(match[1]), ppid: Number(match[2]), command: match[3] }))+ const nothingOrphaned = async (masterPid: number) => {+ // A probe is over within the update; give one a moment to end.+ await until('only the master\'s own children', () => daemonProcesses().every((process) => process.ppid === masterPid && process.command.endsWith(' __run')), 10_000, 200)+ expect(daemonProcesses().filter((process) => process.command.endsWith(' __run'))).toHaveLength(1)+ }++ beforeAll(async () => {+ scratch = mkdtempSync(join(tmpdir(), 'harnessd-reexec-'))+ const out = join(scratch, 'build')+ execFileSync(process.execPath, ['build-bundle.mjs'], {+ cwd: CLI_ROOT, env: { ...process.env, ADAPTER_VERSION: FIRST, BUNDLE_OUT_DIR: out }, stdio: 'pipe',+ })+ const first = readFileSync(join(out, 'cli.js'), 'utf8')+ notify = readFileSync(join(out, 'notify.mjs'))+ expect(count(first, FIRST)).toBeGreaterThanOrEqual(2)+ const release = (version: string, inject = ''): void => {+ // The version is baked in at build time; the others are the same bytes with it swapped, and a+ // fault put in at the top where a test needs one.+ const source = first.replaceAll(FIRST, version).replace('\n', `\n${inject}\n`)+ const file = join(out, `cli-${version}.js`)+ writeFileSync(file, source)+ expect(execFileSync(process.execPath, [file, 'version'], { encoding: 'utf8' }).trim()).toBe(version)+ releases.set(version, Buffer.from(source))+ }+ release(FIRST)+ release('42.0.2')+ // Passes the updater's canary (`cli.js version`), and its master does not answer its probe.+ release('42.0.3', 'if(process.argv[2]==="__harnessd-probe")process.exit(3);')+ release('42.0.4')+ // Answers its probe, and dies as soon as a master re-executes on it.+ release('42.0.5', 'if(process.argv[2]==="__harnessd"&&process.env.HARNESSD_RESUME)process.exit(4);')++ server = createServer((request, response) => {+ const url = request.url ?? ''+ const origin = `http://127.0.0.1:${(server!.address() as { port: number }).port}`+ if (url === '/metadata.json') {+ const cli = releases.get(offered)!+ response.end(JSON.stringify({ cli: {+ version: offered,+ cli: { url: `${origin}/cli-${offered}.js`, sha256: sha(cli), size: cli.length },+ notify: { url: `${origin}/notify.mjs`, sha256: sha(notify), size: notify.length },+ } }))+ return+ }+ const asked = /^\/cli-(.+)\.js$/.exec(url)?.[1]+ if (asked && releases.has(asked)) { response.end(releases.get(asked)); return }+ if (url === '/notify.mjs') { response.end(notify); return }+ response.statusCode = 404+ response.end()+ })+ await new Promise<void>((done) => server!.listen(0, '127.0.0.1', done))+ const local = `http://127.0.0.1:${(server.address() as { port: number }).port}`++ mkdirSync(cliDir())+ writeFileSync(join(cliDir(), 'cli.js'), releases.get(FIRST)!)+ writeFileSync(join(cliDir(), 'notify.mjs'), notify)+ writeFileSync(join(cliDir(), 'package.json'), '{"type":"module"}\n')+ daemon = await IsolatedDaemon.create({+ scriptPath: join(cliDir(), 'cli.js'),+ env: {+ ADAPTER_CLI_DIR: cliDir(),+ ADAPTER_UPDATE_DISABLE: 'false',+ ADAPTER_UPDATE_URL: `${local}/metadata.json`,+ ADAPTER_UPDATE_CHECK_MS: '1000',+ ADAPTER_UPDATE_SLOT_SEC: '-1',+ HARNESSD_UPDATE_PROBATION_MS: '3000',+ HARNESSD_INITIAL_BACKOFF_MS: '100',+ HARNESS_TUI_MANIFEST_URL: `${local}/tui/metadata.json`,+ ADAPTER_RUNTIME_METADATA_URL: `${local}/runtime/metadata.json`,+ ADAPTER_GRID_RUNTIME_METADATA_URL: `${local}/grid/metadata.json`,+ },+ })+ await daemon.start()+ master = daemon.pid+ expect(await builds()).toMatchObject({ core: FIRST, master: FIRST, masterPid: master, reexecs: 0 })++ // An agent at work, whose pane lives in tmux through every swap below.+ const client = await LocalClient.connect(daemon)+ const cwd = join(daemon.projectsDir, 'reexec')+ mkdirSync(cwd, { recursive: true })+ const created = await client.request('agent_create', { engine: 'claude', cwd, bypassPermission: true }, 90_000)+ agent = await until('the agent to bind its conversation', async () => {+ const row = (await rows(client)).find((candidate) => candidate.id === created.agent.id)+ return row?.sessionId && row.status === 'active' ? row : null+ }, 60_000, 500)+ client.close()+ })++ afterAll(async () => {+ await daemon?.close()+ await new Promise<void>((done) => server ? server.close(() => done()) : done())+ if (scratch) rmSync(scratch, { recursive: true, force: true })+ })++ const showLog = () => onTestFailed(() => { console.log(`---- daemon log\n${daemon?.log().split('\n').slice(-150).join('\n')}`) })++ it('replaces the master with the new build in its own process: same pid, and the core and the agent carry on', async () => {+ showLog()+ offered = '42.0.2'+ const now = await on('42.0.2')+ expect(now).toMatchObject({ masterPid: master, reexecs: 1 })+ expect(daemon!.pid).toBe(master)+ expect(IsolatedDaemon.alive(master)).toBe(true)+ expect(daemon!.log()).toContain(`[harnessd] master re-executed (pid ${master}) · now v42.0.2`)+ await until('the master to keep the update', () => daemon!.log().includes('the update stayed up — keeping it'), 30_000)+ expect(installed()).toEqual(releases.get('42.0.2'))+ expect(existsSync(marker())).toBe(false)+ await agentCarriesOn()+ await nothingOrphaned(master!)+ })++ it('rolls back a build whose master does not answer its probe, and remembers it; the master stays as it was', async () => {+ showLog()+ const from = daemon!.log().length+ offered = '42.0.3'+ await until('the rollback', () => daemon!.log().slice(from).includes('the new bundle\'s master would not start — rolled back to the previous bundle'), 90_000)+ expect(daemon!.log().slice(from)).toContain('did not answer its probe')+ expect(await on('42.0.2')).toMatchObject({ masterPid: master, reexecs: 1 })+ expect(installed()).toEqual(releases.get('42.0.2'))+ expect(rejected()).toEqual(['42.0.3'])+ expect(existsSync(join(cliDir(), 'update-pending.json'))).toBe(false)+ // A check every second: a few more of them, and the rejected build is not staged again.+ await until('the core to say it waits for a newer build', () => daemon!.log().slice(from).includes('42.0.3 was rolled back on this machine'), 30_000)+ const cores = daemon!.coresStarted()+ await sleep(4_000)+ expect(daemon!.coresStarted()).toBe(cores)+ await agentCarriesOn()+ await nothingOrphaned(master!)+ })++ it('moves on when a newer build is published', async () => {+ showLog()+ offered = '42.0.4'+ expect(await on('42.0.4')).toMatchObject({ masterPid: master, reexecs: 2 })+ await until('the master to keep the update', () => count(daemon!.log(), 'the update stayed up — keeping it') === 2, 30_000)+ expect(rejected()).toEqual(['42.0.3'])+ await agentCarriesOn()+ await nothingOrphaned(master!)+ })++ it('rolls back, at the next start, a build whose master died after re-executing on it', async () => {+ showLog()+ offered = '42.0.5'+ await until('the master that re-executed on 42.0.5 to die', () => !IsolatedDaemon.alive(master), 90_000, 200)+ expect(readFileSync(marker(), 'utf8')).toContain(`"pid":${master}`)+ expect(installed()).toEqual(releases.get('42.0.5'))+ // The core it replaced had exited for the update; nothing was left running in its name.+ expect(daemonProcesses()).toEqual([])++ // What the desktop app's `harness start`, or launchd restarting the master, would do.+ const from = daemon!.log().length+ await daemon!.start()+ const restarted = daemon!.pid!+ expect(restarted).not.toBe(master)+ expect(daemon!.log().slice(from)).toContain(`the master that re-executed on the new bundle (pid ${master}) never brought a core up — rolled back to the previous bundle`)+ expect(await on('42.0.4')).toMatchObject({ masterPid: restarted, reexecs: 1 })+ expect(installed()).toEqual(releases.get('42.0.4'))+ expect(rejected()).toEqual(['42.0.3', '42.0.5'])+ await until('the marker to be gone', () => !existsSync(marker()), 30_000)+ await agentCarriesOn()+ await nothingOrphaned(restarted)+ })+})++/**+ * A released master and this one, each over the other's core (protocol.ts only grows). Run when+ * MIGRATION_FROM names a released bundle, as e2e/migration.e2e.ts is: one from before re-execution.+ *+ * An older master cannot re-execute: after an update it supervises the newer core until it is+ * restarted, as every machine will once this ships. A newer master that finds the older bundle back on+ * disk does not re-execute on it either: the older bundle does not know the probe, and is never run as+ * a master to find out. Its core runs under the newer master.+ */+const RELEASED = process.env.MIGRATION_FROM+describe.skipIf(!RELEASED)('a released master and this one, each over the other\'s core', () => {+ const NEXT = '0.99.0'+ let scratch = ''+ let server: Server | undefined+ let daemon: IsolatedDaemon | undefined+ let offered: 'released' | 'next' = 'released'+ let releasedVersion = ''+ const bundles = new Map<'released' | 'next', { cli: Buffer; notify: Buffer; version: string }>()+ const cliDir = () => join(scratch, 'cli')+ const status = async (): Promise<Record<string, any> | null> =>+ fetch(`http://127.0.0.1:${daemon!.port}/api/status`).then((response) => response.json()).catch(() => null)++ beforeAll(async () => {+ scratch = mkdtempSync(join(tmpdir(), 'harnessd-reexec-compat-'))+ const out = join(scratch, 'build')+ execFileSync(process.execPath, ['build-bundle.mjs'], { cwd: CLI_ROOT, env: { ...process.env, ADAPTER_VERSION: NEXT, BUNDLE_OUT_DIR: out }, stdio: 'pipe' })+ bundles.set('next', { cli: readFileSync(join(out, 'cli.js')), notify: readFileSync(join(out, 'notify.mjs')), version: NEXT })+ releasedVersion = execFileSync(process.execPath, [RELEASED!, 'version'], { encoding: 'utf8' }).trim()+ bundles.set('released', { cli: readFileSync(RELEASED!), notify: readFileSync(join(dirname(RELEASED!), 'notify.mjs')), version: releasedVersion })+ server = createServer((request, response) => {+ const origin = `http://127.0.0.1:${(server!.address() as { port: number }).port}`+ const bundle = bundles.get(offered)!+ if (request.url === '/metadata.json') {+ response.end(JSON.stringify({ cli: {+ version: bundle.version,+ cli: { url: `${origin}/cli.js`, sha256: sha(bundle.cli), size: bundle.cli.length },+ notify: { url: `${origin}/notify.mjs`, sha256: sha(bundle.notify), size: bundle.notify.length },+ } }))+ return+ }+ if (request.url === '/cli.js') { response.end(bundle.cli); return }+ if (request.url === '/notify.mjs') { response.end(bundle.notify); return }+ response.statusCode = 404+ response.end()+ })+ await new Promise<void>((done) => server!.listen(0, '127.0.0.1', done))+ const local = `http://127.0.0.1:${(server.address() as { port: number }).port}`+ mkdirSync(cliDir())+ writeFileSync(join(cliDir(), 'cli.js'), bundles.get('released')!.cli)+ writeFileSync(join(cliDir(), 'notify.mjs'), bundles.get('released')!.notify)+ writeFileSync(join(cliDir(), 'package.json'), '{"type":"module"}\n')+ daemon = await IsolatedDaemon.create({+ scriptPath: join(cliDir(), 'cli.js'),+ env: {+ ADAPTER_CLI_DIR: cliDir(), ADAPTER_UPDATE_DISABLE: 'false', ADAPTER_UPDATE_URL: `${local}/metadata.json`,+ ADAPTER_UPDATE_CHECK_MS: '1000', ADAPTER_UPDATE_SLOT_SEC: '-1', HARNESSD_UPDATE_PROBATION_MS: '3000', HARNESSD_INITIAL_BACKOFF_MS: '100',+ HARNESS_TUI_MANIFEST_URL: `${local}/tui/metadata.json`, ADAPTER_RUNTIME_METADATA_URL: `${local}/runtime/metadata.json`,+ ADAPTER_GRID_RUNTIME_METADATA_URL: `${local}/grid/metadata.json`,+ },+ })+ await daemon.start()+ })++ afterAll(async () => {+ await daemon?.close()+ await new Promise<void>((done) => server ? server.close(() => done()) : done())+ if (scratch) rmSync(scratch, { recursive: true, force: true })+ })++ it('an older master supervises the newer core an update brings, until it restarts; then this master runs', async () => {+ onTestFailed(() => { console.log(`---- daemon log\n${daemon?.log().split('\n').slice(-120).join('\n')}`) })+ const master = daemon!.pid+ expect((await status())?.version).toBe(releasedVersion)+ offered = 'next'+ const now = await until('the newer core under the released master', async () => {+ const current = await status()+ return current?.version === NEXT ? current : null+ }, 90_000, 250)+ expect(now.harnessd.masterPid).toBe(master)+ expect(now.harnessd.masterVersion).toBeUndefined()+ expect(daemon!.log()).not.toContain('re-executing')+ await until('the released master to keep the update', () => daemon!.log().includes('the update stayed up — keeping it'), 30_000)+ await daemon!.restart()+ expect((await status())?.harnessd).toMatchObject({ masterVersion: NEXT, masterPid: daemon!.pid })+ })++ it('this master runs the older bundle\'s core when it finds it back on disk, without re-executing on it', async () => {+ onTestFailed(() => { console.log(`---- daemon log\n${daemon?.log().split('\n').slice(-120).join('\n')}`) })+ offered = 'released' // not newer: the older core's updater leaves it alone+ const master = daemon!.pid+ const from = daemon!.log().length+ writeFileSync(join(cliDir(), 'cli.js'), bundles.get('released')!.cli)+ process.kill((await status())!.corePid as number, 'SIGKILL')+ const now = await until('the older core under this master', async () => {+ const current = await status()+ return current?.version === releasedVersion ? current : null+ }, 60_000, 250)+ expect(now.harnessd).toMatchObject({ masterVersion: NEXT, masterPid: master, reexecs: 0 })+ expect(daemon!.log().slice(from)).toContain('did not answer its probe (Unknown command: __harnessd-probe) — keeping this master')+ })+})diff --git a/cli/src/cli.ts b/cli/src/cli.tsindex 8b9f66391..2ece36e23 100644--- a/cli/src/cli.ts+++ b/cli/src/cli.ts@@ -177,7 +177,7 @@ import { STORE_REQUESTS, startStore } from './services/store.js'import { startViewers } from './services/viewers.js'import { MODELS_REQUESTS, startModels } from './services/models.js'import { startWorkspaces } from './services/workspaces.js'-import { describeMasterStatus, readStatusFile, runMaster } from './harnessd/master.js'+import { describeMasterStatus, probeMaster, readStatusFile, runMaster } from './harnessd/master.js'import { CORE_EXIT_STOP, CORE_EXIT_UPDATE } from './harnessd/protocol.js'import { isLocalSocketName, localSocketPath, refuseServedDataFolder, type LocalSocketServer } from './lib/localSocket.js'import { legacyDaemonStatus, localDaemonStatus, saveDaemonPort } from './lib/daemonEndpoint.js'@@ -242,6 +242,8 @@ import {// Daemon stdout/stderr. Capped at LOG_MAX_BYTES — see prepareLogFile/trimLogFile in lib/log.ts.const LOG_FILE = DAEMON_LOG_FILE+/** Where a master re-executing on a new bundle leaves word of it, until that master's core is up. */+const HARNESSD_REEXEC_FILE = join(env.ADAPTER_DATA_DIR, 'harnessd-reexec.json')// Pre-rename name. Adopted (renamed, keeping the inode) the first time a daemon opens the log, so a// machine that updates mid-run keeps its history instead of stranding it in a file nobody tails.// The log has had three names; this slot holds the OLDEST. The middle one (`machine.log`) is adopted@@ -5628,8 +5630,13 @@ switch (cmd) {logFile: LOG_FILE,restoreUpdate: () => restoreUpdate(env.ADAPTER_CLI_DIR),confirmUpdate: () => confirmUpdate(env.ADAPTER_CLI_DIR),+ version: VERSION,+ reexecMarkerFile: HARNESSD_REEXEC_FILE,})break+ case '__harnessd-probe': // internal: a master about to re-execute on this bundle asks it first (harnessd/reexec.ts)+ process.exitCode = probeMaster({ env: process.env, execArgv: process.execArgv, version: VERSION })+ breakcase '__service': { // internal: a service harnessd's master runs in its own process (services/process.ts)const socketPath = localSocketPath(env.ADAPTER_DATA_DIR, env.PORT)const runService = rest[0] === 'search' ? runSearchService : rest[0] === 'viewers' ? runViewersServicediff --git a/cli/src/harnessd/AGENTS.md b/cli/src/harnessd/AGENTS.mdindex 896b8b179..41a63ec79 100644--- a/cli/src/harnessd/AGENTS.md+++ b/cli/src/harnessd/AGENTS.md@@ -20,3 +20,6 @@ It keeps the core and the services running, and nothing else.Under systemd, never stop the unit with a stop job (`systemctl stop`, `restart`, `disable --now`):tmux built with systemd support makes every pane PartOf the unit that started its server, so a stopjob ends every agent. `stop()` signals the master instead. The desktop app does not use it yet.+6. **The master replaces itself only in the gap before a core starts** (`reexec.ts`): no core running,+ services stopped and reaped first, in place so the pid stays. Never exec with a child running: it+ would lose its channel, and become an orphan or a zombie no one reaps.diff --git a/cli/src/harnessd/master.spec.ts b/cli/src/harnessd/master.spec.tsindex 49e406c11..87a37e5e9 100644--- a/cli/src/harnessd/master.spec.ts+++ b/cli/src/harnessd/master.spec.ts@@ -5,10 +5,11 @@ import { join } from 'node:path'import { spawn, type ChildProcess } from 'node:child_process'import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'import {- coreExecArgv, coreHandle, describeMasterStatus, heapLimitInArgv, masterDefaults, onProcessSignal, processExit,+ coreExecArgv, coreHandle, describeMasterStatus, heapLimitInArgv, masterDefaults, onProcessSignal, probeMaster, processExecve, processExit,readStatusFile, runMaster, supervisorOptions, trimLogEvery, writeStatusFile, type MasterStatusFile,} from './master.js'-import { DEFAULT_SUPERVISOR_OPTIONS } from './supervisor.js'+import { PROBE_ANSWER, RESUME_ENV, decodeResume, encodeResume, fingerprint, readMarker, writeMarker } from './reexec.js'+import { DEFAULT_SUPERVISOR_OPTIONS, type ResumeState } from './supervisor.js'describe('supervisorOptions', () => {it('reads valid overrides and keeps the defaults for anything unset or invalid', () => {@@ -60,7 +61,7 @@ describe('the status file', () => {afterEach(() => rmSync(dir, { recursive: true, force: true }))const status = (over: Partial<MasterStatusFile> = {}): MasterStatusFile => ({state: 'running', corePid: 7, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null,- protocol: 2, since: 1, masterPid: 42, ...over,+ protocol: 2, since: 1, masterVersion: null, reexecs: 0, masterPid: 42, ...over,})it('is written whole and read back only for the master that wrote it', () => {@@ -137,10 +138,46 @@ describe('trimLogEvery', () => {describe('masterDefaults', () => {it('takes what the config gives, and this process for the rest', () => {const base = { nodePath: 'n', execArgv: [], scriptPath: 's', pidFile: 'p', restoreUpdate: () => {}, confirmUpdate: () => {} }- expect(masterDefaults(base)).toEqual({ env: process.env, exit: processExit, onSignal: onProcessSignal })+ expect(masterDefaults(base)).toEqual({ env: process.env, exit: processExit, onSignal: onProcessSignal, execve: expect.any(Function) })const exit = () => {}const onSignal = () => {}- expect(masterDefaults({ ...base, env: { A: '1' }, exit, onSignal })).toEqual({ env: { A: '1' }, exit, onSignal })+ const execve = () => {}+ expect(masterDefaults({ ...base, env: { A: '1' }, exit, onSignal, execve })).toEqual({ env: { A: '1' }, exit, onSignal, execve })+ expect(masterDefaults({ ...base, execve: null }).execve).toBeNull()+ })++ it('replaces the process through process.execve where this Node has it, and has no way to where it has not', () => {+ expect(processExecve({})).toBeNull()+ const calls: unknown[][] = []+ const proc = { execve(this: unknown, ...args: unknown[]) { calls.push([this === proc, ...args]) } }+ processExecve(proc)!('/node', ['/node', 'cli.js'], { A: '1' })+ expect(calls).toEqual([[true, '/node', ['/node', 'cli.js'], { A: '1' }]])+ // The Node these tests run on has it (22.15 and 23.11 on).+ expect(processExecve()).toEqual(expect.any(Function))+ })+})++describe('the probe a re-executing master asks a bundle', () => {+ it('answers when the master builds, from the state it would be handed or from none', () => {+ const said: string[] = []+ expect(probeMaster({ env: {}, execArgv: [], version: '9.9.9' }, (line) => said.push(line))).toBe(0)+ const resume: ResumeState = { restarts: 1, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 1, unproven: 1 }+ expect(probeMaster({ env: { [RESUME_ENV]: encodeResume(resume) }, execArgv: [], version: '9.9.9' }, (line) => said.push(line))).toBe(0)+ expect(said).toEqual([`${PROBE_ANSWER} · protocol 2 · v9.9.9`, `${PROBE_ANSWER} · protocol 2 · v9.9.9`])+ })++ it('refuses a state it cannot read, and says so on stdout by default', () => {+ const log = vi.spyOn(console, 'log').mockImplementation(() => {})+ try {+ expect(probeMaster({ env: { [RESUME_ENV]: '{"restarts":"many"}' }, execArgv: [], version: '9.9.9' })).toBe(1)+ expect(log).toHaveBeenCalledWith('harnessd-probe failed: the state it would be handed is not one this master can read')+ } finally { log.mockRestore() }+ // A failure that is not an Error is reported all the same: here, the answer itself could not be said.+ const said: string[] = []+ let first = true+ const say = (line: string) => { if (first) { first = false; throw 'stdout closed' } said.push(line) }+ expect(probeMaster({ env: { HARNESSD_SERVICES: 'search' }, execArgv: [], version: '' }, say)).toBe(1)+ expect(said).toEqual(['harnessd-probe failed: stdout closed'])})})@@ -170,7 +207,7 @@ describe('coreHandle', () => {const messages: unknown[] = []handle.onMessage((message) => messages.push(message))child.emit('message', { type: 'x' })- handle.send({ type: 'harnessd:status', status: { state: 'running', corePid: 1, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null, protocol: 2, since: 0 } })+ handle.send({ type: 'harnessd:status', status: { state: 'running', corePid: 1, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null, protocol: 2, since: 0, masterVersion: null, reexecs: 0 } })handle.kill('SIGTERM')expect(handle.pid).toBe(4242)expect(messages).toEqual([{ type: 'x' }])@@ -386,4 +423,126 @@ describe('runMaster', () => {expect(await exited).toBe(0)expect(output).toContain('[harnessd] SIGTERM — stopping')})++ describe('re-executing on the bundle on disk', () => {+ const DEAD_PID = 2_000_000_000+ /** One script plays the bundle: as a core it binds, is ready and beats; as a probe it answers, or not. */+ const bundle = (core: string, probe = `console.log(${JSON.stringify(PROBE_ANSWER)} + ' · test'); process.exit(0)`) => {+ const file = join(dir, 'bundle.cjs')+ writeFileSync(file, `+ const fs = require('node:fs')+ if (process.argv[2] === '__harnessd-probe') { ${probe} }+ else {+ process.send({ type: 'harnessd:bound', protocol: 2, port: 1 })+ process.send({ type: 'harnessd:ready' })+ setInterval(() => process.send({ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1, loopDelayMs: 0 }), 50)+ process.on('SIGTERM', () => process.exit(0))+ ${core}+ }+ `)+ return file+ }+ const start = (scriptPath: string, over: Record<string, unknown> = {}) => {+ const execs: Array<{ file: string; args: string[]; env: NodeJS.ProcessEnv }> = []+ const exits: number[] = []+ const calls: string[] = []+ const signals = new Map<string, () => void>()+ const supervisor = runMaster({+ nodePath: process.execPath, execArgv: [], scriptPath, pidFile: join(dir, 'adapter.pid'), statusFile: join(dir, 'harnessd-status.json'),+ restoreUpdate: () => calls.push('restore'), confirmUpdate: () => calls.push('confirm'),+ env: { ...process.env, HARNESSD_INITIAL_BACKOFF_MS: '10', HARNESSD_UPDATE_PROBATION_MS: '100' },+ exit: (code) => exits.push(code), onSignal: (signal, listener) => signals.set(signal, listener),+ version: '1.0.0', reexecMarkerFile: join(dir, 'harnessd-reexec.json'),+ execve: (file: string, args: string[], env: NodeJS.ProcessEnv) => { execs.push({ file, args, env }) },+ ...over,+ })+ const stop = async () => { signals.get('SIGTERM')!(); await until('the master to finish', () => exits.length > 0) }+ return { supervisor, execs, exits, calls, stop }+ }+ const marker = () => join(dir, 'harnessd-reexec.json')++ it('replaces itself with the bundle its core staged before it exited for the update, leaving its marker and handing on its state', async () => {+ // A newer bundle written over this one, then the update exit: what the core's updater does.+ const script = bundle(`if (process.env.HARNESSD_RESTARTS === '0') setTimeout(() => { fs.appendFileSync(__filename, '\\n// a newer build\\n'); process.exit(75) }, 100)`)+ const own = fingerprint(script)+ const master = start(script)+ await until('the master to re-execute', () => master.execs.length > 0)+ expect(master.execs[0].file).toBe(process.execPath)+ expect(master.execs[0].args).toEqual([process.execPath, script, '__harnessd'])+ expect(decodeResume(master.execs[0].env[RESUME_ENV])).toEqual({ restarts: 1, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 1, unproven: 1 })+ expect(readMarker(marker())).toEqual({ pid: process.pid, from: own, to: fingerprint(script), at: expect.any(Number) })+ expect(master.supervisor.status()).toMatchObject({ state: 'restarting', corePid: null, masterVersion: '1.0.0' })+ await master.stop()+ expect(master.exits).toEqual([0])+ })++ it('rolls an update back when the new bundle\'s master does not answer its probe, and keeps its own code', async () => {+ const script = bundle(`if (process.env.HARNESSD_RESTARTS === '0') setTimeout(() => { fs.appendFileSync(__filename, '\\n// a newer build\\n'); process.exit(75) }, 100)`,+ `console.error('Unknown command: ' + process.argv[2]); process.exit(1)`)+ const master = start(script)+ await until('the rollback', () => master.calls.includes('restore'))+ await until('a core on the bundle restored', () => master.supervisor.status().state === 'running')+ expect(master.execs).toEqual([])+ expect(readMarker(marker())).toBeNull()+ await master.stop()+ })++ it('carries on as itself, without its marker, when the exec fails', async () => {+ const script = bundle(`if (process.env.HARNESSD_RESTARTS === '0') setTimeout(() => { fs.appendFileSync(__filename, '\\n// a newer build\\n'); process.exit(75) }, 100)`)+ const master = start(script, { execve: () => { throw new Error('E2BIG') } })+ await until('a core on the new bundle, under this master', () => master.supervisor.status().state === 'running' && master.supervisor.status().restarts === 1)+ expect(readMarker(marker())).toBeNull()+ await master.stop()+ })++ it('abandons a probe still running when it is stopped, and replaces nothing', async () => {+ const script = bundle(`if (process.env.HARNESSD_RESTARTS === '0') setTimeout(() => { fs.appendFileSync(__filename, '\\n// a newer build\\n'); process.exit(75) }, 100)`,+ `fs.writeFileSync(${JSON.stringify(join(dir, 'probing'))}, String(process.pid)); setInterval(() => {}, 1000)`)+ const master = start(script)+ await until('the probe to be running', () => existsSync(join(dir, 'probing')))+ const probe = Number(readFileSync(join(dir, 'probing'), 'utf8'))+ await master.stop()+ await until('the probe to be gone', () => { try { process.kill(probe, 0); return false } catch { return true } })+ expect(master.execs).toEqual([])+ })++ it('carries on as the master it re-executed from, without handing that state to its children', async () => {+ const seen = join(dir, 'seen')+ const script = bundle(`fs.appendFileSync(${JSON.stringify(seen)}, JSON.stringify({ restarts: process.env.HARNESSD_RESTARTS, resume: process.env[${JSON.stringify(RESUME_ENV)}] ?? null }) + '\\n')`)+ const resume: ResumeState = { restarts: 3, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 1, unproven: 1 }+ const lines: string[] = []+ const log = vi.spyOn(console, 'log').mockImplementation((line: string) => { lines.push(line) })+ try {+ const master = start(script, { env: { ...process.env, HARNESSD_UPDATE_PROBATION_MS: '100', [RESUME_ENV]: encodeResume(resume) } })+ await until('the update to be kept', () => master.calls.includes('confirm'))+ expect(JSON.parse(readFileSync(seen, 'utf8').trim().split('\n')[0])).toEqual({ restarts: '3', resume: null })+ expect(readStatusFile(join(dir, 'harnessd-status.json'), process.pid)).toMatchObject({ masterVersion: '1.0.0', reexecs: 1, restarts: 3 })+ expect(lines.some((line) => line.endsWith(`[harnessd] master re-executed (pid ${process.pid}) · now v1.0.0`))).toBe(true)+ await master.stop()+ // Without a version it says so without one.+ const plain = start(script, { version: undefined, env: { ...process.env, [RESUME_ENV]: encodeResume(resume) } })+ await until('its core', () => plain.supervisor.status().state === 'running')+ expect(lines.some((line) => line.endsWith(`[harnessd] master re-executed (pid ${process.pid})`))).toBe(true)+ await plain.stop()+ } finally { log.mockRestore() }+ })++ it('rolls back a re-execution that never brought a core up, then moves onto the bundle restored', async () => {+ const script = bundle('')+ writeMarker(marker(), { pid: DEAD_PID, from: 'the one before', to: fingerprint(script)!, at: 1 })+ const master = start(script, { restoreUpdate: () => { writeFileSync(script, readFileSync(script, 'utf8') + '\n// the build before\n') } })+ await until('the master to re-execute on the bundle restored', () => master.execs.length > 0)+ expect(readMarker(marker())).toMatchObject({ pid: process.pid, to: fingerprint(script) })+ await master.stop()+ })++ it('removes its marker once its core is up, and leaves alone what it does not judge', async () => {+ const script = bundle('')+ writeMarker(marker(), { pid: process.pid, from: 'the one before', to: fingerprint(script)!, at: 1 })+ const master = start(script)+ await until('its marker to go', () => !existsSync(marker()))+ expect(master.calls).toEqual([])+ await master.stop()+ })+ })})diff --git a/cli/src/harnessd/master.ts b/cli/src/harnessd/master.tsindex f737eb2bc..769959763 100644--- a/cli/src/harnessd/master.ts+++ b/cli/src/harnessd/master.ts@@ -1,7 +1,7 @@/*** `harness __harnessd`: the master process `harness start` launches (see ./supervisor.ts for what it* does). This file is only its wiring to the operating system: the core child, the pid file, the status- * file, the log's size, signals.+ * file, the log's size, signals, and re-executing on a new bundle (./reexec.ts).*/import { spawn, type ChildProcess } from 'node:child_process'import { randomBytes } from 'node:crypto'@@ -10,8 +10,13 @@ import { performance } from 'node:perf_hooks'import { trimLogFile } from '../lib/log.js'import { platformFromEnv, type PlatformName } from './platform.js'import type { MasterMessage } from './protocol.js'+import {+ PROBE_ANSWER, RESUME_ENV, createReexec, decodeResume, fingerprint, readMarker, recoverFailedReexec, removeMarker, runProbe, writeMarker,+} from './reexec.js'import { KNOWN_SERVICES, ServiceSupervisor, serviceOptions, serviceSpecs } from './services.js'-import { DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorOptions, type SupervisorStatus } from './supervisor.js'+import {+ DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorDeps, type SupervisorOptions, type SupervisorStatus,+} from './supervisor.js'/** How often the master keeps the daemon's log under its cap. */export const LOG_TRIM_INTERVAL_MS = 60_000@@ -34,15 +39,32 @@ export interface MasterConfig {exit?: (code: number) => void/** Where SIGTERM, SIGINT and SIGHUP are listened for; defaults to this process. */onSignal?: (signal: NodeJS.Signals, listener: () => void) => void+ /** This bundle's version, for the status (`harness status`, `/api/status`). */+ version?: string+ /** Where a master about to re-execute on the bundle on disk leaves its marker (./reexec.ts). Left+ * out, the master never re-executes, as before. */+ reexecMarkerFile?: string+ /** Replace this process (`process.execve`); defaults to this Node's, null where it has none. */+ execve?: Execve | null}+export type Execve = (file: string, args: string[], env: NodeJS.ProcessEnv) => void+/** The defaults `runMaster` acts on this process with. */export const processExit = (code: number): void => { process.exit(code) }export const onProcessSignal = (signal: NodeJS.Signals, listener: () => void): void => { process.on(signal, listener) }+/** `process.execve` (Node 22.15 and 23.11 on, and not yet in the typings this builds with); null without it. */+export function processExecve(proc: object = process): Execve | null {+ const execve = (proc as { execve?: unknown }).execve+ return typeof execve === 'function' ? (file, args, env) => { execve.call(proc, file, args, env) } : null+}/** What a config leaves out, taken from this process. Pure: choosing a default does not act on one. */-export function masterDefaults(config: MasterConfig): Required<Pick<MasterConfig, 'env' | 'exit' | 'onSignal'>> {- return { env: config.env ?? process.env, exit: config.exit ?? processExit, onSignal: config.onSignal ?? onProcessSignal }+export function masterDefaults(config: MasterConfig): Required<Pick<MasterConfig, 'env' | 'exit' | 'onSignal'>> & { execve: Execve | null } {+ return {+ env: config.env ?? process.env, exit: config.exit ?? processExit, onSignal: config.onSignal ?? onProcessSignal,+ execve: config.execve === undefined ? processExecve() : config.execve,+ }}/** The heap limit `--max-old-space-size` sets in these flags, MiB; null when none does. */@@ -165,9 +187,37 @@ export function coreHandle(child: ChildProcess): CoreHandle {}}+/**+ * `harness __harnessd-probe`: what a master about to re-execute on this bundle asks it first+ * (./reexec.ts). Everything a master does before it starts its children, with the state it would be+ * handed, and nothing after: the whole bundle loads, the state reads, a master is built. Exit 0 and+ * the answer, or 1 and why.+ */+export function probeMaster(config: { env: NodeJS.ProcessEnv; execArgv: string[]; version: string }, say: (line: string) => void = console.log): number {+ try {+ const handed = config.env[RESUME_ENV]+ const resume = decodeResume(handed)+ if (handed !== undefined && !resume) throw new Error('the state it would be handed is not one this master can read')+ serviceSpecs(config.env, KNOWN_SERVICES)+ const inert = { wallClock: () => Date.now() } as unknown as SupervisorDeps+ const status = new Supervisor(inert, supervisorOptions(config.env, config.execArgv), { version: config.version, resume }).status()+ say(`${PROBE_ANSWER} · protocol ${status.protocol} · v${status.masterVersion}`)+ return 0+ } catch (error) {+ say(`harnessd-probe failed: ${error instanceof Error ? error.message : String(error)}`)+ return 1+ }+}+export function runMaster(config: MasterConfig): Supervisor {- const { env, exit, onSignal } = masterDefaults(config)+ const { env: given, exit, onSignal, execve } = masterDefaults(config)process.title = 'harnessd'+ // What the master this process was a moment ago handed on, if it was one (./reexec.ts). Taken out of+ // the environment every child is given: a core or service has no use for it, and a later re-exec+ // writes its own.+ const resume = decodeResume(given[RESUME_ENV])+ const env: NodeJS.ProcessEnv = { ...given }+ delete env[RESUME_ENV]const readPid = (): number | null => {try { return Number.parseInt(readFileSync(config.pidFile, 'utf8').trim(), 10) || null } catch { return null }}@@ -180,6 +230,22 @@ export function runMaster(config: MasterConfig): Supervisor {// Set by the launchd agent or systemd unit `harness service install` writes (./platform.ts).const platform = platformFromEnv(env)if (platform) log(`[harnessd] run by ${platform}, which starts this master again if it dies`)+ if (resume) log(`[harnessd] master re-executed (pid ${process.pid})${config.version ? ` · now v${config.version}` : ''}`)+ // The bundle this master's code came from, read once, as it starts; the bundle on disk is read+ // again before every core starts.+ const own = fingerprint(config.scriptPath)+ const bundle = () => fingerprint(config.scriptPath)+ const markerFile = config.reexecMarkerFile+ const alive = (pid: number): boolean => { try { process.kill(pid, 0); return true } catch { return false } }+ if (markerFile) {+ // A re-execution before this start that never brought a core up: back to the previous bundle,+ // which the supervisor then re-executes this master on before it starts anything.+ recoverFailedReexec({+ marker: readMarker(markerFile), pid: process.pid, alive, current: bundle,+ restoreUpdate: config.restoreUpdate, removeMarker: () => removeMarker(markerFile), log,+ })+ }+ let stopping = false// A new one every boot, given to the core and to each service: how the core knows a service// connection is one this master started, and no other local process.const token = randomBytes(24).toString('hex')@@ -194,6 +260,18 @@ export function runMaster(config: MasterConfig): Supervisor {clearTimer: (timer) => clearTimeout(timer as ReturnType<typeof setTimeout>),log,}, serviceOptions(env), { HARNESSD_SERVICE_TOKEN: token })+ const reexec = markerFile ? createReexec({+ own, current: bundle, nodePath: config.nodePath, execArgv: config.execArgv, scriptPath: config.scriptPath, env, pid: process.pid,+ execve,+ probe: (args, probeEnv) => runProbe(config.nodePath, args, probeEnv),+ stopChildren: (done) => services.stop(done),+ startChildren: () => services.start(),+ writeMarker: (marker) => writeMarker(markerFile, marker),+ removeMarker: () => removeMarker(markerFile),+ stopping: () => stopping,+ now: () => Date.now(),+ log,+ }) : nullconst supervisor = new Supervisor({spawnCore: (extra) => coreHandle(spawn(config.nodePath, [...execArgv, config.scriptPath, '__run'], {env: { ...env, ...extra, HARNESSD_SERVICE_TOKEN: token },@@ -210,13 +288,29 @@ export function runMaster(config: MasterConfig): Supervisor {confirmUpdate: config.confirmUpdate,log,// The services go with the master, after the core: none is left holding the core's socket.- exit: (code) => services.stop(() => {- stopTrimming()- exit(code)- }),- }, options)+ exit: (code) => {+ stopping = true+ reexec?.cancel()+ services.stop(() => {+ stopTrimming()+ exit(code)+ })+ },+ ...(reexec && markerFile ? {+ reexec: reexec.reexec,+ // The re-executed master has brought a core up: its marker has served.+ coreUp: () => removeMarker(markerFile),+ } : {}),+ }, options, { version: config.version, resume })// Services stop beside the core, inside the same grace `harness stop` gives the whole daemon.- for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP'] as const) onSignal(signal, () => { services.stop(() => {}); supervisor.stop(signal) })+ for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP'] as const) {+ onSignal(signal, () => {+ stopping = true+ reexec?.cancel()+ services.stop(() => {})+ supervisor.stop(signal)+ })+ }supervisor.start()// Not waiting on the core: a service reaches it through its socket, and retries until it answers.services.start()diff --git a/cli/src/harnessd/reexec.spec.ts b/cli/src/harnessd/reexec.spec.tsnew file mode 100644index 000000000..aaffa5e21--- /dev/null+++ b/cli/src/harnessd/reexec.spec.ts@@ -0,0 +1,284 @@+import { EventEmitter } from 'node:events'+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'+import { tmpdir } from 'node:os'+import { join } from 'node:path'+import type { ChildProcess } from 'node:child_process'+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'+import {+ PROBE_ANSWER, PROBE_COMMAND, PROBE_TIMEOUT_MS, REEXEC_LIMIT, RESUME_ENV, createReexec, decodeResume, encodeResume, fingerprint,+ readMarker, recoverFailedReexec, removeMarker, runProbe, writeMarker, type ProbeResult, type ReexecDeps,+} from './reexec.js'+import type { ReexecOutcome, ResumeState } from './supervisor.js'++const state = (over: Partial<ResumeState> = {}): ResumeState => ({+ restarts: 2, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 0, unproven: 0, ...over,+})++describe('which bundle', () => {+ let dir: string+ beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'harnessd-reexec-')) })+ afterEach(() => rmSync(dir, { recursive: true, force: true }))++ it('is told by the sha256 of its bytes, or null when it cannot be read', () => {+ const file = join(dir, 'cli.js')+ writeFileSync(file, 'one')+ const one = fingerprint(file)+ expect(one).toMatch(/^[0-9a-f]{64}$/)+ writeFileSync(file, 'two')+ expect(fingerprint(file)).not.toBe(one)+ expect(fingerprint(join(dir, 'none.js'))).toBeNull()+ expect(fingerprint('x', () => Buffer.from('one'))).toBe(one)+ })+})++describe('the state a master hands on', () => {+ it('reads back what was written', () => {+ expect(decodeResume(encodeResume(state()))).toEqual(state())+ expect(decodeResume(encodeResume(state({ lastExit: null, lastExitReason: null, update: null, claimed: false })))).toEqual(state({ lastExit: null, lastExitReason: null, update: null, claimed: false }))+ })++ it('is none when absent, and refused when it is not one this code can read', () => {+ expect(decodeResume(undefined)).toBeNull()+ expect(decodeResume('{not json')).toBeNull()+ for (const wrong of [+ { restarts: -1 }, { restarts: 1.5 }, { lastExit: 3 }, { lastExitReason: 'bored' }, { update: 'probation' }, { claimed: 'yes' },+ { reexecs: -1 }, { reexecs: '1' }, { unproven: 0.5 }, { unproven: null },+ ]) expect(decodeResume(JSON.stringify({ ...state(), ...wrong })), JSON.stringify(wrong)).toBeNull()+ })+})++describe('the marker', () => {+ let dir: string+ beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'harnessd-marker-')) })+ afterEach(() => rmSync(dir, { recursive: true, force: true }))++ it('is written whole, read back, and removed', () => {+ const file = join(dir, 'harnessd-reexec.json')+ expect(readMarker(file)).toBeNull()+ writeMarker(file, { pid: 42, from: 'a', to: 'b', at: 1 })+ expect(readMarker(file)).toEqual({ pid: 42, from: 'a', to: 'b', at: 1 })+ writeMarker(file, { pid: 42, from: null, to: 'b', at: 1 })+ expect(readMarker(file)).toEqual({ pid: 42, from: null, to: 'b', at: 1 })+ removeMarker(file)+ removeMarker(file)+ expect(readMarker(file)).toBeNull()+ })++ it('is no marker when it is not one', () => {+ const file = join(dir, 'harnessd-reexec.json')+ for (const wrong of [{ pid: 'x', to: 'b', at: 1, from: null }, { pid: 1, to: 2, at: 1, from: null }, { pid: 1, to: 'b', at: 'now', from: null }, { pid: 1, to: 'b', at: 1, from: 3 }]) {+ writeFileSync(file, JSON.stringify(wrong))+ expect(readMarker(file), JSON.stringify(wrong)).toBeNull()+ }+ })++ it('is removed even when something odd sits where it was, without throwing', () => {+ const odd = join(dir, 'harnessd-reexec.json')+ mkdirSync(odd)+ expect(() => removeMarker(odd)).not.toThrow()+ })+})++describe('a re-execution that never came up', () => {+ const recover = (over: Partial<Parameters<typeof recoverFailedReexec>[0]> = {}) => {+ const calls: string[] = []+ const result = recoverFailedReexec({+ marker: { pid: 7, from: 'old', to: 'new', at: 1 }, pid: 99, alive: () => false, current: () => 'new',+ restoreUpdate: () => calls.push('restore'), removeMarker: () => calls.push('remove'), log: (line) => calls.push(line), ...over,+ })+ return { result, calls }+ }++ it('is rolled back when the process that re-executed is gone and its bundle is still on disk', () => {+ expect(recover()).toEqual({ result: true, calls: ['remove', 'restore', '[harnessd] the master that re-executed on the new bundle (pid 7) never brought a core up — rolled back to the previous bundle'] })+ })++ it('is let be when there is none, when it is this process\'s own, when its process lives, or when the bundle moved on', () => {+ expect(recover({ marker: null })).toEqual({ result: false, calls: [] })+ expect(recover({ pid: 7 })).toEqual({ result: false, calls: [] })+ expect(recover({ alive: () => true })).toEqual({ result: false, calls: [] })+ expect(recover({ current: () => 'newer' })).toEqual({ result: false, calls: ['remove'] })+ })+})++describe('the probe', () => {+ let dir: string+ beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'harnessd-probe-')) })+ afterEach(() => rmSync(dir, { recursive: true, force: true }))+ const script = (body: string) => {+ const file = join(dir, 'bundle.cjs')+ writeFileSync(file, body)+ return file+ }++ it('passes a bundle whose master answers, and says why one that does not fails', async () => {+ const ok = await runProbe(process.execPath, [script(`console.log('loading'); console.log(${JSON.stringify(PROBE_ANSWER)} + ' · protocol 2')`), PROBE_COMMAND], process.env).result+ expect(ok).toEqual({ ok: true, detail: `${PROBE_ANSWER} · protocol 2` })+ // A bundle from before probes: its usage on stdout, the reason on stderr, which is what is said.+ expect(await runProbe(process.execPath, [script(`console.error('Unknown command: ' + process.argv[2]); console.log('Usage: harness …\\n harness --help'); process.exit(1)`), PROBE_COMMAND], process.env).result)+ .toEqual({ ok: false, detail: `Unknown command: ${PROBE_COMMAND}` })+ // An answer on stderr is no answer.+ expect(await runProbe(process.execPath, [script(`console.error(${JSON.stringify(PROBE_ANSWER)})`), PROBE_COMMAND], process.env).result)+ .toEqual({ ok: false, detail: PROBE_ANSWER })+ expect(await runProbe(process.execPath, [script('console.log("something else")'), PROBE_COMMAND], process.env).result)+ .toEqual({ ok: false, detail: 'something else' })+ expect(await runProbe(process.execPath, [script('process.exit(4)')], process.env).result).toEqual({ ok: false, detail: 'exit 4' })+ expect(await runProbe(process.execPath, [script('process.kill(process.pid, "SIGKILL")')], process.env).result).toEqual({ ok: false, detail: 'signal SIGKILL' })+ expect(await runProbe(join(dir, 'no-node'), [], process.env).result).toEqual({ ok: false, detail: expect.stringContaining('ENOENT') })+ })++ it('gives up on one that does not answer in time, and can be abandoned', async () => {+ const hung = script('setInterval(() => {}, 1000)')+ expect(await runProbe(process.execPath, [hung], process.env, 300).result).toEqual({ ok: false, detail: 'no answer within 300 ms' })+ const running = runProbe(process.execPath, [hung], process.env)+ running.cancel()+ expect(await running.result).toEqual({ ok: false, detail: 'signal SIGKILL' })+ expect(PROBE_TIMEOUT_MS).toBeGreaterThanOrEqual(10_000)+ })++ it('survives a child that has no output to read and cannot be killed', async () => {+ const child = Object.assign(new EventEmitter(), { stdout: null, stderr: null, kill: () => { throw new Error('ESRCH') } }) as unknown as ChildProcess+ vi.useFakeTimers()+ try {+ const running = runProbe('node', [], {}, 100, () => child)+ running.cancel()+ vi.advanceTimersByTime(100)+ expect(await running.result).toEqual({ ok: false, detail: 'no answer within 100 ms' })+ child.emit('close', 0, null)+ } finally { vi.useRealTimers() }+ })+})++describe('re-executing', () => {+ let calls: string[]+ let probes: Array<{ args: string[]; env: NodeJS.ProcessEnv; settle: (result: ProbeResult) => void; cancelled: boolean }>+ let outcomes: ReexecOutcome[]+ let stopping: boolean+ let onDisk: string | null+ let execs: Array<{ file: string; args: string[]; env: NodeJS.ProcessEnv }>+ let stopped: Array<() => void>++ const make = (over: Partial<ReexecDeps> = {}) => createReexec({+ own: 'old', current: () => onDisk, nodePath: '/opt/node', execArgv: ['--flag'], scriptPath: '/opt/cli.js',+ env: { KEEP: '1' }, pid: 4242,+ execve: (file, args, env) => { execs.push({ file, args, env }) },+ probe: (args, env) => {+ let settle!: (result: ProbeResult) => void+ const entry = { args, env, settle: (result: ProbeResult) => settle(result), cancelled: false }+ probes.push(entry)+ return { result: new Promise<ProbeResult>((resolve) => { settle = resolve }), cancel: () => { entry.cancelled = true } }+ },+ stopChildren: (done) => { calls.push('stop children'); stopped.push(done) },+ startChildren: () => calls.push('start children'),+ writeMarker: (marker) => calls.push(`marker ${JSON.stringify(marker)}`),+ removeMarker: () => calls.push('remove marker'),+ stopping: () => stopping,+ now: () => 1_000,+ log: (line) => calls.push(line),+ ...over,+ })+ const settle = async (result: ProbeResult) => { probes.at(-1)!.settle(result); await new Promise((done) => setImmediate(done)) }++ beforeEach(() => {+ calls = []+ probes = []+ outcomes = []+ stopping = false+ onDisk = 'new'+ execs = []+ stopped = []+ })++ it('does nothing when the bundle on disk is this master\'s own, or cannot be read', () => {+ const { reexec } = make()+ onDisk = 'old'+ reexec(state(), (outcome) => outcomes.push(outcome))+ onDisk = null+ reexec(state(), (outcome) => outcomes.push(outcome))+ expect(outcomes).toEqual(['same', 'same'])+ expect(calls).toEqual([])+ })++ it('keeps this master where this Node cannot re-execute, and says so once', () => {+ const { reexec } = make({ execve: null })+ reexec(state(), (outcome) => outcomes.push(outcome))+ reexec(state(), (outcome) => outcomes.push(outcome))+ expect(outcomes).toEqual(['kept', 'kept'])+ expect(calls).toEqual(['[harnessd] the bundle on disk is newer than this master, but this Node cannot re-execute a process: the master keeps its code until it restarts'])+ })++ it('keeps this master after re-executing again and again without a core coming up', () => {+ const { reexec } = make()+ reexec(state({ unproven: REEXEC_LIMIT }), (outcome) => outcomes.push(outcome))+ expect(outcomes).toEqual(['kept'])+ expect(calls).toEqual([`[harnessd] this master re-executed ${REEXEC_LIMIT} times without a core coming up — keeping its code; something keeps rewriting the bundle`])+ })++ it('stops its children, probes the bundle, leaves its marker and replaces itself, handing its state on', async () => {+ const { reexec } = make()+ reexec(state({ reexecs: 2, unproven: 1 }), (outcome) => outcomes.push(outcome))+ expect(calls).toEqual(['[harnessd] the bundle on disk is not this master\'s code — re-executing on it', 'stop children'])+ expect(probes).toEqual([])+ stopped[0]()+ expect(probes[0].args).toEqual(['--flag', '/opt/cli.js', PROBE_COMMAND])+ expect(decodeResume(probes[0].env[RESUME_ENV])).toEqual(state({ reexecs: 3, unproven: 2 }))+ expect(probes[0].env.KEEP).toBe('1')+ await settle({ ok: true, detail: PROBE_ANSWER })+ expect(calls.at(-1)).toBe(`marker ${JSON.stringify({ pid: 4242, from: 'old', to: 'new', at: 1_000 })}`)+ expect(execs).toEqual([{ file: '/opt/node', args: ['/opt/node', '--flag', '/opt/cli.js', '__harnessd'], env: probes[0].env }])+ expect(outcomes).toEqual([])+ })++ it('carries on as itself when the probe fails or the exec does, with its children back', async () => {+ const refused = make()+ refused.reexec(state(), (outcome) => outcomes.push(outcome))+ stopped[0]()+ await settle({ ok: false, detail: 'Unknown command: __harnessd-probe' })+ expect(outcomes).toEqual(['refused'])+ expect(calls.slice(-2)).toEqual(['[harnessd] the new bundle\'s master did not answer its probe (Unknown command: __harnessd-probe) — keeping this master', 'start children'])+ expect(execs).toEqual([])++ calls = []+ const failing = make({ execve: () => { throw new Error('E2BIG') } })+ failing.reexec(state(), (outcome) => outcomes.push(outcome))+ stopped[1]()+ await settle({ ok: true, detail: PROBE_ANSWER })+ expect(outcomes).toEqual(['refused', 'kept'])+ expect(calls.slice(-3)).toEqual(['remove marker', '[harnessd] could not re-execute this master (E2BIG) — keeping it', 'start children'])++ const odd = make({ execve: () => { throw 'odd' } })+ odd.reexec(state(), (outcome) => outcomes.push(outcome))+ stopped[2]()+ await settle({ ok: true, detail: PROBE_ANSWER })+ expect(calls).toContain('[harnessd] could not re-execute this master (odd) — keeping it')+ })++ it('does nothing more once the master is stopping, at any step, and abandons its probe', async () => {+ const first = make()+ first.reexec(state(), (outcome) => outcomes.push(outcome))+ stopping = true+ stopped[0]()+ expect(probes).toEqual([])++ stopping = false+ const second = make()+ second.reexec(state(), (outcome) => outcomes.push(outcome))+ stopped[1]()+ stopping = true+ second.cancel()+ expect(probes[0].cancelled).toBe(true)+ await settle({ ok: true, detail: PROBE_ANSWER })+ expect(execs).toEqual([])++ stopping = false+ const third = make()+ third.reexec(state(), (outcome) => outcomes.push(outcome))+ stopped[2]()+ stopping = true+ await settle({ ok: false, detail: 'no' })+ expect(outcomes).toEqual([])+ expect(calls).not.toContain('start children')+ third.cancel()+ make().cancel()+ })+})diff --git a/cli/src/harnessd/reexec.ts b/cli/src/harnessd/reexec.tsnew file mode 100644index 000000000..70d34ce71--- /dev/null+++ b/cli/src/harnessd/reexec.ts@@ -0,0 +1,251 @@+/**+ * The master replacing itself with the bundle on disk (docs/design/2026-10-03-harnessd.md, "Updates").+ *+ * An update replaced the core but not the master: the core staged the new bundle and exited 75, the+ * master started a core on it, and went on running its own old code until something restarted it, and+ * so did every master fix. Now, in the gap before a core starts, a master whose code is not the bundle+ * on disk re-executes itself on it (`process.execve`).+ *+ * In place, keeping its pid, rather than handing over to a fresh master: launchd and systemd follow the+ * pid they started (a master that exits is, to them, a job that ended, and its successor runs+ * unsupervised), and so do the pid file, `harness stop` and the desktop app's owner check. Only in the+ * gap: with no core running and the services stopped and reaped first, nothing loses its channel to+ * the master and nothing is orphaned, and the agents live in tmux, out of reach of all of it.+ *+ * What exec gives up is the old master as a fallback, so two things stand in for it:+ * - a probe: the bundle's own master is asked first (`__harnessd-probe`, `probeMaster` in ./master.ts)+ * to load and to build itself from the state it would be handed. A bundle from before this answers+ * "Unknown command", so it is never started as a probe, which could have started a second daemon;+ * one that fails the probe fails the update, which is rolled back and remembered;+ * - a marker (`harnessd-reexec.json`): written before the exec, removed once the new master's first+ * core is up. A master that starts and finds one left by a process that is gone knows that re-exec+ * never came up, and puts the previous bundle back (`recoverFailedReexec`). Under launchd or systemd,+ * which restart a master that died, that happens by itself; otherwise at the next `harness start`.+ *+ * Everything that touches the operating system is injected (`ReexecDeps`).+ */+import { spawn, type ChildProcess } from 'node:child_process'+import { createHash } from 'node:crypto'+import { readFileSync, renameSync, rmSync, writeFileSync } from 'node:fs'+import type { ExitReason, ReexecOutcome, ResumeState } from './supervisor.js'++/** The environment variable that hands a master's state to the one it re-executes as. */+export const RESUME_ENV = 'HARNESSD_RESUME'+/** The command a bundle's master answers its probe on, and what it answers. */+export const PROBE_COMMAND = '__harnessd-probe'+export const PROBE_ANSWER = 'harnessd-probe ok'+/** How long a probe may take: the updater gives its canary as long. */+export const PROBE_TIMEOUT_MS = 15_000+/**+ * This many re-executions in a row without a core coming up, and the master keeps its code: each new+ * master found yet another bundle before it could start one, and following them would be a loop. An+ * update that is rolled back takes two (onto the bundle, and back), each brought up a core or tried.+ */+export const REEXEC_LIMIT = 3++/** A file's sha256: which bundle a master runs, or which is on disk; null when it cannot be read. */+export function fingerprint(path: string, read: (path: string) => Buffer = readFileSync): string | null {+ try { return createHash('sha256').update(read(path)).digest('hex') } catch { return null }+}++const EXIT_REASONS: readonly ExitReason[] = ['crashed', 'hung', 'did-not-bind', 'not-ready', 'memory', 'update', 'stopped']++export function encodeResume(state: ResumeState): string {+ return JSON.stringify(state)+}++/** The state a master was handed, or null when there is none or it is not one this code can read. */+export function decodeResume(text: string | undefined): ResumeState | null {+ if (text === undefined) return null+ try {+ const value = JSON.parse(text) as Partial<ResumeState>+ const ok = Number.isInteger(value.restarts) && value.restarts! >= 0+ && (value.lastExit === null || typeof value.lastExit === 'string')+ && (value.lastExitReason === null || EXIT_REASONS.includes(value.lastExitReason as ExitReason))+ && (value.update === null || value.update === 'pending')+ && typeof value.claimed === 'boolean'+ && Number.isInteger(value.reexecs) && value.reexecs! >= 0+ && Number.isInteger(value.unproven) && value.unproven! >= 0+ return ok ? value as ResumeState : null+ } catch {+ return null+ }+}++/** What a master writes before it re-executes: who, from which bundle to which, and when. */+export interface ReexecMarker {+ pid: number+ from: string | null+ to: string+ at: number+}++export function writeMarker(file: string, marker: ReexecMarker): void {+ const temp = `${file}.${marker.pid}.tmp`+ writeFileSync(temp, `${JSON.stringify(marker)}\n`)+ renameSync(temp, file)+}++export function readMarker(file: string): ReexecMarker | null {+ try {+ const marker = JSON.parse(readFileSync(file, 'utf8')) as Partial<ReexecMarker>+ return Number.isInteger(marker.pid) && typeof marker.to === 'string' && typeof marker.at === 'number'+ && (marker.from === null || typeof marker.from === 'string')+ ? marker as ReexecMarker+ : null+ } catch {+ return null+ }+}++export function removeMarker(file: string): void {+ try { rmSync(file, { force: true }) } catch { /* nothing more to do */ }+}++/**+ * At a master's start: did a re-execution before it fail? A marker left by a process that is gone,+ * with the bundle it moved to still on disk, means the master that process became never brought a core+ * up. The previous bundle is put back (`restoreUpdate` also remembers the version, so the updater does+ * not stage it again) and true says so: this master runs that bundle's code and must move off it before+ * it starts anything. A marker that is this process's own is a re-execution in progress, and one whose+ * process lives on is not this master's to judge.+ */+export function recoverFailedReexec(deps: {+ marker: ReexecMarker | null+ pid: number+ alive(pid: number): boolean+ /** The bundle on disk now. */+ current(): string | null+ restoreUpdate(): void+ removeMarker(): void+ log(line: string): void+}): boolean {+ const { marker } = deps+ if (!marker || marker.pid === deps.pid || deps.alive(marker.pid)) return false+ deps.removeMarker()+ if (deps.current() !== marker.to) return false+ deps.restoreUpdate()+ deps.log(`[harnessd] the master that re-executed on the new bundle (pid ${marker.pid}) never brought a core up — rolled back to the previous bundle`)+ return true+}++export interface ProbeResult { ok: boolean; detail: string }++/** Run a bundle's master probe: `node <flags> <bundle> __harnessd-probe`, which must answer in time. */+export function runProbe(+ nodePath: string, args: string[], env: NodeJS.ProcessEnv, timeoutMs = PROBE_TIMEOUT_MS,+ start: (file: string, args: string[], env: NodeJS.ProcessEnv) => ChildProcess = (file, argv, environment) => spawn(file, argv, { env: environment, stdio: ['ignore', 'pipe', 'pipe'] }),+): { result: Promise<ProbeResult>; cancel(): void } {+ const child = start(nodePath, args, env)+ // Apart: a bundle from before probes answers with its usage on stdout and the reason on stderr.+ let out = ''+ let err = ''+ child.stdout?.on('data', (chunk: Buffer) => { out += chunk.toString('utf8') })+ child.stderr?.on('data', (chunk: Buffer) => { err += chunk.toString('utf8') })+ const lastLine = (text: string): string => { const lines = text.trim().split('\n'); return lines[lines.length - 1].trim() }+ const result = new Promise<ProbeResult>((resolve) => {+ let settled = false+ const done = (ok: boolean, detail: string) => {+ if (settled) return+ settled = true+ clearTimeout(timer)+ resolve({ ok, detail })+ }+ const timer = setTimeout(() => {+ try { child.kill('SIGKILL') } catch { /* gone */ }+ done(false, `no answer within ${timeoutMs} ms`)+ }, timeoutMs)+ child.on('error', (error) => done(false, error.message))+ child.on('close', (code, signal) => {+ done(code === 0 && out.includes(PROBE_ANSWER), lastLine(err) || lastLine(out) || (signal ? `signal ${signal}` : `exit ${code}`))+ })+ })+ return { result, cancel: () => { try { child.kill('SIGKILL') } catch { /* gone */ } } }+}++export interface ReexecDeps {+ /** The bundle this master was started from: its code. */+ own: string | null+ /** The bundle on disk now. */+ current(): string | null+ /** What a re-executed master runs: this node, these flags, this bundle. */+ nodePath: string+ execArgv: string[]+ scriptPath: string+ /** The master's own environment; the resume state is added to it. */+ env: NodeJS.ProcessEnv+ pid: number+ /** `process.execve`, where this Node has it (22.15 and 23.11 on). */+ execve: ((file: string, args: string[], env: NodeJS.ProcessEnv) => void) | null+ probe(args: string[], env: NodeJS.ProcessEnv): { result: Promise<ProbeResult>; cancel(): void }+ /** Stop the services and wait for them to be gone, so none is left a zombie or an orphan. */+ stopChildren(done: () => void): void+ /** Start them again, when this master carries on as itself. */+ startChildren(): void+ writeMarker(marker: ReexecMarker): void+ removeMarker(): void+ /** The master is stopping: nothing is replaced or restarted then. */+ stopping(): boolean+ /** The wall clock, ms. */+ now(): number+ log(line: string): void+}++/** The supervisor's `reexec`, and a way to abandon a probe still running when the master stops. */+export function createReexec(deps: ReexecDeps): {+ reexec(state: ResumeState, proceed: (outcome: ReexecOutcome) => void): void+ cancel(): void+} {+ let warned = false+ let probe: { cancel(): void } | null = null+ return {+ reexec: (state, proceed) => {+ const next = deps.current()+ if (next === null || next === deps.own) { proceed('same'); return }+ if (!deps.execve) {+ if (!warned) deps.log('[harnessd] the bundle on disk is newer than this master, but this Node cannot re-execute a process: the master keeps its code until it restarts')+ warned = true+ proceed('kept')+ return+ }+ if (state.unproven >= REEXEC_LIMIT) {+ deps.log(`[harnessd] this master re-executed ${state.unproven} times without a core coming up — keeping its code; something keeps rewriting the bundle`)+ proceed('kept')+ return+ }+ const now = deps.now()+ const env = { ...deps.env, [RESUME_ENV]: encodeResume({ ...state, reexecs: state.reexecs + 1, unproven: state.unproven + 1 }) }+ const args = [...deps.execArgv, deps.scriptPath]+ // Called only once the master is known not to be stopping: its children come back, and its core.+ const carryOn = (outcome: ReexecOutcome) => {+ deps.startChildren()+ proceed(outcome)+ }+ deps.log('[harnessd] the bundle on disk is not this master\'s code — re-executing on it')+ deps.stopChildren(() => {+ if (deps.stopping()) return+ const running = deps.probe([...args, PROBE_COMMAND], env)+ probe = running+ void running.result.then(({ ok, detail }) => {+ probe = null+ if (deps.stopping()) return+ if (!ok) {+ deps.log(`[harnessd] the new bundle's master did not answer its probe (${detail}) — keeping this master`)+ carryOn('refused')+ return+ }+ deps.writeMarker({ pid: deps.pid, from: deps.own, to: next, at: now })+ try {+ // Never returns: from here this process is the new bundle's master, with the same pid.+ deps.execve!(deps.nodePath, [deps.nodePath, ...args, '__harnessd'], env)+ } catch (error) {+ deps.removeMarker()+ deps.log(`[harnessd] could not re-execute this master (${error instanceof Error ? error.message : String(error)}) — keeping it`)+ carryOn('kept')+ }+ })+ })+ },+ cancel: () => { probe?.cancel(); probe = null },+ }+}diff --git a/cli/src/harnessd/supervisor.spec.ts b/cli/src/harnessd/supervisor.spec.tsindex a34e36ca5..13de403af 100644--- a/cli/src/harnessd/supervisor.spec.ts+++ b/cli/src/harnessd/supervisor.spec.ts@@ -1,6 +1,9 @@import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'import { CORE_EXIT_STOP, CORE_EXIT_UPDATE, HARNESSD_PROTOCOL, type MasterMessage } from './protocol.js'-import { DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorOptions, type SupervisorStatus } from './supervisor.js'+import {+ DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type ReexecOutcome, type ResumeState, type SupervisorDeps, type SupervisorOptions,+ type SupervisorStatus,+} from './supervisor.js'const MIB = 1024 * 1024@@ -49,7 +52,7 @@ describe('Supervisor', () => {let pidOf: (index: number) => number | undefinedconst core = () => cores[cores.length - 1]- const make = (overrides: Partial<SupervisorOptions> = {}) => new Supervisor({+ const make = (overrides: Partial<SupervisorOptions> = {}, more: Partial<SupervisorDeps> = {}, identity: ConstructorParameters<typeof Supervisor>[2] = {}) => new Supervisor({spawnCore: (env) => {const next = new FakeCore(pidOf(cores.length), env)cores.push(next)@@ -66,7 +69,8 @@ describe('Supervisor', () => {writeStatus: (status) => statuses.push(status),log: (line) => lines.push(line),exit: (code) => exited.push(code),- }, { ...options, ...overrides })+ ...more,+ }, { ...options, ...overrides }, identity)/** Let time pass for a running core, beating as a healthy one does. */const live = (ms: number) => { for (let left = ms; left > 0; left -= 1_000) { vi.advanceTimersByTime(Math.min(1_000, left)); core().beat() } }@@ -87,7 +91,7 @@ describe('Supervisor', () => {it('starts one core, claims the pid file once it is bound, and runs it once it is ready', () => {const supervisor = make()expect(new Supervisor({ wallClock: () => 7 } as never).status())- .toEqual({ state: 'idle', corePid: null, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null, protocol: HARNESSD_PROTOCOL, since: 7 })+ .toEqual({ state: 'idle', corePid: null, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null, protocol: HARNESSD_PROTOCOL, since: 7, masterVersion: null, reexecs: 0 })supervisor.start()supervisor.start()expect(cores).toHaveLength(1)@@ -102,7 +106,7 @@ describe('Supervisor', () => {core().ready()expect(supervisor.status()).toEqual({state: 'running', corePid: 1000, restarts: 0, lastExit: null, lastExitReason: null, safeMode: null,- protocol: HARNESSD_PROTOCOL, since: Date.now(),+ protocol: HARNESSD_PROTOCOL, since: Date.now(), masterVersion: null, reexecs: 0,})expect(lines.at(-1)).toBe('[harnessd] core ready (pid 1000)')// Every change is written for `harness status`, and told to the bound core.@@ -490,4 +494,118 @@ describe('Supervisor', () => {expect(calls).toEqual(['claim', 'release'])})})++ describe('re-executing on the bundle on disk', () => {+ /** A `reexec` that records what it was asked, and answers when the test says. */+ const reexecs = () => {+ const asked: Array<{ state: ResumeState; proceed: (outcome: ReexecOutcome) => void }> = []+ return { asked, deps: { reexec: (state: ResumeState, proceed: (outcome: ReexecOutcome) => void) => { asked.push({ state, proceed }) }, coreUp: () => calls.push('up') } }+ }++ it('asks before every core it starts, and starts it when the master stays as it is', () => {+ const { asked, deps } = reexecs()+ const supervisor = make({}, deps)+ supervisor.start()+ expect(cores).toHaveLength(0)+ expect(asked[0].state).toEqual({ restarts: 0, lastExit: null, lastExitReason: null, update: null, claimed: false, reexecs: 0, unproven: 0 })+ asked[0].proceed('same')+ expect(cores).toHaveLength(1)+ core().up()+ expect(calls).toEqual(['claim', 'up'])+ crash()+ expect(asked).toHaveLength(2)+ asked[1].proceed('kept')+ expect(cores).toHaveLength(2)+ expect(supervisor.status().state).toBe('restarting')+ })++ it('hands an update on probation, and the pid file, to the master it becomes; and starts no core itself', () => {+ const { asked, deps } = reexecs()+ const supervisor = make({}, deps)+ supervisor.start()+ asked[0].proceed('same')+ core().up()+ core().exit(CORE_EXIT_UPDATE)+ vi.advanceTimersByTime(0)+ expect(asked[1].state).toEqual({ restarts: 1, lastExit: `code ${CORE_EXIT_UPDATE}`, lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 0, unproven: 0 })+ // The process is replaced from here: nothing more of this master runs.+ expect(cores).toHaveLength(1)+ expect(supervisor.status()).toMatchObject({ state: 'restarting', corePid: null })+ })++ it('rolls an update back when the new bundle\'s master would not start, and starts the core on the bundle restored', () => {+ const { asked, deps } = reexecs()+ const supervisor = make({}, deps)+ supervisor.start()+ asked[0].proceed('same')+ core().up()+ core().exit(CORE_EXIT_UPDATE)+ vi.advanceTimersByTime(0)+ asked[1].proceed('refused')+ expect(calls).toEqual(['claim', 'up', 'restore'])+ expect(lines.at(-1)).toBe('[harnessd] the new bundle\'s master would not start — rolled back to the previous bundle; restarting')+ expect(asked[2].state.update).toBeNull()+ asked[2].proceed('same')+ core().up()+ vi.runOnlyPendingTimers()+ expect(calls).toEqual(['claim', 'up', 'restore', 'up'])+ // A bundle replaced by hand has no update to fail: its core starts under this master all the same.+ crash()+ asked[3].proceed('refused')+ expect(cores).toHaveLength(3)+ })++ it('rolls a failed update back and asks again, so the master moves back to the bundle restored', () => {+ const { asked, deps } = reexecs()+ const supervisor = make({}, deps)+ supervisor.start()+ asked[0].proceed('same')+ core().up()+ core().exit(CORE_EXIT_UPDATE)+ vi.advanceTimersByTime(0)+ asked[1].proceed('same')+ core().exit(3)+ expect(calls).toEqual(['claim', 'up', 'restore'])+ vi.advanceTimersByTime(0)+ expect(asked[2].state).toMatchObject({ update: null, restarts: 2, lastExit: 'code 3', lastExitReason: 'crashed' })+ expect(supervisor.status().restarts).toBe(2)+ })++ it('starts nothing when it was stopped while deciding', () => {+ const { asked, deps } = reexecs()+ const supervisor = make({}, deps)+ supervisor.start()+ supervisor.stop('SIGTERM')+ expect(exited).toEqual([0])+ asked[0].proceed('same')+ asked[0].proceed('refused')+ expect(cores).toHaveLength(0)+ })++ it('carries on as the master it was, once re-executed: its counts, its claim, its update on probation', () => {+ const { asked, deps } = reexecs()+ const resume: ResumeState = { restarts: 4, lastExit: 'code 75', lastExitReason: 'update', update: 'pending', claimed: true, reexecs: 2, unproven: 1 }+ const supervisor = make({}, deps, { version: '9.9.9', resume })+ expect(supervisor.status()).toMatchObject({ restarts: 4, lastExit: 'code 75', lastExitReason: 'update', masterVersion: '9.9.9', reexecs: 2 })+ supervisor.start()+ asked[0].proceed('same')+ expect(core().env.HARNESSD_RESTARTS).toBe('4')+ core().up()+ // Its pid file was claimed by the master it was: the same process.+ expect(calls).toEqual(['up'])+ vi.advanceTimersByTime(options.updateProbationMs)+ expect(calls).toEqual(['up', 'confirm'])+ crash()+ // A core came up: the re-executions so far are proven, and no longer count towards a loop.+ expect(asked[1].state).toMatchObject({ reexecs: 2, unproven: 0 })+ })++ it('counts a protocol 1 core as up once it is bound', () => {+ const { asked, deps } = reexecs()+ make({}, deps).start()+ asked[0].proceed('same')+ core().bind(1)+ expect(calls).toEqual(['claim', 'up'])+ })+ })})diff --git a/cli/src/harnessd/supervisor.ts b/cli/src/harnessd/supervisor.tsindex 4817d63c3..966eca7c2 100644--- a/cli/src/harnessd/supervisor.ts+++ b/cli/src/harnessd/supervisor.ts@@ -49,6 +49,46 @@ export interface SupervisorDeps {writeStatus(status: SupervisorStatus): voidlog(line: string): voidexit(code: number): void+ /**+ * Replace this master with the bundle on disk before a core is started on it, carrying `state`+ * (./reexec.ts): asked in the gap before every core start, when no core is running, so the swap+ * orphans nothing. `proceed` is called when it did not replace itself, and why. Left out, never.+ */+ reexec?(state: ResumeState, proceed: (outcome: ReexecOutcome) => void): void+ /** A core is up: ready, or bound for a protocol 1 core. A re-executed master has proved itself. */+ coreUp?(): void+}++/**+ * What a master hands the one that replaces it in its own process (./reexec.ts). The crash times and+ * the backoff are left behind: they are on the old process's monotonic clock, which a new one restarts.+ */+export interface ResumeState {+ restarts: number+ lastExit: string | null+ lastExitReason: ExitReason | null+ /** `pending`: the core about to start runs an update, on probation once it is ready. */+ update: 'pending' | null+ /** The pid file is this process's already. */+ claimed: boolean+ /** How many times this process has re-executed. */+ reexecs: number+ /** Of those, how many since a core last came up: a master that keeps re-executing without one is+ * in a loop, and stops (./reexec.ts `REEXEC_LIMIT`). */+ unproven: number+}++/**+ * Why a master did not replace itself: the bundle on disk is its own code (`same`); it could not or+ * would not, for a reason that is not the bundle's (`kept`); or the bundle's master did not answer its+ * probe (`refused`), which fails an update to it.+ */+export type ReexecOutcome = 'same' | 'kept' | 'refused'++/** Who this master is: its code's version, and what the master it replaced handed on. */+export interface SupervisorIdentity {+ version?: string+ resume?: ResumeState | null}export interface SupervisorOptions {@@ -112,6 +152,10 @@ export interface SupervisorStatus {protocol: number/** When the state last changed, wall clock ms. */since: number+ /** The version of the code this master runs, when it knows it. */+ masterVersion: string | null+ /** How many times this master's process has replaced itself with a newer or restored bundle. */+ reexecs: number}type TimerName = 'bindTimer' | 'readyTimer' | 'heartbeatTimer' | 'killTimer' | 'restartTimer' | 'probationTimer'@@ -151,10 +195,30 @@ export class Supervisor {private heartbeatTimer: unknown = nullprivate killTimer: unknown = nullprivate restartTimer: unknown = null+ private readonly masterVersion: string | null+ private reexecs = 0+ private unproven = 0- constructor(private readonly deps: SupervisorDeps, private readonly options: SupervisorOptions = DEFAULT_SUPERVISOR_OPTIONS) {+ constructor(+ private readonly deps: SupervisorDeps,+ private readonly options: SupervisorOptions = DEFAULT_SUPERVISOR_OPTIONS,+ identity: SupervisorIdentity = {},+ ) {this.backoff = options.initialBackoffMsthis.since = deps.wallClock()+ this.masterVersion = identity.version ?? null+ const resume = identity.resume+ if (resume) {+ // The master this process was a moment ago, carried on: its core exited for an update (or was+ // rolled back), and the one about to start is judged as it would have judged it.+ this.restarts = resume.restarts+ this.lastExit = resume.lastExit+ this.lastExitReason = resume.lastExitReason+ this.update = resume.update+ this.claimed = resume.claimed+ this.reexecs = resume.reexecs+ this.unproven = resume.unproven+ }}status(): SupervisorStatus {@@ -167,12 +231,45 @@ export class Supervisor {safeMode: this.coreSafeMode ?? this.safeMode,protocol: HARNESSD_PROTOCOL,since: this.since,+ masterVersion: this.masterVersion,+ reexecs: this.reexecs,}}start(): void {if (this.state !== 'idle') return- this.spawn()+ this.next()+ }++ /** What this master hands on if it replaces itself now. */+ private resumeState(): ResumeState {+ return {+ restarts: this.restarts, lastExit: this.lastExit, lastExitReason: this.lastExitReason,+ update: this.update === 'pending' ? 'pending' : null, claimed: this.claimed, reexecs: this.reexecs, unproven: this.unproven,+ }+ }++ /**+ * Start the next core, in the one moment no core is running: first, if the bundle on disk is not+ * this master's code (an update the core staged, a rollback that put the previous one back, a bundle+ * replaced by hand), the master replaces itself with it and the new master starts the core. A master+ * left running old code supervised every core after it with that code until something restarted it.+ */+ private next(): void {+ if (!this.deps.reexec) { this.spawn(); return }+ this.deps.reexec(this.resumeState(), (outcome) => {+ // Stopped while it was deciding: the stop has finished this master already.+ if (this.state === 'stopping' || this.state === 'stopped') return+ if (outcome === 'refused' && this.update === 'pending') {+ // A bundle whose master cannot start is an update that failed, as one whose core cannot is.+ this.update = null+ this.deps.restoreUpdate()+ this.deps.log('[harnessd] the new bundle\'s master would not start — rolled back to the previous bundle; restarting')+ this.next()+ return+ }+ this.spawn()+ })}/** Stop the core and the master. A second call while stopping kills the core outright. */@@ -241,7 +338,7 @@ export class Supervisor {this.armTimer('readyTimer', () => this.kill(core, 'not-ready', `bound but not ready within ${this.options.readyTimeoutMs} ms`), this.options.readyTimeoutMs)}if (this.state !== 'stopping') this.setState(readiness ? 'listening' : 'running')- if (!readiness) this.beginProbation()+ if (!readiness) this.up()return}case 'harnessd:ready':@@ -256,7 +353,7 @@ export class Supervisor {this.coreSafeMode = message.safeMode ?? nullthis.deps.log(`[harnessd] core ready (pid ${core.pid ?? '?'})${message.safeMode === undefined ? '' : ` · in safe mode: ${message.safeMode}`}`)this.setState('running')- this.beginProbation()+ this.up()returncase 'harnessd:heartbeat': {if (!this.bound) return@@ -273,6 +370,13 @@ export class Supervisor {}}+ /** A core is up: its update goes on probation, and a re-executed master has proved itself. */+ private up(): void {+ this.beginProbation()+ this.unproven = 0+ this.deps.coreUp?.()+ }+private restartForMemory(why: string): void {this.deps.log(`[harnessd] core ${why} — restarting it`)this.killReason = 'memory'@@ -358,7 +462,7 @@ export class Supervisor {private scheduleSpawn(delay: number): void {this.setState('restarting')- this.armTimer('restartTimer', () => { this.restartTimer = null; this.spawn() }, delay)+ this.armTimer('restartTimer', () => { this.restartTimer = null; this.next() }, delay)}/** Kill a core that broke a promise — to bind, to be ready, to beat — and remember which. */diff --git a/docs/design/2026-10-03-harnessd.md b/docs/design/2026-10-03-harnessd.mdindex a5d05f148..ae207e4b4 100644--- a/docs/design/2026-10-03-harnessd.md+++ b/docs/design/2026-10-03-harnessd.md@@ -325,11 +325,13 @@ the parts still to build say so.with a deadline the master enforces; `harness stop` waits for the master instead of killing itafter a fixed three seconds. A fast stop leaves agents running in tmux; that is the only kind.- **Updates.** The core's updater stages a bundle and exits 75; the master restarts the core on it,- on probation from `ready`, and rolls back if it fails. Next: when the master's own code changed,- it re-executes itself in the gap between the old core's exit and the new core's start, as- Syncthing's monitor does — re-executing with a core running would cost that core its channel.- Bundles move to versioned directories behind one pointer, so a swap is one rename, and a version- that was rolled back is not offered again.+ on probation from `ready`, and rolls back if it fails. When the master's own code changed, it+ re-executes itself in the gap between the old core's exit and the new core's start, as Syncthing's+ monitor does — re-executing with a core running would cost that core its channel. Built+ (`harnessd/reexec.ts`): in place, keeping its pid, so launchd, systemd, the pid file and the app's+ owner check never see the master change; a probe of the bundle's own master first; a marker that+ rolls back a master that died after re-executing. Next: bundles move to versioned directories+ behind one pointer, so a swap is one rename. A version that was rolled back is not offered again.- **Something supervises the supervisor.** On macOS a launchd user agent (registered through`SMAppService`, which asks the user once) and on Linux a systemd user unit run the master in theforeground and restart it; the desktop app then watches instead of respawning, so there is one@@ -562,6 +564,31 @@ Done so far:`HARNESSD_TEST_PLATFORM`) see start and stop go through them; another data folder's definition isignored.+- **The master re-executes itself on the bundle on disk** (`cli/src/harnessd/reexec.ts`, at 100%).+ Before, an update replaced the core and left the master on its old code until something restarted+ it. Now, in the gap before any core starts (an update's exit 75, a rollback, a bundle replaced by+ hand), a master whose code is not the bundle on disk replaces itself with it (`process.execve`).+ - In place, not by handing over to a fresh master: launchd and systemd follow the pid they started,+ and to them a master that exits is a job that ended; the pid file, `harness stop` and the desktop+ app's owner check follow it too. Only in the gap, with the services stopped and reaped first: no+ core loses its channel and nothing is orphaned. The agents live in tmux, out of reach of it all.+ - The state handed on (`HARNESSD_RESUME`, kept from every child): restarts, the last exit, the update+ on probation, the pid file's claim, and re-executions since a core last came up (after three, the+ master keeps its code). The status and `/api/status` gain the master's version and that count.+ - Exec gives up the old master as a fallback, so: a probe (`harness __harnessd-probe`) asks the+ bundle's own master first, and one that fails it fails the update, rolled back and remembered in+ `update-rejected.json` like a failing core; a bundle from before this answers "Unknown command"+ and is never started as a probe. And a marker (`harnessd-reexec.json`), removed once the new+ master's first core is up: a master that starts and finds one left by a process that is gone rolls+ back and moves onto the bundle before. Without `process.execve` (before Node 22.15 and 23.11) the+ master keeps its code, as before.+ - `e2e/reexec.e2e.ts`, on real bundles: the pid stays through every swap while the master's version+ follows the bundle; the core and an agent at work carry on; a build whose master fails its probe is+ rolled back and not staged again; one whose master dies after re-executing is rolled back at the+ next start; every core is the master's child throughout. With a released bundle+ (`MIGRATION_FROM`): an older master supervises the newer core an update brings until it restarts,+ and this master runs an older bundle's core without re-executing on it.+Next, in order:1. **Removals:** requests kept only to refuse retired features, once no shipped app sends them.@@ -574,8 +601,8 @@ Next, in order:to in-process for one release: devices first, then relay (with the self-hosting settings),models, workspaces, search, viewers, teams, the light host, Companion and Memory.5. **Platform supervision by default** (built as an opt-in above; the desktop app watching instead of- respawning, registered through `SMAppService` on macOS) and the master re-executing itself on- update.+ respawning, registered through `SMAppService` on macOS). The other half of this step, the master+ re-executing itself on update, is built (above).6. **State directory move,** last, once nothing else is changing.### The core boundary, step by step