Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

harnessd: services in their own processes, starting with search; rules for coding agents and a test that holds the walls (#757)

Deeed780a6982c8
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 000000000..6a4029179
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,2 @@
+<!-- One set of rules for every agent: Codex reads AGENTS.md, and Claude Code reads it through this import. Claude Code reads AGENTS.md by itself only from v2.1.277, and only when no CLAUDE.md is in the working directory or above it, so without this a session started in a folder with a CLAUDE.md would miss these rules. -->
diff --git a/cli/AGENTS.md b/cli/AGENTS.md
new file mode 100644
index 000000000..3f3788152
--- /dev/null
+++ b/cli/AGENTS.md
@@ -0,0 +1,60 @@
+# The Harness CLI and daemon (harnessd)
+
+Read this before changing anything under `cli/`. The repository-wide process rules (validation,
+merging, releases) are in [../AGENTS.md](../AGENTS.md). The design, and why, is in
+[../docs/design/2026-10-03-harnessd.md](../docs/design/2026-10-03-harnessd.md). The one-page picture
+for new contributors is [../docs/design/2026-10-04-harnessd-before-after.md](../docs/design/2026-10-04-harnessd-before-after.md).
+
+## The shape: master, core, services
+
+```
+MASTER src/harnessd/ supervises: starts, watches, restarts. No feature code, no network.
+CORE src/core/ owns sessions: agents, terminals, transcripts, turns, input, questions.
+ Must never go down. Grows only for what every session needs.
+SERVICES src/services/ everything else: search, viewers, models, workspaces, … and new features.
+ Reaches the core only through core/api.ts; can fail without the core.
+```
+
+`src/cli.ts` `runForeground()` is the composition root: it creates the modules and wires them
+together. `src/backendSocket.ts` is the transport: it receives frames and dispatches them.
+
+## Where new code goes
+
+| You are adding | Put it in | Not in |
+|---|---|---|
+| A new feature (anything a session can run without) | a new service, `src/services/<name>.ts` | the core, `cli.ts`, `backendSocket.ts` |
+| Behaviour of agents, terminals, transcripts, turns, input or questions | the module under `src/core/` that owns it | `cli.ts` |
+| Support for an engine (Claude Code, Codex, …) | `src/engines/<engine>/` | the core |
+| A pure helper with no daemon state | `src/lib/` | the core |
+| Supervision of processes | `src/harnessd/` | anywhere else |
+
+## Rules
+
+1. **No logic in `runForeground` or the `backendSocket.ts` request switch.** Wiring and dispatch only:
+ a handler there is one call into a module or service. `src/architecture.spec.ts` fails when either
+ grows; move the logic out instead of raising the budget.
+2. **A feature is a service.** It runs against `CoreApi` and is reached through a port in `CorePorts`,
+ both in `src/core/api.ts`. A service never imports core modules, the registry, `cli.ts` or
+ `backendSocket.ts` (`src/architecture.spec.ts` checks it). See [src/services/AGENTS.md](src/services/AGENTS.md).
+3. **The core does not wait on a service and does not crash with one.** Services start through
+ `serviceHost.start()`; every port declares fallbacks beside it in `core/api.ts`.
+4. **100% coverage, per file,** for `src/core/`, `src/services/` (`npm run test:core`) and `src/harnessd/`
+ (`npm run test:harnessd`). CI enforces both. Write the test that fails without your change.
+5. **End to end for every user-facing flow** (`npm run test:e2e`, `e2e/`): the real daemon, a private
+ tmux server and fake Claude Code and Codex engines (`e2e/harness/fakeEngine.mjs`). Prefer extending
+ the fake engine faithfully over loosening a test.
+6. **Never test against the developer's own machine.** Tests use a throwaway home and data folder:
+ never the real `~/.claude`, `~/.codex`, daemon (port 18473) or tmux server. In tmux tests unset
+ `TMUX` and `TMUX_PANE`, and point `TMUX_TMPDIR` at a folder that exists (`src/testing/isolatedTmux.ts`).
+7. **Comments say why, in plain sentences.** Name the incident or the measurement that made the code
+ the way it is; that history is what keeps the next change from undoing it.
+
+## Commands
+
+```bash
+npm run typecheck # tsc
+npm test # the unit suite
+npm run test:core # src/core and src/services at 100%
+npm run test:harnessd # src/harnessd at 100%
+npm run test:e2e # the real daemon, end to end
+```
diff --git a/cli/CLAUDE.md b/cli/CLAUDE.md
new file mode 100644
index 000000000..6a4029179
--- /dev/null
+++ b/cli/CLAUDE.md
@@ -0,0 +1,2 @@
+<!-- One set of rules for every agent: Codex reads AGENTS.md, and Claude Code reads it through this import. Claude Code reads AGENTS.md by itself only from v2.1.277, and only when no CLAUDE.md is in the working directory or above it, so without this a session started in a folder with a CLAUDE.md would miss these rules. -->
diff --git a/cli/e2e/serviceProcesses.e2e.ts b/cli/e2e/serviceProcesses.e2e.ts
new file mode 100644
index 000000000..d6d65d745
--- /dev/null
+++ b/cli/e2e/serviceProcesses.e2e.ts
@@ -0,0 +1,153 @@
+/**
+ * Services in their own processes, for Claude Code and Codex, on the real daemon: harnessd's master runs
+ * search beside the core (`HARNESSD_SERVICES=search`), and whatever happens to search — killed outright,
+ * hung, leaking memory, crashing on every start — costs search alone. The core never restarts, every
+ * agent keeps working, a search asked while it is down is answered SERVICE_UNAVAILABLE at once, and the
+ * master brings search back (or parks it, when it keeps crashing).
+ */
+import { execFileSync } from 'node:child_process'
+import { mkdirSync } from 'node:fs'
+import { join } from 'node:path'
+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'
+import { LocalClient, type Frame } from './harness/client.js'
+import { IsolatedDaemon, until } from './harness/daemon.js'
+
+type Engine = 'claude' | 'codex'
+
+const row = async (client: LocalClient, agentId: string) =>
+ ((await client.request<{ agents: Array<Record<string, any>> }>('agents_list', { includeStopped: true }, 30_000)).agents)
+ .find((agent) => agent.id === agentId)
+async function create(daemon: IsolatedDaemon, client: LocalClient, engine: Engine, folder: string): Promise<Record<string, any>> {
+ const cwd = join(daemon.projectsDir, folder)
+ mkdirSync(cwd, { recursive: true })
+ const created = await client.request('agent_create', { engine, cwd, bypassPermission: true }, 90_000)
+ expect(created.error, `${folder}: ${JSON.stringify(created)}`).toBeUndefined()
+ return until(`${folder} to bind its conversation`, async () => {
+ const agent = await row(client, created.agent.id)
+ return agent?.sessionId && agent.status === 'active' ? agent : null
+ }, 60_000, 500)
+}
+const isTurn = (type: string, agentId: string) => (frame: Frame) => frame.type === type && frame.agentId === agentId
+async function turn(client: LocalClient, agentId: string, content: string): Promise<void> {
+ const ended = client.next(isTurn('turn_ended', agentId), 45_000, `turn_ended (${content})`)
+ client.send('message', { agentId, content })
+ await ended
+}
+/** The search service's own process, from the process table: it is `harnessd search`. */
+function searchPids(): number[] {
+ const table = execFileSync('ps', ['-A', '-o', 'pid=,command=']).toString().trim().split('\n')
+ return table.map((line) => line.trim().match(/^(\d+)\s+(.*)$/)).filter((match): match is RegExpMatchArray => !!match)
+ .filter(([, , command]) => command.trim() === 'harnessd search').map(([, pid]) => Number(pid))
+}
+const finds = async (client: LocalClient, word: string, sessionId: string): Promise<boolean> =>
+ JSON.stringify(await client.request('session_search', { query: word }, 30_000)).includes(sessionId)
+
+describe('services in their own processes', () => {
+ let daemon: IsolatedDaemon | undefined
+ afterEach(async () => { await daemon?.close(); daemon = undefined })
+ const fresh = async (env: Record<string, string> = {}) => {
+ const d = await IsolatedDaemon.create({ env: {
+ HARNESSD_SERVICES: 'search',
+ HARNESSD_SERVICE_INITIAL_BACKOFF_MS: '200',
+ HARNESSD_SERVICE_MAX_BACKOFF_MS: '1000',
+ ...env,
+ } })
+ daemon = d
+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-150).join('\n')}`) })
+ await d.start()
+ return d
+ }
+ const restarts = (d: IsolatedDaemon) => d.log().split('\n').filter((line) => /\[harnessd\] service search started .* restart \d+/.test(line)).length
+
+ it('search runs in its own process and answers through the core, for both engines', async () => {
+ const d = await fresh()
+ const client = await LocalClient.connect(d)
+ await until('search to connect to the core', () => d.log().includes('[services] search connected') || null, 30_000, 200)
+ expect(searchPids().length).toBeGreaterThanOrEqual(1)
+ for (const engine of ['claude', 'codex'] as const) {
+ const agent = await create(d, client, engine, `search-out-${engine}`)
+ const word = `axolotl${engine}`
+ await turn(client, agent.id, `about the ${word}`)
+ await until(`search to find the ${engine} conversation`, () => finds(client, word, agent.sessionId) || null, 60_000, 1_000)
+ }
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ })
+
+ it('search killed outright: asked meanwhile it says so at once, agents go on, and the master brings it back', async () => {
+ const d = await fresh()
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, 'claude', 'search-killed')
+ await turn(client, agent.id, 'remember the pangolin')
+ await until('search to find it', () => finds(client, 'pangolin', agent.sessionId) || null, 60_000, 1_000)
+ const before = searchPids()
+ expect(before.length).toBeGreaterThanOrEqual(1)
+ for (const pid of before) process.kill(pid, 'SIGKILL')
+ // Asked while it is down: an answer, not a hang.
+ const answer = await client.request('session_search', { query: 'pangolin' }, 10_000)
+ if (answer.error) expect(answer).toMatchObject({ error: 'SERVICE_UNAVAILABLE', service: 'search', retryable: true })
+ // The agents never noticed.
+ await turn(client, agent.id, 'while search was gone')
+ await until('the master to restart search', () => restarts(d) >= 1 || null, 30_000, 200)
+ await until('search to answer again', () => finds(client, 'pangolin', agent.sessionId) || null, 60_000, 1_000)
+ expect(searchPids().some((pid) => !before.includes(pid))).toBe(true)
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ })
+
+ it('a hung search is killed by the master\'s heartbeat watch and started again; the core goes on', async () => {
+ const d = await fresh({ HARNESSD_SERVICE_HEARTBEAT_TIMEOUT_MS: '2000' })
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, 'codex', 'search-hung')
+ await until('search to connect', () => d.log().includes('[services] search connected') || null, 30_000, 200)
+ // Hung: a stopped process neither beats nor answers, as one stuck in a loop would not.
+ for (const pid of searchPids()) process.kill(pid, 'SIGSTOP')
+ await until('the master to find search hung', () => d.log().includes('[harnessd] service search sent no heartbeat') || null, 30_000, 200)
+ await until('search to be started again', () => restarts(d) >= 1 || null, 30_000, 200)
+ await turn(client, agent.id, 'the core never waited on search')
+ const answer = await client.request('session_search', { query: 'anything' }, 40_000)
+ expect(answer.error === undefined || answer.error === 'SERVICE_UNAVAILABLE', JSON.stringify(answer)).toBe(true)
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ })
+
+ it('a leaking search is restarted at its memory budget, before it can hurt anything else', async () => {
+ const d = await fresh({ HARNESSD_TEST_FAULTS: 'search.leak', HARNESSD_SERVICE_HEAP_LIMIT_MIB: '128' })
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, 'claude', 'search-leaking')
+ await until('the master to restart search for memory', () => /\[harnessd\] service search: (its heap is at|it is using)/.test(d.log()) || null, 60_000, 250)
+ await until('search to be started again', () => restarts(d) >= 1 || null, 30_000, 200)
+ await turn(client, agent.id, 'the leak was search\'s alone')
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ })
+
+ it('a search that crashes on every start is parked, and asked meanwhile says so; agents go on', async () => {
+ const d = await fresh({ HARNESSD_TEST_FAULTS: 'search.crash', HARNESSD_SERVICE_PARK_CRASHES: '3' })
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, 'codex', 'search-crash-loop')
+ await until('the master to park search', () => d.log().includes('[harnessd] service search ended 3 times') || null, 60_000, 250)
+ expect(await client.request('session_search', { query: 'anything' }, 10_000)).toMatchObject({ error: 'SERVICE_UNAVAILABLE', service: 'search', retryable: true })
+ await turn(client, agent.id, 'search is parked and nothing else cares')
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ })
+
+ it('a purge while search is down is not forgotten: search forgets the conversation once it is back', async () => {
+ const d = await fresh()
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, 'claude', 'purge-while-down')
+ await turn(client, agent.id, 'remember the quokka')
+ await until('search to find it', () => finds(client, 'quokka', agent.sessionId) || null, 60_000, 1_000)
+ // Down: killed, and the purge happens before the master has it back.
+ for (const pid of searchPids()) process.kill(pid, 'SIGSTOP')
+ const createdAt = Date.parse(agent.createdAt)
+ const review = await client.request('agent_purge', { agentId: agent.id, sessionId: agent.sessionId, createdAt, mode: 'inspect' }, 60_000)
+ for (const pid of searchPids()) process.kill(pid, 'SIGKILL')
+ const deleted = await client.request('agent_purge', { agentId: agent.id, sessionId: agent.sessionId, createdAt, mode: 'delete', reviewId: review.reviewId }, 90_000)
+ expect(deleted, JSON.stringify(deleted)).toMatchObject({ deleted: true })
+ await until('search to be back', () => restarts(d) >= 1 && d.log().split('[services] search connected').length >= 3 || null, 60_000, 250)
+ await until('search to have forgotten it', async () => !(await finds(client, 'quokka', agent.sessionId)) || null, 30_000, 1_000)
+ client.close()
+ })
+})
diff --git a/cli/src/architecture.spec.ts b/cli/src/architecture.spec.ts
new file mode 100644
index 000000000..41a3f8680
--- /dev/null
+++ b/cli/src/architecture.spec.ts
@@ -0,0 +1,119 @@
+/**
+ * The daemon's shape, checked: master, core, services (cli/AGENTS.md). People and their coding agents
+ * build features in parallel on it, and a rule nobody checks is a rule the next change breaks quietly.
+ * So the boundaries are tests: a service reaches the core only through `core/api.ts`, the core never
+ * reaches into a service, the master holds no feature code, and the two files every change used to land
+ * in — `runForeground` and the socket's request switch — may not grow back.
+ *
+ * When this fails, the message says where the code belongs. Move it there; do not widen the rule.
+ */
+import { readdirSync, readFileSync } from 'node:fs'
+import { join, relative } from 'node:path'
+import ts from 'typescript'
+import { describe, expect, it } from 'vitest'
+
+const SRC = __dirname
+
+interface Import {
+ file: string
+ from: string
+ typeOnly: boolean
+}
+
+/** Every import and re-export in a folder's source (not its tests), and whether it is types only. */
+function importsIn(folder: string): Import[] {
+ const found: Import[] = []
+ const walk = (dir: string): void => {
+ for (const entry of readdirSync(dir, { withFileTypes: true })) {
+ const path = join(dir, entry.name)
+ if (entry.isDirectory()) { walk(path); continue }
+ if (!entry.name.endsWith('.ts') || entry.name.endsWith('.spec.ts') || entry.name.endsWith('.test.ts')) continue
+ const source = ts.createSourceFile(path, readFileSync(path, 'utf8'), ts.ScriptTarget.Latest, true)
+ for (const statement of source.statements) {
+ if ((ts.isImportDeclaration(statement) || ts.isExportDeclaration(statement)) && statement.moduleSpecifier && ts.isStringLiteral(statement.moduleSpecifier)) {
+ let typeOnly = false
+ if (ts.isImportDeclaration(statement)) {
+ const clause = statement.importClause
+ const named = clause?.namedBindings && ts.isNamedImports(clause.namedBindings) ? clause.namedBindings.elements : null
+ typeOnly = !!clause && (clause.isTypeOnly || (!clause.name && !!named && named.length > 0 && named.every((element) => element.isTypeOnly)))
+ } else {
+ typeOnly = statement.isTypeOnly
+ }
+ found.push({ file: relative(SRC, path), from: statement.moduleSpecifier.text, typeOnly })
+ }
+ }
+ }
+ }
+ walk(join(SRC, folder))
+ return found
+}
+
+/** The lines `runForeground` spans in cli.ts. */
+function runForegroundLines(): number {
+ const lines = readFileSync(join(SRC, 'cli.ts'), 'utf8').split('\n')
+ const start = lines.findIndex((line) => line.startsWith('async function runForeground('))
+ const end = lines.findIndex((line, index) => index > start && line === '}')
+ return end - start
+}
+
+/**
+ * The most each may grow to. Today's size and a little room for wiring. Raising a budget needs a reason
+ * a reviewer agrees with; the usual one is wrong, and the code belongs in a module or a service.
+ */
+const RUN_FOREGROUND_BUDGET = 2_700
+const BACKEND_SOCKET_BUDGET = 3_600
+
+/** Exceptions, each with its reason. Keep this short. */
+const SERVICE_MAY_IMPORT: Record<string, string> = {
+ // The search process builds, in its own process, the core API search runs on; this reader is a pure
+ // function of a session row, the same one the core hands search through CoreApi.
+ 'services/searchProcess.ts → ../core/transcripts/databaseHistory.js': 'the core API search runs on, built in its own process',
+ // A pure function of a session row. Move it out of registry.ts when workspaces leaves the core's process.
+ 'services/workspaces.ts → ../lib/registry.js': 'sessionDisplayTitle, a pure helper',
+}
+
+describe('the daemon\'s shape', () => {
+ it('a service reaches the core only through core/api.ts: never a core module, the registry, cli.ts or the socket', () => {
+ const wrong = importsIn('services').filter(({ file, from, typeOnly }) => {
+ if (SERVICE_MAY_IMPORT[`${file} → ${from}`]) return false
+ if (/(^|\/)core\//.test(from)) return from !== '../core/api.js'
+ if (/(^|\/)(cli|backendSocket|localWsServer)\.js$/.test(from)) return true
+ if (/(^|\/)lib\/registry\.js$/.test(from)) return !typeOnly
+ return false
+ }).map(({ file, from }) => `${file} imports ${from}`)
+ expect(wrong, 'A service may use the core only through CoreApi (src/core/api.ts). If CoreApi lacks it, add it there in its own change (src/services/AGENTS.md).').toEqual([])
+ })
+
+ it('the core never reaches into a service, cli.ts or the socket, but for types', () => {
+ const wrong = importsIn('core').filter(({ from, typeOnly }) =>
+ /(^|\/)services\//.test(from) || (/(^|\/)(cli|backendSocket|localWsServer)\.js$/.test(from) && !typeOnly))
+ .map(({ file, from }) => `${file} imports ${from}`)
+ expect(wrong, 'The core calls services only through CorePorts, and is handed the socket\'s pieces as dependencies (src/core/AGENTS.md).').toEqual([])
+ })
+
+ it('the master holds no feature code: Node itself, its own folder, and the log trimmer', () => {
+ const wrong = importsIn('harnessd').filter(({ from }) => !from.startsWith('node:') && !from.startsWith('./') && from !== '../lib/log.js')
+ .map(({ file, from }) => `${file} imports ${from}`)
+ expect(wrong, 'The master is the one process that must not fail: no feature code in it (src/harnessd/AGENTS.md).').toEqual([])
+ })
+
+ it('runForeground and the socket\'s request switch do not grow back', () => {
+ const runForeground = runForegroundLines()
+ const backendSocket = readFileSync(join(SRC, 'backendSocket.ts'), 'utf8').split('\n').length
+ expect(runForeground, `runForeground is ${runForeground} lines, over its ${RUN_FOREGROUND_BUDGET}: it wires modules together. Put behaviour in a core module (src/core/) or a service (src/services/).`).toBeLessThanOrEqual(RUN_FOREGROUND_BUDGET)
+ expect(backendSocket, `backendSocket.ts is ${backendSocket} lines, over its ${BACKEND_SOCKET_BUDGET}: it is transport. A request's handler is one call into a module or a service.`).toBeLessThanOrEqual(BACKEND_SOCKET_BUDGET)
+ })
+
+ it('finds what it checks: imports of every kind, in every folder', () => {
+ const services = importsIn('services')
+ expect(services.some(({ from, typeOnly }) => from === '../core/api.js' && typeOnly)).toBe(true)
+ expect(services.some(({ from, typeOnly }) => from === '../core/api.js' && !typeOnly)).toBe(true)
+ expect(importsIn('core').some(({ from, typeOnly }) => /backendSocket\.js$/.test(from) && typeOnly)).toBe(true)
+ expect(importsIn('harnessd').some(({ from }) => from.startsWith('node:'))).toBe(true)
+ expect(runForegroundLines()).toBeGreaterThan(1_000)
+ for (const exception of Object.keys(SERVICE_MAY_IMPORT)) {
+ const [file, from] = exception.split(' → ')
+ expect(services.some((found) => found.file === file && found.from === from), `${exception} is no longer needed: remove it`).toBe(true)
+ }
+ })
+})
diff --git a/cli/src/backendSocket.ts b/cli/src/backendSocket.ts
index 09491f346..d94b61ba0 100644
--- a/cli/src/backendSocket.ts
+++ b/cli/src/backendSocket.ts
@@ -1436,6 +1436,9 @@ export class BackendSocket {
localClientIds(): string[] { return [...this.localClients.keys()].filter((connId) => !this.toolClients.has(connId)) }
/** A window (or `hn`) on this computer attached or went away — the pair brain thinks only while one is here. */
+ /** Routes a request to the out-of-process service that answers it (core/serviceLinks.ts `route`):
+ * false when none does and the socket answers it itself. */
+ serviceRouter: ((type: string, payload: Record<string, unknown>, reply: (result: Record<string, unknown>) => void) => boolean) | null = null
onLocalClient: ((connId: string, attached: boolean) => void) | null = null
/** Release all connection-scoped state when the loopback WebSocket closes. */
@@ -2064,6 +2067,10 @@ export class BackendSocket {
return
}
+ // A request a service answers in its own process (core/serviceLinks.ts): routed to it, or answered
+ // SERVICE_UNAVAILABLE while it is down. Never waited on in line: the next frame is not held for it.
+ if (this.serviceRouter?.(type, payload, (result) => reply(type, requestId, result))) return
+
try {
switch (type) {
case 'machine_resources':
diff --git a/cli/src/cli.ts b/cli/src/cli.ts
index 7a379e8ca..d5bf9ca96 100644
--- a/cli/src/cli.ts
+++ b/cli/src/cli.ts
@@ -164,6 +164,9 @@ import { createCursorTaskHooks } from './core/engines/cursorTasks.js'
import { databaseHistory } from './core/transcripts/databaseHistory.js'
import { createCoreApi, emptyPorts, MODELS_FALLBACKS, SEARCH_FALLBACKS, TEAMS_FALLBACKS, VIEWERS_FALLBACKS, WORKSPACES_FALLBACKS, type TeamsPort } from './core/api.js'
import { createServiceHost, testFaults } from './core/serviceHost.js'
+import { createServiceLinks } from './core/serviceLinks.js'
+import { KNOWN_SERVICES, SERVICE_REQUESTS, serviceSpecs } from './harnessd/services.js'
+import { runSearchService } from './services/searchProcess.js'
import { startSearch } from './services/search.js'
import { startViewers } from './services/viewers.js'
import { startModels } from './services/models.js'
@@ -2147,11 +2150,40 @@ async function runForeground(session: AuthSession | null): Promise<void> {
})
backend.handoffProvider = (req) => prepareAgentHandoff(handoffDeps, req)
- // Session search (services/search.ts).
- serviceHost.start('search', startSearch, coreApi, SEARCH_FALLBACKS)
+ // The services harnessd's master runs in their own processes (harnessd/services.ts, `HARNESSD_SERVICES`):
+ // only under a master, which is what gives this core the token they connect with. Their requests are
+ // routed to them, and answered SERVICE_UNAVAILABLE while they are down (core/serviceLinks.ts).
+ const serviceToken = process.env.HARNESSD_SUPERVISED === '1' ? process.env.HARNESSD_SERVICE_TOKEN : undefined
+ const outOfProcess = new Set(serviceToken ? serviceSpecs(process.env, KNOWN_SERVICES).map((spec) => spec.name) : [])
+ const serviceLinks = createServiceLinks({
+ token: serviceToken,
+ owned: Object.fromEntries([...outOfProcess].map((name) => [name, SERVICE_REQUESTS[name] ?? []])),
+ // Every agent, live then stopped, with the name the apps show for it: what search indexes.
+ answer: (_service, query) => query === 'agents'
+ ? { agents: coreApi.agents.all().map((session) => ({ ...session, displayName: coreApi.agents.displayName(session) })) }
+ : { error: 'UNKNOWN_QUERY' },
+ })
+ backend.serviceRouter = (type, payload, reply) => serviceLinks.route(type, payload, reply)
+
+ // Session search (services/search.ts): in this process, or in its own (services/searchProcess.ts),
+ // where the core tells it what changed. A purge's forgetting waits for it if it is down.
+ if (outOfProcess.has('search')) {
+ ports.search = {
+ touch: (sessionId) => { serviceLinks.notify('search', { type: 'service_event', payload: { kind: 'touch', sessionId } }) },
+ deleteHistory: (sessionId) => { serviceLinks.notify('search', { type: 'service_event', payload: { kind: 'deleteHistory', sessionId } }, { untilDelivered: true }) },
+ session: () => undefined,
+ // Never asked: its requests are routed to its process before they reach a handler here.
+ search: () => { throw new Error('search runs in its own process') },
+ tail: () => Promise.reject(new Error('search runs in its own process')),
+ stop: () => {},
+ }
+ } else {
+ serviceHost.start('search', startSearch, coreApi, SEARCH_FALLBACKS)
+ }
const sessionSearch = ports.search
- backend.sessionSearchProvider = sessionSearch ? (query, options) => sessionSearch.search(query, options) : null
- backend.sessionTailProvider = sessionSearch ? (sessionId, options) => sessionSearch.tail(sessionId, options) : null
+ const searchHere = sessionSearch && !outOfProcess.has('search') ? sessionSearch : null
+ backend.sessionSearchProvider = searchHere ? (query, options) => searchHere.search(query, options) : null
+ backend.sessionTailProvider = searchHere ? (sessionId, options) => searchHere.tail(sessionId, options) : null
// Search switched off: both requests say search is unavailable, as on a Node without an index.
serviceHost.onOff('search', () => { backend.sessionSearchProvider = null; backend.sessionTailProvider = null })
@@ -2908,6 +2940,7 @@ async function runForeground(session: AuthSession | null): Promise<void> {
const localWsServer = attachLocalWsServer(hookServer, {
localSocketServer: localSocket?.server ?? null,
shareRelay,
+ services: serviceLinks,
onSelectionReply: (connId, machineId, payload) => devices('window', () => windowSelection.reply(connId, machineId, payload)),
onVisitReply: (connId, machineId, payload) => devices('window', () => windowVisit.reply(connId, machineId, payload)),
onFormReply: (connId, machineId, payload) => devices('window', () => windowForm.reply(connId, machineId, payload)),
@@ -5511,6 +5544,16 @@ switch (cmd) {
confirmUpdate: () => confirmUpdate(env.ADAPTER_CLI_DIR),
})
break
+ case '__service': { // internal: a service harnessd's master runs in its own process (services/process.ts)
+ const socketPath = localSocketPath(env.ADAPTER_DATA_DIR, env.PORT)
+ if (rest[0] !== 'search' || !socketPath) {
+ console.error(`[service] ${rest[0] ?? '(none)'}: ${socketPath ? 'no such service in this build' : 'the core has no local socket to reach'}`)
+ process.exit(2)
+ }
+ process.title = `harnessd ${rest[0]}`
+ runSearchService({ dataDir: env.ADAPTER_DATA_DIR, socketPath, machineId: computerId(), token: process.env.HARNESSD_SERVICE_TOKEN ?? '' })
+ break
+ }
case '__run': // internal: the detached daemon child reads the durable SSO session — or runs without one
// NOT `onError`: a daemon that dies here can never be updated. See `enterSafeMode`.
runForeground(readAuthSession()).catch(enterSafeMode)
diff --git a/cli/src/core/AGENTS.md b/cli/src/core/AGENTS.md
new file mode 100644
index 000000000..38affafe9
--- /dev/null
+++ b/cli/src/core/AGENTS.md
@@ -0,0 +1,33 @@
+# The core
+
+The part of the daemon that must never go down: agents, terminals, transcripts, turns, input and
+questions. Everything else is a service ([../services/AGENTS.md](../services/AGENTS.md)).
+
+## Rules for a core module
+
+1. **One factory per module, explicit dependencies:** `createX(deps)` returns the module's functions.
+ Dependencies are passed in, never imported as live state, so the module is tested without a daemon.
+2. **Only what every session needs.** A feature a session can run without belongs in a service. Ask
+ before growing the core; the answer is usually a service and, at most, a new `CoreApi` member.
+3. **Never import a service, `cli.ts` or `backendSocket.ts`,** except types
+ (`import type { BackendSocket }`). The core calls services only through `CorePorts`, and every call
+ has a fallback (`api.ts`). `src/architecture.spec.ts` checks it.
+4. **Never wait on anything slow in line.** A request handler that reads a large file, runs a
+ subprocess or calls the network must be bounded in time and memory: the October 3 crash was an
+ attach that read an 803 MB transcript whole.
+5. **100% coverage, per file** (`npm run test:core`), and an end-to-end test (`../../e2e/`) for any
+ behaviour a person can see. The end-to-end suite has found every race fixed here so far: run it.
+6. **Evidence must be newer than the state it contradicts.** A scan, probe or read that began before
+ a change must not be used to undo that change (see `terminalAgentReconciler.ts` `probedAs`,
+ `routeTouched`, and `transcripts/relaunch.ts`). Most of the races found end to end were this.
+
+## Where things are
+
+- `agents/`: create, fork, restart, retarget, stop, resume, close, discovery, binding.
+- `transcripts/`: attach (bounded reads from the end), ingest, live tail, relaunch marks, normalizers.
+- `turns/`: working/idle, the event funnel, cancel, recaps, heartbeats, hooks.
+- `terminals/`: who controls a pane (the control lease).
+- `engines/`: the engines' hooks.
+- `input.ts`, `questions.ts`: messages into a pane; an agent's question and its answer.
+- `api.ts`: the contract with services. `serviceHost.ts`: services in this process.
+ `serviceLinks.ts`: services in their own processes.
diff --git a/cli/src/core/CLAUDE.md b/cli/src/core/CLAUDE.md
new file mode 100644
index 000000000..6a4029179
--- /dev/null
+++ b/cli/src/core/CLAUDE.md
@@ -0,0 +1,2 @@
+<!-- One set of rules for every agent: Codex reads AGENTS.md, and Claude Code reads it through this import. Claude Code reads AGENTS.md by itself only from v2.1.277, and only when no CLAUDE.md is in the working directory or above it, so without this a session started in a folder with a CLAUDE.md would miss these rules. -->
diff --git a/cli/src/core/serviceLinks.spec.ts b/cli/src/core/serviceLinks.spec.ts
new file mode 100644
index 000000000..c9a533b5d
--- /dev/null
+++ b/cli/src/core/serviceLinks.spec.ts
@@ -0,0 +1,185 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { createServiceLinks, HELD_MAX, type ServiceFrame } from './serviceLinks.js'
+
+const TOKEN = 'a'.repeat(48)
+
+function sink(accepting = true) {
+ const sent: ServiceFrame[] = []
+ return { sent, sendFrame: vi.fn((frame: ServiceFrame) => { sent.push(frame); return accepting }) }
+}
+
+describe('service links', () => {
+ let ids = 0
+ let lines: string[]
+ const make = (over: Partial<Parameters<typeof createServiceLinks>[0]> = {}) => createServiceLinks({
+ token: TOKEN,
+ owned: { search: ['session_search', 'session_tail'] },
+ answer: (service, query) => ({ service, query, agents: [] }),
+ timeoutMs: 5_000,
+ log: (line) => lines.push(line),
+ newId: () => `route-${++ids}`,
+ ...over,
+ })
+
+ beforeEach(() => { vi.useFakeTimers(); ids = 0; lines = [] })
+ afterEach(() => vi.useRealTimers())
+
+ it('lets in only a service it runs out of process, with the master\'s token', () => {
+ const links = make()
+ expect(links.accept('search', 'wrong-token-of-the-same-length-xxxxxxxxxxxxxxxx', sink(), vi.fn())).toBeNull()
+ expect(links.accept('search', 'short', sink(), vi.fn())).toBeNull()
+ expect(links.accept('devices', TOKEN, sink(), vi.fn())).toBeNull()
+ expect(links.accept('toString', TOKEN, sink(), vi.fn())).toBeNull()
+ expect(links.connected('search')).toBe(false)
+ expect(links.accept('search', TOKEN, sink(), vi.fn())).not.toBeNull()
+ expect(links.connected('search')).toBe(true)
+ expect(lines.filter((line) => line.includes('refused'))).toHaveLength(4)
+ // No token at all (a core no master started): no service may connect.
+ expect(make({ token: undefined }).accept('search', TOKEN, sink(), vi.fn())).toBeNull()
+ })
+
+ it('routes a request to its service and relays the answer under the asker\'s own request', async () => {
+ const links = make()
+ const search = sink()
+ const link = links.accept('search', TOKEN, search, vi.fn())!
+ const reply = vi.fn()
+ expect(links.route('session_search', { query: 'zebra', requestId: 'client-1' }, reply)).toBe(true)
+ expect(search.sent).toEqual([{ type: 'session_search', payload: { query: 'zebra', requestId: 'route-1' } }])
+ // Answers that are not this request's are ignored: another type, an unknown id, no id.
+ link.receive({ type: 'session_tail_result', payload: { requestId: 'route-1', rows: [] } })
+ link.receive({ type: 'session_search_result', payload: { requestId: 'route-9', hits: [] } })
+ link.receive({ type: 'session_search_result' })
+ expect(reply).not.toHaveBeenCalled()
+ link.receive({ type: 'session_search_result', payload: { requestId: 'route-1', hits: ['one'] } })
+ expect(reply).toHaveBeenCalledWith({ hits: ['one'] })
+ // Answered once.
+ link.receive({ type: 'session_search_result', payload: { requestId: 'route-1', hits: ['again'] } })
+ expect(reply).toHaveBeenCalledOnce()
+ // What no service owns is the core's own.
+ expect(links.route('agents_list', {}, reply)).toBe(false)
+ })
+
+ it('answers SERVICE_UNAVAILABLE at once while the service is down, and when it cannot be sent to', () => {
+ const links = make()
+ const reply = vi.fn()
+ expect(links.route('session_tail', {}, reply)).toBe(true)
+ expect(reply).toHaveBeenCalledWith({ error: 'SERVICE_UNAVAILABLE', service: 'search', retryable: true })
+ links.accept('search', TOKEN, sink(false), vi.fn())
+ const again = vi.fn()
+ links.route('session_tail', {}, again)
+ expect(again).toHaveBeenCalledWith({ error: 'SERVICE_UNAVAILABLE', service: 'search', retryable: true })
+ })
+
+ it('answers SERVICE_UNAVAILABLE when the service does not answer in time', () => {
+ const links = make()
+ links.accept('search', TOKEN, sink(), vi.fn())
+ const reply = vi.fn()
+ links.route('session_search', {}, reply)
+ vi.advanceTimersByTime(4_999)
+ expect(reply).not.toHaveBeenCalled()
+ vi.advanceTimersByTime(1)
+ expect(reply).toHaveBeenCalledWith({ error: 'SERVICE_UNAVAILABLE', service: 'search', retryable: true })
+ })
+
+ it('answers what was waiting on a service that goes, and lets a newer connection replace an older one', () => {
+ const links = make({ owned: { search: ['session_search'], devices: ['harness_devices_list'] } })
+ links.accept('devices', TOKEN, sink(), vi.fn())
+ const elsewhere = vi.fn()
+ links.route('harness_devices_list', {}, elsewhere)
+ const closeOld = vi.fn()
+ const old = links.accept('search', TOKEN, sink(), closeOld)!
+ const waiting = vi.fn()
+ links.route('session_search', {}, waiting)
+ const newer = links.accept('search', TOKEN, sink(), vi.fn())!
+ expect(closeOld).toHaveBeenCalledWith(4409, 'replaced by a newer connection')
+ // The old connection's end does not take the newer one with it.
+ old.closed()
+ expect(links.connected('search')).toBe(true)
+ expect(waiting).not.toHaveBeenCalled()
+ const pending = vi.fn()
+ links.route('session_search', {}, pending)
+ newer.closed()
+ expect(links.connected('search')).toBe(false)
+ expect(pending).toHaveBeenCalledWith({ error: 'SERVICE_UNAVAILABLE', service: 'search', retryable: true })
+ // What waits on another service is that service's business.
+ expect(elsewhere).not.toHaveBeenCalled()
+ expect(lines).toContain('[services] search disconnected')
+ })
+
+ it('answers a service\'s questions, and says when one could not be answered', async () => {
+ const links = make({
+ answer: (_service, query) => {
+ if (query === 'boom') throw new Error('no')
+ return Promise.resolve({ agents: [{ agentId: 'a' }] })
+ },
+ })
+ const search = sink()
+ const link = links.accept('search', TOKEN, search, vi.fn())!
+ link.receive({ type: 'service_query', payload: { requestId: 'q1', query: 'agents' } })
+ link.receive({ type: 'service_query', payload: { requestId: 'q2', query: 'boom' } })
+ link.receive({ type: 'service_query', payload: { requestId: 'q3', query: 7 } })
+ await vi.waitFor(() => expect(search.sent).toHaveLength(3))
+ expect(search.sent).toEqual(expect.arrayContaining([
+ { type: 'service_query_result', payload: { agents: [{ agentId: 'a' }], requestId: 'q1' } },
+ { type: 'service_query_result', payload: { error: 'QUERY_FAILED', requestId: 'q2' } },
+ { type: 'service_query_result', payload: { agents: [{ agentId: 'a' }], requestId: 'q3' } },
+ ]))
+ })
+
+ it('tells a connected service what it needs to know, and says when none is listening', () => {
+ const links = make()
+ expect(links.notify('search', { type: 'service_event', payload: { kind: 'touch' } })).toBe(false)
+ const search = sink()
+ links.accept('search', TOKEN, search, vi.fn())
+ expect(links.notify('search', { type: 'service_event', payload: { kind: 'touch' } })).toBe(true)
+ expect(search.sent).toEqual([{ type: 'service_event', payload: { kind: 'touch' } }])
+ })
+
+ it('holds what a service must hear while it is down, and says it, in order, when it connects', () => {
+ const links = make()
+ const forget = (sessionId: string) => ({ type: 'service_event', payload: { kind: 'deleteHistory', sessionId } })
+ expect(links.notify('search', forget('one'), { untilDelivered: true })).toBe(false)
+ expect(links.notify('search', forget('two'), { untilDelivered: true })).toBe(false)
+ // Not every notification is owed: a turn boundary missed is caught up by the service itself.
+ expect(links.notify('search', { type: 'service_event', payload: { kind: 'touch' } })).toBe(false)
+ // Nor to a service this core does not run out of process.
+ expect(links.notify('devices', forget('three'), { untilDelivered: true })).toBe(false)
+ // Connected but not hearing it: owed all the same.
+ const deaf = links.accept('search', TOKEN, sink(false), vi.fn())!
+ expect(links.notify('search', forget('three'), { untilDelivered: true })).toBe(false)
+ deaf.closed()
+ const search = sink()
+ links.accept('search', TOKEN, search, vi.fn())
+ expect(search.sent).toEqual([forget('three')])
+ // Said once: a later connection is not told again.
+ const again = sink()
+ links.accept('search', TOKEN, again, vi.fn())
+ expect(again.sent).toEqual([])
+ })
+
+ it('holds a bounded number for a service that stays down, dropping the oldest', () => {
+ const links = make()
+ for (let n = 0; n < HELD_MAX + 5; n++) links.notify('search', { type: 'service_event', payload: { n } }, { untilDelivered: true })
+ const search = sink()
+ links.accept('search', TOKEN, search, vi.fn())
+ expect(search.sent).toHaveLength(HELD_MAX)
+ expect(search.sent[0]).toEqual({ type: 'service_event', payload: { n: 5 } })
+ })
+
+ it('works with its own clock, ids and log by default', () => {
+ vi.useRealTimers()
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
+ const links = createServiceLinks({ token: TOKEN, owned: { search: ['session_search'] }, answer: () => ({}) })
+ const search = sink()
+ links.accept('search', TOKEN, search, vi.fn())
+ const reply = vi.fn()
+ links.route('session_search', {}, reply)
+ expect(String(search.sent[0].payload?.requestId)).toMatch(/^[0-9a-f-]{36}$/)
+ expect(warn).toHaveBeenCalledWith('[services] search connected')
+ // Answered, so its timer is cleared rather than left to fire.
+ const link = links.accept('search', TOKEN, search, vi.fn())!
+ link.closed()
+ expect(reply).toHaveBeenCalledWith({ error: 'SERVICE_UNAVAILABLE', service: 'search', retryable: true })
+ warn.mockRestore()
+ })
+})
diff --git a/cli/src/core/serviceLinks.ts b/cli/src/core/serviceLinks.ts
new file mode 100644
index 000000000..0e9d476df
--- /dev/null
+++ b/cli/src/core/serviceLinks.ts
@@ -0,0 +1,166 @@
+/**
+ * The core's end of the services that run in their own processes (harnessd/services.ts starts them).
+ *
+ * A service reaches the core the way a window does — the local socket — under a service role: its
+ * `machine_select` names the service and carries the token the master gave the core and the service
+ * alike, so no other local process can stand in for one. The core then routes the requests the service
+ * answers to it and relays each answer back to the client that asked; a request that finds the service
+ * down, or not answering in time, is answered SERVICE_UNAVAILABLE with `retryable: true` — the core never
+ * waits on a service in line. The core tells a service what it needs to know with `notify`, and answers
+ * the few questions a service may ask it (`service_query`), nothing more.
+ */
+import { randomUUID, timingSafeEqual } from 'node:crypto'
+
+export interface ServiceFrame {
+ type: string
+ payload?: Record<string, unknown>
+ [key: string]: unknown
+}
+
+/** Where frames to one service's connection go. */
+export interface ServiceSink {
+ sendFrame(frame: ServiceFrame): boolean
+}
+
+/** What the local socket hands back for a service connection: its frames in, its end. */
+export interface ServiceLink {
+ receive(frame: ServiceFrame): void
+ closed(): void
+}
+
+export interface ServiceLinksOptions {
+ /** The token the master started this core with; without one, no service may connect. */
+ token: string | undefined
+ /** The request types each out-of-process service answers. Only these services may connect. */
+ owned: Readonly<Record<string, readonly string[]>>
+ /** The core's answer to a service's question (`service_query`): `query` names it. */
+ answer(service: string, query: string, payload: Record<string, unknown>): Record<string, unknown> | Promise<Record<string, unknown>>
+ /** How long a routed request may wait for its service before it is answered SERVICE_UNAVAILABLE. */
+ timeoutMs?: number
+ log?: (line: string) => void
+ newId?: () => string
+ setTimer?: (run: () => void, ms: number) => unknown
+ clearTimer?: (timer: unknown) => void
+}
+
+interface Connected {
+ sink: ServiceSink
+ close: (code: number, reason: string) => void
+}
+
+interface Waiting {
+ service: string
+ type: string
+ reply: (result: Record<string, unknown>) => void
+ timer: unknown
+}
+
+/** The most notifications held for one service while it is down. */
+export const HELD_MAX = 1_000
+
+export function createServiceLinks(options: ServiceLinksOptions) {
+ const timeoutMs = options.timeoutMs ?? 30_000
+ const log = options.log ?? ((line: string) => console.warn(line))
+ const newId = options.newId ?? randomUUID
+ const setTimer = options.setTimer ?? ((run, ms) => setTimeout(run, ms))
+ const clearTimer = options.clearTimer ?? ((timer) => clearTimeout(timer as ReturnType<typeof setTimeout>))
+ const ownerOf = new Map<string, string>()
+ for (const [service, types] of Object.entries(options.owned)) for (const type of types) ownerOf.set(type, service)
+ const links = new Map<string, Connected>()
+ const waiting = new Map<string, Waiting>()
+ /** What a service must hear even if it is down when it is said (a purge's forgetting), delivered on
+ * its next connection; bounded, the oldest dropped first. */
+ const held = new Map<string, ServiceFrame[]>()
+ const unavailable = (service: string) => ({ error: 'SERVICE_UNAVAILABLE', service, retryable: true })
+
+ const tokenMatches = (offered: string): boolean => {
+ if (!options.token) return false
+ const expected = Buffer.from(options.token)
+ const given = Buffer.from(offered)
+ return given.length === expected.length && timingSafeEqual(given, expected)
+ }
+
+ const settle = (id: string, entry: Waiting, result: Record<string, unknown>): void => {
+ waiting.delete(id)
+ clearTimer(entry.timer)
+ entry.reply(result)
+ }
+
+ return {
+ /** A service connecting. Null — and the socket closes it — unless the master started it. */
+ accept(service: string, token: string, sink: ServiceSink, close: (code: number, reason: string) => void): ServiceLink | null {
+ if (!Object.hasOwn(options.owned, service) || !tokenMatches(token)) {
+ log(`[services] a connection as service "${service.slice(0, 40)}" was refused`)
+ return null
+ }
+ // A service reconnecting (the core restarted, its socket dropped) replaces its old connection.
+ links.get(service)?.close(4409, 'replaced by a newer connection')
+ const connected: Connected = { sink, close }
+ links.set(service, connected)
+ log(`[services] ${service} connected`)
+ const owed = held.get(service) ?? []
+ held.delete(service)
+ for (const frame of owed) sink.sendFrame(frame)
+ return {
+ receive: (frame) => {
+ const payload = frame.payload ?? {}
+ if (frame.type === 'service_query') {
+ const requestId = payload.requestId
+ const query = typeof payload.query === 'string' ? payload.query : ''
+ void Promise.resolve()
+ .then(() => options.answer(service, query, payload))
+ .catch(() => ({ error: 'QUERY_FAILED' }))
+ .then((result) => { sink.sendFrame({ type: 'service_query_result', payload: { ...result, requestId } }) })
+ return
+ }
+ const id = typeof payload.requestId === 'string' ? payload.requestId : ''
+ const entry = waiting.get(id)
+ // Only the answer to a request routed to THIS service, under the type it was asked as.
+ if (!entry || entry.service !== service || frame.type !== `${entry.type}_result`) return
+ const { requestId: _routed, ...result } = payload
+ settle(id, entry, result)
+ },
+ closed: () => {
+ if (links.get(service) !== connected) return
+ links.delete(service)
+ log(`[services] ${service} disconnected`)
+ for (const [id, entry] of waiting) if (entry.service === service) settle(id, entry, unavailable(service))
+ },
+ }
+ },
+
+ /** Route a request a service owns: false when no service owns `type`, and the core answers it itself. */
+ route(type: string, payload: Record<string, unknown>, reply: (result: Record<string, unknown>) => void): boolean {
+ const service = ownerOf.get(type)
+ if (!service) return false
+ const link = links.get(service)
+ if (!link) { reply(unavailable(service)); return true }
+ const id = newId()
+ const entry: Waiting = { service, type, reply, timer: null }
+ // Cleared whenever the entry is settled, so it only ever fires for one still waiting.
+ entry.timer = setTimer(() => settle(id, entry, unavailable(service)), timeoutMs)
+ waiting.set(id, entry)
+ if (!link.sink.sendFrame({ type, payload: { ...payload, requestId: id } })) settle(id, entry, unavailable(service))
+ return true
+ },
+
+ /** Tell a service something it needs to know; false when it is not connected to hear it. With
+ * `untilDelivered`, what it misses is said again when it next connects. */
+ notify(service: string, frame: ServiceFrame, opts: { untilDelivered?: boolean } = {}): boolean {
+ if (links.get(service)?.sink.sendFrame(frame)) return true
+ if (opts.untilDelivered && Object.hasOwn(options.owned, service)) {
+ const owed = held.get(service) ?? []
+ owed.push(frame)
+ if (owed.length > HELD_MAX) owed.shift()
+ held.set(service, owed)
+ }
+ return false
+ },
+
+ connected(service: string): boolean {
+ return links.has(service)
+ },
+ }
+}
+
+export type ServiceLinks = ReturnType<typeof createServiceLinks>
diff --git a/cli/src/harnessd/AGENTS.md b/cli/src/harnessd/AGENTS.md
new file mode 100644
index 000000000..82ce990f6
--- /dev/null
+++ b/cli/src/harnessd/AGENTS.md
@@ -0,0 +1,16 @@
+# harnessd's master
+
+The process `harness start` launches. It keeps the core and the services running, and nothing else.
+
+## Rules
+
+1. **No feature code, no network, no sockets.** Everything here could otherwise fail, and the master is
+ the one process that must not. Import only Node built-ins and the files in this folder (and the log
+ trimmer); `src/architecture.spec.ts` checks it.
+2. **Everything that touches the operating system is injected** (`SupervisorDeps`,
+ `ServiceSupervisorDeps`), so every decision is a unit test with fake timers. 100% coverage per file
+ (`npm run test:harnessd`).
+3. **The spawn protocol only grows** (`protocol.ts`): messages are added, never changed or removed, and
+ unknown ones are ignored, because during an update an older master supervises a newer core.
+4. **The core is always restarted; a service may be parked.** Never let a failing service stop the
+ core or the master.
diff --git a/cli/src/harnessd/CLAUDE.md b/cli/src/harnessd/CLAUDE.md
new file mode 100644
index 000000000..6a4029179
--- /dev/null
+++ b/cli/src/harnessd/CLAUDE.md
@@ -0,0 +1,2 @@
+<!-- One set of rules for every agent: Codex reads AGENTS.md, and Claude Code reads it through this import. Claude Code reads AGENTS.md by itself only from v2.1.277, and only when no CLAUDE.md is in the working directory or above it, so without this a session started in a folder with a CLAUDE.md would miss these rules. -->
diff --git a/cli/src/harnessd/master.spec.ts b/cli/src/harnessd/master.spec.ts
index 58b36a06f..17552b76c 100644
--- a/cli/src/harnessd/master.spec.ts
+++ b/cli/src/harnessd/master.spec.ts
@@ -262,6 +262,41 @@ describe('runMaster', () => {
expect(readStatusFile(statusFile, process.pid)).toMatchObject({ state: 'stopped', lastExitReason: 'stopped' })
})
+ it('runs the services it is told to beside the core, with one token for both, restarts them, and stops them with it', async () => {
+ const pidFile = join(dir, 'adapter.pid')
+ const seen = join(dir, 'seen')
+ const script = join(dir, 'daemon.cjs')
+ writeFileSync(script, `
+ const { appendFileSync } = require('node:fs')
+ const role = process.argv[2] === '__service' ? 'service:' + process.argv[3] : process.argv[2]
+ appendFileSync(${JSON.stringify(seen)}, JSON.stringify({ role, token: process.env.HARNESSD_SERVICE_TOKEN, name: process.env.HARNESSD_SERVICE, restarts: process.env.HARNESSD_RESTARTS, flags: process.execArgv }) + '\\n')
+ if (role === '__run') { process.send({ type: 'harnessd:bound', protocol: 2, port: 1 }); process.send({ type: 'harnessd:ready' }) }
+ setInterval(() => process.send({ type: 'harnessd:heartbeat', rssBytes: 1, heapUsedBytes: 1, loopDelayMs: 0 }), 50)
+ process.on('SIGTERM', () => process.exit(0))
+ if (role === 'service:search' && process.env.HARNESSD_RESTARTS === '0') setTimeout(() => process.exit(1), 100)
+ `)
+ const exits: number[] = []
+ const signals = new Map<string, () => void>()
+ runMaster({
+ nodePath: process.execPath, execArgv: [], scriptPath: script, pidFile,
+ restoreUpdate: () => {}, confirmUpdate: () => {},
+ env: { ...process.env, HARNESSD_SERVICES: 'search,unknown', HARNESSD_SERVICE_INITIAL_BACKOFF_MS: '10' },
+ exit: (code) => exits.push(code), onSignal: (signal, listener) => signals.set(signal, listener),
+ })
+ const lines = (): Array<Record<string, any>> => existsSync(seen) ? readFileSync(seen, 'utf8').trim().split('\n').map((line) => JSON.parse(line)) : []
+ await until('the crashed service to be restarted', () => lines().filter((line) => line.role === 'service:search').length === 2)
+ const core = lines().find((line) => line.role === '__run')!
+ const services = lines().filter((line) => line.role === 'service:search')
+ expect(services.map((service) => service.restarts)).toEqual(['0', '1'])
+ expect(services[0]).toMatchObject({ name: 'search', flags: ['--max-old-space-size=1024'] })
+ expect(services[0].token).toMatch(/^[0-9a-f]{48}$/)
+ expect(core.token).toBe(services[0].token)
+ expect(lines().some((line) => line.role === 'service:unknown')).toBe(false)
+ signals.get('SIGTERM')!()
+ await until('the master to finish', () => exits.length > 0)
+ expect(exits).toEqual([0])
+ })
+
it.each([
['is gone', (pidFile: string) => rmSync(pidFile, { force: true })],
['holds no number', (pidFile: string) => writeFileSync(pidFile, 'garbage\n')],
diff --git a/cli/src/harnessd/master.ts b/cli/src/harnessd/master.ts
index 5bcbcf8f0..9fcb7b800 100644
--- a/cli/src/harnessd/master.ts
+++ b/cli/src/harnessd/master.ts
@@ -4,10 +4,12 @@
* file, the log's size, signals.
*/
import { spawn, type ChildProcess } from 'node:child_process'
+import { randomBytes } from 'node:crypto'
import { readFileSync, renameSync, rmSync, writeFileSync } from 'node:fs'
import { performance } from 'node:perf_hooks'
import { trimLogFile } from '../lib/log.js'
import type { MasterMessage } from './protocol.js'
+import { KNOWN_SERVICES, ServiceSupervisor, serviceOptions, serviceSpecs } from './services.js'
import { DEFAULT_SUPERVISOR_OPTIONS, Supervisor, type CoreHandle, type SupervisorOptions, type SupervisorStatus } from './supervisor.js'
/** How often the master keeps the daemon's log under its cap. */
@@ -171,9 +173,24 @@ export function runMaster(config: MasterConfig): Supervisor {
// Trimmed here rather than by the core: the master outlives every core, and two trimmers rewriting
// one file in place would race.
const stopTrimming = trimLogEvery(config.logFile)
+ const log = (line: string) => console.log(`${new Date().toISOString().replace('T', ' ').slice(0, 23)} ${line}`)
+ // A new one every boot, given to the core and to each service: how the core knows a service
+ // connection is one this master started, and no other local process.
+ const token = randomBytes(24).toString('hex')
+ const services = new ServiceSupervisor(serviceSpecs(env, KNOWN_SERVICES), {
+ spawnService: (spec, extra) => coreHandle(spawn(config.nodePath, [...coreExecArgv(config.execArgv, spec.heapLimitMiB), config.scriptPath, '__service', spec.name], {
+ env: { ...env, ...extra },
+ stdio: ['ignore', 'inherit', 'inherit', 'ipc'],
+ })),
+ now: () => performance.now(),
+ wallClock: () => Date.now(),
+ setTimer: (run, ms) => setTimeout(run, ms),
+ clearTimer: (timer) => clearTimeout(timer as ReturnType<typeof setTimeout>),
+ log,
+ }, serviceOptions(env), { HARNESSD_SERVICE_TOKEN: token })
const supervisor = new Supervisor({
spawnCore: (extra) => coreHandle(spawn(config.nodePath, [...execArgv, config.scriptPath, '__run'], {
- env: { ...env, ...extra },
+ env: { ...env, ...extra, HARNESSD_SERVICE_TOKEN: token },
stdio: ['ignore', 'inherit', 'inherit', 'ipc'],
})),
now: () => performance.now(),
@@ -185,14 +202,18 @@ export function runMaster(config: MasterConfig): Supervisor {
releasePidFile: () => { if (readPid() === process.pid) rmSync(config.pidFile, { force: true }) },
restoreUpdate: config.restoreUpdate,
confirmUpdate: config.confirmUpdate,
- log: (line) => console.log(`${new Date().toISOString().replace('T', ' ').slice(0, 23)} ${line}`),
- exit: (code) => {
+ log,
+ // The services go with the master, after the core: none is left holding the core's socket.
+ exit: (code) => services.stop(() => {
stopTrimming()
exit(code)
- },
+ }),
}, options)
- for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP'] as const) onSignal(signal, () => supervisor.stop(signal))
+ // Services stop beside the core, inside the same grace `harness stop` gives the whole daemon.
+ for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP'] as const) onSignal(signal, () => { services.stop(() => {}); supervisor.stop(signal) })
supervisor.start()
+ // Not waiting on the core: a service reaches it through its socket, and retries until it answers.
+ services.start()
return supervisor
}
diff --git a/cli/src/harnessd/services.spec.ts b/cli/src/harnessd/services.spec.ts
new file mode 100644
index 000000000..282aade83
--- /dev/null
+++ b/cli/src/harnessd/services.spec.ts
@@ -0,0 +1,285 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { DEFAULT_SERVICE_OPTIONS, ServiceSupervisor, serviceOptions, serviceSpecs, type ServiceSpec, type ServiceSupervisorOptions } from './services.js'
+import type { CoreHandle } from './supervisor.js'
+
+const MIB = 1024 * 1024
+
+class FakeService implements CoreHandle {
+ readonly kills: NodeJS.Signals[] = []
+ private messageListeners: Array<(message: unknown) => void> = []
+ private exitListeners: Array<(code: number | null, signal: NodeJS.Signals | null) => void> = []
+ constructor(readonly pid: number | undefined, readonly spec: ServiceSpec, readonly env: Record<string, string>) {}
+ send(): void {}
+ kill(signal: NodeJS.Signals): void { this.kills.push(signal) }
+ onMessage(listener: (message: unknown) => void): void { this.messageListeners.push(listener) }
+ onExit(listener: (code: number | null, signal: NodeJS.Signals | null) => void): void { this.exitListeners.push(listener) }
+ say(message: unknown): void { for (const listener of this.messageListeners) listener(message) }
+ beat(rssBytes = 100 * MIB, heapUsedBytes = rssBytes / 2): void { this.say({ type: 'harnessd:heartbeat', rssBytes, heapUsedBytes, loopDelayMs: 2 }) }
+ exit(code: number | null, signal: NodeJS.Signals | null = null): void { for (const listener of this.exitListeners) listener(code, signal) }
+}
+
+const options: ServiceSupervisorOptions = {
+ ...DEFAULT_SERVICE_OPTIONS,
+ heartbeatTimeoutMs: 6_000,
+ stopGraceMs: 1_000,
+ initialBackoffMs: 1_000,
+ maxBackoffMs: 4_000,
+ backoffResetMs: 20_000,
+ heapRestartPercent: 75,
+ parkCrashes: 3,
+ parkWindowMs: 60_000,
+ parkRetryMs: 120_000,
+}
+const search: ServiceSpec = { name: 'search', heapLimitMiB: 512, rssLimitMiB: 1_024 }
+const devices: ServiceSpec = { name: 'devices', heapLimitMiB: 256, rssLimitMiB: 0 }
+
+describe('ServiceSupervisor', () => {
+ let children: FakeService[]
+ let lines: string[]
+ const latest = (name = 'search') => children.filter((child) => child.spec.name === name).at(-1)!
+ const make = (specs: ServiceSpec[] = [search], overrides: Partial<ServiceSupervisorOptions> = {}, env: Record<string, string> = {}) =>
+ new ServiceSupervisor(specs, {
+ spawnService: (spec, extra) => {
+ const child = new FakeService(2000 + children.length, spec, extra)
+ children.push(child)
+ return child
+ },
+ now: () => performance.now(),
+ wallClock: () => Date.now(),
+ setTimer: (run, ms) => setTimeout(run, ms),
+ clearTimer: (timer) => clearTimeout(timer as ReturnType<typeof setTimeout>),
+ log: (line) => lines.push(line),
+ }, { ...options, ...overrides }, env)
+
+ beforeEach(() => {
+ vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'Date', 'performance'] })
+ children = []
+ lines = []
+ })
+ afterEach(() => vi.useRealTimers())
+
+ it('starts every service with its name, its token and its watchdog, and none waits on another', () => {
+ const supervisor = make([search, devices], {}, { HARNESSD_SERVICE_TOKEN: 'secret' })
+ supervisor.start()
+ expect(children.map((child) => child.spec.name)).toEqual(['search', 'devices'])
+ expect(latest().env).toEqual({ HARNESSD_SERVICE_TOKEN: 'secret', HARNESSD_SERVICE: 'search', HARNESSD_RESTARTS: '0', HARNESSD_WATCHDOG_MS: '6000' })
+ expect(supervisor.status().map((status) => status.state)).toEqual(['starting', 'starting'])
+ latest().beat()
+ expect(supervisor.status()[0]).toMatchObject({ name: 'search', state: 'running', pid: 2000, restarts: 0, lastExit: null })
+ // Anything that is not a heartbeat is ignored.
+ latest().say({ type: 'harnessd:bound', protocol: 2, port: 1 })
+ latest().say('noise')
+ expect(supervisor.status()[0].state).toBe('running')
+ })
+
+ it('restarts a crashed service with a backoff that doubles, caps, and is forgiven after a long run', () => {
+ const supervisor = make([search], { parkCrashes: 99 })
+ supervisor.start()
+ latest().beat()
+ latest().exit(1)
+ expect(supervisor.status()[0]).toMatchObject({ state: 'restarting', restarts: 1, lastExit: 'code 1', lastExitReason: 'crashed', pid: null })
+ vi.advanceTimersByTime(999)
+ expect(children).toHaveLength(1)
+ vi.advanceTimersByTime(1)
+ expect(children).toHaveLength(2)
+ expect(latest().env.HARNESSD_RESTARTS).toBe('1')
+ expect(supervisor.status()[0].state).toBe('restarting')
+ latest().beat()
+ expect(supervisor.status()[0].state).toBe('running')
+ for (const delay of [2_000, 4_000, 4_000]) {
+ latest().exit(null, 'SIGSEGV')
+ vi.advanceTimersByTime(delay - 1)
+ const before = children.length
+ vi.advanceTimersByTime(1)
+ expect(children.length).toBe(before + 1)
+ }
+ expect(lines.some((line) => line.includes('exited (signal SIGSEGV, crashed) — restarting in 4000 ms'))).toBe(true)
+ // Up long enough: the next crash restarts after the initial delay again.
+ for (let left = 21_000; left > 0; left -= 1_000) { vi.advanceTimersByTime(1_000); latest().beat() }
+ latest().exit(1)
+ const before = children.length
+ vi.advanceTimersByTime(1_000)
+ expect(children.length).toBe(before + 1)
+ })
+
+ it('kills a service that stops beating, from its start too, and restarts it as hung', () => {
+ const supervisor = make()
+ supervisor.start()
+ // Never beat at all: hung from the start.
+ vi.advanceTimersByTime(6_000)
+ expect(latest().kills).toEqual(['SIGKILL'])
+ latest().exit(null, 'SIGKILL')
+ expect(supervisor.status()[0]).toMatchObject({ lastExitReason: 'hung', state: 'restarting' })
+ vi.advanceTimersByTime(1_000)
+ const second = latest()
+ for (let i = 0; i < 3; i++) { vi.advanceTimersByTime(5_000); second.beat() }
+ expect(second.kills).toEqual([])
+ vi.advanceTimersByTime(6_000)
+ expect(second.kills).toEqual(['SIGKILL'])
+ expect(lines.some((line) => line.includes('sent no heartbeat for 6000 ms — it is hung'))).toBe(true)
+ })
+
+ it('restarts a service that outgrows its heap share or its resident budget, and counts it like a crash', () => {
+ const supervisor = make([search, devices], { parkCrashes: 99 })
+ supervisor.start()
+ latest('search').beat(200 * MIB, 380 * MIB)
+ expect(latest('search').kills).toEqual([])
+ latest('search').beat(200 * MIB, 390 * MIB)
+ expect(latest('search').kills).toEqual(['SIGTERM'])
+ // While it is going, more beats do not ask again.
+ latest('search').beat(200 * MIB, 500 * MIB)
+ expect(latest('search').kills).toEqual(['SIGTERM'])
+ latest('search').exit(0)
+ expect(supervisor.status()[0]).toMatchObject({ lastExitReason: 'memory', restarts: 1 })
+ vi.advanceTimersByTime(1_000)
+ latest('search').beat(1_100 * MIB, 10 * MIB)
+ expect(latest('search').kills).toEqual(['SIGTERM'])
+ expect(lines.some((line) => line.includes('over its 1024 MiB budget'))).toBe(true)
+ // No resident budget: only the heap share counts.
+ latest('devices').beat(9_000 * MIB, 10 * MIB)
+ expect(latest('devices').kills).toEqual([])
+ expect(lines.some((line) => line.includes('past 75% of its 512 MiB limit'))).toBe(true)
+ })
+
+ it('SIGKILLs a service that outlives its grace to stop for memory', () => {
+ make().start()
+ latest().beat(100 * MIB, 500 * MIB)
+ expect(latest().kills).toEqual(['SIGTERM'])
+ vi.advanceTimersByTime(1_000)
+ expect(latest().kills).toEqual(['SIGTERM', 'SIGKILL'])
+ expect(lines.some((line) => line.includes('outlived its 1000 ms to stop'))).toBe(true)
+ })
+
+ it('parks a service that keeps crashing, says so, and tries it again later', () => {
+ const supervisor = make()
+ supervisor.start()
+ for (let crash = 1; crash <= 2; crash++) {
+ latest().exit(1)
+ vi.advanceTimersByTime(4_000)
+ }
+ latest().exit(1)
+ expect(supervisor.status()[0]).toMatchObject({ state: 'parked', restarts: 3 })
+ expect(lines.some((line) => line.includes('ended 3 times in 1 min (code 1, crashed) — parked; trying again in 2 min'))).toBe(true)
+ const parked = children.length
+ vi.advanceTimersByTime(119_999)
+ expect(children.length).toBe(parked)
+ vi.advanceTimersByTime(1)
+ expect(children.length).toBe(parked + 1)
+ // A fresh start: its next crash is one, not the fourth.
+ latest().exit(1)
+ expect(supervisor.status()[0].state).toBe('restarting')
+ })
+
+ it('crashes far apart never park a service', () => {
+ const supervisor = make()
+ supervisor.start()
+ for (let crash = 0; crash < 5; crash++) {
+ latest().exit(1)
+ vi.advanceTimersByTime(61_000)
+ }
+ expect(supervisor.status()[0].state).not.toBe('parked')
+ })
+
+ it('stops every service, SIGKILLing one that outlives its grace, and says when all are gone', () => {
+ const supervisor = make([search, devices])
+ supervisor.start()
+ const done = vi.fn()
+ supervisor.stop(done)
+ expect(supervisor.status().map((status) => status.state)).toEqual(['stopping', 'stopping'])
+ expect(latest('search').kills).toEqual(['SIGTERM'])
+ latest('search').exit(0)
+ expect(done).not.toHaveBeenCalled()
+ vi.advanceTimersByTime(1_000)
+ expect(latest('devices').kills).toEqual(['SIGTERM', 'SIGKILL'])
+ latest('devices').exit(null, 'SIGKILL')
+ expect(done).toHaveBeenCalledOnce()
+ expect(supervisor.status()).toEqual([
+ expect.objectContaining({ name: 'search', state: 'stopped', lastExitReason: 'stopped', lastExit: 'code 0' }),
+ expect.objectContaining({ name: 'devices', state: 'stopped', lastExitReason: 'stopped', lastExit: 'signal SIGKILL' }),
+ ])
+ // Stopping again, or a late exit from a process already let go, changes nothing.
+ const again = vi.fn()
+ supervisor.stop(again)
+ expect(again).toHaveBeenCalledOnce()
+ })
+
+ it('a second stop while stopping waits for the same end, and sends no second signal', () => {
+ const supervisor = make()
+ supervisor.start()
+ const first = vi.fn()
+ const second = vi.fn()
+ supervisor.stop(first)
+ supervisor.stop(second)
+ expect(latest().kills).toEqual(['SIGTERM'])
+ latest().exit(0)
+ expect(first).toHaveBeenCalledOnce()
+ expect(second).toHaveBeenCalledOnce()
+ })
+
+ it('stops a service waiting to restart, or parked, without starting it', () => {
+ const supervisor = make([search, devices])
+ supervisor.start()
+ latest('search').beat()
+ for (let crash = 0; crash < 3; crash++) { latest('devices').exit(1); vi.advanceTimersByTime(4_000); latest('search').beat() }
+ latest('search').exit(1)
+ expect(supervisor.status().map((status) => status.state)).toEqual(['restarting', 'parked'])
+ const started = children.length
+ const done = vi.fn()
+ supervisor.stop(done)
+ expect(done).toHaveBeenCalledOnce()
+ vi.advanceTimersByTime(300_000)
+ expect(children.length).toBe(started)
+ expect(supervisor.status().map((status) => status.state)).toEqual(['stopped', 'stopped'])
+ })
+
+ it('with no services, starting does nothing and stopping is done at once', () => {
+ const supervisor = make([])
+ supervisor.start()
+ const done = vi.fn()
+ supervisor.stop(done)
+ expect(done).toHaveBeenCalledOnce()
+ expect(supervisor.status()).toEqual([])
+ })
+
+ it('names a process with no pid, and ignores what a process it let go of says', () => {
+ const supervisor = new ServiceSupervisor([search], {
+ spawnService: (spec, env) => { const child = new FakeService(undefined, spec, env); children.push(child); return child },
+ now: () => performance.now(),
+ wallClock: () => Date.now(),
+ setTimer: (run, ms) => setTimeout(run, ms),
+ clearTimer: (timer) => clearTimeout(timer as ReturnType<typeof setTimeout>),
+ log: (line) => lines.push(line),
+ }, options)
+ supervisor.start()
+ expect(lines[0]).toContain('started (pid ?)')
+ const first = latest()
+ first.exit(1)
+ vi.advanceTimersByTime(1_000)
+ first.beat(9_000 * MIB, 9_000 * MIB)
+ first.exit(1)
+ expect(latest().kills).toEqual([])
+ expect(supervisor.status()[0].restarts).toBe(1)
+ })
+
+ it('reads its timings from the environment, keeping the defaults for what is unset or invalid', () => {
+ expect(serviceOptions({})).toEqual(DEFAULT_SERVICE_OPTIONS)
+ expect(serviceOptions({
+ HARNESSD_SERVICE_HEARTBEAT_TIMEOUT_MS: '2000', HARNESSD_SERVICE_STOP_GRACE_MS: '50', HARNESSD_SERVICE_INITIAL_BACKOFF_MS: '10',
+ HARNESSD_SERVICE_MAX_BACKOFF_MS: '20', HARNESSD_SERVICE_BACKOFF_RESET_MS: '30', HARNESSD_SERVICE_HEAP_RESTART_PERCENT: '150',
+ HARNESSD_SERVICE_PARK_CRASHES: '2', HARNESSD_SERVICE_PARK_WINDOW_MS: '40', HARNESSD_SERVICE_PARK_RETRY_MS: '60',
+ })).toEqual({ heartbeatTimeoutMs: 2_000, stopGraceMs: 50, initialBackoffMs: 10, maxBackoffMs: 20, backoffResetMs: 30, heapRestartPercent: 100, parkCrashes: 2, parkWindowMs: 40, parkRetryMs: 60 })
+ // A heartbeat patience under a second, or a value that is not a number, keeps the default.
+ expect(serviceOptions({ HARNESSD_SERVICE_HEARTBEAT_TIMEOUT_MS: '10', HARNESSD_SERVICE_PARK_CRASHES: 'many' }))
+ .toEqual(DEFAULT_SERVICE_OPTIONS)
+ })
+
+ it('reads the services to run from the environment, keeping only known ones, once each', () => {
+ const known = { search: { heapLimitMiB: 512, rssLimitMiB: 1_024 }, devices: { heapLimitMiB: 256, rssLimitMiB: 0 } }
+ expect(serviceSpecs({ HARNESSD_SERVICES: ' search, nope ,devices,search,' }, known)).toEqual([search, devices])
+ expect(serviceSpecs({}, known)).toEqual([])
+ expect(serviceSpecs({ HARNESSD_SERVICES: 'toString,constructor' }, known)).toEqual([])
+ // One heap limit for every service, when given as a whole number of MiB.
+ expect(serviceSpecs({ HARNESSD_SERVICES: 'search', HARNESSD_SERVICE_HEAP_LIMIT_MIB: '96' }, known)).toEqual([{ ...search, heapLimitMiB: 96 }])
+ expect(serviceSpecs({ HARNESSD_SERVICES: 'search', HARNESSD_SERVICE_HEAP_LIMIT_MIB: 'lots' }, known)).toEqual([search])
+ })
+})
diff --git a/cli/src/harnessd/services.ts b/cli/src/harnessd/services.ts
new file mode 100644
index 000000000..2c182407c
--- /dev/null
+++ b/cli/src/harnessd/services.ts
@@ -0,0 +1,326 @@
+/**
+ * harnessd's services, each in its own process, kept running by the master beside the core.
+ *
+ * A service is a feature the core can run without (search, devices, models, …). In the core's process a
+ * service's exception is caught (core/serviceHost.ts), but a native crash, a hung event loop or a leak
+ * would still take the core with it. Out here a service can only take itself down: the master restarts
+ * it with backoff, kills it when it stops beating (hung) or outgrows its memory budget (leaking), parks it
+ * when it keeps crashing, and the core answers its requests SERVICE_UNAVAILABLE meanwhile.
+ *
+ * The same shape as ./supervisor.ts, for many children and fewer promises: a service has no port to
+ * bind and no update to prove; it beats, or it is restarted. Everything that touches the operating
+ * system is injected, so every decision here is tested without one.
+ */
+import { isCoreMessage } from './protocol.js'
+import type { CoreHandle } from './supervisor.js'
+
+export interface ServiceSpec {
+ /** The service's name, as `harness __service <name>` runs it. */
+ name: string
+ /** The V8 heap limit it runs with, MiB; its budget is a share of it. */
+ heapLimitMiB: number
+ /** Resident memory past which it is restarted, MiB; 0: off. */
+ rssLimitMiB: number
+}
+
+export interface ServiceSupervisorDeps {
+ /** Start a service's process with these extra environment variables. */
+ spawnService(spec: ServiceSpec, env: Record<string, string>): CoreHandle
+ /** A monotonic clock, in ms. */
+ now(): number
+ /** The wall clock, in ms: what the status says things happened at. */
+ wallClock(): number
+ setTimer(run: () => void, ms: number): unknown
+ clearTimer(timer: unknown): void
+ log(line: string): void
+}
+
+export interface ServiceSupervisorOptions {
+ /** A service that sends no heartbeat for this long — from its start, too — is hung: killed, restarted. */
+ heartbeatTimeoutMs: number
+ /** How long a service gets to stop after SIGTERM before SIGKILL. */
+ stopGraceMs: number
+ /** Restart delay: starts here, doubles per crash, caps at `maxBackoffMs`. */
+ initialBackoffMs: number
+ maxBackoffMs: number
+ /** A service that stayed up this long earns the next crash the initial delay again. */
+ backoffResetMs: number
+ /** Past this share of its heap limit a service is restarted cleanly, before V8 aborts it. */
+ heapRestartPercent: number
+ /** This many crashes inside `parkWindowMs` park the service: reported, and not respawned in a loop. */
+ parkCrashes: number
+ parkWindowMs: number
+ /** A parked service is tried again after this long. */
+ parkRetryMs: number
+}
+
+export const DEFAULT_SERVICE_OPTIONS: ServiceSupervisorOptions = {
+ heartbeatTimeoutMs: 30_000,
+ stopGraceMs: 1_500,
+ initialBackoffMs: 1_000,
+ maxBackoffMs: 60_000,
+ backoffResetMs: 60_000,
+ heapRestartPercent: 75,
+ parkCrashes: 5,
+ parkWindowMs: 600_000,
+ parkRetryMs: 1_800_000,
+}
+
+export type ServiceState = 'starting' | 'running' | 'restarting' | 'parked' | 'stopping' | 'stopped'
+export type ServiceExitReason = 'crashed' | 'hung' | 'memory' | 'stopped'
+
+export interface ServiceStatus {
+ name: string
+ state: ServiceState
+ pid: number | null
+ restarts: number
+ lastExit: string | null
+ lastExitReason: ServiceExitReason | null
+ /** When the state last changed, wall clock ms. */
+ since: number
+}
+
+const MIB = 1024 * 1024
+const describeExit = (code: number | null, signal: NodeJS.Signals | null): string =>
+ signal ? `signal ${signal}` : `code ${code}`
+
+/** One service's process, restarted for as long as the master runs. */
+class Service {
+ state: ServiceState = 'starting'
+ private since: number
+ private child: CoreHandle | null = null
+ private restarts = 0
+ private backoff: number
+ private upAt = 0
+ private crashes: number[] = []
+ private lastExit: string | null = null
+ private lastExitReason: ServiceExitReason | null = null
+ /** Why the master killed the running process, when it did. */
+ private killReason: ServiceExitReason | null = null
+ private ending = false
+ private heartbeatTimer: unknown = null
+ private killTimer: unknown = null
+ private restartTimer: unknown = null
+ private onStopped: Array<() => void> = []
+
+ constructor(
+ readonly spec: ServiceSpec,
+ private readonly deps: ServiceSupervisorDeps,
+ private readonly options: ServiceSupervisorOptions,
+ private readonly env: Record<string, string>,
+ ) {
+ this.backoff = options.initialBackoffMs
+ this.since = deps.wallClock()
+ }
+
+ status(): ServiceStatus {
+ return {
+ name: this.spec.name,
+ state: this.state,
+ pid: this.child?.pid ?? null,
+ restarts: this.restarts,
+ lastExit: this.lastExit,
+ lastExitReason: this.lastExitReason,
+ since: this.since,
+ }
+ }
+
+ start(): void {
+ const child = this.deps.spawnService(this.spec, {
+ ...this.env,
+ HARNESSD_SERVICE: this.spec.name,
+ HARNESSD_RESTARTS: String(this.restarts),
+ HARNESSD_WATCHDOG_MS: String(this.options.heartbeatTimeoutMs),
+ })
+ this.child = child
+ this.killReason = null
+ this.ending = false
+ this.upAt = this.deps.now()
+ child.onMessage((message) => { if (this.child === child) this.onMessage(child, message) })
+ child.onExit((code, signal) => { if (this.child === child) this.onExit(code, signal) })
+ this.deps.log(`[harnessd] service ${this.spec.name} started (pid ${child.pid ?? '?'})${this.restarts ? ` · restart ${this.restarts}` : ''}`)
+ this.setState(this.restarts === 0 ? 'starting' : 'restarting')
+ this.watchHeartbeat(child)
+ }
+
+ /** SIGTERM, then SIGKILL after the grace; `done` once it is gone. A parked or waiting one is simply stopped. */
+ stop(done: () => void): void {
+ if (this.state === 'stopped') { done(); return }
+ this.onStopped.push(done)
+ if (this.state === 'stopping') return
+ this.clearTimer('restartTimer')
+ if (!this.child) { this.setState('stopped'); this.stopped(); return }
+ this.setState('stopping')
+ this.end('SIGTERM')
+ }
+
+ private stopped(): void {
+ const waiting = this.onStopped
+ this.onStopped = []
+ for (const done of waiting) done()
+ }
+
+ private onMessage(child: CoreHandle, message: unknown): void {
+ if (!isCoreMessage(message) || message.type !== 'harnessd:heartbeat') return
+ this.watchHeartbeat(child)
+ if (this.state === 'starting' || this.state === 'restarting') this.setState('running')
+ if (this.ending) return
+ const heapBudget = this.spec.heapLimitMiB * MIB * this.options.heapRestartPercent / 100
+ if (heapBudget && message.heapUsedBytes > heapBudget) {
+ this.restartForMemory(`its heap is at ${Math.round(message.heapUsedBytes / MIB)} MiB, past ${this.options.heapRestartPercent}% of its ${this.spec.heapLimitMiB} MiB limit`)
+ } else if (this.spec.rssLimitMiB && message.rssBytes > this.spec.rssLimitMiB * MIB) {
+ this.restartForMemory(`it is using ${Math.round(message.rssBytes / MIB)} MiB, over its ${this.spec.rssLimitMiB} MiB budget`)
+ }
+ }
+
+ private restartForMemory(why: string): void {
+ this.deps.log(`[harnessd] service ${this.spec.name}: ${why} — restarting it`)
+ this.killReason = 'memory'
+ this.end('SIGTERM')
+ }
+
+ private onExit(code: number | null, signal: NodeJS.Signals | null): void {
+ const exit = describeExit(code, signal)
+ this.clearTimer('heartbeatTimer')
+ this.clearTimer('killTimer')
+ this.child = null
+ this.lastExit = exit
+ if (this.state === 'stopping') {
+ this.lastExitReason = 'stopped'
+ this.deps.log(`[harnessd] service ${this.spec.name} stopped (${exit})`)
+ this.setState('stopped')
+ this.stopped()
+ return
+ }
+ const reason = this.killReason ?? 'crashed'
+ this.lastExitReason = reason
+ this.restarts++
+ const now = this.deps.now()
+ if (this.upAt && now - this.upAt >= this.options.backoffResetMs) this.backoff = this.options.initialBackoffMs
+ // A memory restart counts like a crash: a service over budget from the start must not spin.
+ this.crashes = [...this.crashes.filter((at) => now - at < this.options.parkWindowMs), now]
+ if (this.crashes.length >= this.options.parkCrashes) {
+ this.crashes = []
+ this.backoff = this.options.initialBackoffMs
+ this.deps.log(`[harnessd] service ${this.spec.name} ended ${this.options.parkCrashes} times in ${Math.round(this.options.parkWindowMs / 60_000)} min (${exit}, ${reason}) — parked; trying again in ${Math.round(this.options.parkRetryMs / 60_000)} min`)
+ this.setState('parked')
+ this.armTimer('restartTimer', () => { this.restartTimer = null; this.start() }, this.options.parkRetryMs)
+ return
+ }
+ const delay = this.backoff
+ this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs)
+ this.deps.log(`[harnessd] service ${this.spec.name} exited (${exit}, ${reason}) — restarting in ${delay} ms`)
+ this.setState('restarting')
+ this.armTimer('restartTimer', () => { this.restartTimer = null; this.start() }, delay)
+ }
+
+ private end(signal: NodeJS.Signals): void {
+ const child = this.child!
+ this.ending = true
+ this.clearTimer('heartbeatTimer')
+ child.kill(signal)
+ this.armTimer('killTimer', () => {
+ this.deps.log(`[harnessd] service ${this.spec.name} outlived its ${this.options.stopGraceMs} ms to stop — killing it`)
+ child.kill('SIGKILL')
+ }, this.options.stopGraceMs)
+ }
+
+ private watchHeartbeat(child: CoreHandle): void {
+ this.armTimer('heartbeatTimer', () => {
+ this.deps.log(`[harnessd] service ${this.spec.name} sent no heartbeat for ${this.options.heartbeatTimeoutMs} ms — it is hung; killing it`)
+ this.killReason = 'hung'
+ child.kill('SIGKILL')
+ }, this.options.heartbeatTimeoutMs)
+ }
+
+ private setState(state: ServiceState): void {
+ this.state = state
+ this.since = this.deps.wallClock()
+ }
+
+ private armTimer(name: 'heartbeatTimer' | 'killTimer' | 'restartTimer', run: () => void, ms: number): void {
+ this.clearTimer(name)
+ this[name] = this.deps.setTimer(run, ms)
+ }
+
+ private clearTimer(name: 'heartbeatTimer' | 'killTimer' | 'restartTimer'): void {
+ if (this[name] !== null) this.deps.clearTimer(this[name])
+ this[name] = null
+ }
+}
+
+/** Every enabled service's process, started together and stopped together. */
+export class ServiceSupervisor {
+ private readonly services: Service[]
+
+ constructor(
+ specs: readonly ServiceSpec[],
+ deps: ServiceSupervisorDeps,
+ options: ServiceSupervisorOptions = DEFAULT_SERVICE_OPTIONS,
+ /** What every service is started with: the token that lets the core know it, for one. */
+ env: Record<string, string> = {},
+ ) {
+ this.services = specs.map((spec) => new Service(spec, deps, options, env))
+ }
+
+ /** Start every service; none waits on another, or on the core. */
+ start(): void {
+ for (const service of this.services) service.start()
+ }
+
+ /** Stop every service; `done` once all are gone. */
+ stop(done: () => void): void {
+ let left = this.services.length
+ if (!left) { done(); return }
+ for (const service of this.services) service.stop(() => { if (--left === 0) done() })
+ }
+
+ status(): ServiceStatus[] {
+ return this.services.map((service) => service.status())
+ }
+}
+
+/**
+ * The services this build can run in their own processes, with their memory budgets. Which of them do
+ * is `HARNESSD_SERVICES` (`search`): off until named, while each one beds in, and the core runs a
+ * service that is not out here in its own process, as before.
+ */
+export const KNOWN_SERVICES: Readonly<Record<string, Omit<ServiceSpec, 'name'>>> = {
+ search: { heapLimitMiB: 1_024, rssLimitMiB: 2_048 },
+}
+
+/** The requests each of them answers, which the core routes to its process (core/serviceLinks.ts). */
+export const SERVICE_REQUESTS: Readonly<Record<string, readonly string[]>> = {
+ search: ['session_search', 'session_tail'],
+}
+
+/** Service timings from the environment (for tests and support); anything unset or invalid keeps its default. */
+export function serviceOptions(env: NodeJS.ProcessEnv): ServiceSupervisorOptions {
+ const read = (name: string, fallback: number, min: number): number => {
+ const value = Number(env[name])
+ return env[name] !== undefined && Number.isFinite(value) && value >= min ? value : fallback
+ }
+ const d = DEFAULT_SERVICE_OPTIONS
+ return {
+ // A second at least: below that a GC pause reads as a hang.
+ heartbeatTimeoutMs: read('HARNESSD_SERVICE_HEARTBEAT_TIMEOUT_MS', d.heartbeatTimeoutMs, 1_000),
+ stopGraceMs: read('HARNESSD_SERVICE_STOP_GRACE_MS', d.stopGraceMs, 1),
+ initialBackoffMs: read('HARNESSD_SERVICE_INITIAL_BACKOFF_MS', d.initialBackoffMs, 0),
+ maxBackoffMs: read('HARNESSD_SERVICE_MAX_BACKOFF_MS', d.maxBackoffMs, 0),
+ backoffResetMs: read('HARNESSD_SERVICE_BACKOFF_RESET_MS', d.backoffResetMs, 0),
+ heapRestartPercent: Math.min(100, read('HARNESSD_SERVICE_HEAP_RESTART_PERCENT', d.heapRestartPercent, 1)),
+ parkCrashes: read('HARNESSD_SERVICE_PARK_CRASHES', d.parkCrashes, 1),
+ parkWindowMs: read('HARNESSD_SERVICE_PARK_WINDOW_MS', d.parkWindowMs, 0),
+ parkRetryMs: read('HARNESSD_SERVICE_PARK_RETRY_MS', d.parkRetryMs, 0),
+ }
+}
+
+/** The services to run, from `HARNESSD_SERVICES` (`search,devices`): only names this build knows.
+ * `HARNESSD_SERVICE_HEAP_LIMIT_MIB` gives every one the same heap limit instead (tests, support). */
+export function serviceSpecs(env: NodeJS.ProcessEnv, known: Readonly<Record<string, Omit<ServiceSpec, 'name'>>>): ServiceSpec[] {
+ const names = (env.HARNESSD_SERVICES ?? '').split(',').map((name) => name.trim()).filter(Boolean)
+ const heap = Number(env.HARNESSD_SERVICE_HEAP_LIMIT_MIB)
+ return [...new Set(names)].filter((name) => Object.hasOwn(known, name)).map((name) => ({
+ name, ...known[name], ...(Number.isInteger(heap) && heap > 0 ? { heapLimitMiB: heap } : {}),
+ }))
+}
diff --git a/cli/src/localWsServer.ts b/cli/src/localWsServer.ts
index eb1e39c2a..8fb9960f6 100644
--- a/cli/src/localWsServer.ts
+++ b/cli/src/localWsServer.ts
@@ -112,6 +112,11 @@ export interface LocalWsServerOptions {
/** Questions still waiting on the user, as the `commander_question` frames that announced them. A window
* that connects after one was asked is handed them, so a terminal opened late still sees who is blocked. */
openQuestions?: () => Frame[]
+ /** The core's end of its out-of-process services (core/serviceLinks.ts): a `machine_select` with
+ * `role: "service"` is handed here, and the connection is the service's from then on. */
+ services?: {
+ accept(service: string, token: string, sink: LocalClientSink, close: (code: number, reason: string) => void): { receive(frame: Frame): void; closed(): void } | null
+ }
/**
* The window answering a `voice_route_request` — words spoken into the dial that IT was asked to route.
*
@@ -327,6 +332,8 @@ export function attachLocalWsServer(server: http.Server, options: LocalWsServerO
let sentPanes = false
let sentSwarms = false
let chain = Promise.resolve()
+ /** Set when this connection is one of the core's services, not a window. */
+ let serviceLink: { receive(frame: Frame): void; closed(): void } | null = null
const sink: LocalClientSink = {
sendFrame: (frame) => {
@@ -375,6 +382,16 @@ export function attachLocalWsServer(server: http.Server, options: LocalWsServerO
}
return
}
+ // One of the core's own services, started by harnessd's master: its frames go to its link, and it
+ // never joins the windows' event stream.
+ if (requestedMachineId === options.machineId && payload.role === 'service') {
+ const link = options.services?.accept(String(payload.service ?? ''), String(payload.token ?? ''), sink, close) ?? null
+ if (!link) { close(4401, 'service refused'); return }
+ serviceLink = link
+ selected = true
+ sink.sendFrame({ type: 'connected', payload: { machineId: options.machineId, transport: 'local', localProtocolVersion: LOCAL_WS_PROTOCOL_VERSION, service: payload.service } })
+ return
+ }
if (requestedMachineId === options.machineId) {
tool = payload.tool === true
if (!options.backend.registerLocalClient(connId, terminalSink, tool ? { tool: true } : {})) {
@@ -438,6 +455,12 @@ export function attachLocalWsServer(server: http.Server, options: LocalWsServerO
ws.on('message', (raw, isBinary) => {
chain = chain.then(() => {
if (!selected) return selectMachine(raw, isBinary)
+ if (serviceLink) {
+ const frame = isBinary ? null : jsonFrame(raw)
+ if (!frame) { close(4400, 'invalid json frame'); return }
+ serviceLink.receive(frame)
+ return
+ }
// Parsed ONCE. Every sniff below used to re-run JSON.parse on the same bytes — up to seven
// times for a frame that matched none of them, which is what a terminal_ack (every 16ms of
@@ -737,7 +760,8 @@ export function attachLocalWsServer(server: http.Server, options: LocalWsServerO
if (sentPanes) options.onAppPanes?.([], false)
if (sentSwarms) options.onAppSwarms?.(null)
if (sentSwarms) options.onAppTabAgents?.(connId, null)
- if (relay) { relay.detach(); relay = null }
+ if (serviceLink) { serviceLink.closed(); serviceLink = null }
+ else if (relay) { relay.detach(); relay = null }
else if (selected) void options.backend.unregisterLocalClient(connId)
selected = false
}
diff --git a/cli/src/services/AGENTS.md b/cli/src/services/AGENTS.md
new file mode 100644
index 000000000..8737d653e
--- /dev/null
+++ b/cli/src/services/AGENTS.md
@@ -0,0 +1,41 @@
+# Services
+
+A service is a feature the core can run without. It may fail; the core may not. These rules are what
+let several people build features at once without touching the core or each other.
+
+## Adding a service
+
+1. **Write `start<Name>(core: CoreApi, ports: CorePorts)` in `src/services/<name>.ts`.** Read the core
+ only through `core` (`src/core/api.ts`). If `CoreApi` lacks something you need, add it to `CoreApi`
+ in a separate, reviewed change; never import a core module, the registry, `cli.ts` or
+ `backendSocket.ts`.
+2. **If the core must call the service, give it a port.** Add `<Name>Port` and its fallbacks
+ (`<NAME>_FALLBACKS`) to `src/core/api.ts`. A fallback is what the core does while the service is
+ off or failing: `undefined` for nothing, a value, or `FAIL` to answer that one request
+ `SERVICE_UNAVAILABLE`. The core only ever calls the port, never the service.
+3. **Start it in `src/cli.ts` with `serviceHost.start('<name>', start<Name>, coreApi, <NAME>_FALLBACKS)`.**
+ Never call `start<Name>` directly: the host leaves a service off when its start throws, guards every
+ call, and switches it off after five failures in a minute.
+4. **Test it to 100%** with `fakeCore()` (`src/testing/fakeCore.ts`); `npm run test:core` covers this
+ folder. Prove failure isolation end to end with `HARNESSD_TEST_FAULTS=<name>` (its start fails) and
+ `<name>.<member>` (one call fails); see `e2e/services.e2e.ts`.
+
+## Running in its own process
+
+A service that can crash natively, hang or leak should run in its own process, where it costs only
+itself. Search does: `HARNESSD_SERVICES=search`.
+
+- `src/harnessd/services.ts` runs it (`KNOWN_SERVICES`, with its memory budget) and lists the
+ requests it answers (`SERVICE_REQUESTS`).
+- `src/services/process.ts` is the process's side: heartbeats to the master, the connection to the
+ core with the master's token, reconnecting after core restarts.
+- `src/core/serviceLinks.ts` is the core's side: it routes those requests to the process, answers
+ `SERVICE_UNAVAILABLE` while it is down, and holds what the service must not miss.
+- `src/services/searchProcess.ts` is the example to copy. `e2e/serviceProcesses.e2e.ts` is the proof
+ to copy: killed, hung (SIGSTOP), leaking, crashing on every start.
+
+## Do not
+
+- Do not keep state the core needs. If the core would break without your data, it is not a service.
+- Do not hold credentials. Ask `core.account`.
+- Do not write to tmux, the registry or another service's files.
diff --git a/cli/src/services/CLAUDE.md b/cli/src/services/CLAUDE.md
new file mode 100644
index 000000000..6a4029179
--- /dev/null
+++ b/cli/src/services/CLAUDE.md
@@ -0,0 +1,2 @@
+<!-- One set of rules for every agent: Codex reads AGENTS.md, and Claude Code reads it through this import. Claude Code reads AGENTS.md by itself only from v2.1.277, and only when no CLAUDE.md is in the working directory or above it, so without this a session started in a folder with a CLAUDE.md would miss these rules. -->
diff --git a/cli/src/services/process.spec.ts b/cli/src/services/process.spec.ts
new file mode 100644
index 000000000..8b818377b
--- /dev/null
+++ b/cli/src/services/process.spec.ts
@@ -0,0 +1,237 @@
+import { EventEmitter } from 'node:events'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import WebSocket from 'ws'
+import type { MasterChannel } from '../harnessd/coreLink.js'
+import { runServiceProcess, serviceFaults, type ServiceProcessOptions } from './process.js'
+
+/** A socket that records what it is sent, and is told what the core says. */
+class FakeSocket extends EventEmitter {
+ readyState: number = WebSocket.CONNECTING
+ readonly sent: Array<{ type: string; payload: Record<string, unknown> }> = []
+ closed = false
+ constructor(readonly url: string, private readonly failSend = false) { super() }
+ send(data: string): void {
+ if (this.failSend) throw new Error('gone')
+ this.sent.push(JSON.parse(data))
+ }
+ close(): void { this.closed = true }
+ open(): void { this.readyState = WebSocket.OPEN; this.emit('open') }
+ say(frame: unknown): void { this.emit('message', Buffer.from(typeof frame === 'string' ? frame : JSON.stringify(frame))) }
+ drop(): void { this.readyState = WebSocket.CLOSED; this.emit('error', new Error('reset')); this.emit('close') }
+}
+
+class FakeChannel extends EventEmitter implements MasterChannel {
+ readonly beats: unknown[] = []
+ readonly parentPid = 1
+ send = (message: unknown) => { this.beats.push(message) }
+ memoryUsage() { return { rss: 10, heapUsed: 5 } }
+}
+
+describe('a service in its own process', () => {
+ let sockets: FakeSocket[]
+ let exits: number[]
+ let lines: string[]
+ const socket = () => sockets[sockets.length - 1]
+ const run = (over: Partial<ServiceProcessOptions> = {}) => runServiceProcess({
+ name: 'search',
+ socketPath: '/data/daemon-1.sock',
+ machineId: 'machine-1',
+ token: 'token-1',
+ requests: {},
+ channel: new FakeChannel(),
+ env: {},
+ connect: (url) => { const next = new FakeSocket(url); sockets.push(next); return next as unknown as WebSocket },
+ exit: (code) => exits.push(code),
+ loopDelay: { take: () => 4, stop: vi.fn() },
+ log: (line) => lines.push(line),
+ initialBackoffMs: 100,
+ maxBackoffMs: 400,
+ ...over,
+ })
+
+ beforeEach(() => { vi.useFakeTimers(); sockets = []; exits = []; lines = [] })
+ afterEach(() => vi.useRealTimers())
+
+ it('connects to the core\'s socket as the service it is, with the master\'s token', () => {
+ const service = run()
+ expect(socket().url).toBe('ws+unix:///data/daemon-1.sock:/api/local-ws')
+ socket().open()
+ expect(socket().sent).toEqual([{ type: 'machine_select', payload: { machineId: 'machine-1', localProtocolVersion: 1, role: 'service', service: 'search', token: 'token-1' } }])
+ service.stop()
+ expect(socket().closed).toBe(true)
+ })
+
+ it('beats to the master as the core does, and exits when the master goes', () => {
+ const channel = new FakeChannel()
+ const loopDelay = { take: () => 4, stop: vi.fn() }
+ run({ channel, loopDelay, env: { HARNESSD_WATCHDOG_MS: '3000' } })
+ expect(channel.beats).toEqual([{ type: 'harnessd:heartbeat', rssBytes: 10, heapUsedBytes: 5, loopDelayMs: 4 }])
+ vi.advanceTimersByTime(1_000)
+ expect(channel.beats).toHaveLength(2)
+ channel.emit('disconnect')
+ expect(exits).toEqual([0])
+ expect(loopDelay.stop).toHaveBeenCalled()
+ vi.advanceTimersByTime(10_000)
+ expect(channel.beats).toHaveLength(2)
+ })
+
+ it('without a master to beat to, runs all the same', () => {
+ const channel = new FakeChannel()
+ ;(channel as { send?: unknown }).send = undefined
+ const service = run({ channel })
+ expect(channel.beats).toEqual([])
+ service.stop()
+ })
+
+ it('answers the requests routed to it under their own request, failures included', async () => {
+ const requests = {
+ session_search: vi.fn((payload: Record<string, unknown>) => ({ hits: [payload.query] })),
+ session_tail: vi.fn(async () => { throw new Error('index gone') }),
+ session_other: vi.fn(() => { throw 'not an error' }),
+ }
+ run({ requests })
+ socket().open()
+ socket().say({ type: 'session_search', payload: { query: 'zebra', requestId: 'r1' } })
+ socket().say({ type: 'session_tail', payload: { requestId: 'r2' } })
+ socket().say({ type: 'session_other', payload: { requestId: 'r3' } })
+ // Not a request it answers, not JSON, no payload: nothing.
+ socket().say({ type: 'agents_list', payload: { requestId: 'r4' } })
+ socket().say('{not json')
+ socket().say({ type: 'toString' })
+ socket().say({ type: 7, payload: { requestId: 'r5' } })
+ await vi.waitFor(() => expect(socket().sent).toHaveLength(4))
+ expect(socket().sent.slice(1)).toEqual(expect.arrayContaining([
+ { type: 'session_search_result', payload: { hits: ['zebra'], requestId: 'r1' } },
+ { type: 'session_tail_result', payload: { error: 'SERVICE_FAILED', detail: 'index gone', requestId: 'r2' } },
+ { type: 'session_other_result', payload: { error: 'SERVICE_FAILED', detail: 'not an error', requestId: 'r3' } },
+ ]))
+ })
+
+ it('hears what the core tells it, and asks the core what it needs to know', async () => {
+ const onEvent = vi.fn()
+ let core: Parameters<NonNullable<ServiceProcessOptions['onConnected']>>[0] | null = null
+ run({ onEvent, onConnected: (connection) => { core = connection } })
+ socket().open()
+ socket().say({ type: 'connected', payload: { service: 'search' } })
+ expect(lines).toContain('[service search] connected to the core')
+ socket().say({ type: 'service_event', payload: { kind: 'touch', sessionId: 's1' } })
+ socket().say({ type: 'service_event' })
+ expect(onEvent).toHaveBeenCalledWith({ kind: 'touch', sessionId: 's1' })
+ expect(onEvent).toHaveBeenCalledWith({})
+ const asked = core!.query('agents')
+ expect(socket().sent.at(-1)).toEqual({ type: 'service_query', payload: { query: 'agents', requestId: 'search-1' } })
+ // An answer to nothing it asked is ignored.
+ socket().say({ type: 'service_query_result', payload: { requestId: 'search-9', agents: [] } })
+ socket().say({ type: 'service_query_result', payload: { requestId: 'search-1', agents: [{ agentId: 'a' }] } })
+ await expect(asked).resolves.toEqual({ agents: [{ agentId: 'a' }] })
+ // Asked with no connection, or when the connection goes first: rejected, not left hanging.
+ const pending = core!.query('agents', { since: 1 })
+ socket().drop()
+ await expect(pending).rejects.toThrow('the core went away')
+ await expect(core!.query('agents')).rejects.toThrow('not connected to the core')
+ })
+
+ it('reconnects with a backoff that doubles and caps while the core is away, and resets once connected', () => {
+ const service = run()
+ socket().drop()
+ expect(sockets).toHaveLength(1)
+ vi.advanceTimersByTime(100)
+ expect(sockets).toHaveLength(2)
+ socket().drop()
+ vi.advanceTimersByTime(199)
+ expect(sockets).toHaveLength(2)
+ vi.advanceTimersByTime(1)
+ expect(sockets).toHaveLength(3)
+ socket().drop()
+ vi.advanceTimersByTime(400)
+ socket().drop()
+ vi.advanceTimersByTime(400)
+ expect(sockets).toHaveLength(5)
+ socket().open()
+ socket().say({ type: 'connected' })
+ socket().drop()
+ vi.advanceTimersByTime(100)
+ expect(sockets).toHaveLength(6)
+ // Stopped: no reconnecting, and a socket that closes afterwards changes nothing.
+ service.stop()
+ socket().drop()
+ vi.advanceTimersByTime(10_000)
+ expect(sockets).toHaveLength(6)
+ })
+
+ it('stops a reconnect it had scheduled, and stopping twice is stopping once', () => {
+ const service = run()
+ socket().drop()
+ service.stop()
+ service.stop()
+ vi.advanceTimersByTime(10_000)
+ expect(sockets).toHaveLength(1)
+ })
+
+ it('a socket that cannot be written to costs nothing: the core answers for it', async () => {
+ run({
+ requests: { session_search: () => ({ hits: [] }) },
+ connect: (url) => { const next = new FakeSocket(url, true); sockets.push(next); return next as unknown as WebSocket },
+ })
+ socket().open()
+ socket().say({ type: 'session_search', payload: { requestId: 'r1' } })
+ await vi.advanceTimersByTimeAsync(0)
+ expect(socket().sent).toEqual([])
+ })
+
+ it('reads its test faults from the environment, its own only', () => {
+ expect([...serviceFaults({ HARNESSD_TEST_FAULTS: 'search.crash, search.hang,search.leak,devices.crash,search.other,search' }, 'search')]).toEqual(['crash', 'leak'])
+ expect(serviceFaults({}, 'search').size).toBe(0)
+ })
+
+ it('crashes or leaks on purpose when a test asks it to', () => {
+ run({ env: { HARNESSD_TEST_FAULTS: 'search.crash' } })
+ vi.advanceTimersByTime(200)
+ expect(exits).toEqual([1])
+ const leaking = run({ env: { HARNESSD_TEST_FAULTS: 'search.leak' } })
+ vi.advanceTimersByTime(300)
+ leaking.stop()
+ // Without a fault, connecting is just connecting.
+ run({ env: {} })
+ socket().open()
+ socket().say({ type: 'connected' })
+ expect(lines.at(-1)).toBe('[service search] connected to the core')
+ })
+
+ it('says it connected on the console by default', () => {
+ const log = vi.spyOn(console, 'log').mockImplementation(() => {})
+ const service = runServiceProcess({
+ name: 'search', socketPath: '/data/daemon-1.sock', machineId: 'm', token: 't', requests: {}, env: {},
+ channel: Object.assign(new FakeChannel(), { send: undefined }),
+ connect: (url) => { const next = new FakeSocket(url); sockets.push(next); return next as unknown as WebSocket },
+ loopDelay: { take: () => 0, stop: () => {} },
+ })
+ socket().open()
+ socket().say({ type: 'connected' })
+ expect(log).toHaveBeenCalledWith('[service search] connected to the core')
+ service.stop()
+ log.mockRestore()
+ })
+
+ it('runs on this process by default: its channel, its exit, its sockets', () => {
+ // No beat may reach the test runner's own channel: this process's `send` is set aside.
+ const send = process.send
+ ;(process as { send?: unknown }).send = undefined
+ const on = vi.spyOn(process, 'once').mockImplementation(() => process)
+ const exit = vi.spyOn(process, 'exit').mockImplementation((() => undefined) as never)
+ const log = vi.spyOn(console, 'log').mockImplementation(() => {})
+ try {
+ const service = runServiceProcess({ name: 'search', socketPath: '/nonexistent/daemon.sock', machineId: 'm', token: 't', requests: {} })
+ expect(on).toHaveBeenCalledWith('disconnect', expect.any(Function))
+ const leave = on.mock.calls.find(([event]) => event === 'disconnect')![1] as () => void
+ leave()
+ expect(exit).toHaveBeenCalledWith(0)
+ service.stop()
+ } finally {
+ ;(process as { send?: unknown }).send = send
+ on.mockRestore()
+ exit.mockRestore()
+ log.mockRestore()
+ }
+ })
+})
diff --git a/cli/src/services/process.ts b/cli/src/services/process.ts
new file mode 100644
index 000000000..ad0ae2fef
--- /dev/null
+++ b/cli/src/services/process.ts
@@ -0,0 +1,175 @@
+/**
+ * A service in its own process (`harness __service <name>`), started and watched by harnessd's master
+ * (harnessd/services.ts).
+ *
+ * It tells the master it is alive over the spawn channel, as the core does, and exits when the master
+ * goes, so a crash never leaves one behind. It reaches the core the way a window does — the core's local
+ * socket — under the service role, with the token the master gave both (core/serviceLinks.ts), and
+ * reconnects with backoff whenever the core restarts; the service keeps its own state meanwhile. It
+ * answers the requests the core routes to it, hears what the core tells it, and asks the core what it
+ * needs to know.
+ */
+import WebSocket from 'ws'
+import { heartbeatInterval, processLoopDelay, type LoopDelay, type MasterChannel } from '../harnessd/coreLink.js'
+
+type Payload = Record<string, unknown>
+
+/** The core, as a connected service reaches it. */
+export interface CoreConnection {
+ /** Ask the core something (`service_query`); rejects when the connection goes before it answers. */
+ query(query: string, payload?: Payload): Promise<Payload>
+}
+
+export interface ServiceProcessOptions {
+ name: string
+ /** The core's local socket. */
+ socketPath: string
+ /** This machine, as the core's `machine_select` expects it. */
+ machineId: string
+ token: string
+ /** The requests the core routes here, by type: each answered under `<type>_result`. */
+ requests: Readonly<Record<string, (payload: Payload) => Payload | Promise<Payload>>>
+ /** What the core tells this service (`service_event`). */
+ onEvent?: (payload: Payload) => void
+ /** Each time it is connected to a core: the first time, and after every core restart. */
+ onConnected?: (core: CoreConnection) => void
+ /** The master's spawn channel; this process when absent. */
+ channel?: MasterChannel
+ env?: NodeJS.ProcessEnv
+ connect?: (url: string) => WebSocket
+ exit?: (code: number) => void
+ loopDelay?: LoopDelay
+ log?: (line: string) => void
+ /** The reconnect delay starts here and doubles to `maxBackoffMs`. */
+ initialBackoffMs?: number
+ maxBackoffMs?: number
+}
+
+/**
+ * `HARNESSD_TEST_FAULTS` for a service's own process, as the end-to-end suite uses them: `<name>.crash`
+ * exits soon after start, `<name>.leak` keeps allocating memory it never lets go. Each is what the master
+ * must survive. (A hang needs no fault: a stopped process — SIGSTOP — beats no more than a hung one.)
+ */
+export function serviceFaults(env: NodeJS.ProcessEnv, name: string): Set<'crash' | 'leak'> {
+ const faults = new Set<'crash' | 'leak'>()
+ for (const entry of (env.HARNESSD_TEST_FAULTS ?? '').split(',')) {
+ const [service, fault] = entry.trim().split('.')
+ if (service === name && (fault === 'crash' || fault === 'leak')) faults.add(fault)
+ }
+ return faults
+}
+
+export interface ServiceProcess {
+ stop(): void
+}
+
+
+export function runServiceProcess(options: ServiceProcessOptions): ServiceProcess {
+ const channel: MasterChannel = options.channel ?? (process as unknown as MasterChannel)
+ const env = options.env ?? process.env
+ const connect = options.connect ?? ((url: string) => new WebSocket(url))
+ const exit = options.exit ?? ((code: number) => process.exit(code))
+ const log = options.log ?? ((line: string) => console.log(line))
+ const initialBackoffMs = options.initialBackoffMs ?? 250
+ const maxBackoffMs = options.maxBackoffMs ?? 5_000
+ let stopped = false
+ let backoff = initialBackoffMs
+ let socket: WebSocket | null = null
+ let reconnectTimer: ReturnType<typeof setTimeout> | null = null
+ let queries = 0
+ const pending = new Map<string, { resolve: (payload: Payload) => void; reject: (error: Error) => void }>()
+
+ // Alive, to the master: the same beat the core sends, so the same watchdog and budgets apply.
+ const loopDelay = options.loopDelay ?? processLoopDelay()
+ const beat = (): void => {
+ const memory = channel.memoryUsage()
+ channel.send?.({ type: 'harnessd:heartbeat', rssBytes: memory.rss, heapUsedBytes: memory.heapUsed, loopDelayMs: loopDelay.take() })
+ }
+ const beating = channel.send ? setInterval(beat, heartbeatInterval(env)) : null
+ if (channel.send) beat()
+ const faults = serviceFaults(env, options.name)
+ const leaked: Buffer[] = []
+ if (faults.has('crash')) setTimeout(() => exit(1), 200)
+ const leaking = faults.has('leak') ? setInterval(() => { leaked.push(Buffer.alloc(8 * 1024 * 1024, 1)); leaked.push(Buffer.from(new Array(200_000).fill('x').join(''))) }, 100) : null
+ // The master is gone: so is this service, rather than an orphan holding what it holds.
+ channel.once('disconnect', () => { stop(); exit(0) })
+
+ const send = (frame: { type: string; payload: Payload }): void => {
+ try { socket?.send(JSON.stringify(frame)) } catch { /* the socket is going; the core answers for us */ }
+ }
+
+ const core: CoreConnection = {
+ query: (query, payload = {}) => new Promise<Payload>((resolve, reject) => {
+ if (!socket || socket.readyState !== WebSocket.OPEN) { reject(new Error('not connected to the core')); return }
+ const requestId = `${options.name}-${++queries}`
+ pending.set(requestId, { resolve, reject })
+ send({ type: 'service_query', payload: { ...payload, query, requestId } })
+ }),
+ }
+
+ const onFrame = (raw: WebSocket.RawData): void => {
+ let frame: { type?: unknown; payload?: Payload }
+ try { frame = JSON.parse(raw.toString()) as typeof frame } catch { return }
+ const payload = frame.payload ?? {}
+ if (frame.type === 'connected') {
+ backoff = initialBackoffMs
+ log(`[service ${options.name}] connected to the core`)
+ options.onConnected?.(core)
+ return
+ }
+ if (frame.type === 'service_event') { options.onEvent?.(payload); return }
+ if (frame.type === 'service_query_result') {
+ const waiting = pending.get(String(payload.requestId))
+ if (!waiting) return
+ pending.delete(String(payload.requestId))
+ const { requestId: _id, ...answer } = payload
+ waiting.resolve(answer)
+ return
+ }
+ const type = typeof frame.type === 'string' ? frame.type : ''
+ const handle = Object.hasOwn(options.requests, type) ? options.requests[type] : undefined
+ if (!handle) return
+ const requestId = payload.requestId
+ // A request that fails here is answered as failed, never left for the core's timeout.
+ void Promise.resolve()
+ .then(() => handle(payload))
+ .catch((error: unknown) => ({ error: 'SERVICE_FAILED', detail: error instanceof Error ? error.message : String(error) }))
+ .then((result) => send({ type: `${type}_result`, payload: { ...result, requestId } }))
+ }
+
+ // Only ever run at start and from the reconnect timer, which `stop` clears.
+ const dial = (): void => {
+ const ws = connect(`ws+unix://${options.socketPath}:/api/local-ws`)
+ socket = ws
+ ws.on('open', () => {
+ // Through `send`, which a socket failing as it opens cannot turn into a crash: `close` follows.
+ send({ type: 'machine_select', payload: {
+ machineId: options.machineId, localProtocolVersion: 1, role: 'service', service: options.name, token: options.token,
+ } })
+ })
+ ws.on('message', onFrame)
+ ws.on('error', () => { /* `close` follows, and reconnects */ })
+ ws.on('close', () => {
+ // A new socket is dialled only after this one closes, so this is always the current one.
+ socket = null
+ for (const [id, waiting] of pending) { pending.delete(id); waiting.reject(new Error('the core went away')) }
+ if (stopped) return
+ const delay = backoff
+ backoff = Math.min(backoff * 2, maxBackoffMs)
+ reconnectTimer = setTimeout(() => { reconnectTimer = null; dial() }, delay)
+ })
+ }
+
+ const stop = (): void => {
+ if (stopped) return
+ stopped = true
+ if (beating) clearInterval(beating)
+ if (leaking) clearInterval(leaking)
+ loopDelay.stop()
+ if (reconnectTimer) clearTimeout(reconnectTimer)
+ try { socket?.close() } catch { /* already closed */ }
+ }
+
+ dial()
+ return { stop }
+}
diff --git a/cli/src/services/searchProcess.spec.ts b/cli/src/services/searchProcess.spec.ts
new file mode 100644
index 000000000..e5792b7f1
--- /dev/null
+++ b/cli/src/services/searchProcess.spec.ts
@@ -0,0 +1,142 @@
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { afterEach, describe, expect, it, vi } from 'vitest'
+import { runServiceProcess, type CoreConnection, type ServiceProcessOptions } from './process.js'
+import { runSearchService, searchCoreApi } from './searchProcess.js'
+
+// The real defaults would reach this machine's own conversations and a real socket: never in a test.
+vi.mock('../lib/sessionSearch/externals/index.js', () => ({ externalProviders: () => [] }))
+vi.mock('./process.js', () => ({ runServiceProcess: vi.fn(() => ({ stop: vi.fn() })) }))
+
+const SESSION = '0b0c1d2e-3f40-4a5b-8c6d-7e8f90a1b2c3'
+
+describe('search in its own process', () => {
+ const dirs: string[] = []
+ const services: Array<{ stop(): void }> = []
+ afterEach(() => {
+ for (const service of services.splice(0)) service.stop()
+ for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true })
+ })
+
+ const setup = (over: Partial<Parameters<typeof runSearchService>[0]> = {}) => {
+ const dataDir = mkdtempSync(join(tmpdir(), 'search-process-'))
+ dirs.push(dataDir)
+ let options: ServiceProcessOptions | null = null
+ const stop = vi.fn()
+ const service = runSearchService({
+ dataDir, socketPath: '/data/daemon-1.sock', machineId: 'm', token: 't', providers: [],
+ run: (given) => { options = given; return { stop } },
+ ...over,
+ })
+ services.push(service)
+ return { dataDir, options: options!, stop, service }
+ }
+ /** A Claude Code conversation on disk, and its agent as the core describes it. */
+ const conversation = (dataDir: string, words: string) => {
+ const folder = join(dataDir, 'claude')
+ mkdirSync(folder, { recursive: true })
+ const transcriptPath = join(folder, `${SESSION}.jsonl`)
+ const at = new Date().toISOString()
+ writeFileSync(transcriptPath, [
+ { type: 'user', uuid: 'u1', timestamp: at, sessionId: SESSION, message: { role: 'user', content: words } },
+ { type: 'assistant', uuid: 'a1', timestamp: at, sessionId: SESSION, message: { role: 'assistant', content: [{ type: 'text', text: `about ${words}` }] } },
+ ].map((line) => JSON.stringify(line)).join('\n') + '\n')
+ return {
+ schemaVersion: 2, active: true, agentId: 'agent-1', sessionId: SESSION, engine: 'claude', transcriptPath, cwd: dataDir,
+ title: 'a chat', registeredAt: 1, boundAt: 1, lastTranscriptAt: Date.now(), lastHookAt: 0, runtimes: [], displayName: 'My agent',
+ }
+ }
+ const core = (agents: unknown[]): CoreConnection => ({ query: vi.fn(async () => ({ agents })) })
+
+ it('indexes what the core\'s agents said once told of a turn, finds it, and forgets it when purged', async () => {
+ const { dataDir, options } = setup()
+ const agent = conversation(dataDir, 'a zebrafish named Ines')
+ // Told of a turn before it is connected: indexed once it knows the agents.
+ options.onEvent!({ kind: 'touch', sessionId: SESSION })
+ options.onConnected!(core([agent]))
+ options.onEvent!({ kind: 'touch', sessionId: SESSION })
+ await vi.waitFor(async () => {
+ expect(JSON.stringify(await options.requests.session_search({ query: 'zebrafish' }))).toContain(SESSION)
+ }, { timeout: 15_000, interval: 250 })
+ const tail = await options.requests.session_tail({ sessionId: SESSION, beforeTurn: 5, maxChars: 2_000 })
+ expect(tail.error).toBeUndefined()
+ // Something it is told that it does not act on changes nothing.
+ options.onEvent!({ kind: 'renamed', sessionId: SESSION })
+ expect(JSON.stringify(await options.requests.session_search({ query: 'zebrafish' }))).toContain(SESSION)
+ options.onEvent!({ kind: 'deleteHistory', sessionId: SESSION })
+ expect(JSON.stringify(await options.requests.session_search({ query: 'zebrafish', limit: 5 }))).not.toContain(SESSION)
+ })
+
+ it('answers what it cannot serve the way the core does', async () => {
+ const { options } = setup()
+ expect(await options.requests.session_tail({})).toEqual({ error: 'BAD_SESSION' })
+ expect(await options.requests.session_tail({ sessionId: 'never-indexed' })).toEqual({ error: 'NOT_INDEXED', sessionId: 'never-indexed' })
+ expect((await options.requests.session_search({ query: 7, limit: 'x', from: Number.NaN })).error).toBeUndefined()
+ // An event naming no session is nothing to act on.
+ options.onEvent!({ kind: 'touch' })
+ })
+
+ it('keeps the agents it knew when the core cannot answer for them', async () => {
+ const { dataDir, options } = setup()
+ const agent = conversation(dataDir, 'a heron')
+ options.onConnected!(core([agent]))
+ options.onConnected!({ query: vi.fn(async () => { throw new Error('the core went away') }) })
+ options.onConnected!({ query: vi.fn(async () => ({ error: 'UNKNOWN_QUERY' })) })
+ options.onEvent!({ kind: 'touch', sessionId: SESSION })
+ await vi.waitFor(async () => {
+ expect(JSON.stringify(await options.requests.session_search({ query: 'heron' }))).toContain(SESSION)
+ }, { timeout: 15_000, interval: 250 })
+ })
+
+ it('without an index (no node:sqlite) says search is unavailable, as the core would', async () => {
+ const { options } = setup({ start: () => {} })
+ expect(await options.requests.session_search({ query: 'anything' })).toEqual({ error: 'SEARCH_UNAVAILABLE' })
+ expect(await options.requests.session_tail({ sessionId: SESSION })).toEqual({ error: 'SEARCH_UNAVAILABLE' })
+ options.onEvent!({ kind: 'touch', sessionId: SESSION })
+ })
+
+ it('stops its process and its index together', () => {
+ const { stop, service } = setup()
+ service.stop()
+ expect(stop).toHaveBeenCalledOnce()
+ const unindexed = setup({ start: () => {} })
+ unindexed.service.stop()
+ expect(unindexed.stop).toHaveBeenCalledOnce()
+ })
+
+ it('runs on a core API of the agents the core last named, answering what search never asks as nothing', async () => {
+ const live = { agentId: 'live', active: true, displayName: 'Live one' }
+ const stopped = { agentId: 'stopped', active: false }
+ const api = searchCoreApi('/data', () => [live, stopped] as never, [])
+ expect(api.dataDir).toBe('/data')
+ expect(api.agents.all()).toEqual([live, stopped])
+ expect(api.agents.live()).toEqual([live])
+ expect(api.agents.byAgent('stopped')).toBe(stopped)
+ expect(api.agents.displayName(live as never)).toBe('Live one')
+ expect(api.agents.displayName(stopped as never)).toBe('')
+ expect(api.agents.advertised()).toEqual([])
+ expect(api.agents.terminalAvailable('live')).toBe(false)
+ api.agents.sync(live as never)
+ expect(api.transcripts.databaseHistory({ engine: 'claude' } as never)).toBeUndefined()
+ expect(api.external.sessions.list()).toEqual([])
+ await expect(api.account.mintGridName()).resolves.toBeNull()
+ await expect(api.account.accessToken()).rejects.toThrow('search holds no credential')
+ api.clients.viewerChanged('live')
+ api.clients.gridNamed('grid')
+ })
+
+ it('runs as a real service process, over the conversations this machine has, by default', () => {
+ const dataDir = mkdtempSync(join(tmpdir(), 'search-process-'))
+ dirs.push(dataDir)
+ const service = runSearchService({ dataDir, socketPath: '/data/daemon-1.sock', machineId: 'm', token: 't' })
+ services.push(service)
+ expect(runServiceProcess).toHaveBeenCalledWith(expect.objectContaining({ name: 'search', socketPath: '/data/daemon-1.sock' }))
+ })
+
+ it('reaches the core as `search`, through the socket and token it was given', () => {
+ const { options } = setup()
+ expect(options).toMatchObject({ name: 'search', socketPath: '/data/daemon-1.sock', machineId: 'm', token: 't' })
+ expect(Object.keys(options.requests).sort()).toEqual(['session_search', 'session_tail'])
+ })
+})
diff --git a/cli/src/services/searchProcess.ts b/cli/src/services/searchProcess.ts
new file mode 100644
index 000000000..7b6110eb2
--- /dev/null
+++ b/cli/src/services/searchProcess.ts
@@ -0,0 +1,117 @@
+/**
+ * Session search in its own process (`harness __service search`, with `HARNESSD_SERVICES=search`).
+ *
+ * The same index as in the core's process (services/search.ts, lib/sessionSearch/), on a core API built
+ * here: the agents come from the core (`service_query agents`, refreshed on every turn boundary it is
+ * told of), and everything else — the database readers, the conversations Harness did not start — is
+ * read here, as the core would read it. Its SQLite index, its memory and its native code are this
+ * process's alone: a crash or a leak in search costs search, and the master starts it again.
+ */
+import type { CoreApi } from '../core/api.js'
+import { emptyPorts } from '../core/api.js'
+import { databaseHistory } from '../core/transcripts/databaseHistory.js'
+import { ExternalSessions, OpenSessions } from '../lib/sessionSearch/external.js'
+import { externalProviders } from '../lib/sessionSearch/externals/index.js'
+import type { RegisteredSession } from '../lib/registry.js'
+import { runServiceProcess, type CoreConnection, type ServiceProcess } from './process.js'
+import { startSearch } from './search.js'
+
+type Payload = Record<string, unknown>
+const number = (value: unknown) => typeof value === 'number' && Number.isFinite(value) ? value : undefined
+const integer = (value: unknown) => typeof value === 'number' && Number.isInteger(value) ? value : undefined
+
+export interface SearchServiceOptions {
+ dataDir: string
+ socketPath: string
+ machineId: string
+ token: string
+ /** Swapped in tests for one that does not touch the real socket or process. */
+ run?: typeof runServiceProcess
+ /** Swapped in tests for an index that cannot open, as on a Node without `node:sqlite`. */
+ start?: typeof startSearch
+ /** Where conversations Harness did not start are found; swapped in tests for none. */
+ providers?: ReturnType<typeof externalProviders>
+}
+
+/**
+ * The core API search runs on in its own process: the agents the core last named, and everything else
+ * read here, as the core would read it. What search never asks — windows, sign-in — answers as nothing.
+ */
+export function searchCoreApi(
+ dataDir: string,
+ agents: () => Array<RegisteredSession & { displayName?: string }>,
+ providers: ReturnType<typeof externalProviders>,
+): CoreApi {
+ return {
+ dataDir,
+ agents: {
+ all: agents,
+ live: () => agents().filter((agent) => agent.active),
+ displayName: (session) => (session as { displayName?: string }).displayName ?? '',
+ byAgent: (agentId) => agents().find((agent) => agent.agentId === agentId),
+ advertised: () => [],
+ terminalAvailable: () => false,
+ sync: () => {},
+ },
+ transcripts: { databaseHistory },
+ external: {
+ sessions: new ExternalSessions({ providers, excluded: [dataDir], log: console.warn }),
+ open: new OpenSessions({ providers, log: console.warn }),
+ },
+ // Search holds no credential and talks to no window: these are never asked of it.
+ account: { mintGridName: async () => null, accessToken: () => Promise.reject(new Error('search holds no credential')) },
+ clients: { viewerChanged: () => {}, gridNamed: () => {} },
+ }
+}
+
+export function runSearchService(options: SearchServiceOptions): ServiceProcess {
+ let agents: Array<RegisteredSession & { displayName?: string }> = []
+ let core: CoreConnection | null = null
+ const refresh = async (): Promise<void> => {
+ if (!core) return
+ const answer = await core.query('agents').catch(() => null)
+ if (Array.isArray(answer?.agents)) agents = answer.agents as typeof agents
+ }
+ const api = searchCoreApi(options.dataDir, () => agents, options.providers ?? externalProviders())
+ const ports = emptyPorts()
+ ;(options.start ?? startSearch)(api, ports)
+ const index = ports.search
+ const run = options.run ?? runServiceProcess
+ const service = run({
+ name: 'search',
+ socketPath: options.socketPath,
+ machineId: options.machineId,
+ token: options.token,
+ // The same answers the core's own handlers give (backendSocket.ts `session_search`, `session_tail`).
+ requests: {
+ session_search: (payload: Payload) => index
+ ? { ...index.search(typeof payload.query === 'string' ? payload.query.slice(0, 500) : '', { limit: number(payload.limit), from: number(payload.from), to: number(payload.to) }) }
+ : { error: 'SEARCH_UNAVAILABLE' },
+ session_tail: async (payload: Payload) => {
+ if (!index) return { error: 'SEARCH_UNAVAILABLE' }
+ const sessionId = typeof payload.sessionId === 'string' ? payload.sessionId.slice(0, 200) : ''
+ if (!sessionId) return { error: 'BAD_SESSION' }
+ const tail = await index.tail(sessionId, { beforeTurn: integer(payload.beforeTurn), maxChars: integer(payload.maxChars) })
+ return tail ? { ...tail } : { error: 'NOT_INDEXED', sessionId }
+ },
+ },
+ onEvent: (payload) => {
+ const sessionId = typeof payload.sessionId === 'string' ? payload.sessionId : ''
+ if (!sessionId || !index) return
+ // A turn boundary: the agents may have changed (a new one, a new conversation), and this
+ // session has new turns to index.
+ if (payload.kind === 'touch') void refresh().then(() => index.touch(sessionId))
+ else if (payload.kind === 'deleteHistory') index.deleteHistory(sessionId)
+ },
+ onConnected: (connection) => {
+ core = connection
+ void refresh()
+ },
+ })
+ return {
+ stop: () => {
+ service.stop()
+ index?.stop()
+ },
+ }
+}
diff --git a/docs/design/2026-10-03-harnessd.md b/docs/design/2026-10-03-harnessd.md
index d631285b8..3eb40f940 100644
--- a/docs/design/2026-10-03-harnessd.md
+++ b/docs/design/2026-10-03-harnessd.md
@@ -60,6 +60,9 @@ somewhere to run.
## Processes
+The design the rollout below works toward. What runs today, and what changed since the last release,
+is in [2026-10-04-harnessd-before-after.md](2026-10-04-harnessd-before-after.md).
+
```text
desktop app · hn · `harness` · engine hooks phone · web · other machines
│ │
@@ -418,6 +421,26 @@ Done so far:
The funnel calls the teams and the device service the same way. `HARNESSD_TEST_FAULTS` injects
faults, and `e2e/services.e2e.ts` proves on the real daemon that agents bind, run their turns and
come back after a restart with every service failing to start or failing on every call.
+- **Services in their own processes** (step 4, started; off until `HARNESSD_SERVICES` names one). The
+ master runs each named service as `harness __service <name>` beside the core
+ (`cli/src/harnessd/services.ts`): it restarts one that crashes, with backoff from 1 s to 60 s; kills
+ one that stops beating (hung) or outgrows its memory budget (leaking); and parks one that ends five
+ times in ten minutes, trying it again after thirty. A service reaches the core through its local
+ socket under a service role, with a token the master gives both on every boot
+ (`cli/src/core/serviceLinks.ts`, `cli/src/services/process.ts`). The core routes the service's
+ requests to it and answers `SERVICE_UNAVAILABLE` at once while it is down, holds what it must hear
+ (a purge's forgetting) until it is back, and never waits on it. Search is the first:
+ `cli/src/services/searchProcess.ts` runs the same index on a core API built in its own process.
+ `e2e/serviceProcesses.e2e.ts` proves on the real daemon that search killed, hung (stopped), leaking
+ or crashing on every start costs search alone: the core never restarts and agents keep working.
+- **The walls, written down and checked.** The rules live in `AGENTS.md` files that coding agents read:
+ `cli/` and one per layer (`src/core/`, `src/services/`, `src/harnessd/`). A `CLAUDE.md` beside each
+ imports it, so Claude Code reads the same rules as Codex. On its own, Claude Code reads `AGENTS.md`
+ only from v2.1.277, and only when no `CLAUDE.md` is in the working directory or above it.
+ `cli/src/architecture.spec.ts` enforces the walls in the unit suite. A service may reach the core
+ only through `core/api.ts`. The core may import a service, `cli.ts` or the socket only for types.
+ The master may use only Node and its own folder. `runForeground` and `backendSocket.ts` have
+ size budgets.
- **What the daemon survives** (`e2e/robustness.e2e.ts`): hostile local clients, strangers and
malformed bodies at the hook door, a pane killed from outside, a turn that outlives a restart, a
turn that ends while the daemon is down, and garbage state files at start.
diff --git a/docs/design/2026-10-04-harnessd-before-after.md b/docs/design/2026-10-04-harnessd-before-after.md
new file mode 100644
index 000000000..77b98ed8c
--- /dev/null
+++ b/docs/design/2026-10-04-harnessd-before-after.md
@@ -0,0 +1,139 @@
+# harnessd: before and after the refactor
+
+A one-page picture of what changed in the daemon between the last release (v0.3.57, 2 October 2026)
+and the harnessd rebuild, for explaining the refactor to the team. The full design, its rules and
+its plan are in [2026-10-03-harnessd.md](2026-10-03-harnessd.md). Paths are under `cli/src/`.
+
+## Before: one process, one function
+
+```
+ desktop app · hn · harness CLI · engine hooks · phone/web (via relay)
+ │
+┌──────────────────────────────────────┴──────────────────────────────────────┐
+│ ONE PROCESS · harness __run │
+│ │
+│ cli.ts runForeground() one function, 5,306 lines, shared variables │
+│ backendSocket.ts ~70 request types, 53 callbacks wired by hand │
+│ │
+│ agents · tmux · discovery · hooks · transcripts · turns · input · questions│
+│ relay · E2EE · sharing · dial · Wi-Fi device · search · viewers · grid │
+│ workspaces · teams · recap model pool · … │
+│ everything inside one function: every part can touch every other │
+│ │
+│ ✗ one bug anywhere (exception, hang, memory) takes the whole daemon down │
+│ ✗ no supervisor: back only when the desktop app respawned it (up to ~70 s) │
+│ ✗ no part can be tested alone: it exists only inside that function │
+│ ✗ every feature edits the same file: conflicts and risky reviews │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+On 3 October an 803 MB Codex transcript was read whole on attach, ran the process out of memory,
+and it crash-looped: every agent on the machine lost its daemon at once.
+
+## After: three layers, with walls between them
+
+```
+ desktop app · hn · harness CLI · engine hooks · phone/web (via relay)
+ │
+┌──────────────────────────────────────┴──────────────────────────────────────┐
+│ ① MASTER: the supervisor src/harnessd/ harness __harnessd │
+│ supervisor.ts starts, watches and restarts the core · memory budget · │
+│ crash-loop safe mode · update probation and rollback │
+│ no feature code, no network: almost nothing in it can fail │
+└──────────────────────────────────────┬──────────────────────────────────────┘
+ │ spawn channel (harnessd/protocol.ts)
+┌──────────────────────────────────────┴──────────────────────────────────────┐
+│ ② CORE: must never go down src/core/ harness __run │
+│ │
+│ core/agents/ create · fork · restart · stop · resume · close · │
+│ discovery · bind 15 files │
+│ core/transcripts/ attach · live tail · relaunch marks · normalizers 6 │
+│ core/turns/ working/idle · event funnel · cancel · recaps 7 │
+│ core/terminals/ who controls a tmux pane (the control lease) 1 │
+│ core/engines/ engine hooks (13 engines) 2 │
+│ core/input.ts typed messages into the pane, in order │
+│ core/questions.ts an agent's question to you, and your answer back │
+│ │
+│ cli.ts runForeground() only wires these together: 5,306 → 2,636 lines │
+│ each module: one factory, explicit dependencies, 100% test coverage │
+│ │
+│ core/api.ts the contract between core and services: │
+│ CoreApi = what a service may ask of the core │
+│ CorePorts = what the core asks of a service │
+│ core/serviceHost.ts the circuit breaker: a service that fails to start is │
+│ left off · every call is guarded, with a fallback · │
+│ 5 failures in 60 s switch it off │
+└──────────────────────────────────────┬──────────────────────────────────────┘
+ │ only through CoreApi / CorePorts
+┌──────────────────────────────────────┴──────────────────────────────────────┐
+│ ③ SERVICES: the extendable part src/services/ │
+│ in the core's process behind the host, or each in its own process │
+│ (HARNESSD_SERVICES=search: harnessd/services.ts runs it, the core routes │
+│ to it through core/serviceLinks.ts) — search is the first │
+│ │
+│ services/search.ts session search uses lib/sessionSearch/ │
+│ services/viewers.ts harness viewers (DSH) uses dsh/ │
+│ services/models.ts Grid and local models uses lib/grid* │
+│ services/workspaces.ts worktrees, branch names │
+│ teams prompt scopes teams/ via ports.teams │
+│ ─ not yet behind the host ─ │
+│ devices dial, Wi-Fi device cable/, lib/autonomous-device/ │
+│ every call into them is guarded │
+│ relay backend link, E2EE backendSocket.ts, lib/e2ee/ │
+│ sharing shared harnesses sharing/ next to move │
+└─────────────────────────────────────────────────────────────────────────────┘
+ agents live in tmux: they keep running through every restart
+```
+
+## Why the subsystems are better than the monolith
+
+| What happens | Before (monolith) | After (subsystems) |
+|---|---|---|
+| A bug in search, viewers, models, workspaces or teams | The whole daemon goes down, for every agent. | That request gets a retryable error (`SERVICE_UNAVAILABLE`). After 5 failures the service is switched off, and agents are unaffected. |
+| The core crashes, hangs or runs out of memory | Down until the desktop app respawns it, up to about 70 s. | The master restarts it in about a second. Agents keep running in tmux. |
+| A bad release | Every machine crash-loops. | The update is rolled back and never tried again. |
+| One huge transcript | It was read whole, which caused the October 3 crash loop. | Reads are bounded, proven under a 256 MiB memory cap. |
+| Testing a part | Only through the whole daemon. | Each module is unit-tested to 100% (`npm run test:core`). An end-to-end suite of 160 tests runs on the real daemon (`npm run test:e2e`). |
+| Building a feature | Everyone edits the same 5,300-line function. | A feature is a new service with its own folder and tests, written against `CoreApi`, and the core doesn't change. |
+
+## How to add a service
+
+1. Put it in `services/<name>.ts` as `start<Name>(core: CoreApi, ports: CorePorts)`. It may only
+ ask the core what `CoreApi` offers.
+2. If the core must call it, add a port to `CorePorts` in `core/api.ts`, with fallbacks beside it:
+ what the core does while the service is off or failing.
+3. Start it in `cli.ts` through `serviceHost.start(...)`, never directly. The host guards its start
+ and every call, and switches it off when it keeps failing.
+4. Hold the file to 100% (`npm run test:core` covers `src/services/`). For the end-to-end suite,
+ `HARNESSD_TEST_FAULTS=<name>` makes its start fail, and `<name>.<member>` makes one call fail.
+
+The same rules are written for coding agents in `AGENTS.md` files, one per layer: `cli/`, `src/core/`,
+`src/services/` and `src/harnessd/`. Codex reads them directly. Claude Code reads them through a
+`CLAUDE.md` beside each one that imports it, so every agent gets one set of rules. `src/architecture.spec.ts`
+fails the build when code crosses a wall. A service that imports a core module fails it. So does a core
+module that imports a service, feature code in the master, or a `runForeground` or `backendSocket.ts`
+that grows past its budget. Each failure message says where the code belongs.
+
+## What the end-to-end suite has found
+
+15 real bugs, each fixed with a test that fails without the fix. They include:
+
+- a corrupt registry that blocked every agent create;
+- a pane killed from outside that stayed "active" forever;
+- a message sent just after a resume or restart that was lost;
+- rapid messages typed into one prompt, out of order;
+- an agent created alongside others that was retired the moment its engine started;
+- a turn taken just after a resume or restart that no window ever saw;
+- a Stop pressed during start that was refused, and the ghost agent it could leave;
+- a compaction that showed every window its turn starting twice;
+- a Codex fork that never bound its conversation after a daemon restart.
+
+The full list is in the design doc's "Found while mapping".
+
+## Next
+
+Search already runs in its own process when `HARNESSD_SERVICES=search` is set. Killed, hung, leaking
+or crash-looping, it costs search alone (`e2e/serviceProcesses.e2e.ts`). It is off by default until
+it has been run for real. The other services follow the same way, devices next, so that a native crash
+or a memory leak in any of them cannot touch the core either. After that come platform supervision
+(launchd, systemd) and the state directory move.
diff --git a/store/CLAUDE.md b/store/CLAUDE.md
new file mode 100644
index 000000000..6a4029179
--- /dev/null
+++ b/store/CLAUDE.md
@@ -0,0 +1,2 @@
+<!-- One set of rules for every agent: Codex reads AGENTS.md, and Claude Code reads it through this import. Claude Code reads AGENTS.md by itself only from v2.1.277, and only when no CLAUDE.md is in the working directory or above it, so without this a session started in a folder with a CLAUDE.md would miss these rules. -->