Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

harnessd: models' process runs only once grid is in use here or a request needs it (#971)

Deee8976a5aa783
diff --git a/cli/AGENTS.md b/cli/AGENTS.md
index 9b217de5d..f87596be4 100644
--- a/cli/AGENTS.md
+++ b/cli/AGENTS.md
@@ -29,7 +29,7 @@ they show as `harnessd`, `harnessd-core` and `harnessd-<process>` (`src/harnessd
| viewers | the harness viewers, their remote streams and rendered surfaces, and the Store | always |
| edge | workspaces, usage, the monitor, the project readers, the change-agent handoff, the recaps | always |
| gateway | the relay and its E2EE (`src/gateway/`) | always |
-| models | grid, local models, the Model Manager | always |
+| models | grid, local models, the Model Manager | on demand: once grid is in use here (a managed grid, saved grid pictures, local models), or on a models request |
| updater | checks, downloads and stages a new build (`src/services/updaterProcess.ts`) | by the master, for the installed copy only |
| devices | the dials, the window bridges, the fleet, the voice router, the Wi-Fi device | on demand: once there is a device |
| orchestrator, teams (with Tab collaboration), sharing, commandBar | the experiments | on demand: on a request, or saved state at start |
diff --git a/cli/e2e/lean.e2e.ts b/cli/e2e/lean.e2e.ts
index 06bb90682..505c7ff28 100644
--- a/cli/e2e/lean.e2e.ts
+++ b/cli/e2e/lean.e2e.ts
@@ -57,6 +57,9 @@ describe('harnessd\'s master and services lean', () => {
const fresh = async (env: Record<string, string> = {}, scriptPath = bundle) => {
const d = await IsolatedDaemon.create({ scriptPath, env })
daemon = d
+ // Grid in use here, so that models' process starts with the others and is held lean too (core/modelsWake.ts).
+ mkdirSync(join(d.dataDir, 'local-models'), { recursive: true })
+ writeFileSync(join(d.dataDir, 'local-models', 'operations.json'), '[]')
onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-120).join('\n')}`) })
await d.start()
await until('every service process', () => PROCESSES.every((name) => new RegExp(`service ${name} started`).test(d.log())) || null, 60_000, 200)
diff --git a/cli/e2e/models.e2e.ts b/cli/e2e/models.e2e.ts
index 8831cc7db..feee2fb38 100644
--- a/cli/e2e/models.e2e.ts
+++ b/cli/e2e/models.e2e.ts
@@ -142,13 +142,12 @@ describe('switching a Codex agent model and effort', () => {
it('lists the choices from models\' own process, and switches with models killed: the switch is the core\'s', async () => {
const d = await fresh({ codexModel: 'gpt-5.5', env: { HARNESSD_SERVICE_INITIAL_BACKOFF_MS: '8000', HARNESSD_SERVICE_MAX_BACKOFF_MS: '8000' } })
- await until('models to connect to the core', () => d.log().includes('[services] models connected') || null, 30_000, 200)
const client = await LocalClient.connect(d)
const agent = await create(d, client, 'apart')
await turn(client, agent.id, 'hello')
await runningOn(client, agent.id, 'gpt-5.5', 'high')
// The picker's choices come from models, in its own process (harnessd/services.ts), which asks the core
- // for the agent's catalog.
+ // for the agent's catalog. It is started by this first ask, which waits for it (core/modelsWake.ts).
const xhigh = profile(agent.id, 'gpt-6-luna', 'xhigh')
const offered = (await client.request<{ models: Array<{ id: string }> }>('models_list', { agentId: agent.id }, 30_000)).models
expect(offered.map((option) => option.id)).toContain(xhigh)
diff --git a/cli/e2e/modelsOnDemand.e2e.ts b/cli/e2e/modelsOnDemand.e2e.ts
new file mode 100644
index 000000000..e4cfb3ed6
--- /dev/null
+++ b/cli/e2e/modelsOnDemand.e2e.ts
@@ -0,0 +1,91 @@
+/**
+ * Models' process on demand (core/modelsWake.ts, harnessd/services.ts `models`), on the real daemon, signed out:
+ * a computer that uses no grid runs no models' process, about 70 MiB it never pays, and the first request that
+ * needs models starts it, waits for it and is answered by it.
+ * - Idle with no grid: no process, while the others run.
+ * - A models request (the saved APIs, the picker's list, the Codex profiles): the first starts it, those that
+ * come while it starts wait with it, in order, and are answered by it; killed, it comes back.
+ * - Grid in use here (a managed grid, saved grid pictures, local models): asked for as the core starts.
+ * - Named in `HARNESSD_SERVICES` (tests, support), it starts with the daemon, as before.
+ * A core too old to ask has it started as it binds: e2e/reexec.e2e.ts, under a released core.
+ */
+import { mkdirSync, writeFileSync } from 'node:fs'
+import { join } from 'node:path'
+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'
+import { LocalClient } from './harness/client.js'
+import { IsolatedDaemon, until } from './harness/daemon.js'
+
+const starts = (d: IsolatedDaemon) => [...d.log().matchAll(/\[harnessd\] service models started/g)].length
+const connections = (d: IsolatedDaemon) => d.log().split('\n').filter((line) => line.includes('[services] models connected')).length
+const pidsOf = (d: IsolatedDaemon) => [...d.log().matchAll(/\[harnessd\] service models started \(pid (\d+)\)/g)].map((match) => Number(match[1]))
+
+describe('models\' process, once grid is in use or asked for', () => {
+ let daemon: IsolatedDaemon | undefined
+ afterEach(async () => { await daemon?.close(); daemon = undefined })
+ const fresh = async (opts: { env?: Record<string, string>; before?: (d: IsolatedDaemon) => void } = {}) => {
+ const d = await IsolatedDaemon.create({ env: { HARNESSD_SERVICE_INITIAL_BACKOFF_MS: '200', ...opts.env } })
+ daemon = d
+ opts.before?.(d)
+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-150).join('\n')}`) })
+ await d.start()
+ // The others are up, and the core has had time to ask for anything it would ask for as it starts.
+ await until('the services to connect', () => d.log().includes('[services] search connected') || null, 30_000, 200)
+ await new Promise((resolve) => setTimeout(resolve, 3_000))
+ return d
+ }
+
+ it('runs no models\' process on a computer that uses no grid; its first request starts it, and the requests meanwhile wait with it', async () => {
+ const d = await fresh()
+ expect(starts(d), 'a models\' process on a computer that uses no grid').toBe(0)
+ expect(d.log()).not.toContain('asking for models\' process')
+ const window = await LocalClient.connect(d)
+ // Three requests while it starts: each waits for it, and each is its answer, not SERVICE_UNAVAILABLE.
+ const [apis, profiles, grid] = await Promise.all([
+ window.request('api_connections', { action: 'list' }, 30_000),
+ window.request('codex_profiles_list', {}, 30_000),
+ window.request('grid_models_list', {}, 30_000),
+ ])
+ expect(apis.error, JSON.stringify(apis)).toBeUndefined()
+ expect(apis).toHaveProperty('connections')
+ expect(profiles.error, JSON.stringify(profiles)).toBeUndefined()
+ expect(profiles).toHaveProperty('profiles')
+ expect(grid.error, JSON.stringify(grid)).toBeUndefined()
+ expect(starts(d)).toBe(1)
+ expect(connections(d)).toBe(1)
+ // Started, it stays: the next request is answered by the same process.
+ expect((await window.request('codex_profiles_list', {}, 30_000)).error).toBeUndefined()
+ expect(starts(d)).toBe(1)
+ window.close()
+ })
+
+ it('killed once started, comes back, and is answered by again', async () => {
+ const d = await fresh()
+ const window = await LocalClient.connect(d)
+ expect((await window.request('codex_profiles_list', {}, 30_000)).error).toBeUndefined()
+ const [pid] = pidsOf(d)
+ process.kill(pid, 'SIGKILL')
+ await until('the master to start models again', () => d.log().includes('[harnessd] service models started (pid') && starts(d) >= 2 || null, 30_000, 200)
+ await until('models to connect again', () => connections(d) >= 2 || null, 30_000, 200)
+ expect((await window.request('codex_profiles_list', {}, 30_000)).error).toBeUndefined()
+ expect(d.coresStarted()).toBe(1)
+ window.close()
+ })
+
+ for (const [what, lay] of [
+ ['a managed grid', (d: IsolatedDaemon) => { mkdirSync(d.env.ADAPTER_RUNTIME_DIR!, { recursive: true }); writeFileSync(join(d.env.ADAPTER_RUNTIME_DIR!, 'current-grid'), join(d.env.ADAPTER_RUNTIME_DIR!, 'grid-0', 'bin', 'grid')) }],
+ ['local models', (d: IsolatedDaemon) => { mkdirSync(join(d.dataDir, 'local-models'), { recursive: true }); writeFileSync(join(d.dataDir, 'local-models', 'operations.json'), '[]') }],
+ ] as const) {
+ it(`asks for it as the core starts when grid is in use here: ${what}`, async () => {
+ const d = await fresh({ before: lay })
+ expect(d.log()).toContain(`[models] ${what}: asking for models' process`)
+ await until('models to connect', () => connections(d) >= 1 || null, 30_000, 200)
+ expect(starts(d)).toBe(1)
+ })
+ }
+
+ it('named in HARNESSD_SERVICES, it starts with the daemon, with no grid', async () => {
+ const d = await fresh({ env: { HARNESSD_SERVICES: 'search,models' } })
+ await until('models to connect', () => connections(d) >= 1 || null, 30_000, 200)
+ expect(d.log()).not.toContain('asking for models\' process')
+ })
+})
diff --git a/cli/e2e/reexec.e2e.ts b/cli/e2e/reexec.e2e.ts
index 96f1bf2bd..7dae75022 100644
--- a/cli/e2e/reexec.e2e.ts
+++ b/cli/e2e/reexec.e2e.ts
@@ -377,5 +377,8 @@ describe.skipIf(!RELEASED)('a released master and this one, each over the other\
expect(daemon!.log().slice(from)).toContain(releasedCanReexec
? `now v${releasedVersion}`
: 'did not answer its probe (Unknown command: __harnessd-probe) — keeping this master')
+ // This master over a core too old to ask for a process on demand starts each as that core binds
+ // (harnessd/services.ts `unasked`): models, which a core of protocol 4 asks for only once grid is in use.
+ if (!releasedCanReexec) await until('this master to start models for the older core', () => daemon!.log().slice(from).includes('[harnessd] service models started') || null, 30_000, 200)
})
})
diff --git a/cli/e2e/services.e2e.ts b/cli/e2e/services.e2e.ts
index 3d607274b..b6acf463b 100644
--- a/cli/e2e/services.e2e.ts
+++ b/cli/e2e/services.e2e.ts
@@ -8,7 +8,7 @@
* (core/serviceHost.ts).
*/
import { randomUUID } from 'node:crypto'
-import { mkdirSync } from 'node:fs'
+import { mkdirSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'
import { afterEach, describe, expect, it, onTestFailed } from 'vitest'
import { LocalClient, type Frame } from './harness/client.js'
@@ -64,6 +64,9 @@ describe('a failing service never takes the core down', () => {
daemon = await IsolatedDaemon.create({ env: { HARNESSD_TEST_FAULTS: EVERY_PROCESS_FAILING, ...QUICK_TO_PARK } })
const d = daemon
onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-120).join('\n')}`) })
+ // Grid in use here, so that models is asked for as the core starts and fails with the others (core/modelsWake.ts).
+ mkdirSync(join(d.dataDir, 'local-models'), { recursive: true })
+ writeFileSync(join(d.dataDir, 'local-models', 'operations.json'), '[]')
await d.start()
// Each process fails as it starts, again and again, until the master parks it; the core never waits for one.
for (const service of SERVICE_PROCESSES) {
@@ -76,6 +79,8 @@ describe('a failing service never takes the core down', () => {
expect(await client.request('dsh_list', {})).toMatchObject({ error: 'SERVICE_UNAVAILABLE', service: 'store', retryable: true })
expect(await client.request('fs_list_dir', { path: d.projectsDir })).toMatchObject({ error: 'SERVICE_UNAVAILABLE', service: 'projects', retryable: true })
expect(await client.request('machine_resources', {})).toMatchObject({ error: 'SERVICE_UNAVAILABLE', service: 'monitor', retryable: true })
+ // Models is on demand and never came up: its request waits no longer than a start (core/serviceLinks.ts
+ // `ON_DEMAND_START_MS`), and every one after it is answered at once.
expect(await client.request('grid_fleet_models_list', {})).toMatchObject({ error: 'SERVICE_UNAVAILABLE', service: 'models', retryable: true })
// A create on a grid model, with models parked, is refused as the grid being unavailable, at once.
const onGrid = await client.request('agent_create', { engine: 'codex', cwd: d.projectsDir, gridModel: 'Qwen-35B', gridName: 'mine', bypassPermission: true }, 30_000)
@@ -312,8 +317,7 @@ describe('a failing service never takes the core down', () => {
const d = daemon
onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-80).join('\n')}`) })
await d.start()
- // In its own process: asked before it has connected, it is answered SERVICE_UNAVAILABLE, retryable.
- await until('models to connect', () => d.log().includes('[services] models connected') || null, 30_000, 200)
+ // In its own process, on demand: the first request starts it and waits for it (core/modelsWake.ts).
const client = await LocalClient.connect(daemon)
expect(await client.request('grid_models_list', {})).toMatchObject({ error: 'SERVICE_FAILED', service: 'models' })
// The October 6 bundled run received the RPC before the master's stdout delivered this same
diff --git a/cli/src/core/AGENTS.md b/cli/src/core/AGENTS.md
index 7040e3d69..1df14a26f 100644
--- a/cli/src/core/AGENTS.md
+++ b/cli/src/core/AGENTS.md
@@ -71,3 +71,4 @@ questions. Everything else is a service ([../services/AGENTS.md](../services/AGE
sends a device); `wifiAgents.ts`: its doors into the core (the agents as it lists them, a prompt, an
agent made for a Store harness); `wifiLink.ts`: its service in the devices' process. `devicesWake.ts`: when
that process is asked for, once there is a device (a dial's port in /dev, a paired Wi-Fi device).
+ `modelsWake.ts`: when models' process is asked for as the core starts, once grid is in use here.
diff --git a/cli/src/core/main.ts b/cli/src/core/main.ts
index f8a5b0476..196d413b7 100644
--- a/cli/src/core/main.ts
+++ b/cli/src/core/main.ts
@@ -150,6 +150,7 @@ import { BackendSocket, isLocalClientId } from '../backendSocket.js'
import { createGatewayLink, laneOf } from './gatewayLink.js'
import { createWifiCore, WIFI_START_MS } from './wifi.js'
import { createDevicesWake, DEVICES_ON_DEMAND } from './devicesWake.js'
+import { wakeModels } from './modelsWake.js'
import { createWifiLink } from './wifiLink.js'
import { wifiDoors } from './wifiAgents.js'
import { autonomousDeviceLocalRequest } from '../lib/autonomous-device/localApi.js'
@@ -1137,8 +1138,8 @@ async function runForeground(session: AuthSession | null): Promise<void> {
token: serviceToken,
owned: Object.fromEntries([...outOfProcess].map((name) => [name, requestsOf[name] ?? []])),
waits: LONG_ANSWERS,
- // An experiment's process runs once it is on, the devices' once there is one: a request for one off asks for it.
- onDemand: new Set([...experiments, ...DEVICES_ON_DEMAND]),
+ // An experiment's process runs once it is on, the devices' once there is one, models' once grid is used: a request asks.
+ onDemand: new Set([...experiments, ...DEVICES_ON_DEMAND, 'models']),
want: (service) => experimentHooks.want(service),
// The gateway's first: its `backend` reads (the device key log) were refused below as NOT_AN_EXPERIMENT.
answer: async (service, query, payload) => (service === 'gateway' && gatewayLink ? gatewayLink.answer(query, payload) : null) ?? deliveries.answer(service, query, payload)
@@ -1761,6 +1762,7 @@ async function runForeground(session: AuthSession | null): Promise<void> {
// bound core alone (core/experiments.ts). The others wait for their first request.
wakeExperiments({ dataDir: env.ADAPTER_DATA_DIR, experiments: EXPERIMENTS, outOfProcess, want: (service) => experimentHooks.want(service) })
devicesWake.start(outOfProcess)
+ wakeModels({ outOfProcess, dataDir: env.ADAPTER_DATA_DIR, runtimeDir: env.ADAPTER_RUNTIME_DIR, want: (service) => coreLink.want(service) })
} else {
try { writeFileSync(PID_FILE, String(process.pid) + '\n') } catch { /* best effort */ }
}
diff --git a/cli/src/core/modelsWake.spec.ts b/cli/src/core/modelsWake.spec.ts
new file mode 100644
index 000000000..85b638219
--- /dev/null
+++ b/cli/src/core/modelsWake.spec.ts
@@ -0,0 +1,77 @@
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { gridInUse, wakeModels } from './modelsWake.js'
+
+describe('grid in use on this computer, which starts models\' process with the core', () => {
+ let root: string
+ let dataDir: string
+ let runtimeDir: string
+ beforeEach(() => {
+ root = mkdtempSync(join(tmpdir(), 'models-wake-'))
+ dataDir = join(root, 'data')
+ runtimeDir = join(root, 'runtime')
+ mkdirSync(dataDir)
+ mkdirSync(runtimeDir)
+ })
+ afterEach(() => rmSync(root, { recursive: true, force: true }))
+
+ it('is none on a computer that never used grid: no managed grid, no pictures, no local models', () => {
+ expect(gridInUse({ dataDir, runtimeDir })).toBeNull()
+ // Folders that are there and hold nothing of grid's are no use of it either.
+ mkdirSync(join(dataDir, 'grid-pictures'))
+ writeFileSync(join(dataDir, 'grid-pictures', 'notes.txt'), '')
+ mkdirSync(join(dataDir, 'local-models'))
+ expect(gridInUse({ dataDir, runtimeDir })).toBeNull()
+ })
+
+ it('is a managed grid, whose pin models follows', () => {
+ writeFileSync(join(runtimeDir, 'current-grid'), '/runtime/grid-1/bin/grid\n')
+ expect(gridInUse({ dataDir, runtimeDir })).toBe('a managed grid')
+ })
+
+ it('is a saved grid picture, which agents\' grid notes are read from', () => {
+ mkdirSync(join(dataDir, 'grid-pictures'))
+ writeFileSync(join(dataDir, 'grid-pictures', 'own.json'), '{}')
+ expect(gridInUse({ dataDir, runtimeDir })).toBe('saved grid pictures')
+ })
+
+ it('is the Model Manager\'s local models', () => {
+ mkdirSync(join(dataDir, 'local-models'))
+ writeFileSync(join(dataDir, 'local-models', 'operations.json'), '[]')
+ expect(gridInUse({ dataDir, runtimeDir })).toBe('local models')
+ })
+
+ it('reads through what it is given, and a folder it cannot list holds nothing', () => {
+ expect(gridInUse({ dataDir: '/d', runtimeDir: '/r', exists: () => false, list: () => { throw new Error('EACCES') } })).toBeNull()
+ expect(gridInUse({ dataDir: '/d', runtimeDir: '/r', exists: () => false, list: (folder) => (folder === '/d/local-models' ? ['a'] : []) })).toBe('local models')
+ })
+})
+
+describe('models\' process asked for as the core starts', () => {
+ const inUse = { dataDir: '/d', runtimeDir: '/r', exists: (path: string) => path === '/r/current-grid', list: () => [] }
+
+ it('when it runs out here and grid is in use, saying why', () => {
+ const wanted: string[] = []
+ const lines: string[] = []
+ wakeModels({ ...inUse, outOfProcess: new Set(['models']), want: (service) => wanted.push(service), log: (line) => lines.push(line) })
+ expect(wanted).toEqual(['models'])
+ expect(lines).toEqual(['[models] a managed grid: asking for models\' process'])
+ })
+
+ it('not when grid is not in use, nor when models runs in the core\'s process', () => {
+ const wanted: string[] = []
+ wakeModels({ ...inUse, exists: () => false, outOfProcess: new Set(['models']), want: (service) => wanted.push(service) })
+ wakeModels({ ...inUse, outOfProcess: new Set(['search']), want: (service) => wanted.push(service) })
+ expect(wanted).toEqual([])
+ })
+
+ it('says why on the console by default', () => {
+ const log = vi.spyOn(console, 'log').mockImplementation(() => {})
+ try {
+ wakeModels({ ...inUse, outOfProcess: new Set(['models']), want: () => {} })
+ expect(log).toHaveBeenCalledWith('[models] a managed grid: asking for models\' process')
+ } finally { log.mockRestore() }
+ })
+})
diff --git a/cli/src/core/modelsWake.ts b/cli/src/core/modelsWake.ts
new file mode 100644
index 000000000..b9bac93c2
--- /dev/null
+++ b/cli/src/core/modelsWake.ts
@@ -0,0 +1,30 @@
+/**
+ * When models' process starts (harnessd/services.ts `models`, on demand since protocol 4): as the core starts when
+ * models has work of its own here, and otherwise at the first request that needs it (core/serviceLinks.ts
+ * `onDemand`), which waits for it. A computer that uses no grid never pays its 70 MiB.
+ *
+ * Its own work, with no request coming in: a managed grid's pin, followed at every start of models and every ten
+ * minutes (`current-grid` in the runtime folder), and the saved grid pictures it reads back and tells the core,
+ * which reads agents' grid notes and the keystroke prewarm from them (`grid-pictures/`). The Model Manager's local
+ * models (`local-models/`) count too: a computer running models of its own uses grid.
+ */
+import { existsSync, readdirSync } from 'node:fs'
+import { join } from 'node:path'
+
+export interface GridInUseDeps { dataDir: string; runtimeDir: string; exists?: (path: string) => boolean; list?: (folder: string) => string[] }
+
+/** Why grid is in use on this computer, as the log says it; null when it is not. */
+export function gridInUse(deps: GridInUseDeps): string | null {
+ const list = (folder: string): string[] => { try { return (deps.list ?? readdirSync)(folder) } catch { return [] } }
+ if ((deps.exists ?? existsSync)(join(deps.runtimeDir, 'current-grid'))) return 'a managed grid'
+ if (list(join(deps.dataDir, 'grid-pictures')).some((name) => name.endsWith('.json'))) return 'saved grid pictures'
+ return list(join(deps.dataDir, 'local-models')).length ? 'local models' : null
+}
+
+/** At the core's start: models' process asked for when it runs out here and grid is in use. */
+export function wakeModels(deps: GridInUseDeps & { outOfProcess: ReadonlySet<string>; want(service: string): void; log?(line: string): void }): void {
+ const why = deps.outOfProcess.has('models') ? gridInUse(deps) : null
+ if (!why) return
+ ;(deps.log ?? console.log)(`[models] ${why}: asking for models' process`)
+ deps.want('models')
+}
diff --git a/cli/src/core/serviceLinks.spec.ts b/cli/src/core/serviceLinks.spec.ts
index aaaee7d60..9e2781281 100644
--- a/cli/src/core/serviceLinks.spec.ts
+++ b/cli/src/core/serviceLinks.spec.ts
@@ -1,5 +1,5 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
-import { createServiceLinks, HELD_MAX, type ServiceFrame } from './serviceLinks.js'
+import { createServiceLinks, HELD_MAX, ON_DEMAND_START_MS, type ServiceFrame } from './serviceLinks.js'
const TOKEN = 'a'.repeat(48)
const ASKER = { local: false, owner: true }
@@ -138,6 +138,35 @@ describe('service links', () => {
expect(reply).toHaveBeenCalledWith({ link: 'ready' })
})
+ it('waits for one on demand to start no longer than its start wait, then for its answer as long as the request may', () => {
+ // Models' requests wait up to fifteen minutes for their answers: a models that cannot start must not hold
+ // one that long, and one that starts must have the whole wait to answer in.
+ const waits = { models: { grid_fleet_model_download: 900_000 } }
+ const links = make({ owned: { models: ['grid_fleet_model_download'] }, onDemand: new Set(['models']), want: vi.fn(), waits, startWaitMs: 20_000 })
+ const never = vi.fn()
+ links.route('grid_fleet_model_download', {}, ASKER, never)
+ vi.advanceTimersByTime(19_999)
+ expect(never).not.toHaveBeenCalled()
+ vi.advanceTimersByTime(1)
+ expect(never).toHaveBeenCalledWith({ error: 'SERVICE_UNAVAILABLE', service: 'models', retryable: true })
+
+ const fresh = make({ owned: { models: ['grid_fleet_model_download'] }, onDemand: new Set(['models']), want: vi.fn(), waits })
+ const slow = vi.fn(), lost = vi.fn()
+ fresh.route('grid_fleet_model_download', { modelId: 'a' }, ASKER, slow)
+ fresh.route('grid_fleet_model_download', { modelId: 'b' }, ASKER, lost)
+ vi.advanceTimersByTime(ON_DEMAND_START_MS - 1)
+ const models = sink()
+ const link = fresh.accept('models', TOKEN, models, vi.fn())!
+ expect(models.sent).toHaveLength(2)
+ // Well past the start wait, it is still waited for: the download takes as long as it takes.
+ vi.advanceTimersByTime(600_000)
+ link.receive({ type: 'grid_fleet_model_download_result', payload: { requestId: 'route-2', done: true } })
+ expect(slow).toHaveBeenCalledWith({ done: true })
+ // And the one never answered is let go at its own wait.
+ vi.advanceTimersByTime(300_000)
+ expect(lost).toHaveBeenCalledWith({ error: 'SERVICE_UNAVAILABLE', service: 'models', retryable: true })
+ })
+
it('answers SERVICE_UNAVAILABLE for one on demand that does not come in time, then at once until it comes, or will not take the request', () => {
const want = vi.fn()
const links = make({ owned: { orchestrator: ['orchestrator'], devices: ['harness_devices_list'], search: ['session_search'] }, onDemand: new Set(['orchestrator', 'devices']), want })
diff --git a/cli/src/core/serviceLinks.ts b/cli/src/core/serviceLinks.ts
index 34ad9e498..3c7e07290 100644
--- a/cli/src/core/serviceLinks.ts
+++ b/cli/src/core/serviceLinks.ts
@@ -57,11 +57,14 @@ export interface ServiceLinksOptions {
/** Longer waits for the answers that take longer, by service and then by type (core/api.ts
* `LONG_ANSWERS`): a grid command, grid's set-up, a harness's install. */
waits?: Readonly<Record<string, Readonly<Record<string, number>>>>
- /** The services whose process runs only once asked for: the experiments (core/api.ts `EXPERIMENTS`) and the
- * devices (core/devicesWake.ts). A request for one that has not connected yet asks for it (`want`) and waits
- * for it to connect, within the same time. One that has connected and is down again is answered as any
- * service is, at once; so is one that did not connect within a request's wait, until it does. */
+ /** The services whose process runs only once asked for: the experiments (core/api.ts `EXPERIMENTS`), the
+ * devices (core/devicesWake.ts) and models (core/modelsWake.ts). A request for one that has not connected yet
+ * asks for it (`want`) and waits for it to connect, for its own wait or `startWaitMs`, whichever is shorter,
+ * and then for its answer, for its own wait. One that has connected and is down again is answered as any
+ * service is, at once; so is one that did not connect in time, until it does. */
onDemand?: ReadonlySet<string>
+ /** How long a request waits for its service on demand to connect (`ON_DEMAND_START_MS`). */
+ startWaitMs?: number
/** Ask the master for a process on demand (harnessd/coreLink.ts `want`). */
want?(service: string): void
log?: (line: string) => void
@@ -80,6 +83,8 @@ interface Waiting {
type: string
reply: (result: Record<string, unknown>) => void
timer: unknown
+ /** How long it waits for its answer once it is sent. */
+ wait: number
/** Sent once its service connects: an experiment's request that woke it. */
frame?: ServiceFrame
}
@@ -87,11 +92,18 @@ interface Waiting {
/** The most notifications held for one service while it is down. */
export const HELD_MAX = 1_000
+/** How long a request waits for its service on demand to start and connect, at most. A process starts in a
+ * second or two even on a loaded machine; one that has not connected in this long is not starting (it ends as
+ * it starts, and its master parks it). Models' requests wait up to fifteen minutes for their answers (grid's
+ * set-up, a download), which a request for a models that cannot start must not wait on its connection. */
+export const ON_DEMAND_START_MS = 20_000
+
/** Who asks when the core itself asks a service (`call`): this machine's owner, on this machine. */
const THE_CORE: Asker = { local: true, owner: true }
export function createServiceLinks(options: ServiceLinksOptions) {
const timeoutMs = options.timeoutMs ?? 30_000
+ const startWaitMs = options.startWaitMs ?? ON_DEMAND_START_MS
const log = options.log ?? ((line: string) => console.warn(line))
const newId = options.newId ?? randomUUID
const setTimer = options.setTimer ?? ((run, ms) => setTimeout(run, ms))
@@ -138,12 +150,13 @@ export function createServiceLinks(options: ServiceLinksOptions) {
if (!link && !starting) { reply(unavailable(service)); return }
const id = newId()
const frame: ServiceFrame = { type, payload: { ...payload, requestId: id }, asker }
- const entry: Waiting = { service, type, reply, timer: null, ...(starting ? { frame } : {}) }
+ const wait = waitMs ?? waitFor(service, type)
+ const entry: Waiting = { service, type, reply, timer: null, wait, ...(starting ? { frame } : {}) }
// Cleared whenever the entry is settled, so it only ever fires for one still waiting.
entry.timer = setTimer(() => {
if (entry.frame) unstarted.add(service)
settle(id, entry, unavailable(service))
- }, waitMs ?? waitFor(service, type))
+ }, starting ? Math.min(wait, startWaitMs) : wait)
waiting.set(id, entry)
if (starting) { options.want?.(service); return }
if (!link!.sink.sendFrame(frame)) settle(id, entry, unavailable(service))
@@ -174,7 +187,10 @@ export function createServiceLinks(options: ServiceLinksOptions) {
if (entry.service !== service || !entry.frame) continue
const frame = entry.frame
delete entry.frame
- if (!sink.sendFrame(frame)) settle(id, entry, unavailable(service))
+ if (!sink.sendFrame(frame)) { settle(id, entry, unavailable(service)); continue }
+ // Sent: it waits for its answer for its own wait now, as a request to a service that was up does.
+ clearTimer(entry.timer)
+ entry.timer = setTimer(() => settle(id, entry, unavailable(service)), entry.wait)
}
options.connected?.(service)
return {
diff --git a/cli/src/harnessd/protocol.ts b/cli/src/harnessd/protocol.ts
index 133a4d9fc..bfeec360c 100644
--- a/cli/src/harnessd/protocol.ts
+++ b/cli/src/harnessd/protocol.ts
@@ -15,7 +15,8 @@ import type { SupervisorStatus } from './supervisor.js'
/** 2: `ready`, the event-loop delay on the heartbeat, and the exit-code contract below. 3: `want`, and the
* master's updater: the master tells its core to hand over for a build its updater staged (`harnessd:update`).
- * 4: `want` for the devices' process too, which runs only once there is a device (./services.ts `askedSince`). */
+ * 4: `want` for the devices' process too, which runs only once there is a device, and for models', which runs
+ * only once grid is in use here or a request needs it (./services.ts `askedSince`). */
export const HARNESSD_PROTOCOL = 4
/** The command a bundle's master answers its probe on, and what it answers. */
@@ -63,8 +64,8 @@ export type CoreMessage =
* loop was held since the last one (protocol 2). */
| { type: 'harnessd:heartbeat'; rssBytes: number; heapUsedBytes: number; loopDelayMs?: number }
/** Start the process that runs this service (harnessd/services.ts `onDemand`): an experiment that is on now
- * (protocol 3), or the devices, now that there is one (protocol 4). A master from before ignores it, having
- * started every service at once. */
+ * (protocol 3), or the devices, now that there is one, or models, now that grid is in use or asked for
+ * (protocol 4). A master from before ignores it, having started every service at once. */
| { type: 'harnessd:want'; service: string }
export type MasterMessage =
diff --git a/cli/src/harnessd/services.spec.ts b/cli/src/harnessd/services.spec.ts
index 5b2c3a9e5..01a39c344 100644
--- a/cli/src/harnessd/services.spec.ts
+++ b/cli/src/harnessd/services.spec.ts
@@ -346,6 +346,16 @@ describe('ServiceSupervisor', () => {
never.stop(done)
expect(done).toHaveBeenCalledOnce()
expect(never.status()[0].state).toBe('stopped')
+ // Started again with the others (a re-execution the new bundle refused): off, and started when asked.
+ never.start()
+ expect(never.status()[0].state).toBe('off')
+ never.unasked(2)
+ expect(children.map((child) => child.spec.name)).toEqual(['orchestrator', 'devices', 'teams'])
+ // One running then is stopped and started with the others, and waits to be asked for again.
+ never.stop(vi.fn())
+ latest('teams').exit(0)
+ never.start()
+ expect(never.status()[0].state).toBe('off')
})
it('with no services, starting does nothing and stopping is done at once', () => {
@@ -429,6 +439,14 @@ describe('ServiceSupervisor', () => {
expect(named.onDemand).toBeUndefined()
})
+ it('runs models only once grid is in use or asked for, by a core of protocol 4, and from the start when named', () => {
+ const [models] = serviceSpecs({}, SERVICE_HOSTS).filter((spec) => spec.name === 'models')
+ expect(models).toMatchObject({ services: ['models'], onDemand: true, askedSince: 4 })
+ const [named] = serviceSpecs({ HARNESSD_SERVICES: 'models' }, SERVICE_HOSTS)
+ expect(named.services).toEqual(['models'])
+ expect(named.onDemand).toBeUndefined()
+ })
+
it('knows every service each process this build runs hosts, and each in one process only', () => {
const hosted = Object.values(SERVICE_HOSTS).flatMap((host) => host.services)
expect(KNOWN_SERVICES).toEqual(hosted)
diff --git a/cli/src/harnessd/services.ts b/cli/src/harnessd/services.ts
index f2b208a6f..1abe314a9 100644
--- a/cli/src/harnessd/services.ts
+++ b/cli/src/harnessd/services.ts
@@ -163,6 +163,12 @@ class Service {
if (this.state === 'off') this.start()
}
+ /** A process on demand stopped with the others and started again with them: off, for whoever needs it next to
+ * ask. Left stopped, no `want` started it again for the rest of the master's life. */
+ rest(): void {
+ if (this.state === 'stopped') this.setState('off')
+ }
+
/** SIGTERM, then SIGKILL after the grace; `done` once it is gone. A parked or waiting one is simply stopped. */
stop(done: () => void): void {
if (this.state === 'stopped') { done(); return }
@@ -307,9 +313,14 @@ export class ServiceSupervisor {
this.services = specs.map((spec) => new Service(spec, deps, options, env))
}
- /** Start every service but those on demand; none waits on another, or on the core. */
+ /** Start every service but those on demand, which wait to be asked for again; none waits on another, or on the
+ * core. Found end to end (e2e/reexec.e2e.ts): a master whose re-execution was refused stops its services and
+ * starts them again, and a process on demand stayed stopped, so a core too old to ask never got models. */
start(): void {
- for (const service of this.services) if (!service.spec.onDemand) service.start()
+ for (const service of this.services) {
+ if (!service.spec.onDemand) service.start()
+ else service.rest()
+ }
}
/** Start the process on demand that hosts [service], if it is not running yet: the core asked for it (`harnessd:want`). */
@@ -384,8 +395,10 @@ export const SERVICE_HOSTS: Readonly<Record<string, ServiceHostSpec>> = {
// where a fault costs the remote clients and nothing else (docs/design/2026-10-06-core-boundary-next.md).
gateway: { services: ['gateway'], heapLimitMiB: 512, rssLimitMiB: 1_024 },
// Grid's pictures, the Model Manager's catalog and the models found on this machine. Its downloads, model
- // servers and `grid` commands run in processes of their own, outside this budget.
- models: { services: ['models'], heapLimitMiB: 512, rssLimitMiB: 1_024 },
+ // servers and `grid` commands run in processes of their own, outside this budget. Started only once grid is
+ // in use here or a request needs it (core/modelsWake.ts): about 70 MiB at idle, which a computer that uses
+ // no grid never pays.
+ models: { services: ['models'], heapLimitMiB: 512, rssLimitMiB: 1_024, onDemand: true, askedSince: 4 },
// The devices (services/devicesProcess.ts): the dials on USB (pure-JS serial, a frame decoder whose buffer
// is bounded per dial), the window bridges, the fleet's router and its lane, the voice router's engine
// worker (a process of its own, outside this budget). Hardware that speaks whatever its firmware says:
diff --git a/cli/src/services/AGENTS.md b/cli/src/services/AGENTS.md
index dfd6a2746..7565d491e 100644
--- a/cli/src/services/AGENTS.md
+++ b/cli/src/services/AGENTS.md
@@ -176,14 +176,17 @@ like any other. Named in `HARNESSD_SERVICES`, it starts with the others.
`src/core/main.ts`) asks for it and waits for it, within the request's own wait. The service is welcomed
before its queued requests are delivered. Each experiment with saved state in the data folder is
asked for as the core starts (`src/core/experiments.ts`). The devices are asked for once there is a device:
- a dial's port in /dev, a paired Wi-Fi device, or its session (`src/core/devicesWake.ts`).
+ a dial's port in /dev, a paired Wi-Fi device, or its session (`src/core/devicesWake.ts`). Models is asked
+ for as the core starts when grid is in use here: a managed grid, whose pin it follows, saved grid pictures,
+ which agents' grid notes are read from, or local models (`src/core/modelsWake.ts`); otherwise its first
+ request asks for it.
- **Older cores.** `askedSince` is the core protocol (`HARNESSD_PROTOCOL`, `src/harnessd/protocol.ts`) from
which a core asks for this process. A core that speaks an older one never asks, so the master starts the
process as that core binds (`ServiceSupervisor.unasked`). It is 3 by default, the experiments'; the
- devices became on demand at protocol 4. Making another process on demand needs a protocol bump, and that
+ devices and models became on demand at protocol 4. Making another process on demand needs a protocol bump, and that
number in its `askedSince`.
-- **Proof.** `e2e/experiments.e2e.ts` and `e2e/devicesOnDemand.e2e.ts`: off, it has no process; asked for, it
- starts and answers.
+- **Proof.** `e2e/experiments.e2e.ts`, `e2e/devicesOnDemand.e2e.ts` and `e2e/modelsOnDemand.e2e.ts`: off, it has
+ no process; asked for, it starts and answers.
## Experiments