Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Agents controllers update once no session is mid-turn, and move their agents onto the new release (#866)
Louis Amaudruz733551e9f551
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5496d85ce..002b434c2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -3505,6 +3505,26 @@ tags; see RELEASING.md.can run (16 by default).#### Fixed+- **A controller updates to the release Switch names while sessions sit idle.**+ It waited until no session existed on the machine, and a session connected to+ a room counted even with nothing to do, so a machine with long-lived sessions+ stayed `waiting_idle` for good. It now waits only until no session is in the+ middle of a turn. Each agent host reports which of its sessions are busy (one+ that has not said yet counts as busy), and the controller looks again every+ few seconds while it waits rather than at the next status report. Idle+ sessions in the controller's own process are restarted with it and resume+ their conversations; agents running in a process of their own keep running+ through it. An agent host too old to report busy sessions still holds the+ update while it has any session.+- **Agents running in a process of their own move onto the release the+ controller updated to.** Such an agent host (detached, or a systemd unit with+ separate users, as on Switch cloud machines) outlives the controller and kept+ the shared-host bundle it started with, so a fix in it reached the agent only+ when the agent was restarted by hand. Each agent host now records the bundle+ it runs (by SHA-256) in its `health.json`, and the controller restarts one on+ another bundle, or too old to say, once none of its sessions is mid-turn;+ idle sessions resume their conversations. A host that comes back still+ outdated is tried again after 10 minutes.- **A refusal that cannot pass stops the controller instead of retrying itforever.** The server answers `protocol_unsupported` when it no longer speaksthe controller's protocol and `invalid_credential` when it knows no controllerdiff --git a/console/packages/agent-controller/README.md b/console/packages/agent-controller/README.mdindex 15ae74733..103843802 100644--- a/console/packages/agent-controller/README.md+++ b/console/packages/agent-controller/README.md@@ -348,14 +348,22 @@ When Switch names a release for this controller, `run` follows it; otherwise`controller_release: {"version", "url", "sha256"}`. When it names a version otherthan the one running (newer or older: Switch can roll back), the controller:-1. waits until no session runs on the machine (`sessions_running` is 0), checking- at each status report, and reports `update: {"target", "state": "waiting_idle"}`- in its status meanwhile;+1. waits until no session on the machine is in the middle of a turn, checking at+ each status report and every few seconds while it waits, and reports+ `update: {"target", "state": "waiting_idle"}` in its status meanwhile. Each+ agent host reports which of its sessions are busy; one that has not said yet+ counts as busy. Idle sessions do not hold the update back: those in the+ controller's own process are restarted with it and resume their+ conversations, and agents running in a process of their own keep running+ through it (once the controller is back on the new release, it restarts each+ of those onto it as soon as none of its sessions is mid-turn: each records+ the shared-host bundle it runs in its `health.json`). An agent host too old to report busy sessions holds it back+ while it has any session, as `sessions_running` counts them;2. downloads the package into `<data dir>/updates/` (at most 200 MB) and checksits SHA-256 against the one Switch gave, the only one trusted; a package thatdiffers is deleted;3. installs it from that file, never from the URL, checks that the installed CLI- says `--version` is that version, checks again that no session started+ says `--version` is that version, checks again that no turn startedmeanwhile, and switches to it;4. exits with code `7`, and its service starts it again on the new release.diff --git a/console/packages/agent-controller/src/controller.test.ts b/console/packages/agent-controller/src/controller.test.tsindex cff123b23..d7cde0795 100644--- a/console/packages/agent-controller/src/controller.test.ts+++ b/console/packages/agent-controller/src/controller.test.ts@@ -662,7 +662,7 @@ describe('the release Switch names', () => {sha256: 'c'.repeat(64),};- it('is installed while no session runs, and the controller ends to restart on it', async () => {+ it('is installed while no session is mid-turn, and the controller ends to restart on it', async () => {core.setAssignment({ revision: 1, agents: [agent(1)] });core.controllerRelease = release;const { updater, installed } = selfUpdate();@@ -671,7 +671,7 @@ describe('the release Switch names', () => {expect(installed).toEqual(['0.2.0']);});- it('waits while a session runs, and says so in the status', async () => {+ it('waits while an agent host that does not report busy sessions has any, and says so in the status', async () => {core.setAssignment({ revision: 1, agents: [agent(1)] });const { updater, installed } = selfUpdate();running = runController({ ...deps(), selfUpdate: updater }, stop.signal);@@ -689,6 +689,25 @@ describe('the release Switch names', () => {expect(installed).toEqual(['0.2.0']);});+ it('waits only while a session is mid-turn when the agent host says which are', async () => {+ core.setAssignment({ revision: 1, agents: [agent(1)] });+ const { updater, installed } = selfUpdate();+ running = runController({ ...deps(), selfUpdate: updater }, stop.signal);+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the agent started');+ const health = runtime.observation('agent-1').health!;+ health.placements = { 'session-1': 'room-a', 'session-2': 'room-b' };+ health.busy = ['session-1'];+ core.controllerRelease = release;+ await waitFor(+ () => core.statusReports.some((report) => report.update?.state === 'waiting_idle'),+ 'a status saying the update waits'+ );+ expect(installed).toEqual([]);+ health.busy = [];+ expect(await running).toBe('restarting');+ expect(installed).toEqual(['0.2.0']);+ });+it('is left alone by a controller that does not update itself', async () => {core.setAssignment({ revision: 1, agents: [agent(1)] });core.controllerRelease = release;@@ -700,6 +719,53 @@ describe('the release Switch names', () => {});});+describe('an agent host that outlived the controller on another bundle', () => {+ const restarts = () =>+ runtime.launches('agent-1').filter((call) => call.options.restart === true);++ it('is restarted onto this controller’s bundle once none of its sessions is mid-turn, and only once', async () => {+ core.setAssignment({ revision: 1, agents: [agent(1)] });+ running = runController(deps(), stop.signal);+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the agent started');+ const observation = runtime.observation('agent-1');+ observation.outdated = true;+ observation.health!.placements = { 'session-1': 'room-a' };+ observation.health!.busy = ['session-1'];+ await new Promise((resolve) => setTimeout(resolve, 200));+ expect(restarts()).toEqual([]);++ observation.health!.busy = [];+ await waitFor(() => restarts().length === 1, 'the outdated agent host restarted');+ await new Promise((resolve) => setTimeout(resolve, 200));+ expect(restarts()).toHaveLength(1);+ stop.abort();+ expect(await running).toBe('stopped');+ });++ it('is not restarted again while it stays on another bundle, within the retry wait', async () => {+ core.setAssignment({ revision: 1, agents: [agent(1)] });+ running = runController(deps(), stop.signal);+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the agent started');+ runtime.observation('agent-1').outdated = true;+ await waitFor(() => restarts().length === 1, 'the outdated agent host restarted');+ runtime.observation('agent-1').outdated = true;+ await new Promise((resolve) => setTimeout(resolve, 200));+ expect(restarts()).toHaveLength(1);+ stop.abort();+ expect(await running).toBe('stopped');+ });++ it('is left alone when it runs this controller’s bundle', async () => {+ core.setAssignment({ revision: 1, agents: [agent(1)] });+ running = runController(deps(), stop.signal);+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the agent started');+ await new Promise((resolve) => setTimeout(resolve, 200));+ expect(restarts()).toEqual([]);+ stop.abort();+ expect(await running).toBe('stopped');+ });+});+describe('ensureRelayCredentials', () => {const login = {status: 'connected' as const,diff --git a/console/packages/agent-controller/src/controller.ts b/console/packages/agent-controller/src/controller.tsindex 78332a5a6..e3660e9dc 100644--- a/console/packages/agent-controller/src/controller.ts+++ b/console/packages/agent-controller/src/controller.ts@@ -30,6 +30,7 @@ import { type GivenLogin, registerSealingKey, SealedLogins, sealingKeys } from 'import { CONTROLLER_CREDENTIAL, type SecretStore } from './secrets';import type { SelfUpdater } from './self-update';import {+ noSessionMidTurn,type ProviderLocator,ProviderStatuses,StatusCollector,@@ -70,6 +71,13 @@ export const DEFAULT_TIMING: ControllerTiming = {eventBufferLimit: 5_000,};+/**+ * How long an agent host restarted onto this controller's shared-host bundle+ * is left before it is restarted again, when the restart failed or it still+ * runs another bundle.+ */+export const OUTDATED_RESTART_RETRY_MS = 10 * 60 * 1000;+export type ControllerDeps = {store: ControllerStore;secrets: SecretStore;@@ -499,7 +507,16 @@ export async function runController(});};- const idle = async () => (await collector.collect(assignment)).machine.sessions_running === 0;+ const idle = async () =>+ noSessionMidTurn(+ await Promise.all((assignment?.agents ?? []).map((entry) => runtime.observe(entry.agent_id)))+ );+ /** What the last status response named; looked at again while an update waits for idle. */+ let offeredRelease: unknown = null;+ const offerRelease = () => {+ if (deps.selfUpdate && !stop.signal.aborted)+ void deps.selfUpdate.offer(offeredRelease, { idle, restart, signal: stop.signal });+ };reporter = new StatusReporter({collect: async () => {const collected = await collector.collect(assignment);@@ -511,12 +528,8 @@ export async function runController(reportWithinS = answer.report_within_s;if (assignment === null || answer.assignment_revision > assignment.revision)void sync('status says the assignment moved on');- if (deps.selfUpdate && !stop.signal.aborted)- void deps.selfUpdate.offer(answer.controller_release, {- idle,- restart,- signal: stop.signal,- });+ offeredRelease = answer.controller_release;+ offerRelease();},seq: () => store.nextStatusSeq(deps.now()),minGapMs: timing.statusMinGapMs,@@ -544,6 +557,61 @@ export async function runController(});};+ /** When each outdated agent host was last restarted onto this controller's bundle. */+ const outdatedRestarts = new Map<string, number>();+ let outdatedQueued = false;+ /**+ * Restarts each agent host that outlived this controller on another+ * shared-host bundle (`AgentObservation.outdated`) onto this controller's,+ * once none of its sessions is mid-turn: idle sessions restart with it and+ * resume their conversations. An agent host already restarted for it is+ * left for `OUTDATED_RESTART_RETRY_MS` before another try.+ */+ const restartOutdated = (): Promise<void> => {+ if (outdatedQueued) return Promise.resolve();+ outdatedQueued = true;+ return queue.run(async () => {+ outdatedQueued = false;+ if (!assignment || stop.signal.aborted) return;+ try {+ for (const entry of assignment.agents) {+ if (entry.desired_state !== 'running' || definitionProblem(entry)) continue;+ const agentId = entry.agent_id;+ const observation = await runtime.observe(agentId);+ if (!observation.alive || !observation.outdated) continue;+ const last = outdatedRestarts.get(agentId);+ if (last !== undefined && deps.now() - last < OUTDATED_RESTART_RETRY_MS) continue;+ if (!noSessionMidTurn([observation])) continue;+ if (last !== undefined)+ log.error(+ 'An agent host restarted onto this controller’s shared-host bundle is still on another (the restart failed, or it came back on another bundle); restarting it again',+ { agentId, restartedAt: new Date(last).toISOString() }+ );+ outdatedRestarts.set(agentId, deps.now());+ log.warn(+ 'Restarting an agent host that outlived this controller on another shared-host bundle, now that none of its sessions is mid-turn',+ { agentId, version: deps.version }+ );+ await startAgent(+ {+ kind: 'start',+ agentId,+ entry,+ restart: true,+ replaceIdentity: false,+ clearTakenOver: false,+ relaunch: true,+ why: 'agent host on another shared-host bundle',+ },+ reconcileDeps+ );+ }+ } catch (error) {+ failed('Restarting outdated agent hosts', error);+ }+ });+ };+const onFrame = async (frame: ControllerFrame): Promise<void> => {switch (frame.type) {case 'connection_state':@@ -608,6 +676,10 @@ export async function runController(return;}void reporter?.requestIfChanged();+ // A turn ending changes nothing reported, so it would otherwise wait for+ // the next periodic report to let the update through.+ if (deps.selfUpdate?.status()?.state === 'waiting_idle') offerRelease();+ void restartOutdated();}, timing.statusPollMs);let ending: ControllerExit = 'stopped';diff --git a/console/packages/agent-controller/src/detached-runtime.ts b/console/packages/agent-controller/src/detached-runtime.tsindex 2fa39c5e9..c5f167b0c 100644--- a/console/packages/agent-controller/src/detached-runtime.ts+++ b/console/packages/agent-controller/src/detached-runtime.ts@@ -15,6 +15,7 @@ import type { DataLayout } from './paths';import {type AgentObservation,type AgentRunner,+ LaunchedBundle,type LaunchOptions,ownsRoot,OWNER_RECORDS,@@ -38,18 +39,24 @@ const LAUNCH_TIMEOUT_MS = 60_000;* controller's hub, over a WebSocket on the relay's port.** Being its own process, it is not stopped when the controller exits: it- * keeps retrying the hub, and picks up again when the controller is back.+ * keeps retrying the hub, and picks up again when the controller is back,+ * still on the bundle it was started from. One on another bundle than this+ * controller's is `outdated`, and the controller restarts it once idle.* A restart is a stop and a start: the agent host is turned off, waited out,* and launched again from the new template.*/export class DetachedRuntime implements AgentRunner {+ private readonly bundle: LaunchedBundle;+constructor(private readonly deps: {layout: DataLayout;/** The `shared-host-daemon.mjs` bundle from agent-providers. */bundlePath: string;}- ) {}+ ) {+ this.bundle = new LaunchedBundle(deps.bundlePath);+ }async observe(agentId: string): Promise<AgentObservation> {const root = this.deps.layout.watcherRoot(agentId);@@ -67,7 +74,12 @@ export class DetachedRuntime implements AgentRunner {current: parsed.pid !== process.pid && (await ownsRoot(parsed.pid, root)),};}- return { ...(await readRoot(root)), alive: living, health };+ return {+ ...(await readRoot(root)),+ alive: living,+ health,+ outdated: living && (await this.bundle.outdates(health)),+ };}async launch(agentId: string, template: SharedHostConfig, options: LaunchOptions): Promise<void> {diff --git a/console/packages/agent-controller/src/main.ts b/console/packages/agent-controller/src/main.tsindex 92160d4d5..9c1c185f8 100644--- a/console/packages/agent-controller/src/main.ts+++ b/console/packages/agent-controller/src/main.ts@@ -986,7 +986,7 @@ async function updateCommand(args: string[]): Promise<number> {const prefix = installPrefix(cliPath());if (prefix && (await isManagedRelease(prefix)))throw new ConfigurationError(- `This controller runs from ${prefix}, a releases directory install-service set up: it installs the release Switch names itself, once no session runs. To install another by hand, install it with npm and run install-service again.`+ `This controller runs from ${prefix}, a releases directory install-service set up: it installs the release Switch names itself, once no session is mid-turn. To install another by hand, install it with npm and run install-service again.`);const dataDir = resolveDataDir(values['data-dir']);process.stdout.write(`Installing ${latest.version} from ${latest.packageUrl}\n`);diff --git a/console/packages/agent-controller/src/runtime.test.ts b/console/packages/agent-controller/src/runtime.test.tsindex f9b3dc39a..ef3afbd68 100644--- a/console/packages/agent-controller/src/runtime.test.ts+++ b/console/packages/agent-controller/src/runtime.test.ts@@ -9,10 +9,15 @@ import {statSync,writeFileSync,} from 'node:fs';+import { mkdtemp, rm, writeFile } from 'node:fs/promises';import { tmpdir } from 'node:os';import { join } from 'node:path';import { setTimeout as delay } from 'node:timers/promises';-import { type OpenAgentStream, readSharedCredentials } from '@switch-console/agent-providers';+import {+ bundleIdentity,+ type OpenAgentStream,+ readSharedCredentials,+} from '@switch-console/agent-providers';import {SERVICE_ACCOUNT_KEY_FIXTURE,vertexCredentialFixture,@@ -20,7 +25,13 @@ import {import { afterEach, beforeEach, describe, expect, it } from 'vitest';import { silentLogger } from './log';import { dataLayout } from './paths';-import { InProcessRuntime, isInside, loginProbeEnvironment, observeOnDisk } from './runtime';+import {+ InProcessRuntime,+ isInside,+ LaunchedBundle,+ loginProbeEnvironment,+ observeOnDisk,+} from './runtime';import { buildWatcherTemplate } from './template';/** Stands in for the shared-host bundle: answers `--probe`. Sessions are never started here. */@@ -331,6 +342,39 @@ describe('InProcessRuntime', () => {});});+describe('LaunchedBundle', () => {+ const health = (bundle: string | undefined, current = true) => ({+ state: 'connected' as const,+ detail: null,+ since: new Date(0).toISOString(),+ placements: {},+ pid: 4242,+ updatedAt: new Date(0).toISOString(),+ current,+ ...(bundle === undefined ? {} : { bundle }),+ });++ it('says a live agent host on another bundle, or one that names none, is outdated', async () => {+ const dir = await mkdtemp(join(tmpdir(), 'launched-bundle-'));+ try {+ const path = join(dir, 'shared-host.mjs');+ await writeFile(path, 'console.log(1);');+ const launched = new LaunchedBundle(path);+ const ours = await bundleIdentity(path);+ expect(await launched.outdates(health(ours))).toBe(false);+ expect(await launched.outdates(health('another'))).toBe(true);+ expect(await launched.outdates(health(undefined))).toBe(true);+ expect(await launched.outdates(health('another', false))).toBe(false);+ expect(await launched.outdates(null)).toBe(false);++ await writeFile(path, 'console.log(22);');+ expect(await launched.outdates(health(ours))).toBe(true);+ } finally {+ await rm(dir, { recursive: true, force: true });+ }+ });+});+describe('isInside', () => {it('is true strictly below the root', () => {expect(isInside('/data/workspaces', '/data/workspaces/scout')).toBe(true);diff --git a/console/packages/agent-controller/src/runtime.ts b/console/packages/agent-controller/src/runtime.tsindex 5c88bac37..22f435599 100644--- a/console/packages/agent-controller/src/runtime.ts+++ b/console/packages/agent-controller/src/runtime.ts@@ -6,6 +6,7 @@ import { isAbsolute, join, relative, resolve, sep } from 'node:path';import { setTimeout as delay } from 'node:timers/promises';import { promisify } from 'node:util';import {+ bundleIdentity,clearTakenOver,commandLineNames,ensureSharedProcess,@@ -56,6 +57,13 @@ export type AgentObservation = {/** `supervisor/failure.json`: why the agent host stopped and was not restarted. */failure: string | null;takenOver: TakenOver | null;+ /**+ * The agent host outlives this controller (it runs in a process of its own)+ * and runs another shared-host bundle than the one this controller launches+ * agent hosts from, or one too old to say which. Read from a live agent+ * host's current health only; false otherwise.+ */+ outdated: boolean;};/** What an agent with no agent host root at all looks like. */@@ -67,6 +75,7 @@ export function emptyObservation(): AgentObservation {health: null,failure: null,takenOver: null,+ outdated: false,};}@@ -277,6 +286,34 @@ export async function recordedPid(path: string): Promise<number | null> {return typeof pid === 'number' && Number.isSafeInteger(pid) && pid > 0 ? pid : null;}+/**+ * The shared-host bundle a runner launches agent hosts from, by+ * `bundleIdentity`: read again only when the file at its path changes, as an+ * update installed over it or a moved `current` link does.+ */+export class LaunchedBundle {+ private known: { key: string; identity: string } | null = null;++ constructor(private readonly path: string) {}++ async identity(): Promise<string> {+ const info = await stat(this.path);+ const key = `${info.dev}:${info.ino}:${info.size}:${info.mtimeMs}`;+ if (this.known?.key !== key) this.known = { key, identity: await bundleIdentity(this.path) };+ return this.known.identity;+ }++ /**+ * Whether the live agent host whose health this is runs another bundle, or+ * one too old to say which. Health that is not current says nothing.+ */+ async outdates(health: AgentObservation['health']): Promise<boolean> {+ if (!health?.current) return false;+ if (health.bundle === undefined) return true;+ return health.bundle !== (await this.identity());+ }+}+/** The owner records an agent host writes under its root: the worker's, then a detached one's supervisor's. */export const OWNER_RECORDS = ['shared-owner.lock', join('supervisor', 'owner.json')];@@ -293,11 +330,13 @@ export async function observeOnDisk(healthText === null? null: { ...watcherHealthFileSchema.parse(JSON.parse(healthText)), current: running };- return { ...(await readRoot(root)), alive: running, health };+ return { ...(await readRoot(root)), alive: running, health, outdated: false };}-/** Everything about an agent host root but whether it runs. */-export async function readRoot(root: string): Promise<Omit<AgentObservation, 'alive'>> {+/** Everything about an agent host root but whether it runs, and on which bundle. */+export async function readRoot(+ root: string+): Promise<Omit<AgentObservation, 'alive' | 'outdated'>> {const config = await readOptional(join(root, 'config.json'));let configured: AgentObservation['configured'] = null;if (config !== null) {@@ -432,6 +471,7 @@ export class InProcessRuntime implements AgentRuntime {current: true,}: null,+ outdated: false,};}@@ -503,7 +543,7 @@ export class InProcessRuntime implements AgentRuntime {const control = new WatcherControl();const sessions = inProcessSupervision(this.deps.bundlePath, this.links);// For `status`, which runs in another process and reads only disk.- const stopRecording = recordWatcherHealth(root, control);+ const stopRecording = recordWatcherHealth(root, control, null);const done = (async () => {try {await runAgentHost(root, config, signal, sessions, control, this.deps.openStream(agentId));diff --git a/console/packages/agent-controller/src/self-update.ts b/console/packages/agent-controller/src/self-update.tsindex 0b2c7e0e9..7829a6d74 100644--- a/console/packages/agent-controller/src/self-update.ts+++ b/console/packages/agent-controller/src/self-update.ts@@ -13,7 +13,9 @@ import { RELEASE_VERSION } from './update';* Updating this controller to the release Switch names in its status* responses (`controller_release`): fetched, checked against the SHA-256* Switch gave, installed, then switched to by exiting with `EXIT_RESTARTING`- * for the service to start it again. Only while no session runs.+ * for the service to start it again. Only while no session is in the middle+ * of a turn: idle sessions are restarted with it and resume their+ * conversations.*//** The largest package a download is allowed to be. */@@ -322,7 +324,7 @@ export type SelfUpdaterDeps = {/** What the running controller lends the updater for one offer. */export type UpdateHost = {- /** Whether no session runs here now. */+ /** Whether no session here is in the middle of a turn now. */idle: () => Promise<boolean>;/** Ends the controller with `restarting`. */restart: () => void;@@ -332,7 +334,7 @@ export type UpdateHost = {/*** Follows the release Switch names: a version other than this one (newer or- * older, so Switch can roll back) is installed while no session runs, and+ * older, so Switch can roll back) is installed while no session is mid-turn, and* the controller restarts into it. A failed attempt is not repeated for an* hour; what stands in the way is reported (`status()`).*/@@ -447,7 +449,7 @@ export class SelfUpdater {this.report = { target, state: 'waiting_idle' };this.logOnce(`waiting:${target}`, () =>this.deps.log.info(- 'Switch names another controller release; it is installed once no session runs',+ 'Switch names another controller release; it is installed once no session is mid-turn',{from: this.deps.version,to: target,diff --git a/console/packages/agent-controller/src/status.test.ts b/console/packages/agent-controller/src/status.test.tsindex 55c0cef5e..40c696d6e 100644--- a/console/packages/agent-controller/src/status.test.ts+++ b/console/packages/agent-controller/src/status.test.ts@@ -10,6 +10,7 @@ import { type AgentAssignment, type SealedLoginResponse, statusReportSchema } frimport { SealedLogins } from './sealed-logins';import {mapAgentProcess,+ noSessionMidTurn,PathProviderLocator,PROVIDER_TTL_MS,providerStatusFrom,@@ -185,6 +186,34 @@ describe('mapAgentProcess', () => {});});+describe('noSessionMidTurn', () => {+ const reporting = (busy: string[] | undefined, current = true) =>+ observed({ alive: true, health: { ...health('connected', null, current)!, busy } });++ it('holds while an agent host reports a session mid-turn, however many sit idle', () => {+ expect(noSessionMidTurn([reporting([])])).toBe(true);+ expect(noSessionMidTurn([reporting([]), reporting(['session-a'])])).toBe(false);+ });++ it('holds while an agent host too old to report busy sessions has any session', () => {+ expect(noSessionMidTurn([reporting(undefined)])).toBe(false);+ const empty = reporting(undefined);+ empty.health!.placements = {};+ expect(noSessionMidTurn([empty])).toBe(true);+ });++ it('counts no session for an agent host not running or whose report is not its own', () => {+ expect(noSessionMidTurn([reporting(['session-a'], false)])).toBe(true);+ expect(+ noSessionMidTurn([+ observed({ alive: false, health: { ...health('connected')!, busy: ['x'] } }),+ ])+ ).toBe(true);+ expect(noSessionMidTurn([emptyObservation(), observed({ alive: true })])).toBe(true);+ expect(noSessionMidTurn([])).toBe(true);+ });+});+describe('providerStatusFrom', () => {const at = '2026-01-01T00:00:00Z';const located = { path: '/usr/bin/claude', version: '2.0.0', versionProblem: null };diff --git a/console/packages/agent-controller/src/status.ts b/console/packages/agent-controller/src/status.tsindex 7fa169897..4d00f4c95 100644--- a/console/packages/agent-controller/src/status.ts+++ b/console/packages/agent-controller/src/status.ts@@ -512,6 +512,23 @@ export function mapAgentProcess(input: {};}+/**+ * Whether no session here is in the middle of a turn, as each assigned+ * agent's host last reported it: what a controller waits for before it+ * restarts into another release. Idle sessions do not hold it back; they are+ * restarted with it and resume their conversations. An agent host too old to+ * report which sessions are busy holds it back while it has any session at+ * all. An agent host not running, or whose report is not its own, has none.+ */+export function noSessionMidTurn(observations: AgentObservation[]): boolean {+ return observations.every((observation) => {+ const health = observation.alive && observation.health?.current ? observation.health : null;+ if (!health) return true;+ if (health.busy) return health.busy.length === 0;+ return Object.keys(health.placements).length === 0;+ });+}+/*** Builds status reports: the machine, the providers as last checked, and every* assigned agent as its agent host's files describe it. `since` is kept acrossdiff --git a/console/packages/agent-controller/src/systemd-runtime.ts b/console/packages/agent-controller/src/systemd-runtime.tsindex f624f3594..cfbb3ed42 100644--- a/console/packages/agent-controller/src/systemd-runtime.ts+++ b/console/packages/agent-controller/src/systemd-runtime.ts@@ -22,6 +22,7 @@ import {type AgentRuntime,emptyObservation,isInside,+ LaunchedBundle,type LaunchOptions,loginProbeEnvironment,parseRelayCredentials,@@ -94,13 +95,16 @@ type UnitState = { activeState: string; result: string; mainPid: number };* credentials and environment it loads, and starts and stops the unit through* `systemctl`, which polkit allows it for these units only. The agent host* hears its events on the controller's hub, as one in a process of its own- * does, and outlives the controller.+ * does, and outlives the controller. The unit runs the bundle through the+ * releases directory's `current` link, so restarting one that is `outdated`+ * brings it onto the release the controller runs.** Everything an agent writes is read as untrusted: through no link, parsed* against its schema, and never run.*/export class SystemdRuntime implements AgentRuntime {private readonly names: ReturnType<typeof separateUserNames>;+ private readonly bundle: LaunchedBundle;constructor(private readonly deps: {@@ -114,6 +118,7 @@ export class SystemdRuntime implements AgentRuntime {}) {this.names = separateUserNames(deps.config.uid);+ this.bundle = new LaunchedBundle(deps.config.bundle);}/** The agent user the agent runs as, claimed now when it holds none. */@@ -266,7 +271,13 @@ export class SystemdRuntime implements AgentRuntime {(state.activeState === 'failed'? `The agent's unit ${unit} failed (${state.result}); see journalctl -u ${unit}.`: null);- return { ...onDisk, alive, health, failure };+ return {+ ...onDisk,+ alive,+ health,+ failure,+ outdated: alive && (await this.bundle.outdates(health)),+ };}async launch(agentId: string, template: SharedHostConfig, options: LaunchOptions): Promise<void> {diff --git a/console/packages/agent-controller/src/testing/fake-runtime.ts b/console/packages/agent-controller/src/testing/fake-runtime.tsindex 25826b7c3..4a04a043b 100644--- a/console/packages/agent-controller/src/testing/fake-runtime.ts+++ b/console/packages/agent-controller/src/testing/fake-runtime.ts@@ -116,6 +116,7 @@ export class FakeRuntime implements AgentRuntime {updatedAt: new Date(0).toISOString(),current: true,},+ outdated: false,} satisfies Partial<AgentObservation>);}diff --git a/console/packages/agent-providers/src/host/agent-host.test.ts b/console/packages/agent-providers/src/host/agent-host.test.tsindex 2b6c4d2ff..0474a2d29 100644--- a/console/packages/agent-providers/src/host/agent-host.test.ts+++ b/console/packages/agent-providers/src/host/agent-host.test.ts@@ -1646,6 +1646,48 @@ it('brings live sessions in step with a rewritten definition, each once its turnexpect(restarted).toHaveLength(2);});+it('reports which of its sessions are mid-turn, as their hosts start, go busy, go idle and exit', async () => {+ const root = await mkdtemp(join(tmpdir(), 'shared-watch-busy-'));+ roots.push(root);+ paths.root = root;+ const config = await spawning(root);+ const first = await existing(root, config);+ const second = await existing(root, config);+ await assignTo(root, config, 1, 'room', first.sessionId);+ await assignTo(root, config, 2, 'elsewhere', second.sessionId);+ vi.spyOn(console, 'warn').mockImplementation(() => {});+ const hosts = sessionHosts();+ await hosts.start(first.sessionRoot);+ hosts.busy(first.sessionRoot, false);+ const control = new WatcherControl();+ const abort = new AbortController();+ const run = runAgentHost(+ root,+ config,+ abort.signal,+ hosts.supervision,+ control,+ openSwitchStream+ );+ try {+ await eventually(() => streams.length === 1);+ expect(control.health().busy).toEqual([]);+ hosts.busy(first.sessionRoot, true);+ expect(control.health().busy).toEqual([first.sessionId]);+ hosts.busy(first.sessionRoot, false);+ expect(control.health().busy).toEqual([]);+ // A host that has not said yet whether it is in a turn is not taken to be idle.+ await hosts.start(second.sessionRoot);+ expect(control.health().busy).toEqual([second.sessionId]);+ hosts.exit(second.sessionRoot);+ expect(control.health().busy).toEqual([]);+ } finally {+ abort.abort();+ await run;+ }+ expect(control.health().busy).toEqual([]);+});+/** Records, as a session's host does as it starts, the shape of the grants it started with. */async function startedWith(sessionRoot: string, shape: string): Promise<void> {await writeFile(join(sessionRoot, 'services.json'), JSON.stringify({ shape }));diff --git a/console/packages/agent-providers/src/host/agent-host.ts b/console/packages/agent-providers/src/host/agent-host.tsindex 99ba3551d..a527db5d5 100644--- a/console/packages/agent-providers/src/host/agent-host.ts+++ b/console/packages/agent-providers/src/host/agent-host.ts@@ -713,8 +713,31 @@ export async function runAgentHost(token: credentials.SWITCH_API_TOKEN,};const placements = await SessionPlacements.open(root, () => assignments.placements());- control.report({ state: 'connecting', detail: null, placements: placements.snapshot() });- unbind.push(placements.onChange((map) => control.report({ placements: map })));+ /**+ * This agent's sessions whose host runs here and has not said it is idle:+ * the ones an agents controller restarting now would cut off mid-turn.+ */+ const busySessions = (): string[] => {+ const live = new Set(links.live());+ const ids = new Set([+ ...assignments.sessions().map((config) => config.session.sessionId),+ ...Object.keys(placements.snapshot()),+ ]);+ return [...ids].filter((sessionId) => {+ const sessionRoot = sharedSessionRoot(sessionId);+ return live.has(sessionRoot) && links.busy(sessionRoot)?.busy !== false;+ });+ };+ control.report({+ state: 'connecting',+ detail: null,+ placements: placements.snapshot(),+ busy: busySessions(),+ });+ unbind.push(+ placements.onChange((map) => control.report({ placements: map, busy: busySessions() })),+ links.onBusy(() => control.report({ busy: busySessions() }))+ );let stream: AgentEventStream | null = null;let publishing: Promise<void> = Promise.resolve();/**@@ -1537,6 +1560,7 @@ export async function runAgentHost(: String(failure): ending.detail,placements: {},+ busy: [],});await releaseOwner(root, ownerPath, owner);}diff --git a/console/packages/agent-providers/src/host/session-channel.test.ts b/console/packages/agent-providers/src/host/session-channel.test.tsindex 41884a404..dcde538c6 100644--- a/console/packages/agent-providers/src/host/session-channel.test.ts+++ b/console/packages/agent-providers/src/host/session-channel.test.ts@@ -318,7 +318,7 @@ it('keeps what a host last said about being busy, and answers a barrier once itchild.emit('exit', 0, null);await expect(lost).rejects.toThrow(SessionUnavailableError);expect(links.busy('root')).toBeNull();- expect(heard).toEqual(['root', 'root', 'root']);+ expect(heard).toEqual(['root', 'root', 'root', 'root']);});it('answers a busy barrier on the host side with the settled state', async () => {diff --git a/console/packages/agent-providers/src/host/session-channel.ts b/console/packages/agent-providers/src/host/session-channel.tsindex f01309f14..9d20c7bbc 100644--- a/console/packages/agent-providers/src/host/session-channel.ts+++ b/console/packages/agent-providers/src/host/session-channel.ts@@ -279,6 +279,7 @@ export class SessionLinks {link.identity = null;link.failure = null;link.busy = null;+ for (const listener of this.busyListeners) listener(root);child.on('message', (raw) => {const parsed = fromChildSchema.safeParse(raw);if (!parsed.success) {@@ -553,7 +554,7 @@ export class SessionLinks {return [...this.links].flatMap(([root, link]) => (link.child ? [root] : []));}- /** Hear each change in what a host says about being busy, and each host exit. */+ /** Hear each host start and exit, and each change in what a host says about being busy. */onBusy(listener: (root: string) => void): () => void {this.busyListeners.add(listener);return () => this.busyListeners.delete(listener);diff --git a/console/packages/agent-providers/src/host/shared-daemon.ts b/console/packages/agent-providers/src/host/shared-daemon.tsindex dd7c3703c..b9c4e65f0 100644--- a/console/packages/agent-providers/src/host/shared-daemon.ts+++ b/console/packages/agent-providers/src/host/shared-daemon.ts@@ -26,7 +26,7 @@ import {import { hostSessionProcess } from './shared-host';import { onStopRequest } from './stop-request';import { superviseSharedHost } from './supervisor';-import { recordWatcherHealth } from './watcher-health-file';+import { bundleIdentity, recordWatcherHealth } from './watcher-health-file';import { WatcherControl } from './watcher-tools';const [root, configPath, mode] = process.argv.slice(2);@@ -123,6 +123,8 @@ async function main(): Promise<void> {links: null,});} else if (mode === '--watch-worker') {+ // Read before anything else, while the file is the one this process loaded.+ const bundle = await bundleIdentity(process.argv[1]!);const stop = new AbortController();onStopRequest(() => stop.abort(), resolve(root));// The sidecar is the parent of the sessions it runs: it talks to each over@@ -159,7 +161,7 @@ async function main(): Promise<void> {: undefined;// Console reads the watcher's connection state from this file, with the// rest of the host's watcher state, rather than from the control port.- const stopRecording = recordWatcherHealth(resolve(root), control);+ const stopRecording = recordWatcherHealth(resolve(root), control, bundle);// A watcher that stops (disabled, stood down after a takeover, or// signalled) takes the process with it: the control port and every// session host go too, so the supervisor sees a clean exit and does notdiff --git a/console/packages/agent-providers/src/host/watcher-health-file.test.ts b/console/packages/agent-providers/src/host/watcher-health-file.test.tsindex b367ff86c..8a95d5dff 100644--- a/console/packages/agent-providers/src/host/watcher-health-file.test.ts+++ b/console/packages/agent-providers/src/host/watcher-health-file.test.ts@@ -1,8 +1,14 @@-import { mkdtemp, readdir, readFile, rm } from 'node:fs/promises';+import { createHash } from 'node:crypto';+import { mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises';import { tmpdir } from 'node:os';import { join } from 'node:path';import { afterEach, expect, it, vi } from 'vitest';-import { recordWatcherHealth, WATCHER_HEALTH_FILE } from './watcher-health-file';+import {+ bundleIdentity,+ recordWatcherHealth,+ WATCHER_HEALTH_FILE,+ watcherHealthFileSchema,+} from './watcher-health-file';import { WatcherControl } from './watcher-tools';const roots: string[] = [];@@ -18,7 +24,7 @@ it('keeps the watcher’s own account of its connection on the host, as it changconst root = await mkdtemp(join(tmpdir(), 'watcher-health-'));roots.push(root);const control = new WatcherControl();- const stop = recordWatcherHealth(root, control);+ const stop = recordWatcherHealth(root, control, 'bundle-sha');await vi.waitFor(async () =>expect(await written(root)).toMatchObject({ state: 'not-running', pid: process.pid })@@ -32,6 +38,7 @@ it('keeps the watcher’s own account of its connection on the host, as it changdetail: null,placements: { session: 'room' },pid: process.pid,+ bundle: 'bundle-sha',}));// Renamed into place, so nothing half-written is ever left beside it.@@ -42,3 +49,21 @@ it('keeps the watcher’s own account of its connection on the host, as it changawait new Promise((resolve) => setTimeout(resolve, 20));expect((await written(root)).state).toBe('connected');});++it('names the bundle a process started from by its content, and reads a file that names none', async () => {+ const root = await mkdtemp(join(tmpdir(), 'watcher-health-'));+ roots.push(root);+ const bundle = join(root, 'shared-host.mjs');+ await writeFile(bundle, 'console.log(1);');+ expect(await bundleIdentity(bundle)).toBe(+ createHash('sha256').update('console.log(1);').digest('hex')+ );++ const control = new WatcherControl();+ const stop = recordWatcherHealth(root, control, null);+ await vi.waitFor(async () => expect(await written(root)).toMatchObject({ pid: process.pid }));+ stop();+ const parsed = watcherHealthFileSchema.parse(await written(root));+ expect(parsed.bundle).toBeUndefined();+ expect(parsed.busy).toBeUndefined();+});diff --git a/console/packages/agent-providers/src/host/watcher-health-file.ts b/console/packages/agent-providers/src/host/watcher-health-file.tsindex 0d1bcb8cd..233d53706 100644--- a/console/packages/agent-providers/src/host/watcher-health-file.ts+++ b/console/packages/agent-providers/src/host/watcher-health-file.ts@@ -1,5 +1,5 @@-import { randomUUID } from 'node:crypto';-import { writeFile } from 'node:fs/promises';+import { createHash, randomUUID } from 'node:crypto';+import { readFile, writeFile } from 'node:fs/promises';import { join } from 'node:path';import { z } from 'zod';import { fileMode } from './host-permissions';@@ -26,17 +26,39 @@ export const WATCHER_HEALTH_FILE = 'health.json';export const watcherHealthFileSchema = watcherHealthSchema.extend({pid: z.number().int().positive(),updatedAt: z.string(),+ /**+ * The `bundleIdentity` of the shared-host bundle the writing process was+ * started from. Absent from a process that does not run from one (an agents+ * controller's own), and from one too old to say.+ */+ bundle: z.string().optional(),});export type WatcherHealthFile = z.infer<typeof watcherHealthFileSchema>;+/**+ * What identifies a shared-host bundle: the SHA-256 of the file. Its path does+ * not, since an install can replace the file in place.+ */+export async function bundleIdentity(path: string): Promise<string> {+ return createHash('sha256')+ .update(await readFile(path))+ .digest('hex');+}+/*** Keep `health.json` in `root` in step with what the watcher reports to* `control`, from now until the returned function is called. Written whole,* under a temporary name and renamed into place, so a reader never sees half* of it; and in order, so an older state never lands after a newer one.+ * `bundle` is the `bundleIdentity` of the bundle this process started from,+ * or null when it did not start from one.*/-export function recordWatcherHealth(root: string, control: WatcherControl): () => void {+export function recordWatcherHealth(+ root: string,+ control: WatcherControl,+ bundle: string | null+): () => void {const path = join(root, WATCHER_HEALTH_FILE);let writing: Promise<void> = Promise.resolve();const write = () => {@@ -44,6 +66,7 @@ export function recordWatcherHealth(root: string, control: WatcherControl): () =...control.health(),pid: process.pid,updatedAt: new Date().toISOString(),+ ...(bundle ? { bundle } : {}),};writing = writing.then(async () => {diff --git a/console/packages/agent-providers/src/host/watcher-tools.ts b/console/packages/agent-providers/src/host/watcher-tools.tsindex 122c4c349..6dc344348 100644--- a/console/packages/agent-providers/src/host/watcher-tools.ts+++ b/console/packages/agent-providers/src/host/watcher-tools.ts@@ -320,6 +320,11 @@ export const watcherHealthSchema = z.object({since: z.string(),/** Session id → room id, as the watcher routes now. Empty while it is not running. */placements: z.record(z.string(), z.string()),+ /**+ * The agent's sessions whose host runs here and is mid-turn, or has not yet+ * said whether it is. Absent from a watcher too old to report it.+ */+ busy: z.array(z.string()).optional(),});export type WatcherHealth = z.infer<typeof watcherHealthSchema>;@@ -400,6 +405,7 @@ export class WatcherControl {state?: WatcherState;detail?: string | null;placements?: PlacementMap;+ busy?: string[];}): void {const state = change.state ?? this.current.state;const same = state === this.current.state;@@ -408,11 +414,17 @@ export class WatcherControl {detail: change.detail !== undefined ? change.detail : same ? this.current.detail : null,since: same ? this.current.since : new Date().toISOString(),placements: change.placements ? { ...change.placements } : this.current.placements,+ ...(change.busy+ ? { busy: [...change.busy].sort() }+ : this.current.busy+ ? { busy: this.current.busy }+ : {}),};if (same &&next.detail === this.current.detail &&- samePlacements(next.placements, this.current.placements)+ samePlacements(next.placements, this.current.placements) &&+ sameList(next.busy, this.current.busy))return;this.current = next;@@ -426,6 +438,11 @@ export class WatcherControl {}}+function sameList(a: string[] | undefined, b: string[] | undefined): boolean {+ if (!a || !b) return a === b;+ return a.length === b.length && a.every((value, index) => value === b[index]);+}+function samePlacements(a: PlacementMap, b: PlacementMap): boolean {const keys = Object.keys(a);return keys.length === Object.keys(b).length && keys.every((key) => a[key] === b[key]);diff --git a/console/packages/agent-providers/src/index.ts b/console/packages/agent-providers/src/index.tsindex 656bec3e2..b2a2b8125 100644--- a/console/packages/agent-providers/src/index.ts+++ b/console/packages/agent-providers/src/index.ts@@ -94,6 +94,7 @@ export {} from './host/hub-stream';export { clearTakenOver, readTakenOver, type TakenOver } from './host/taken-over';export {+ bundleIdentity,recordWatcherHealth,WATCHER_HEALTH_FILE,type WatcherHealthFile,diff --git a/docs/design/controller-contract-v1.md b/docs/design/controller-contract-v1.mdindex ad4c64ae8..895df03a9 100644--- a/docs/design/controller-contract-v1.md+++ b/docs/design/controller-contract-v1.md@@ -136,7 +136,8 @@ type ControllerRelease = {`_URL`, `_SHA256`; Helm `switchCore.controllerRelease`), and the chart from aSwitch release names that release's controller. A controller running anotherversion downloads the package, refuses it unless its SHA-256 matches, and installs-it and restarts once no agent session is running. It says how that is going in+it and restarts once no agent session is in the middle of a turn (idle sessions+are restarted with it and resume their conversations). It says how that is going inits report's `update` (`{ target, state, problem? }`), which Core stores with therest of the report. Rolling back is pointing the setting at the previousrelease's package. A controller older than the field ignores it. A Switch Console@@ -160,7 +161,7 @@ type StatusReport = {agents: AgentStatus[]update?: { // why it does not run controller_release yet. Absent when it does, and from older controllerstarget: string // the version it is moving to- state: "waiting_idle" | "failed" | "restart_needed"+ state: "waiting_idle" | "failed" | "restart_needed" // waiting_idle: a session on the machine is in the middle of a turnproblem?: string // human-readable, when it failed}}