Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Agents controllers update once no session is mid-turn, and move their agents onto the new release (#866)

Louis Amaudruz733551e9f551
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5496d85ce..002b434c2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3505,6 +3505,26 @@ tags; see RELEASING.md.
can run (16 by default).
#### Fixed
+- **A controller updates to the release Switch names while sessions sit idle.**
+ It waited until no session existed on the machine, and a session connected to
+ a room counted even with nothing to do, so a machine with long-lived sessions
+ stayed `waiting_idle` for good. It now waits only until no session is in the
+ middle of a turn. Each agent host reports which of its sessions are busy (one
+ that has not said yet counts as busy), and the controller looks again every
+ few seconds while it waits rather than at the next status report. Idle
+ sessions in the controller's own process are restarted with it and resume
+ their conversations; agents running in a process of their own keep running
+ through it. An agent host too old to report busy sessions still holds the
+ update while it has any session.
+- **Agents running in a process of their own move onto the release the
+ controller updated to.** Such an agent host (detached, or a systemd unit with
+ separate users, as on Switch cloud machines) outlives the controller and kept
+ the shared-host bundle it started with, so a fix in it reached the agent only
+ when the agent was restarted by hand. Each agent host now records the bundle
+ it runs (by SHA-256) in its `health.json`, and the controller restarts one on
+ another bundle, or too old to say, once none of its sessions is mid-turn;
+ idle sessions resume their conversations. A host that comes back still
+ outdated is tried again after 10 minutes.
- **A refusal that cannot pass stops the controller instead of retrying it
forever.** The server answers `protocol_unsupported` when it no longer speaks
the controller's protocol and `invalid_credential` when it knows no controller
diff --git a/console/packages/agent-controller/README.md b/console/packages/agent-controller/README.md
index 15ae74733..103843802 100644
--- a/console/packages/agent-controller/README.md
+++ b/console/packages/agent-controller/README.md
@@ -348,14 +348,22 @@ When Switch names a release for this controller, `run` follows it; otherwise
`controller_release: {"version", "url", "sha256"}`. When it names a version other
than the one running (newer or older: Switch can roll back), the controller:
-1. waits until no session runs on the machine (`sessions_running` is 0), checking
- at each status report, and reports `update: {"target", "state": "waiting_idle"}`
- in its status meanwhile;
+1. waits until no session on the machine is in the middle of a turn, checking at
+ each status report and every few seconds while it waits, and reports
+ `update: {"target", "state": "waiting_idle"}` in its status meanwhile. Each
+ agent host reports which of its sessions are busy; one that has not said yet
+ counts as busy. Idle sessions do not hold the update back: those in the
+ controller's own process are restarted with it and resume their
+ conversations, and agents running in a process of their own keep running
+ through it (once the controller is back on the new release, it restarts each
+ of those onto it as soon as none of its sessions is mid-turn: each records
+ the shared-host bundle it runs in its `health.json`). An agent host too old to report busy sessions holds it back
+ while it has any session, as `sessions_running` counts them;
2. downloads the package into `<data dir>/updates/` (at most 200 MB) and checks
its SHA-256 against the one Switch gave, the only one trusted; a package that
differs is deleted;
3. installs it from that file, never from the URL, checks that the installed CLI
- says `--version` is that version, checks again that no session started
+ says `--version` is that version, checks again that no turn started
meanwhile, and switches to it;
4. exits with code `7`, and its service starts it again on the new release.
diff --git a/console/packages/agent-controller/src/controller.test.ts b/console/packages/agent-controller/src/controller.test.ts
index cff123b23..d7cde0795 100644
--- a/console/packages/agent-controller/src/controller.test.ts
+++ b/console/packages/agent-controller/src/controller.test.ts
@@ -662,7 +662,7 @@ describe('the release Switch names', () => {
sha256: 'c'.repeat(64),
};
- it('is installed while no session runs, and the controller ends to restart on it', async () => {
+ it('is installed while no session is mid-turn, and the controller ends to restart on it', async () => {
core.setAssignment({ revision: 1, agents: [agent(1)] });
core.controllerRelease = release;
const { updater, installed } = selfUpdate();
@@ -671,7 +671,7 @@ describe('the release Switch names', () => {
expect(installed).toEqual(['0.2.0']);
});
- it('waits while a session runs, and says so in the status', async () => {
+ it('waits while an agent host that does not report busy sessions has any, and says so in the status', async () => {
core.setAssignment({ revision: 1, agents: [agent(1)] });
const { updater, installed } = selfUpdate();
running = runController({ ...deps(), selfUpdate: updater }, stop.signal);
@@ -689,6 +689,25 @@ describe('the release Switch names', () => {
expect(installed).toEqual(['0.2.0']);
});
+ it('waits only while a session is mid-turn when the agent host says which are', async () => {
+ core.setAssignment({ revision: 1, agents: [agent(1)] });
+ const { updater, installed } = selfUpdate();
+ running = runController({ ...deps(), selfUpdate: updater }, stop.signal);
+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the agent started');
+ const health = runtime.observation('agent-1').health!;
+ health.placements = { 'session-1': 'room-a', 'session-2': 'room-b' };
+ health.busy = ['session-1'];
+ core.controllerRelease = release;
+ await waitFor(
+ () => core.statusReports.some((report) => report.update?.state === 'waiting_idle'),
+ 'a status saying the update waits'
+ );
+ expect(installed).toEqual([]);
+ health.busy = [];
+ expect(await running).toBe('restarting');
+ expect(installed).toEqual(['0.2.0']);
+ });
+
it('is left alone by a controller that does not update itself', async () => {
core.setAssignment({ revision: 1, agents: [agent(1)] });
core.controllerRelease = release;
@@ -700,6 +719,53 @@ describe('the release Switch names', () => {
});
});
+describe('an agent host that outlived the controller on another bundle', () => {
+ const restarts = () =>
+ runtime.launches('agent-1').filter((call) => call.options.restart === true);
+
+ it('is restarted onto this controller’s bundle once none of its sessions is mid-turn, and only once', async () => {
+ core.setAssignment({ revision: 1, agents: [agent(1)] });
+ running = runController(deps(), stop.signal);
+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the agent started');
+ const observation = runtime.observation('agent-1');
+ observation.outdated = true;
+ observation.health!.placements = { 'session-1': 'room-a' };
+ observation.health!.busy = ['session-1'];
+ await new Promise((resolve) => setTimeout(resolve, 200));
+ expect(restarts()).toEqual([]);
+
+ observation.health!.busy = [];
+ await waitFor(() => restarts().length === 1, 'the outdated agent host restarted');
+ await new Promise((resolve) => setTimeout(resolve, 200));
+ expect(restarts()).toHaveLength(1);
+ stop.abort();
+ expect(await running).toBe('stopped');
+ });
+
+ it('is not restarted again while it stays on another bundle, within the retry wait', async () => {
+ core.setAssignment({ revision: 1, agents: [agent(1)] });
+ running = runController(deps(), stop.signal);
+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the agent started');
+ runtime.observation('agent-1').outdated = true;
+ await waitFor(() => restarts().length === 1, 'the outdated agent host restarted');
+ runtime.observation('agent-1').outdated = true;
+ await new Promise((resolve) => setTimeout(resolve, 200));
+ expect(restarts()).toHaveLength(1);
+ stop.abort();
+ expect(await running).toBe('stopped');
+ });
+
+ it('is left alone when it runs this controller’s bundle', async () => {
+ core.setAssignment({ revision: 1, agents: [agent(1)] });
+ running = runController(deps(), stop.signal);
+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the agent started');
+ await new Promise((resolve) => setTimeout(resolve, 200));
+ expect(restarts()).toEqual([]);
+ stop.abort();
+ expect(await running).toBe('stopped');
+ });
+});
+
describe('ensureRelayCredentials', () => {
const login = {
status: 'connected' as const,
diff --git a/console/packages/agent-controller/src/controller.ts b/console/packages/agent-controller/src/controller.ts
index 78332a5a6..e3660e9dc 100644
--- a/console/packages/agent-controller/src/controller.ts
+++ b/console/packages/agent-controller/src/controller.ts
@@ -30,6 +30,7 @@ import { type GivenLogin, registerSealingKey, SealedLogins, sealingKeys } from '
import { CONTROLLER_CREDENTIAL, type SecretStore } from './secrets';
import type { SelfUpdater } from './self-update';
import {
+ noSessionMidTurn,
type ProviderLocator,
ProviderStatuses,
StatusCollector,
@@ -70,6 +71,13 @@ export const DEFAULT_TIMING: ControllerTiming = {
eventBufferLimit: 5_000,
};
+/**
+ * How long an agent host restarted onto this controller's shared-host bundle
+ * is left before it is restarted again, when the restart failed or it still
+ * runs another bundle.
+ */
+export const OUTDATED_RESTART_RETRY_MS = 10 * 60 * 1000;
+
export type ControllerDeps = {
store: ControllerStore;
secrets: SecretStore;
@@ -499,7 +507,16 @@ export async function runController(
});
};
- const idle = async () => (await collector.collect(assignment)).machine.sessions_running === 0;
+ const idle = async () =>
+ noSessionMidTurn(
+ await Promise.all((assignment?.agents ?? []).map((entry) => runtime.observe(entry.agent_id)))
+ );
+ /** What the last status response named; looked at again while an update waits for idle. */
+ let offeredRelease: unknown = null;
+ const offerRelease = () => {
+ if (deps.selfUpdate && !stop.signal.aborted)
+ void deps.selfUpdate.offer(offeredRelease, { idle, restart, signal: stop.signal });
+ };
reporter = new StatusReporter({
collect: async () => {
const collected = await collector.collect(assignment);
@@ -511,12 +528,8 @@ export async function runController(
reportWithinS = answer.report_within_s;
if (assignment === null || answer.assignment_revision > assignment.revision)
void sync('status says the assignment moved on');
- if (deps.selfUpdate && !stop.signal.aborted)
- void deps.selfUpdate.offer(answer.controller_release, {
- idle,
- restart,
- signal: stop.signal,
- });
+ offeredRelease = answer.controller_release;
+ offerRelease();
},
seq: () => store.nextStatusSeq(deps.now()),
minGapMs: timing.statusMinGapMs,
@@ -544,6 +557,61 @@ export async function runController(
});
};
+ /** When each outdated agent host was last restarted onto this controller's bundle. */
+ const outdatedRestarts = new Map<string, number>();
+ let outdatedQueued = false;
+ /**
+ * Restarts each agent host that outlived this controller on another
+ * shared-host bundle (`AgentObservation.outdated`) onto this controller's,
+ * once none of its sessions is mid-turn: idle sessions restart with it and
+ * resume their conversations. An agent host already restarted for it is
+ * left for `OUTDATED_RESTART_RETRY_MS` before another try.
+ */
+ const restartOutdated = (): Promise<void> => {
+ if (outdatedQueued) return Promise.resolve();
+ outdatedQueued = true;
+ return queue.run(async () => {
+ outdatedQueued = false;
+ if (!assignment || stop.signal.aborted) return;
+ try {
+ for (const entry of assignment.agents) {
+ if (entry.desired_state !== 'running' || definitionProblem(entry)) continue;
+ const agentId = entry.agent_id;
+ const observation = await runtime.observe(agentId);
+ if (!observation.alive || !observation.outdated) continue;
+ const last = outdatedRestarts.get(agentId);
+ if (last !== undefined && deps.now() - last < OUTDATED_RESTART_RETRY_MS) continue;
+ if (!noSessionMidTurn([observation])) continue;
+ if (last !== undefined)
+ log.error(
+ 'An agent host restarted onto this controller’s shared-host bundle is still on another (the restart failed, or it came back on another bundle); restarting it again',
+ { agentId, restartedAt: new Date(last).toISOString() }
+ );
+ outdatedRestarts.set(agentId, deps.now());
+ log.warn(
+ 'Restarting an agent host that outlived this controller on another shared-host bundle, now that none of its sessions is mid-turn',
+ { agentId, version: deps.version }
+ );
+ await startAgent(
+ {
+ kind: 'start',
+ agentId,
+ entry,
+ restart: true,
+ replaceIdentity: false,
+ clearTakenOver: false,
+ relaunch: true,
+ why: 'agent host on another shared-host bundle',
+ },
+ reconcileDeps
+ );
+ }
+ } catch (error) {
+ failed('Restarting outdated agent hosts', error);
+ }
+ });
+ };
+
const onFrame = async (frame: ControllerFrame): Promise<void> => {
switch (frame.type) {
case 'connection_state':
@@ -608,6 +676,10 @@ export async function runController(
return;
}
void reporter?.requestIfChanged();
+ // A turn ending changes nothing reported, so it would otherwise wait for
+ // the next periodic report to let the update through.
+ if (deps.selfUpdate?.status()?.state === 'waiting_idle') offerRelease();
+ void restartOutdated();
}, timing.statusPollMs);
let ending: ControllerExit = 'stopped';
diff --git a/console/packages/agent-controller/src/detached-runtime.ts b/console/packages/agent-controller/src/detached-runtime.ts
index 2fa39c5e9..c5f167b0c 100644
--- a/console/packages/agent-controller/src/detached-runtime.ts
+++ b/console/packages/agent-controller/src/detached-runtime.ts
@@ -15,6 +15,7 @@ import type { DataLayout } from './paths';
import {
type AgentObservation,
type AgentRunner,
+ LaunchedBundle,
type LaunchOptions,
ownsRoot,
OWNER_RECORDS,
@@ -38,18 +39,24 @@ const LAUNCH_TIMEOUT_MS = 60_000;
* controller's hub, over a WebSocket on the relay's port.
*
* Being its own process, it is not stopped when the controller exits: it
- * keeps retrying the hub, and picks up again when the controller is back.
+ * keeps retrying the hub, and picks up again when the controller is back,
+ * still on the bundle it was started from. One on another bundle than this
+ * controller's is `outdated`, and the controller restarts it once idle.
* A restart is a stop and a start: the agent host is turned off, waited out,
* and launched again from the new template.
*/
export class DetachedRuntime implements AgentRunner {
+ private readonly bundle: LaunchedBundle;
+
constructor(
private readonly deps: {
layout: DataLayout;
/** The `shared-host-daemon.mjs` bundle from agent-providers. */
bundlePath: string;
}
- ) {}
+ ) {
+ this.bundle = new LaunchedBundle(deps.bundlePath);
+ }
async observe(agentId: string): Promise<AgentObservation> {
const root = this.deps.layout.watcherRoot(agentId);
@@ -67,7 +74,12 @@ export class DetachedRuntime implements AgentRunner {
current: parsed.pid !== process.pid && (await ownsRoot(parsed.pid, root)),
};
}
- return { ...(await readRoot(root)), alive: living, health };
+ return {
+ ...(await readRoot(root)),
+ alive: living,
+ health,
+ outdated: living && (await this.bundle.outdates(health)),
+ };
}
async launch(agentId: string, template: SharedHostConfig, options: LaunchOptions): Promise<void> {
diff --git a/console/packages/agent-controller/src/main.ts b/console/packages/agent-controller/src/main.ts
index 92160d4d5..9c1c185f8 100644
--- a/console/packages/agent-controller/src/main.ts
+++ b/console/packages/agent-controller/src/main.ts
@@ -986,7 +986,7 @@ async function updateCommand(args: string[]): Promise<number> {
const prefix = installPrefix(cliPath());
if (prefix && (await isManagedRelease(prefix)))
throw new ConfigurationError(
- `This controller runs from ${prefix}, a releases directory install-service set up: it installs the release Switch names itself, once no session runs. To install another by hand, install it with npm and run install-service again.`
+ `This controller runs from ${prefix}, a releases directory install-service set up: it installs the release Switch names itself, once no session is mid-turn. To install another by hand, install it with npm and run install-service again.`
);
const dataDir = resolveDataDir(values['data-dir']);
process.stdout.write(`Installing ${latest.version} from ${latest.packageUrl}\n`);
diff --git a/console/packages/agent-controller/src/runtime.test.ts b/console/packages/agent-controller/src/runtime.test.ts
index f9b3dc39a..ef3afbd68 100644
--- a/console/packages/agent-controller/src/runtime.test.ts
+++ b/console/packages/agent-controller/src/runtime.test.ts
@@ -9,10 +9,15 @@ import {
statSync,
writeFileSync,
} from 'node:fs';
+import { mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
-import { type OpenAgentStream, readSharedCredentials } from '@switch-console/agent-providers';
+import {
+ bundleIdentity,
+ type OpenAgentStream,
+ readSharedCredentials,
+} from '@switch-console/agent-providers';
import {
SERVICE_ACCOUNT_KEY_FIXTURE,
vertexCredentialFixture,
@@ -20,7 +25,13 @@ import {
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { silentLogger } from './log';
import { dataLayout } from './paths';
-import { InProcessRuntime, isInside, loginProbeEnvironment, observeOnDisk } from './runtime';
+import {
+ InProcessRuntime,
+ isInside,
+ LaunchedBundle,
+ loginProbeEnvironment,
+ observeOnDisk,
+} from './runtime';
import { buildWatcherTemplate } from './template';
/** Stands in for the shared-host bundle: answers `--probe`. Sessions are never started here. */
@@ -331,6 +342,39 @@ describe('InProcessRuntime', () => {
});
});
+describe('LaunchedBundle', () => {
+ const health = (bundle: string | undefined, current = true) => ({
+ state: 'connected' as const,
+ detail: null,
+ since: new Date(0).toISOString(),
+ placements: {},
+ pid: 4242,
+ updatedAt: new Date(0).toISOString(),
+ current,
+ ...(bundle === undefined ? {} : { bundle }),
+ });
+
+ it('says a live agent host on another bundle, or one that names none, is outdated', async () => {
+ const dir = await mkdtemp(join(tmpdir(), 'launched-bundle-'));
+ try {
+ const path = join(dir, 'shared-host.mjs');
+ await writeFile(path, 'console.log(1);');
+ const launched = new LaunchedBundle(path);
+ const ours = await bundleIdentity(path);
+ expect(await launched.outdates(health(ours))).toBe(false);
+ expect(await launched.outdates(health('another'))).toBe(true);
+ expect(await launched.outdates(health(undefined))).toBe(true);
+ expect(await launched.outdates(health('another', false))).toBe(false);
+ expect(await launched.outdates(null)).toBe(false);
+
+ await writeFile(path, 'console.log(22);');
+ expect(await launched.outdates(health(ours))).toBe(true);
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+ });
+});
+
describe('isInside', () => {
it('is true strictly below the root', () => {
expect(isInside('/data/workspaces', '/data/workspaces/scout')).toBe(true);
diff --git a/console/packages/agent-controller/src/runtime.ts b/console/packages/agent-controller/src/runtime.ts
index 5c88bac37..22f435599 100644
--- a/console/packages/agent-controller/src/runtime.ts
+++ b/console/packages/agent-controller/src/runtime.ts
@@ -6,6 +6,7 @@ import { isAbsolute, join, relative, resolve, sep } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
import { promisify } from 'node:util';
import {
+ bundleIdentity,
clearTakenOver,
commandLineNames,
ensureSharedProcess,
@@ -56,6 +57,13 @@ export type AgentObservation = {
/** `supervisor/failure.json`: why the agent host stopped and was not restarted. */
failure: string | null;
takenOver: TakenOver | null;
+ /**
+ * The agent host outlives this controller (it runs in a process of its own)
+ * and runs another shared-host bundle than the one this controller launches
+ * agent hosts from, or one too old to say which. Read from a live agent
+ * host's current health only; false otherwise.
+ */
+ outdated: boolean;
};
/** What an agent with no agent host root at all looks like. */
@@ -67,6 +75,7 @@ export function emptyObservation(): AgentObservation {
health: null,
failure: null,
takenOver: null,
+ outdated: false,
};
}
@@ -277,6 +286,34 @@ export async function recordedPid(path: string): Promise<number | null> {
return typeof pid === 'number' && Number.isSafeInteger(pid) && pid > 0 ? pid : null;
}
+/**
+ * The shared-host bundle a runner launches agent hosts from, by
+ * `bundleIdentity`: read again only when the file at its path changes, as an
+ * update installed over it or a moved `current` link does.
+ */
+export class LaunchedBundle {
+ private known: { key: string; identity: string } | null = null;
+
+ constructor(private readonly path: string) {}
+
+ async identity(): Promise<string> {
+ const info = await stat(this.path);
+ const key = `${info.dev}:${info.ino}:${info.size}:${info.mtimeMs}`;
+ if (this.known?.key !== key) this.known = { key, identity: await bundleIdentity(this.path) };
+ return this.known.identity;
+ }
+
+ /**
+ * Whether the live agent host whose health this is runs another bundle, or
+ * one too old to say which. Health that is not current says nothing.
+ */
+ async outdates(health: AgentObservation['health']): Promise<boolean> {
+ if (!health?.current) return false;
+ if (health.bundle === undefined) return true;
+ return health.bundle !== (await this.identity());
+ }
+}
+
/** The owner records an agent host writes under its root: the worker's, then a detached one's supervisor's. */
export const OWNER_RECORDS = ['shared-owner.lock', join('supervisor', 'owner.json')];
@@ -293,11 +330,13 @@ export async function observeOnDisk(
healthText === null
? null
: { ...watcherHealthFileSchema.parse(JSON.parse(healthText)), current: running };
- return { ...(await readRoot(root)), alive: running, health };
+ return { ...(await readRoot(root)), alive: running, health, outdated: false };
}
-/** Everything about an agent host root but whether it runs. */
-export async function readRoot(root: string): Promise<Omit<AgentObservation, 'alive'>> {
+/** Everything about an agent host root but whether it runs, and on which bundle. */
+export async function readRoot(
+ root: string
+): Promise<Omit<AgentObservation, 'alive' | 'outdated'>> {
const config = await readOptional(join(root, 'config.json'));
let configured: AgentObservation['configured'] = null;
if (config !== null) {
@@ -432,6 +471,7 @@ export class InProcessRuntime implements AgentRuntime {
current: true,
}
: null,
+ outdated: false,
};
}
@@ -503,7 +543,7 @@ export class InProcessRuntime implements AgentRuntime {
const control = new WatcherControl();
const sessions = inProcessSupervision(this.deps.bundlePath, this.links);
// For `status`, which runs in another process and reads only disk.
- const stopRecording = recordWatcherHealth(root, control);
+ const stopRecording = recordWatcherHealth(root, control, null);
const done = (async () => {
try {
await runAgentHost(root, config, signal, sessions, control, this.deps.openStream(agentId));
diff --git a/console/packages/agent-controller/src/self-update.ts b/console/packages/agent-controller/src/self-update.ts
index 0b2c7e0e9..7829a6d74 100644
--- a/console/packages/agent-controller/src/self-update.ts
+++ b/console/packages/agent-controller/src/self-update.ts
@@ -13,7 +13,9 @@ import { RELEASE_VERSION } from './update';
* Updating this controller to the release Switch names in its status
* responses (`controller_release`): fetched, checked against the SHA-256
* Switch gave, installed, then switched to by exiting with `EXIT_RESTARTING`
- * for the service to start it again. Only while no session runs.
+ * for the service to start it again. Only while no session is in the middle
+ * of a turn: idle sessions are restarted with it and resume their
+ * conversations.
*/
/** The largest package a download is allowed to be. */
@@ -322,7 +324,7 @@ export type SelfUpdaterDeps = {
/** What the running controller lends the updater for one offer. */
export type UpdateHost = {
- /** Whether no session runs here now. */
+ /** Whether no session here is in the middle of a turn now. */
idle: () => Promise<boolean>;
/** Ends the controller with `restarting`. */
restart: () => void;
@@ -332,7 +334,7 @@ export type UpdateHost = {
/**
* Follows the release Switch names: a version other than this one (newer or
- * older, so Switch can roll back) is installed while no session runs, and
+ * older, so Switch can roll back) is installed while no session is mid-turn, and
* the controller restarts into it. A failed attempt is not repeated for an
* hour; what stands in the way is reported (`status()`).
*/
@@ -447,7 +449,7 @@ export class SelfUpdater {
this.report = { target, state: 'waiting_idle' };
this.logOnce(`waiting:${target}`, () =>
this.deps.log.info(
- 'Switch names another controller release; it is installed once no session runs',
+ 'Switch names another controller release; it is installed once no session is mid-turn',
{
from: this.deps.version,
to: target,
diff --git a/console/packages/agent-controller/src/status.test.ts b/console/packages/agent-controller/src/status.test.ts
index 55c0cef5e..40c696d6e 100644
--- a/console/packages/agent-controller/src/status.test.ts
+++ b/console/packages/agent-controller/src/status.test.ts
@@ -10,6 +10,7 @@ import { type AgentAssignment, type SealedLoginResponse, statusReportSchema } fr
import { SealedLogins } from './sealed-logins';
import {
mapAgentProcess,
+ noSessionMidTurn,
PathProviderLocator,
PROVIDER_TTL_MS,
providerStatusFrom,
@@ -185,6 +186,34 @@ describe('mapAgentProcess', () => {
});
});
+describe('noSessionMidTurn', () => {
+ const reporting = (busy: string[] | undefined, current = true) =>
+ observed({ alive: true, health: { ...health('connected', null, current)!, busy } });
+
+ it('holds while an agent host reports a session mid-turn, however many sit idle', () => {
+ expect(noSessionMidTurn([reporting([])])).toBe(true);
+ expect(noSessionMidTurn([reporting([]), reporting(['session-a'])])).toBe(false);
+ });
+
+ it('holds while an agent host too old to report busy sessions has any session', () => {
+ expect(noSessionMidTurn([reporting(undefined)])).toBe(false);
+ const empty = reporting(undefined);
+ empty.health!.placements = {};
+ expect(noSessionMidTurn([empty])).toBe(true);
+ });
+
+ it('counts no session for an agent host not running or whose report is not its own', () => {
+ expect(noSessionMidTurn([reporting(['session-a'], false)])).toBe(true);
+ expect(
+ noSessionMidTurn([
+ observed({ alive: false, health: { ...health('connected')!, busy: ['x'] } }),
+ ])
+ ).toBe(true);
+ expect(noSessionMidTurn([emptyObservation(), observed({ alive: true })])).toBe(true);
+ expect(noSessionMidTurn([])).toBe(true);
+ });
+});
+
describe('providerStatusFrom', () => {
const at = '2026-01-01T00:00:00Z';
const located = { path: '/usr/bin/claude', version: '2.0.0', versionProblem: null };
diff --git a/console/packages/agent-controller/src/status.ts b/console/packages/agent-controller/src/status.ts
index 7fa169897..4d00f4c95 100644
--- a/console/packages/agent-controller/src/status.ts
+++ b/console/packages/agent-controller/src/status.ts
@@ -512,6 +512,23 @@ export function mapAgentProcess(input: {
};
}
+/**
+ * Whether no session here is in the middle of a turn, as each assigned
+ * agent's host last reported it: what a controller waits for before it
+ * restarts into another release. Idle sessions do not hold it back; they are
+ * restarted with it and resume their conversations. An agent host too old to
+ * report which sessions are busy holds it back while it has any session at
+ * all. An agent host not running, or whose report is not its own, has none.
+ */
+export function noSessionMidTurn(observations: AgentObservation[]): boolean {
+ return observations.every((observation) => {
+ const health = observation.alive && observation.health?.current ? observation.health : null;
+ if (!health) return true;
+ if (health.busy) return health.busy.length === 0;
+ return Object.keys(health.placements).length === 0;
+ });
+}
+
/**
* Builds status reports: the machine, the providers as last checked, and every
* assigned agent as its agent host's files describe it. `since` is kept across
diff --git a/console/packages/agent-controller/src/systemd-runtime.ts b/console/packages/agent-controller/src/systemd-runtime.ts
index f624f3594..cfbb3ed42 100644
--- a/console/packages/agent-controller/src/systemd-runtime.ts
+++ b/console/packages/agent-controller/src/systemd-runtime.ts
@@ -22,6 +22,7 @@ import {
type AgentRuntime,
emptyObservation,
isInside,
+ LaunchedBundle,
type LaunchOptions,
loginProbeEnvironment,
parseRelayCredentials,
@@ -94,13 +95,16 @@ type UnitState = { activeState: string; result: string; mainPid: number };
* credentials and environment it loads, and starts and stops the unit through
* `systemctl`, which polkit allows it for these units only. The agent host
* hears its events on the controller's hub, as one in a process of its own
- * does, and outlives the controller.
+ * does, and outlives the controller. The unit runs the bundle through the
+ * releases directory's `current` link, so restarting one that is `outdated`
+ * brings it onto the release the controller runs.
*
* Everything an agent writes is read as untrusted: through no link, parsed
* against its schema, and never run.
*/
export class SystemdRuntime implements AgentRuntime {
private readonly names: ReturnType<typeof separateUserNames>;
+ private readonly bundle: LaunchedBundle;
constructor(
private readonly deps: {
@@ -114,6 +118,7 @@ export class SystemdRuntime implements AgentRuntime {
}
) {
this.names = separateUserNames(deps.config.uid);
+ this.bundle = new LaunchedBundle(deps.config.bundle);
}
/** The agent user the agent runs as, claimed now when it holds none. */
@@ -266,7 +271,13 @@ export class SystemdRuntime implements AgentRuntime {
(state.activeState === 'failed'
? `The agent's unit ${unit} failed (${state.result}); see journalctl -u ${unit}.`
: null);
- return { ...onDisk, alive, health, failure };
+ return {
+ ...onDisk,
+ alive,
+ health,
+ failure,
+ outdated: alive && (await this.bundle.outdates(health)),
+ };
}
async launch(agentId: string, template: SharedHostConfig, options: LaunchOptions): Promise<void> {
diff --git a/console/packages/agent-controller/src/testing/fake-runtime.ts b/console/packages/agent-controller/src/testing/fake-runtime.ts
index 25826b7c3..4a04a043b 100644
--- a/console/packages/agent-controller/src/testing/fake-runtime.ts
+++ b/console/packages/agent-controller/src/testing/fake-runtime.ts
@@ -116,6 +116,7 @@ export class FakeRuntime implements AgentRuntime {
updatedAt: new Date(0).toISOString(),
current: true,
},
+ outdated: false,
} satisfies Partial<AgentObservation>);
}
diff --git a/console/packages/agent-providers/src/host/agent-host.test.ts b/console/packages/agent-providers/src/host/agent-host.test.ts
index 2b6c4d2ff..0474a2d29 100644
--- a/console/packages/agent-providers/src/host/agent-host.test.ts
+++ b/console/packages/agent-providers/src/host/agent-host.test.ts
@@ -1646,6 +1646,48 @@ it('brings live sessions in step with a rewritten definition, each once its turn
expect(restarted).toHaveLength(2);
});
+it('reports which of its sessions are mid-turn, as their hosts start, go busy, go idle and exit', async () => {
+ const root = await mkdtemp(join(tmpdir(), 'shared-watch-busy-'));
+ roots.push(root);
+ paths.root = root;
+ const config = await spawning(root);
+ const first = await existing(root, config);
+ const second = await existing(root, config);
+ await assignTo(root, config, 1, 'room', first.sessionId);
+ await assignTo(root, config, 2, 'elsewhere', second.sessionId);
+ vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const hosts = sessionHosts();
+ await hosts.start(first.sessionRoot);
+ hosts.busy(first.sessionRoot, false);
+ const control = new WatcherControl();
+ const abort = new AbortController();
+ const run = runAgentHost(
+ root,
+ config,
+ abort.signal,
+ hosts.supervision,
+ control,
+ openSwitchStream
+ );
+ try {
+ await eventually(() => streams.length === 1);
+ expect(control.health().busy).toEqual([]);
+ hosts.busy(first.sessionRoot, true);
+ expect(control.health().busy).toEqual([first.sessionId]);
+ hosts.busy(first.sessionRoot, false);
+ expect(control.health().busy).toEqual([]);
+ // A host that has not said yet whether it is in a turn is not taken to be idle.
+ await hosts.start(second.sessionRoot);
+ expect(control.health().busy).toEqual([second.sessionId]);
+ hosts.exit(second.sessionRoot);
+ expect(control.health().busy).toEqual([]);
+ } finally {
+ abort.abort();
+ await run;
+ }
+ expect(control.health().busy).toEqual([]);
+});
+
/** Records, as a session's host does as it starts, the shape of the grants it started with. */
async function startedWith(sessionRoot: string, shape: string): Promise<void> {
await writeFile(join(sessionRoot, 'services.json'), JSON.stringify({ shape }));
diff --git a/console/packages/agent-providers/src/host/agent-host.ts b/console/packages/agent-providers/src/host/agent-host.ts
index 99ba3551d..a527db5d5 100644
--- a/console/packages/agent-providers/src/host/agent-host.ts
+++ b/console/packages/agent-providers/src/host/agent-host.ts
@@ -713,8 +713,31 @@ export async function runAgentHost(
token: credentials.SWITCH_API_TOKEN,
};
const placements = await SessionPlacements.open(root, () => assignments.placements());
- control.report({ state: 'connecting', detail: null, placements: placements.snapshot() });
- unbind.push(placements.onChange((map) => control.report({ placements: map })));
+ /**
+ * This agent's sessions whose host runs here and has not said it is idle:
+ * the ones an agents controller restarting now would cut off mid-turn.
+ */
+ const busySessions = (): string[] => {
+ const live = new Set(links.live());
+ const ids = new Set([
+ ...assignments.sessions().map((config) => config.session.sessionId),
+ ...Object.keys(placements.snapshot()),
+ ]);
+ return [...ids].filter((sessionId) => {
+ const sessionRoot = sharedSessionRoot(sessionId);
+ return live.has(sessionRoot) && links.busy(sessionRoot)?.busy !== false;
+ });
+ };
+ control.report({
+ state: 'connecting',
+ detail: null,
+ placements: placements.snapshot(),
+ busy: busySessions(),
+ });
+ unbind.push(
+ placements.onChange((map) => control.report({ placements: map, busy: busySessions() })),
+ links.onBusy(() => control.report({ busy: busySessions() }))
+ );
let stream: AgentEventStream | null = null;
let publishing: Promise<void> = Promise.resolve();
/**
@@ -1537,6 +1560,7 @@ export async function runAgentHost(
: String(failure)
: ending.detail,
placements: {},
+ busy: [],
});
await releaseOwner(root, ownerPath, owner);
}
diff --git a/console/packages/agent-providers/src/host/session-channel.test.ts b/console/packages/agent-providers/src/host/session-channel.test.ts
index 41884a404..dcde538c6 100644
--- a/console/packages/agent-providers/src/host/session-channel.test.ts
+++ b/console/packages/agent-providers/src/host/session-channel.test.ts
@@ -318,7 +318,7 @@ it('keeps what a host last said about being busy, and answers a barrier once it
child.emit('exit', 0, null);
await expect(lost).rejects.toThrow(SessionUnavailableError);
expect(links.busy('root')).toBeNull();
- expect(heard).toEqual(['root', 'root', 'root']);
+ expect(heard).toEqual(['root', 'root', 'root', 'root']);
});
it('answers a busy barrier on the host side with the settled state', async () => {
diff --git a/console/packages/agent-providers/src/host/session-channel.ts b/console/packages/agent-providers/src/host/session-channel.ts
index f01309f14..9d20c7bbc 100644
--- a/console/packages/agent-providers/src/host/session-channel.ts
+++ b/console/packages/agent-providers/src/host/session-channel.ts
@@ -279,6 +279,7 @@ export class SessionLinks {
link.identity = null;
link.failure = null;
link.busy = null;
+ for (const listener of this.busyListeners) listener(root);
child.on('message', (raw) => {
const parsed = fromChildSchema.safeParse(raw);
if (!parsed.success) {
@@ -553,7 +554,7 @@ export class SessionLinks {
return [...this.links].flatMap(([root, link]) => (link.child ? [root] : []));
}
- /** Hear each change in what a host says about being busy, and each host exit. */
+ /** Hear each host start and exit, and each change in what a host says about being busy. */
onBusy(listener: (root: string) => void): () => void {
this.busyListeners.add(listener);
return () => this.busyListeners.delete(listener);
diff --git a/console/packages/agent-providers/src/host/shared-daemon.ts b/console/packages/agent-providers/src/host/shared-daemon.ts
index dd7c3703c..b9c4e65f0 100644
--- a/console/packages/agent-providers/src/host/shared-daemon.ts
+++ b/console/packages/agent-providers/src/host/shared-daemon.ts
@@ -26,7 +26,7 @@ import {
import { hostSessionProcess } from './shared-host';
import { onStopRequest } from './stop-request';
import { superviseSharedHost } from './supervisor';
-import { recordWatcherHealth } from './watcher-health-file';
+import { bundleIdentity, recordWatcherHealth } from './watcher-health-file';
import { WatcherControl } from './watcher-tools';
const [root, configPath, mode] = process.argv.slice(2);
@@ -123,6 +123,8 @@ async function main(): Promise<void> {
links: null,
});
} else if (mode === '--watch-worker') {
+ // Read before anything else, while the file is the one this process loaded.
+ const bundle = await bundleIdentity(process.argv[1]!);
const stop = new AbortController();
onStopRequest(() => stop.abort(), resolve(root));
// The sidecar is the parent of the sessions it runs: it talks to each over
@@ -159,7 +161,7 @@ async function main(): Promise<void> {
: undefined;
// Console reads the watcher's connection state from this file, with the
// rest of the host's watcher state, rather than from the control port.
- const stopRecording = recordWatcherHealth(resolve(root), control);
+ const stopRecording = recordWatcherHealth(resolve(root), control, bundle);
// A watcher that stops (disabled, stood down after a takeover, or
// signalled) takes the process with it: the control port and every
// session host go too, so the supervisor sees a clean exit and does not
diff --git a/console/packages/agent-providers/src/host/watcher-health-file.test.ts b/console/packages/agent-providers/src/host/watcher-health-file.test.ts
index b367ff86c..8a95d5dff 100644
--- a/console/packages/agent-providers/src/host/watcher-health-file.test.ts
+++ b/console/packages/agent-providers/src/host/watcher-health-file.test.ts
@@ -1,8 +1,14 @@
-import { mkdtemp, readdir, readFile, rm } from 'node:fs/promises';
+import { createHash } from 'node:crypto';
+import { mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, expect, it, vi } from 'vitest';
-import { recordWatcherHealth, WATCHER_HEALTH_FILE } from './watcher-health-file';
+import {
+ bundleIdentity,
+ recordWatcherHealth,
+ WATCHER_HEALTH_FILE,
+ watcherHealthFileSchema,
+} from './watcher-health-file';
import { WatcherControl } from './watcher-tools';
const roots: string[] = [];
@@ -18,7 +24,7 @@ it('keeps the watcher’s own account of its connection on the host, as it chang
const root = await mkdtemp(join(tmpdir(), 'watcher-health-'));
roots.push(root);
const control = new WatcherControl();
- const stop = recordWatcherHealth(root, control);
+ const stop = recordWatcherHealth(root, control, 'bundle-sha');
await vi.waitFor(async () =>
expect(await written(root)).toMatchObject({ state: 'not-running', pid: process.pid })
@@ -32,6 +38,7 @@ it('keeps the watcher’s own account of its connection on the host, as it chang
detail: null,
placements: { session: 'room' },
pid: process.pid,
+ bundle: 'bundle-sha',
})
);
// Renamed into place, so nothing half-written is ever left beside it.
@@ -42,3 +49,21 @@ it('keeps the watcher’s own account of its connection on the host, as it chang
await new Promise((resolve) => setTimeout(resolve, 20));
expect((await written(root)).state).toBe('connected');
});
+
+it('names the bundle a process started from by its content, and reads a file that names none', async () => {
+ const root = await mkdtemp(join(tmpdir(), 'watcher-health-'));
+ roots.push(root);
+ const bundle = join(root, 'shared-host.mjs');
+ await writeFile(bundle, 'console.log(1);');
+ expect(await bundleIdentity(bundle)).toBe(
+ createHash('sha256').update('console.log(1);').digest('hex')
+ );
+
+ const control = new WatcherControl();
+ const stop = recordWatcherHealth(root, control, null);
+ await vi.waitFor(async () => expect(await written(root)).toMatchObject({ pid: process.pid }));
+ stop();
+ const parsed = watcherHealthFileSchema.parse(await written(root));
+ expect(parsed.bundle).toBeUndefined();
+ expect(parsed.busy).toBeUndefined();
+});
diff --git a/console/packages/agent-providers/src/host/watcher-health-file.ts b/console/packages/agent-providers/src/host/watcher-health-file.ts
index 0d1bcb8cd..233d53706 100644
--- a/console/packages/agent-providers/src/host/watcher-health-file.ts
+++ b/console/packages/agent-providers/src/host/watcher-health-file.ts
@@ -1,5 +1,5 @@
-import { randomUUID } from 'node:crypto';
-import { writeFile } from 'node:fs/promises';
+import { createHash, randomUUID } from 'node:crypto';
+import { readFile, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { z } from 'zod';
import { fileMode } from './host-permissions';
@@ -26,17 +26,39 @@ export const WATCHER_HEALTH_FILE = 'health.json';
export const watcherHealthFileSchema = watcherHealthSchema.extend({
pid: z.number().int().positive(),
updatedAt: z.string(),
+ /**
+ * The `bundleIdentity` of the shared-host bundle the writing process was
+ * started from. Absent from a process that does not run from one (an agents
+ * controller's own), and from one too old to say.
+ */
+ bundle: z.string().optional(),
});
export type WatcherHealthFile = z.infer<typeof watcherHealthFileSchema>;
+/**
+ * What identifies a shared-host bundle: the SHA-256 of the file. Its path does
+ * not, since an install can replace the file in place.
+ */
+export async function bundleIdentity(path: string): Promise<string> {
+ return createHash('sha256')
+ .update(await readFile(path))
+ .digest('hex');
+}
+
/**
* Keep `health.json` in `root` in step with what the watcher reports to
* `control`, from now until the returned function is called. Written whole,
* under a temporary name and renamed into place, so a reader never sees half
* of it; and in order, so an older state never lands after a newer one.
+ * `bundle` is the `bundleIdentity` of the bundle this process started from,
+ * or null when it did not start from one.
*/
-export function recordWatcherHealth(root: string, control: WatcherControl): () => void {
+export function recordWatcherHealth(
+ root: string,
+ control: WatcherControl,
+ bundle: string | null
+): () => void {
const path = join(root, WATCHER_HEALTH_FILE);
let writing: Promise<void> = Promise.resolve();
const write = () => {
@@ -44,6 +66,7 @@ export function recordWatcherHealth(root: string, control: WatcherControl): () =
...control.health(),
pid: process.pid,
updatedAt: new Date().toISOString(),
+ ...(bundle ? { bundle } : {}),
};
writing = writing
.then(async () => {
diff --git a/console/packages/agent-providers/src/host/watcher-tools.ts b/console/packages/agent-providers/src/host/watcher-tools.ts
index 122c4c349..6dc344348 100644
--- a/console/packages/agent-providers/src/host/watcher-tools.ts
+++ b/console/packages/agent-providers/src/host/watcher-tools.ts
@@ -320,6 +320,11 @@ export const watcherHealthSchema = z.object({
since: z.string(),
/** Session id → room id, as the watcher routes now. Empty while it is not running. */
placements: z.record(z.string(), z.string()),
+ /**
+ * The agent's sessions whose host runs here and is mid-turn, or has not yet
+ * said whether it is. Absent from a watcher too old to report it.
+ */
+ busy: z.array(z.string()).optional(),
});
export type WatcherHealth = z.infer<typeof watcherHealthSchema>;
@@ -400,6 +405,7 @@ export class WatcherControl {
state?: WatcherState;
detail?: string | null;
placements?: PlacementMap;
+ busy?: string[];
}): void {
const state = change.state ?? this.current.state;
const same = state === this.current.state;
@@ -408,11 +414,17 @@ export class WatcherControl {
detail: change.detail !== undefined ? change.detail : same ? this.current.detail : null,
since: same ? this.current.since : new Date().toISOString(),
placements: change.placements ? { ...change.placements } : this.current.placements,
+ ...(change.busy
+ ? { busy: [...change.busy].sort() }
+ : this.current.busy
+ ? { busy: this.current.busy }
+ : {}),
};
if (
same &&
next.detail === this.current.detail &&
- samePlacements(next.placements, this.current.placements)
+ samePlacements(next.placements, this.current.placements) &&
+ sameList(next.busy, this.current.busy)
)
return;
this.current = next;
@@ -426,6 +438,11 @@ export class WatcherControl {
}
}
+function sameList(a: string[] | undefined, b: string[] | undefined): boolean {
+ if (!a || !b) return a === b;
+ return a.length === b.length && a.every((value, index) => value === b[index]);
+}
+
function samePlacements(a: PlacementMap, b: PlacementMap): boolean {
const keys = Object.keys(a);
return keys.length === Object.keys(b).length && keys.every((key) => a[key] === b[key]);
diff --git a/console/packages/agent-providers/src/index.ts b/console/packages/agent-providers/src/index.ts
index 656bec3e2..b2a2b8125 100644
--- a/console/packages/agent-providers/src/index.ts
+++ b/console/packages/agent-providers/src/index.ts
@@ -94,6 +94,7 @@ export {
} from './host/hub-stream';
export { clearTakenOver, readTakenOver, type TakenOver } from './host/taken-over';
export {
+ bundleIdentity,
recordWatcherHealth,
WATCHER_HEALTH_FILE,
type WatcherHealthFile,
diff --git a/docs/design/controller-contract-v1.md b/docs/design/controller-contract-v1.md
index ad4c64ae8..895df03a9 100644
--- a/docs/design/controller-contract-v1.md
+++ b/docs/design/controller-contract-v1.md
@@ -136,7 +136,8 @@ type ControllerRelease = {
`_URL`, `_SHA256`; Helm `switchCore.controllerRelease`), and the chart from a
Switch release names that release's controller. A controller running another
version downloads the package, refuses it unless its SHA-256 matches, and installs
-it and restarts once no agent session is running. It says how that is going in
+it and restarts once no agent session is in the middle of a turn (idle sessions
+are restarted with it and resume their conversations). It says how that is going in
its report's `update` (`{ target, state, problem? }`), which Core stores with the
rest of the report. Rolling back is pointing the setting at the previous
release's package. A controller older than the field ignores it. A Switch Console
@@ -160,7 +161,7 @@ type StatusReport = {
agents: AgentStatus[]
update?: { // why it does not run controller_release yet. Absent when it does, and from older controllers
target: string // the version it is moving to
- state: "waiting_idle" | "failed" | "restart_needed"
+ state: "waiting_idle" | "failed" | "restart_needed" // waiting_idle: a session on the machine is in the middle of a turn
problem?: string // human-readable, when it failed
}
}