Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Stop the agents controller retrying refusals that cannot pass (#714)
Abel Dantas6cbaee154e42
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bef8decb5..2b6907799 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1382,6 +1382,13 @@ version of their own to them without also giving them a release of their own.### [Unreleased]#### Fixed+- **A machine whose controller Switch refuses for good says so instead of+ restarting it forever.** When the server needs a newer agents controller than+ Console carries, "This computer as a machine" shows that Console needs an+ update and stops restarting the controller. When the server no longer knows+ its credential, Console forgets it as it does for a removed machine, and+ enrolls the machine again on the next check. SSH hosts set up by Console+ stop restarting on either and enroll again on the second.- **A Claude Code session is no longer parked while its background subagentsare still working.** The idle timer only looked at turns, so a session whoseturn had ended with subagents still running in the background was stopped@@ -3046,6 +3053,16 @@ tags; see RELEASING.md.### [Unreleased]+#### Fixed+- **A refusal that cannot pass stops the controller instead of retrying it+ forever.** The server answers `protocol_unsupported` when it no longer speaks+ the controller's protocol and `invalid_credential` when it knows no controller+ by its credential, and marks both not retryable. The controller retried them+ every few seconds for as long as it ran. It now stops its agents and exits+ with a new code, `5` (update the controller) or `6` (enroll the machine+ again), and the systemd unit and launchd job it installs do not restart it on+ either.+### [0.1.1] - 2026-10-07#### Fixeddiff --git a/console/apps/switch-console-desktop/src/main/core/agent-migration/agent-migration.ts b/console/apps/switch-console-desktop/src/main/core/agent-migration/agent-migration.tsindex 25f07e34e..208bbec67 100644--- a/console/apps/switch-console-desktop/src/main/core/agent-migration/agent-migration.ts+++ b/console/apps/switch-console-desktop/src/main/core/agent-migration/agent-migration.ts@@ -218,7 +218,7 @@ async function resolveSshHost(const probed = overview.process;const revoked =(overview.remote?.kind === 'ok' && overview.remote.controller?.state === 'revoked') ||- (probed?.kind === 'stopped' && probed.code === 3);+ (probed?.kind === 'stopped' && (probed.code === 3 || probed.code === 6));const controller: TargetLookup['controller'] = overview.enrollment? {controllerId: overview.enrollment.controllerId,@@ -653,7 +653,7 @@ async function healMachine(agent: MigrationAgent): Promise<MachineHealth> {const gone =(remote?.kind === 'ok' &&(remote.controller === null || remote.controller.state === 'revoked')) ||- (running?.kind === 'stopped' && running.code === 3);+ (running?.kind === 'stopped' && (running.code === 3 || running.code === 6));if (gone) {log.warn('Switch revoked or forgot an SSH host’s controller; enrolling it again', {event: 'agent_migration',@@ -702,6 +702,10 @@ async function healMachine(agent: MigrationAgent): Promise<MachineHealth> {throw new Error('Another copy of this computer’s controller connected to Switch and took over, so this one is not started again.');+ if (overview.phase.kind === 'update_required')+ throw new Error(+ 'This Switch server needs a newer agents controller than this Console carries. Update Switch Console, then try again.'+ );if (overview.phase.kind !== 'running' && overview.phase.kind !== 'restarting') {await embeddedControllerService.restart(serverId);return { kind: 'changed' };diff --git a/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.test.ts b/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.test.tsindex b72997722..cab10686e 100644--- a/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.test.ts@@ -66,10 +66,12 @@ describe('ControllerSupervisor', () => {await supervisor.stop(1_000);});- it('hands revocation and takeover over without restarting', async () => {+ it('hands revocation, takeover and refusals that cannot pass over without restarting', async () => {for (const [code, exit] of [[3, 'revoked'],[4, 'taken_over'],+ [5, 'upgrade_required'],+ [6, 'credential_invalid'],] as const) {const { supervisor, calls, finals } = harness();supervisor.start();diff --git a/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.ts b/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.tsindex 9c3e69d94..80eb8f2b9 100644--- a/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.ts+++ b/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.ts@@ -6,6 +6,15 @@ import type { EmbeddedControllerPhase } from '@shared/core/embedded-controller/eexport const EXIT_CONFIGURATION = 2;export const EXIT_REVOKED = 3;export const EXIT_TAKEN_OVER = 4;+export const EXIT_UPGRADE_REQUIRED = 5;+export const EXIT_CREDENTIAL_INVALID = 6;++/** How the controller ended when it must not be started again as it is. */+export type ControllerFinalExit =+ | 'revoked'+ | 'taken_over'+ | 'upgrade_required'+ | 'credential_invalid';/** The part of a spawned `ChildProcess` the supervisor uses. */export interface ControllerChild {@@ -52,8 +61,12 @@ export type SupervisorDeps = {spawn: SpawnController;launch: () => Promise<ControllerLaunch>;onPhase: (phase: EmbeddedControllerPhase) => void;- /** It exited in a way that must not be retried: the server revoked it, or another copy took over. */- onFinal: (exit: 'revoked' | 'taken_over') => void;+ /**+ * It exited in a way that must not be retried: the server revoked it,+ * another copy took over, the server needs a newer controller, or it knows+ * no controller by this credential.+ */+ onFinal: (exit: ControllerFinalExit) => void;/** One line the controller wrote, at the level it wrote it at. */onLine: (level: ControllerLogLevel, line: string) => void;now: () => number;@@ -198,6 +211,8 @@ export class ControllerSupervisor {return;}if (code === EXIT_REVOKED) return this.deps.onFinal('revoked');+ if (code === EXIT_UPGRADE_REQUIRED) return this.deps.onFinal('upgrade_required');+ if (code === EXIT_CREDENTIAL_INVALID) return this.deps.onFinal('credential_invalid');if (code === EXIT_TAKEN_OVER) return this.deps.onFinal('taken_over');if (code === EXIT_CONFIGURATION) {this.deps.onPhase({diff --git a/console/apps/switch-console-desktop/src/main/core/embedded-controller/embedded-controller-service.ts b/console/apps/switch-console-desktop/src/main/core/embedded-controller/embedded-controller-service.tsindex 6b7849b4c..69c5b7a7f 100644--- a/console/apps/switch-console-desktop/src/main/core/embedded-controller/embedded-controller-service.ts+++ b/console/apps/switch-console-desktop/src/main/core/embedded-controller/embedded-controller-service.ts@@ -14,6 +14,7 @@ import {} from './controller-files';import {type ControllerBackoff,+ type ControllerFinalExit,type ControllerLaunch,type ControllerLogLevel,ControllerSupervisor,@@ -495,7 +496,18 @@ export class EmbeddedControllerService {return moved;}- private async onFinal(serverId: string, exit: 'revoked' | 'taken_over'): Promise<void> {+ private async onFinal(serverId: string, exit: ControllerFinalExit): Promise<void> {+ if (exit === 'upgrade_required') {+ this.deps.log.error(+ 'Switch needs a newer agents controller than this Console carries; not restarting it until Console is updated',+ { serverId }+ );+ this.setRunnerPhase(serverId, {+ kind: 'update_required',+ at: new Date(this.deps.now()).toISOString(),+ });+ return;+ }if (exit === 'taken_over') {this.deps.log.error('Another copy of this computer’s controller connected to Switch and took over; not restarting it',@@ -510,9 +522,12 @@ export class EmbeddedControllerService {if (this.busy.has(serverId)) return;const record = await this.deps.records.get(serverId);const at = new Date(this.deps.now()).toISOString();- this.deps.log.warn('Switch removed this computer as a machine; forgetting its credential', {- serverId,- });+ this.deps.log.warn(+ exit === 'credential_invalid'+ ? 'Switch no longer knows this computer’s controller credential; forgetting it'+ : 'Switch removed this computer as a machine; forgetting its credential',+ { serverId }+ );this.runners.delete(serverId);try {await this.deps.secrets.deleteSecret(credentialSecretKey(serverId));diff --git a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-controller-service.test.ts b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-controller-service.test.tsindex e8987108a..89c5ef468 100644--- a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-controller-service.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-controller-service.test.ts@@ -192,7 +192,7 @@ describe('making an SSH host a machine', () => {expect(unit).toContain(`"/home/ada/.local/state/switch/agent-controller/console-${SERVER}"`);expect(unit).toContain('--shared-host-bundle');expect(unit).toContain('Restart=on-failure');- expect(unit).toContain('RestartPreventExitStatus=2 3 4');+ expect(unit).toContain('RestartPreventExitStatus=2 3 4 5 6');expect(unit).toContain('"PATH=/home/ada/.local/bin:/usr/bin"');expect(records.get(`${HOST}|${SERVER}`)).toMatchObject({controllerId: 'controller-7',diff --git a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.test.ts b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.test.tsindex bd9bac693..54d1b59a4 100644--- a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.test.ts@@ -286,6 +286,6 @@ describe('the systemd unit', () => {);expect(unit).toContain('Environment="PATH=/home/ada/.local/bin:/usr/bin"');expect(unit).toContain('Restart=on-failure');- expect(unit).toContain('RestartPreventExitStatus=2 3 4');+ expect(unit).toContain('RestartPreventExitStatus=2 3 4 5 6');});});diff --git a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.ts b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.tsindex 7faafb406..2a2e6386f 100644--- a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.ts+++ b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.ts@@ -111,7 +111,8 @@ export const enrollResultSchema = z.discriminatedUnion('ok', [/*** Keeps the controller running under the supervisor: started again on exit* code 1 with a backoff, and left stopped on 0 (stopped), 2 (configuration),- * 3 (revoked) and 4 (taken over), as the controller's README asks of any+ * 3 (revoked), 4 (taken over), 5 (needs a newer controller) and 6 (credential+ * not known), as the controller's README asks of any* supervisor. Records itself in `console-supervisor.json` in the data* directory. A run that lasted ten minutes resets the backoff.*@@ -134,7 +135,7 @@ const start = () => {child.on('exit', (code, signal) => {child = null;if (stopping) { record({ state: 'stopped' }); process.exit(0); }- if ([0, 2, 3, 4].includes(code)) { record({ state: 'exited', code: code }); process.exit(0); }+ if ([0, 2, 3, 4, 5, 6].includes(code)) { record({ state: 'exited', code: code }); process.exit(0); }if (Date.now() - started > 600000) attempt = 0;attempt += 1;const wait = Math.min(5000 * Math.pow(2, Math.min(attempt - 1, 4)), 60000);@@ -293,7 +294,7 @@ export function systemdUnit(input: {'Environment=SWITCH_CONTROLLER_LOG_LEVEL=info','Restart=on-failure','RestartSec=5',- 'RestartPreventExitStatus=2 3 4',+ 'RestartPreventExitStatus=2 3 4 5 6','','[Install]','WantedBy=default.target',diff --git a/console/apps/switch-console-desktop/src/shared/core/embedded-controller/embedded-controller.ts b/console/apps/switch-console-desktop/src/shared/core/embedded-controller/embedded-controller.tsindex 57f6b4b96..45bc28dc9 100644--- a/console/apps/switch-console-desktop/src/shared/core/embedded-controller/embedded-controller.ts+++ b/console/apps/switch-console-desktop/src/shared/core/embedded-controller/embedded-controller.ts@@ -18,6 +18,8 @@ export type EmbeddedControllerPhase =| { kind: 'removed'; at: string }/** Another copy of this controller connected and took over; this one is not restarted. */| { kind: 'taken_over'; at: string }+ /** The server needs a newer controller than this Console carries; not restarted until Console is updated. */+ | { kind: 'update_required'; at: string }| { kind: 'error'; message: string };export type EmbeddedControllerEnrollment = {diff --git a/console/packages/agent-controller/README.md b/console/packages/agent-controller/README.mdindex 2163103cf..9a198fdad 100644--- a/console/packages/agent-controller/README.md+++ b/console/packages/agent-controller/README.md@@ -111,6 +111,8 @@ switch-agent-controller update [--check]| `2` | A configuration error that starting again unchanged cannot fix. | No, not until the configuration changes || `3` | The server revoked this controller. | No: it must be enrolled again || `4` | Another instance of this controller, with the same identity, opened the controller stream after this one, which took it over. | No |+ | `5` | The server does not speak this controller's protocol (`protocol_unsupported`). | No: update the controller |+ | `6` | The server knows no controller by this credential (`invalid_credential`): it was replaced, or its key was deleted. | No: enroll again |Code `2` covers: an unknown command, option or missing argument; a serverURL that is not one, or is plain `http` to a host that is not loopback; a@@ -206,7 +208,9 @@ To run it under another supervisor, have it run `run` and restart it on exit cod`1` only. Do not restart it on `2`, because it would fail the same way until itsconfiguration is fixed, nor on `3`, because a revoked controller must be enrolledagain, nor on `4`, because two instances would take the stream from each other in-turn. With systemd, `Restart=on-failure` and `RestartPreventExitStatus=2 3 4`.+turn, nor on `5` or `6`, because the server has said it will refuse this controller+the same way again. With systemd, `Restart=on-failure` and+`RestartPreventExitStatus=2 3 4 5 6`.## Run by a parent processdiff --git a/console/packages/agent-controller/src/api.ts b/console/packages/agent-controller/src/api.tsindex f3852b09f..dd1ca59b1 100644--- a/console/packages/agent-controller/src/api.ts+++ b/console/packages/agent-controller/src/api.ts@@ -91,6 +91,23 @@ export function isTakenOver(error: unknown): boolean {return error instanceof ControllerApiError && error.code === 'taken_over';}+/**+ * The server does not speak this controller's protocol. Retrying cannot change+ * that: only a newer controller can, so this one stops and says so.+ */+export function isUpgradeRequired(error: unknown): boolean {+ return error instanceof ControllerApiError && error.code === 'protocol_unsupported';+}++/**+ * The server knows no controller by this credential: it was replaced, or the+ * key behind it was deleted. Retrying the same credential cannot pass, so this+ * one stops; the machine has to be enrolled again.+ */+export function isCredentialRefused(error: unknown): boolean {+ return error instanceof ControllerApiError && error.code === 'invalid_credential';+}+/*** The agent bridge URL as given to `enroll`, checked and without a trailing* slash. The controller's relay forwards every agent's calls to it with thediff --git a/console/packages/agent-controller/src/controller.ts b/console/packages/agent-controller/src/controller.tsindex 1c9e69bff..edf830ef5 100644--- a/console/packages/agent-controller/src/controller.ts+++ b/console/packages/agent-controller/src/controller.ts@@ -1,6 +1,14 @@import type { OpenAgentStream } from '@switch-console/agent-providers';import { AgentHub } from './agent-hub';-import { AccessTokens, ControllerClient, type Fetch, isRevoked, type OpenWebSocket } from './api';+import {+ AccessTokens,+ ControllerClient,+ type Fetch,+ isCredentialRefused,+ isRevoked,+ isUpgradeRequired,+ type OpenWebSocket,+} from './api';import { ConfigurationError } from './errors';import { HubSocket } from './hub-socket';import { errorMessage, type Logger } from './log';@@ -85,8 +93,18 @@ export type ControllerDeps = {* `taken_over`: another instance of this controller opened the controller* stream after this one. Restarting would take it straight back, so this one* ends instead.+ *+ * `upgrade_required` and `credential_invalid`: the server refused this+ * controller's protocol, or knows no controller by its credential. Both are+ * refusals the server marks not retryable, and asking again cannot change+ * them, so this one ends rather than retrying forever.*/-export type ControllerExit = 'stopped' | 'revoked' | 'taken_over';+export type ControllerExit =+ | 'stopped'+ | 'revoked'+ | 'taken_over'+ | 'upgrade_required'+ | 'credential_invalid';/*** Makes an agent's credentials file name the relay, its hub, and a token the@@ -295,11 +313,20 @@ export async function runController(return revocation;};+ /** Set when a refusal that cannot pass ends the run. */+ let refused: 'upgrade_required' | 'credential_invalid' | null = null;+ const refuse = (ending: 'upgrade_required' | 'credential_invalid') => {+ refused ??= ending;+ stop.abort();+ };+const failed = (what: string, error: unknown) => {if (isRevoked(error)) {void revoke();return;}+ if (isUpgradeRequired(error)) return refuse('upgrade_required');+ if (isCredentialRefused(error)) return refuse('credential_invalid');log.warn(`${what} failed; it is retried on the next nudge, reconnect or resync.`, {error: errorMessage(error),});@@ -538,6 +565,15 @@ export async function runController(log.error('Another instance of this controller took over its connection to Switch; this one stops its agents and exits.');+ if (ending === 'upgrade_required' || ending === 'credential_invalid') refused ??= ending;+ if (refused === 'upgrade_required')+ log.error(+ 'Switch does not speak this controller’s protocol; it stops its agents and exits. Update the controller.'+ );+ if (refused === 'credential_invalid')+ log.error(+ 'Switch knows no controller by this credential; it stops its agents and exits. Enroll this machine again.'+ );} finally {clearInterval(resync);clearInterval(poll);@@ -552,7 +588,7 @@ export async function runController(await relay.close();}}- return revocation ? 'revoked' : ending;+ return revocation ? 'revoked' : (refused ?? ending);}/**diff --git a/console/packages/agent-controller/src/exit-codes.ts b/console/packages/agent-controller/src/exit-codes.tsindex 8dd94c598..7fcc77fdc 100644--- a/console/packages/agent-controller/src/exit-codes.ts+++ b/console/packages/agent-controller/src/exit-codes.ts@@ -10,6 +10,10 @@ export const EXIT_CONFIGURATION = 2;export const EXIT_REVOKED = 3;/** Another instance of this controller took its stream over. */export const EXIT_TAKEN_OVER = 4;+/** The server does not speak this controller's protocol. Update the controller. */+export const EXIT_UPGRADE_REQUIRED = 5;+/** The server knows no controller by this credential. It has to be enrolled again. */+export const EXIT_CREDENTIAL_INVALID = 6;/*** The exit code a command that threw `error` ends with: configuration errors,diff --git a/console/packages/agent-controller/src/main.ts b/console/packages/agent-controller/src/main.tsindex 6383a44cb..39125c7e1 100644--- a/console/packages/agent-controller/src/main.ts+++ b/console/packages/agent-controller/src/main.ts@@ -13,7 +13,7 @@ import {normalizeServerUrl,nodeWebSocket,} from './api';-import { DEFAULT_TIMING, runController } from './controller';+import { type ControllerExit, DEFAULT_TIMING, runController } from './controller';import { DetachedRuntime } from './detached-runtime';import { formatChecks, runDoctor } from './doctor';import { readEnvFile } from './env-file';@@ -23,6 +23,8 @@ import {EXIT_OK,EXIT_REVOKED,EXIT_TAKEN_OVER,+ EXIT_UPGRADE_REQUIRED,+ EXIT_CREDENTIAL_INVALID,exitCodeFor,isParseArgsError,} from './exit-codes';@@ -122,9 +124,18 @@ The shared host bundle defaults to SWITCH_CONTROLLER_SHARED_HOST_BUNDLE, thenthe one built in the workspace.Log level: SWITCH_CONTROLLER_LOG_LEVEL (debug, info, warn, error; default info).Exit codes: 0 stopped, 1 error that may pass, 2 configuration error,-3 revoked, 4 taken over by another instance.+3 revoked, 4 taken over by another instance, 5 the server needs a newer+controller, 6 the server knows no controller by this credential.`;+const EXIT_CODE_FOR: Record<ControllerExit, number> = {+ stopped: EXIT_OK,+ revoked: EXIT_REVOKED,+ taken_over: EXIT_TAKEN_OVER,+ upgrade_required: EXIT_UPGRADE_REQUIRED,+ credential_invalid: EXIT_CREDENTIAL_INVALID,+};+function bundlePath(flag: string | undefined): string {return resolveSharedHostBundle(flag, process.env, defaultSharedHostBundle);}@@ -386,7 +397,7 @@ async function runCommand(args: string[]): Promise<number> {},stop.signal);- return exit === 'revoked' ? EXIT_REVOKED : exit === 'taken_over' ? EXIT_TAKEN_OVER : EXIT_OK;+ return EXIT_CODE_FOR[exit];} finally {for (const signal of SIGNALS) process.off(signal, onSignal);store.close();diff --git a/console/packages/agent-controller/src/service.test.ts b/console/packages/agent-controller/src/service.test.tsindex 8cc78c528..a6f067082 100644--- a/console/packages/agent-controller/src/service.test.ts+++ b/console/packages/agent-controller/src/service.test.ts@@ -20,7 +20,7 @@ describe('systemdUnit', () => {);expect(unit).toContain('Environment="PATH=/home/me/.local/bin:/usr/bin:/bin"');expect(unit).toContain('Restart=on-failure');- expect(unit).toContain('RestartPreventExitStatus=2 3 4');+ expect(unit).toContain('RestartPreventExitStatus=2 3 4 5 6');expect(unit).not.toContain('--launchd');});});diff --git a/console/packages/agent-controller/src/service.ts b/console/packages/agent-controller/src/service.tsindex e64cb1af9..783db42bd 100644--- a/console/packages/agent-controller/src/service.ts+++ b/console/packages/agent-controller/src/service.ts@@ -75,7 +75,7 @@ export function systemdUnit(spec: ServiceSpec): string {`Environment=${quote(`PATH=${spec.path}`)}`,'Restart=on-failure','RestartSec=5',- 'RestartPreventExitStatus=2 3 4',+ 'RestartPreventExitStatus=2 3 4 5 6','','[Install]','WantedBy=default.target',@@ -134,7 +134,7 @@ ${strings(runArguments(spec, true))}}/** The exit codes `--launchd` turns into 0, so launchd does not restart into the same failure. */-export const LAUNCHD_FINAL_EXIT_CODES: readonly number[] = [2, 3, 4];+export const LAUNCHD_FINAL_EXIT_CODES: readonly number[] = [2, 3, 4, 5, 6];/** Where the service's files are, for this user. */export type ServicePaths = {diff --git a/console/packages/agent-controller/src/stream.test.ts b/console/packages/agent-controller/src/stream.test.tsindex 44329133e..74f4cc36d 100644--- a/console/packages/agent-controller/src/stream.test.ts+++ b/console/packages/agent-controller/src/stream.test.ts@@ -315,6 +315,32 @@ describe('runControllerStream', () => {expect(refusedAttach.staleTokens).toBe(0);});+ it('stops for good when the server refuses its protocol, on open and on attach', async () => {+ const refusedOpen = new ScriptedClient();+ let opens = 0;+ refusedOpen.open = async () => {+ opens++;+ throw new ControllerApiError(426, 'protocol_unsupported', 'protocol 1', false, null);+ };+ expect(await run(refusedOpen, new AbortController().signal).ending).toBe('upgrade_required');+ expect(opens).toBe(1);++ const refusedAttach = new ScriptedClient();+ refusedAttach.sockets = () => scriptedSocket([refused(426, 'protocol_unsupported')], 4426);+ expect(await run(refusedAttach, new AbortController().signal).ending).toBe('upgrade_required');+ });++ it('stops for good when the server knows no controller by its credential', async () => {+ const scripted = new ScriptedClient();+ let opens = 0;+ scripted.open = async () => {+ opens++;+ throw new ControllerApiError(401, 'invalid_credential', 'not valid', false, null);+ };+ expect(await run(scripted, new AbortController().signal).ending).toBe('credential_invalid');+ expect(opens).toBe(1);+ });+it('drops a token the socket refused as stale, and attaches again', async () => {const scripted = new ScriptedClient();scripted.sockets = () =>diff --git a/console/packages/agent-controller/src/stream.ts b/console/packages/agent-controller/src/stream.tsindex 0285e15ab..531ce24e7 100644--- a/console/packages/agent-controller/src/stream.ts+++ b/console/packages/agent-controller/src/stream.ts@@ -3,8 +3,10 @@ import type { z } from 'zod';import {ControllerApiError,type ControllerClient,+ isCredentialRefused,isRevoked,isTakenOver,+ isUpgradeRequired,type OpenedSocket,} from './api';import { errorMessage, type Logger } from './log';@@ -66,7 +68,12 @@ export type ControllerFrame =| { type: 'credential.revoked'; data: Record<string, never> };/** Why the stream stopped for good. */-export type StreamEnding = 'stopped' | 'revoked' | 'taken_over';+export type StreamEnding =+ | 'stopped'+ | 'revoked'+ | 'taken_over'+ | 'upgrade_required'+ | 'credential_invalid';export type ControllerStreamOptions = {client: Pick<ControllerClient, 'openConnection' | 'openSocket'>;@@ -151,8 +158,11 @@ type Message = { event: string; data: unknown };* Ends with `'revoked'` on `credential.revoked` or a refusal saying* `controller_revoked`, with `'taken_over'` when another instance of this* controller took the connection over (an `evicted` frame or a refusal saying- * so: reopening would take it straight back, so this one stops), and with- * `'stopped'` when `signal` fires. Any other `evicted` opens a new connection.+ * so: reopening would take it straight back, so this one stops), with+ * `'upgrade_required'` when the server refuses this controller's protocol and+ * `'credential_invalid'` when it knows no controller by this credential (the+ * server marks both not retryable, and retrying them used to spin forever),+ * and with `'stopped'` when `signal` fires. Any other `evicted` opens a new connection.*/export function runControllerStream(options: ControllerStreamOptions): Promise<StreamEnding> {return new ControllerStream(options).run();@@ -184,6 +194,8 @@ class ControllerStream {if (!attempt.ending && !signal.aborted) {if (isRevoked(error)) attempt.ending = 'revoked';else if (isTakenOver(error)) attempt.ending = 'taken_over';+ else if (isUpgradeRequired(error)) attempt.ending = 'upgrade_required';+ else if (isCredentialRefused(error)) attempt.ending = 'credential_invalid';else {const reason = attempt.abort.signal.reason;failure = errorMessage(reason instanceof IdleTimeout ? reason : error);