Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Stop the agents controller retrying refusals that cannot pass (#714)

Abel Dantas6cbaee154e42
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bef8decb5..2b6907799 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1382,6 +1382,13 @@ version of their own to them without also giving them a release of their own.
### [Unreleased]
#### Fixed
+- **A machine whose controller Switch refuses for good says so instead of
+ restarting it forever.** When the server needs a newer agents controller than
+ Console carries, "This computer as a machine" shows that Console needs an
+ update and stops restarting the controller. When the server no longer knows
+ its credential, Console forgets it as it does for a removed machine, and
+ enrolls the machine again on the next check. SSH hosts set up by Console
+ stop restarting on either and enroll again on the second.
- **A Claude Code session is no longer parked while its background subagents
are still working.** The idle timer only looked at turns, so a session whose
turn had ended with subagents still running in the background was stopped
@@ -3046,6 +3053,16 @@ tags; see RELEASING.md.
### [Unreleased]
+#### Fixed
+- **A refusal that cannot pass stops the controller instead of retrying it
+ forever.** The server answers `protocol_unsupported` when it no longer speaks
+ the controller's protocol and `invalid_credential` when it knows no controller
+ by its credential, and marks both not retryable. The controller retried them
+ every few seconds for as long as it ran. It now stops its agents and exits
+ with a new code, `5` (update the controller) or `6` (enroll the machine
+ again), and the systemd unit and launchd job it installs do not restart it on
+ either.
+
### [0.1.1] - 2026-10-07
#### Fixed
diff --git a/console/apps/switch-console-desktop/src/main/core/agent-migration/agent-migration.ts b/console/apps/switch-console-desktop/src/main/core/agent-migration/agent-migration.ts
index 25f07e34e..208bbec67 100644
--- a/console/apps/switch-console-desktop/src/main/core/agent-migration/agent-migration.ts
+++ b/console/apps/switch-console-desktop/src/main/core/agent-migration/agent-migration.ts
@@ -218,7 +218,7 @@ async function resolveSshHost(
const probed = overview.process;
const revoked =
(overview.remote?.kind === 'ok' && overview.remote.controller?.state === 'revoked') ||
- (probed?.kind === 'stopped' && probed.code === 3);
+ (probed?.kind === 'stopped' && (probed.code === 3 || probed.code === 6));
const controller: TargetLookup['controller'] = overview.enrollment
? {
controllerId: overview.enrollment.controllerId,
@@ -653,7 +653,7 @@ async function healMachine(agent: MigrationAgent): Promise<MachineHealth> {
const gone =
(remote?.kind === 'ok' &&
(remote.controller === null || remote.controller.state === 'revoked')) ||
- (running?.kind === 'stopped' && running.code === 3);
+ (running?.kind === 'stopped' && (running.code === 3 || running.code === 6));
if (gone) {
log.warn('Switch revoked or forgot an SSH host’s controller; enrolling it again', {
event: 'agent_migration',
@@ -702,6 +702,10 @@ async function healMachine(agent: MigrationAgent): Promise<MachineHealth> {
throw new Error(
'Another copy of this computer’s controller connected to Switch and took over, so this one is not started again.'
);
+ if (overview.phase.kind === 'update_required')
+ throw new Error(
+ 'This Switch server needs a newer agents controller than this Console carries. Update Switch Console, then try again.'
+ );
if (overview.phase.kind !== 'running' && overview.phase.kind !== 'restarting') {
await embeddedControllerService.restart(serverId);
return { kind: 'changed' };
diff --git a/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.test.ts b/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.test.ts
index b72997722..cab10686e 100644
--- a/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.test.ts
@@ -66,10 +66,12 @@ describe('ControllerSupervisor', () => {
await supervisor.stop(1_000);
});
- it('hands revocation and takeover over without restarting', async () => {
+ it('hands revocation, takeover and refusals that cannot pass over without restarting', async () => {
for (const [code, exit] of [
[3, 'revoked'],
[4, 'taken_over'],
+ [5, 'upgrade_required'],
+ [6, 'credential_invalid'],
] as const) {
const { supervisor, calls, finals } = harness();
supervisor.start();
diff --git a/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.ts b/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.ts
index 9c3e69d94..80eb8f2b9 100644
--- a/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.ts
+++ b/console/apps/switch-console-desktop/src/main/core/embedded-controller/controller-supervisor.ts
@@ -6,6 +6,15 @@ import type { EmbeddedControllerPhase } from '@shared/core/embedded-controller/e
export const EXIT_CONFIGURATION = 2;
export const EXIT_REVOKED = 3;
export const EXIT_TAKEN_OVER = 4;
+export const EXIT_UPGRADE_REQUIRED = 5;
+export const EXIT_CREDENTIAL_INVALID = 6;
+
+/** How the controller ended when it must not be started again as it is. */
+export type ControllerFinalExit =
+ | 'revoked'
+ | 'taken_over'
+ | 'upgrade_required'
+ | 'credential_invalid';
/** The part of a spawned `ChildProcess` the supervisor uses. */
export interface ControllerChild {
@@ -52,8 +61,12 @@ export type SupervisorDeps = {
spawn: SpawnController;
launch: () => Promise<ControllerLaunch>;
onPhase: (phase: EmbeddedControllerPhase) => void;
- /** It exited in a way that must not be retried: the server revoked it, or another copy took over. */
- onFinal: (exit: 'revoked' | 'taken_over') => void;
+ /**
+ * It exited in a way that must not be retried: the server revoked it,
+ * another copy took over, the server needs a newer controller, or it knows
+ * no controller by this credential.
+ */
+ onFinal: (exit: ControllerFinalExit) => void;
/** One line the controller wrote, at the level it wrote it at. */
onLine: (level: ControllerLogLevel, line: string) => void;
now: () => number;
@@ -198,6 +211,8 @@ export class ControllerSupervisor {
return;
}
if (code === EXIT_REVOKED) return this.deps.onFinal('revoked');
+ if (code === EXIT_UPGRADE_REQUIRED) return this.deps.onFinal('upgrade_required');
+ if (code === EXIT_CREDENTIAL_INVALID) return this.deps.onFinal('credential_invalid');
if (code === EXIT_TAKEN_OVER) return this.deps.onFinal('taken_over');
if (code === EXIT_CONFIGURATION) {
this.deps.onPhase({
diff --git a/console/apps/switch-console-desktop/src/main/core/embedded-controller/embedded-controller-service.ts b/console/apps/switch-console-desktop/src/main/core/embedded-controller/embedded-controller-service.ts
index 6b7849b4c..69c5b7a7f 100644
--- a/console/apps/switch-console-desktop/src/main/core/embedded-controller/embedded-controller-service.ts
+++ b/console/apps/switch-console-desktop/src/main/core/embedded-controller/embedded-controller-service.ts
@@ -14,6 +14,7 @@ import {
} from './controller-files';
import {
type ControllerBackoff,
+ type ControllerFinalExit,
type ControllerLaunch,
type ControllerLogLevel,
ControllerSupervisor,
@@ -495,7 +496,18 @@ export class EmbeddedControllerService {
return moved;
}
- private async onFinal(serverId: string, exit: 'revoked' | 'taken_over'): Promise<void> {
+ private async onFinal(serverId: string, exit: ControllerFinalExit): Promise<void> {
+ if (exit === 'upgrade_required') {
+ this.deps.log.error(
+ 'Switch needs a newer agents controller than this Console carries; not restarting it until Console is updated',
+ { serverId }
+ );
+ this.setRunnerPhase(serverId, {
+ kind: 'update_required',
+ at: new Date(this.deps.now()).toISOString(),
+ });
+ return;
+ }
if (exit === 'taken_over') {
this.deps.log.error(
'Another copy of this computer’s controller connected to Switch and took over; not restarting it',
@@ -510,9 +522,12 @@ export class EmbeddedControllerService {
if (this.busy.has(serverId)) return;
const record = await this.deps.records.get(serverId);
const at = new Date(this.deps.now()).toISOString();
- this.deps.log.warn('Switch removed this computer as a machine; forgetting its credential', {
- serverId,
- });
+ this.deps.log.warn(
+ exit === 'credential_invalid'
+ ? 'Switch no longer knows this computer’s controller credential; forgetting it'
+ : 'Switch removed this computer as a machine; forgetting its credential',
+ { serverId }
+ );
this.runners.delete(serverId);
try {
await this.deps.secrets.deleteSecret(credentialSecretKey(serverId));
diff --git a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-controller-service.test.ts b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-controller-service.test.ts
index e8987108a..89c5ef468 100644
--- a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-controller-service.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-controller-service.test.ts
@@ -192,7 +192,7 @@ describe('making an SSH host a machine', () => {
expect(unit).toContain(`"/home/ada/.local/state/switch/agent-controller/console-${SERVER}"`);
expect(unit).toContain('--shared-host-bundle');
expect(unit).toContain('Restart=on-failure');
- expect(unit).toContain('RestartPreventExitStatus=2 3 4');
+ expect(unit).toContain('RestartPreventExitStatus=2 3 4 5 6');
expect(unit).toContain('"PATH=/home/ada/.local/bin:/usr/bin"');
expect(records.get(`${HOST}|${SERVER}`)).toMatchObject({
controllerId: 'controller-7',
diff --git a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.test.ts b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.test.ts
index bd9bac693..54d1b59a4 100644
--- a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.test.ts
@@ -286,6 +286,6 @@ describe('the systemd unit', () => {
);
expect(unit).toContain('Environment="PATH=/home/ada/.local/bin:/usr/bin"');
expect(unit).toContain('Restart=on-failure');
- expect(unit).toContain('RestartPreventExitStatus=2 3 4');
+ expect(unit).toContain('RestartPreventExitStatus=2 3 4 5 6');
});
});
diff --git a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.ts b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.ts
index 7faafb406..2a2e6386f 100644
--- a/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.ts
+++ b/console/apps/switch-console-desktop/src/main/core/host-controllers/host-scripts.ts
@@ -111,7 +111,8 @@ export const enrollResultSchema = z.discriminatedUnion('ok', [
/**
* Keeps the controller running under the supervisor: started again on exit
* code 1 with a backoff, and left stopped on 0 (stopped), 2 (configuration),
- * 3 (revoked) and 4 (taken over), as the controller's README asks of any
+ * 3 (revoked), 4 (taken over), 5 (needs a newer controller) and 6 (credential
+ * not known), as the controller's README asks of any
* supervisor. Records itself in `console-supervisor.json` in the data
* directory. A run that lasted ten minutes resets the backoff.
*
@@ -134,7 +135,7 @@ const start = () => {
child.on('exit', (code, signal) => {
child = null;
if (stopping) { record({ state: 'stopped' }); process.exit(0); }
- if ([0, 2, 3, 4].includes(code)) { record({ state: 'exited', code: code }); process.exit(0); }
+ if ([0, 2, 3, 4, 5, 6].includes(code)) { record({ state: 'exited', code: code }); process.exit(0); }
if (Date.now() - started > 600000) attempt = 0;
attempt += 1;
const wait = Math.min(5000 * Math.pow(2, Math.min(attempt - 1, 4)), 60000);
@@ -293,7 +294,7 @@ export function systemdUnit(input: {
'Environment=SWITCH_CONTROLLER_LOG_LEVEL=info',
'Restart=on-failure',
'RestartSec=5',
- 'RestartPreventExitStatus=2 3 4',
+ 'RestartPreventExitStatus=2 3 4 5 6',
'',
'[Install]',
'WantedBy=default.target',
diff --git a/console/apps/switch-console-desktop/src/shared/core/embedded-controller/embedded-controller.ts b/console/apps/switch-console-desktop/src/shared/core/embedded-controller/embedded-controller.ts
index 57f6b4b96..45bc28dc9 100644
--- a/console/apps/switch-console-desktop/src/shared/core/embedded-controller/embedded-controller.ts
+++ b/console/apps/switch-console-desktop/src/shared/core/embedded-controller/embedded-controller.ts
@@ -18,6 +18,8 @@ export type EmbeddedControllerPhase =
| { kind: 'removed'; at: string }
/** Another copy of this controller connected and took over; this one is not restarted. */
| { kind: 'taken_over'; at: string }
+ /** The server needs a newer controller than this Console carries; not restarted until Console is updated. */
+ | { kind: 'update_required'; at: string }
| { kind: 'error'; message: string };
export type EmbeddedControllerEnrollment = {
diff --git a/console/packages/agent-controller/README.md b/console/packages/agent-controller/README.md
index 2163103cf..9a198fdad 100644
--- a/console/packages/agent-controller/README.md
+++ b/console/packages/agent-controller/README.md
@@ -111,6 +111,8 @@ switch-agent-controller update [--check]
| `2` | A configuration error that starting again unchanged cannot fix. | No, not until the configuration changes |
| `3` | The server revoked this controller. | No: it must be enrolled again |
| `4` | Another instance of this controller, with the same identity, opened the controller stream after this one, which took it over. | No |
+ | `5` | The server does not speak this controller's protocol (`protocol_unsupported`). | No: update the controller |
+ | `6` | The server knows no controller by this credential (`invalid_credential`): it was replaced, or its key was deleted. | No: enroll again |
Code `2` covers: an unknown command, option or missing argument; a server
URL that is not one, or is plain `http` to a host that is not loopback; a
@@ -206,7 +208,9 @@ To run it under another supervisor, have it run `run` and restart it on exit cod
`1` only. Do not restart it on `2`, because it would fail the same way until its
configuration is fixed, nor on `3`, because a revoked controller must be enrolled
again, nor on `4`, because two instances would take the stream from each other in
-turn. With systemd, `Restart=on-failure` and `RestartPreventExitStatus=2 3 4`.
+turn, nor on `5` or `6`, because the server has said it will refuse this controller
+the same way again. With systemd, `Restart=on-failure` and
+`RestartPreventExitStatus=2 3 4 5 6`.
## Run by a parent process
diff --git a/console/packages/agent-controller/src/api.ts b/console/packages/agent-controller/src/api.ts
index f3852b09f..dd1ca59b1 100644
--- a/console/packages/agent-controller/src/api.ts
+++ b/console/packages/agent-controller/src/api.ts
@@ -91,6 +91,23 @@ export function isTakenOver(error: unknown): boolean {
return error instanceof ControllerApiError && error.code === 'taken_over';
}
+/**
+ * The server does not speak this controller's protocol. Retrying cannot change
+ * that: only a newer controller can, so this one stops and says so.
+ */
+export function isUpgradeRequired(error: unknown): boolean {
+ return error instanceof ControllerApiError && error.code === 'protocol_unsupported';
+}
+
+/**
+ * The server knows no controller by this credential: it was replaced, or the
+ * key behind it was deleted. Retrying the same credential cannot pass, so this
+ * one stops; the machine has to be enrolled again.
+ */
+export function isCredentialRefused(error: unknown): boolean {
+ return error instanceof ControllerApiError && error.code === 'invalid_credential';
+}
+
/**
* The agent bridge URL as given to `enroll`, checked and without a trailing
* slash. The controller's relay forwards every agent's calls to it with the
diff --git a/console/packages/agent-controller/src/controller.ts b/console/packages/agent-controller/src/controller.ts
index 1c9e69bff..edf830ef5 100644
--- a/console/packages/agent-controller/src/controller.ts
+++ b/console/packages/agent-controller/src/controller.ts
@@ -1,6 +1,14 @@
import type { OpenAgentStream } from '@switch-console/agent-providers';
import { AgentHub } from './agent-hub';
-import { AccessTokens, ControllerClient, type Fetch, isRevoked, type OpenWebSocket } from './api';
+import {
+ AccessTokens,
+ ControllerClient,
+ type Fetch,
+ isCredentialRefused,
+ isRevoked,
+ isUpgradeRequired,
+ type OpenWebSocket,
+} from './api';
import { ConfigurationError } from './errors';
import { HubSocket } from './hub-socket';
import { errorMessage, type Logger } from './log';
@@ -85,8 +93,18 @@ export type ControllerDeps = {
* `taken_over`: another instance of this controller opened the controller
* stream after this one. Restarting would take it straight back, so this one
* ends instead.
+ *
+ * `upgrade_required` and `credential_invalid`: the server refused this
+ * controller's protocol, or knows no controller by its credential. Both are
+ * refusals the server marks not retryable, and asking again cannot change
+ * them, so this one ends rather than retrying forever.
*/
-export type ControllerExit = 'stopped' | 'revoked' | 'taken_over';
+export type ControllerExit =
+ | 'stopped'
+ | 'revoked'
+ | 'taken_over'
+ | 'upgrade_required'
+ | 'credential_invalid';
/**
* Makes an agent's credentials file name the relay, its hub, and a token the
@@ -295,11 +313,20 @@ export async function runController(
return revocation;
};
+ /** Set when a refusal that cannot pass ends the run. */
+ let refused: 'upgrade_required' | 'credential_invalid' | null = null;
+ const refuse = (ending: 'upgrade_required' | 'credential_invalid') => {
+ refused ??= ending;
+ stop.abort();
+ };
+
const failed = (what: string, error: unknown) => {
if (isRevoked(error)) {
void revoke();
return;
}
+ if (isUpgradeRequired(error)) return refuse('upgrade_required');
+ if (isCredentialRefused(error)) return refuse('credential_invalid');
log.warn(`${what} failed; it is retried on the next nudge, reconnect or resync.`, {
error: errorMessage(error),
});
@@ -538,6 +565,15 @@ export async function runController(
log.error(
'Another instance of this controller took over its connection to Switch; this one stops its agents and exits.'
);
+ if (ending === 'upgrade_required' || ending === 'credential_invalid') refused ??= ending;
+ if (refused === 'upgrade_required')
+ log.error(
+ 'Switch does not speak this controller’s protocol; it stops its agents and exits. Update the controller.'
+ );
+ if (refused === 'credential_invalid')
+ log.error(
+ 'Switch knows no controller by this credential; it stops its agents and exits. Enroll this machine again.'
+ );
} finally {
clearInterval(resync);
clearInterval(poll);
@@ -552,7 +588,7 @@ export async function runController(
await relay.close();
}
}
- return revocation ? 'revoked' : ending;
+ return revocation ? 'revoked' : (refused ?? ending);
}
/**
diff --git a/console/packages/agent-controller/src/exit-codes.ts b/console/packages/agent-controller/src/exit-codes.ts
index 8dd94c598..7fcc77fdc 100644
--- a/console/packages/agent-controller/src/exit-codes.ts
+++ b/console/packages/agent-controller/src/exit-codes.ts
@@ -10,6 +10,10 @@ export const EXIT_CONFIGURATION = 2;
export const EXIT_REVOKED = 3;
/** Another instance of this controller took its stream over. */
export const EXIT_TAKEN_OVER = 4;
+/** The server does not speak this controller's protocol. Update the controller. */
+export const EXIT_UPGRADE_REQUIRED = 5;
+/** The server knows no controller by this credential. It has to be enrolled again. */
+export const EXIT_CREDENTIAL_INVALID = 6;
/**
* The exit code a command that threw `error` ends with: configuration errors,
diff --git a/console/packages/agent-controller/src/main.ts b/console/packages/agent-controller/src/main.ts
index 6383a44cb..39125c7e1 100644
--- a/console/packages/agent-controller/src/main.ts
+++ b/console/packages/agent-controller/src/main.ts
@@ -13,7 +13,7 @@ import {
normalizeServerUrl,
nodeWebSocket,
} from './api';
-import { DEFAULT_TIMING, runController } from './controller';
+import { type ControllerExit, DEFAULT_TIMING, runController } from './controller';
import { DetachedRuntime } from './detached-runtime';
import { formatChecks, runDoctor } from './doctor';
import { readEnvFile } from './env-file';
@@ -23,6 +23,8 @@ import {
EXIT_OK,
EXIT_REVOKED,
EXIT_TAKEN_OVER,
+ EXIT_UPGRADE_REQUIRED,
+ EXIT_CREDENTIAL_INVALID,
exitCodeFor,
isParseArgsError,
} from './exit-codes';
@@ -122,9 +124,18 @@ The shared host bundle defaults to SWITCH_CONTROLLER_SHARED_HOST_BUNDLE, then
the one built in the workspace.
Log level: SWITCH_CONTROLLER_LOG_LEVEL (debug, info, warn, error; default info).
Exit codes: 0 stopped, 1 error that may pass, 2 configuration error,
-3 revoked, 4 taken over by another instance.
+3 revoked, 4 taken over by another instance, 5 the server needs a newer
+controller, 6 the server knows no controller by this credential.
`;
+const EXIT_CODE_FOR: Record<ControllerExit, number> = {
+ stopped: EXIT_OK,
+ revoked: EXIT_REVOKED,
+ taken_over: EXIT_TAKEN_OVER,
+ upgrade_required: EXIT_UPGRADE_REQUIRED,
+ credential_invalid: EXIT_CREDENTIAL_INVALID,
+};
+
function bundlePath(flag: string | undefined): string {
return resolveSharedHostBundle(flag, process.env, defaultSharedHostBundle);
}
@@ -386,7 +397,7 @@ async function runCommand(args: string[]): Promise<number> {
},
stop.signal
);
- return exit === 'revoked' ? EXIT_REVOKED : exit === 'taken_over' ? EXIT_TAKEN_OVER : EXIT_OK;
+ return EXIT_CODE_FOR[exit];
} finally {
for (const signal of SIGNALS) process.off(signal, onSignal);
store.close();
diff --git a/console/packages/agent-controller/src/service.test.ts b/console/packages/agent-controller/src/service.test.ts
index 8cc78c528..a6f067082 100644
--- a/console/packages/agent-controller/src/service.test.ts
+++ b/console/packages/agent-controller/src/service.test.ts
@@ -20,7 +20,7 @@ describe('systemdUnit', () => {
);
expect(unit).toContain('Environment="PATH=/home/me/.local/bin:/usr/bin:/bin"');
expect(unit).toContain('Restart=on-failure');
- expect(unit).toContain('RestartPreventExitStatus=2 3 4');
+ expect(unit).toContain('RestartPreventExitStatus=2 3 4 5 6');
expect(unit).not.toContain('--launchd');
});
});
diff --git a/console/packages/agent-controller/src/service.ts b/console/packages/agent-controller/src/service.ts
index e64cb1af9..783db42bd 100644
--- a/console/packages/agent-controller/src/service.ts
+++ b/console/packages/agent-controller/src/service.ts
@@ -75,7 +75,7 @@ export function systemdUnit(spec: ServiceSpec): string {
`Environment=${quote(`PATH=${spec.path}`)}`,
'Restart=on-failure',
'RestartSec=5',
- 'RestartPreventExitStatus=2 3 4',
+ 'RestartPreventExitStatus=2 3 4 5 6',
'',
'[Install]',
'WantedBy=default.target',
@@ -134,7 +134,7 @@ ${strings(runArguments(spec, true))}
}
/** The exit codes `--launchd` turns into 0, so launchd does not restart into the same failure. */
-export const LAUNCHD_FINAL_EXIT_CODES: readonly number[] = [2, 3, 4];
+export const LAUNCHD_FINAL_EXIT_CODES: readonly number[] = [2, 3, 4, 5, 6];
/** Where the service's files are, for this user. */
export type ServicePaths = {
diff --git a/console/packages/agent-controller/src/stream.test.ts b/console/packages/agent-controller/src/stream.test.ts
index 44329133e..74f4cc36d 100644
--- a/console/packages/agent-controller/src/stream.test.ts
+++ b/console/packages/agent-controller/src/stream.test.ts
@@ -315,6 +315,32 @@ describe('runControllerStream', () => {
expect(refusedAttach.staleTokens).toBe(0);
});
+ it('stops for good when the server refuses its protocol, on open and on attach', async () => {
+ const refusedOpen = new ScriptedClient();
+ let opens = 0;
+ refusedOpen.open = async () => {
+ opens++;
+ throw new ControllerApiError(426, 'protocol_unsupported', 'protocol 1', false, null);
+ };
+ expect(await run(refusedOpen, new AbortController().signal).ending).toBe('upgrade_required');
+ expect(opens).toBe(1);
+
+ const refusedAttach = new ScriptedClient();
+ refusedAttach.sockets = () => scriptedSocket([refused(426, 'protocol_unsupported')], 4426);
+ expect(await run(refusedAttach, new AbortController().signal).ending).toBe('upgrade_required');
+ });
+
+ it('stops for good when the server knows no controller by its credential', async () => {
+ const scripted = new ScriptedClient();
+ let opens = 0;
+ scripted.open = async () => {
+ opens++;
+ throw new ControllerApiError(401, 'invalid_credential', 'not valid', false, null);
+ };
+ expect(await run(scripted, new AbortController().signal).ending).toBe('credential_invalid');
+ expect(opens).toBe(1);
+ });
+
it('drops a token the socket refused as stale, and attaches again', async () => {
const scripted = new ScriptedClient();
scripted.sockets = () =>
diff --git a/console/packages/agent-controller/src/stream.ts b/console/packages/agent-controller/src/stream.ts
index 0285e15ab..531ce24e7 100644
--- a/console/packages/agent-controller/src/stream.ts
+++ b/console/packages/agent-controller/src/stream.ts
@@ -3,8 +3,10 @@ import type { z } from 'zod';
import {
ControllerApiError,
type ControllerClient,
+ isCredentialRefused,
isRevoked,
isTakenOver,
+ isUpgradeRequired,
type OpenedSocket,
} from './api';
import { errorMessage, type Logger } from './log';
@@ -66,7 +68,12 @@ export type ControllerFrame =
| { type: 'credential.revoked'; data: Record<string, never> };
/** Why the stream stopped for good. */
-export type StreamEnding = 'stopped' | 'revoked' | 'taken_over';
+export type StreamEnding =
+ | 'stopped'
+ | 'revoked'
+ | 'taken_over'
+ | 'upgrade_required'
+ | 'credential_invalid';
export type ControllerStreamOptions = {
client: Pick<ControllerClient, 'openConnection' | 'openSocket'>;
@@ -151,8 +158,11 @@ type Message = { event: string; data: unknown };
* Ends with `'revoked'` on `credential.revoked` or a refusal saying
* `controller_revoked`, with `'taken_over'` when another instance of this
* controller took the connection over (an `evicted` frame or a refusal saying
- * so: reopening would take it straight back, so this one stops), and with
- * `'stopped'` when `signal` fires. Any other `evicted` opens a new connection.
+ * so: reopening would take it straight back, so this one stops), with
+ * `'upgrade_required'` when the server refuses this controller's protocol and
+ * `'credential_invalid'` when it knows no controller by this credential (the
+ * server marks both not retryable, and retrying them used to spin forever),
+ * and with `'stopped'` when `signal` fires. Any other `evicted` opens a new connection.
*/
export function runControllerStream(options: ControllerStreamOptions): Promise<StreamEnding> {
return new ControllerStream(options).run();
@@ -184,6 +194,8 @@ class ControllerStream {
if (!attempt.ending && !signal.aborted) {
if (isRevoked(error)) attempt.ending = 'revoked';
else if (isTakenOver(error)) attempt.ending = 'taken_over';
+ else if (isUpgradeRequired(error)) attempt.ending = 'upgrade_required';
+ else if (isCredentialRefused(error)) attempt.ending = 'credential_invalid';
else {
const reason = attempt.abort.signal.reason;
failure = errorMessage(reason instanceof IdleTimeout ? reason : error);