Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Drop machine slots: a cloud machine boots from its user data (#740)

Louis Amaudruz6173528dd5b8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2b6907799..16ce07a54 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -45,6 +45,34 @@ version of their own to them without also giving them a release of their own.
### [Unreleased]
#### Added
+- **Switch cloud machines that run the agents controller sleep when idle,
+ and wake on a message.** Such a machine is stopped once its status reports
+ say no session has run for `HOSTED_IDLE_STOP_MINUTES` (released, as before,
+ when no agent is placed on it). A message addressed to one of its agents
+ starts it again; the room is told once that the machine is starting, and
+ Switch keeps the message for the agent until it is back. Placing an agent on
+ a sleeping machine starts it too, and is judged by its last report. A
+ machine its owner stopped stays stopped, and says so.
+- **Switch cloud machines can run the agents controller.** With
+ `HOSTED_MACHINE_RUNTIME=controller` (Helm `switchCore.hostedMachineRuntime`)
+ and agent management on, a cloud machine claimed from then on enrolls the
+ standard `switch-agent-controller` with a one-time code Core hands the hosted
+ controller at prepare (the same code on every retry of a revision, valid 30
+ minutes), instead of running the hosted worker. The controller it enrolls as
+ becomes the machine's (`hosted_machines.controller_id`, kind `ec2`, "Switch
+ cloud"); its status reports make the machine ready; the managed agents placed
+ on it are the machine's agents, and keep it from being retained. Launches are
+ refused in that mode, and such machines are not stopped when idle yet.
+ Machines claimed before keep the worker runtime. Migration `e7a2c4b9d013`.
+- **Provider logins sealed to a machine.** A controller registers an X25519
+ public key (at enrollment, or once afterwards); its owner gives the machine a
+ provider login sealed to that key, which Switch stores and relays as
+ ciphertext it cannot open (`PUT/GET/DELETE
+ /gateway/management/controllers/{id}/provider-logins/{provider}`, and
+ `GET /v1/management/controllers/{id}/provider-credentials/{provider}` for the
+ controller). Giving one queues a `provider.login` operation the machine takes
+ it up with; revoking the machine deletes its logins. Migration
+ `d41c7a9e2b58` adds `sealed_provider_logins`.
- **The Helm chart turns agent management on from its values.**
`switchCore.agentManagement.enabled` and `secrets.controllerTokenSecret` (or
`CONTROLLER_TOKEN_SECRET` in `secrets.existingSecret`) set
@@ -1381,6 +1409,20 @@ version of their own to them without also giving them a release of their own.
### [Unreleased]
+#### Added
+- **Switch cloud on the agents controller.** On a server whose cloud machines
+ run the agents controller, choosing Switch cloud in the New agent form starts
+ your cloud machine and waits for it to come online (its first start takes a
+ few minutes), then creates the agent as a managed agent on it, like on any
+ machine. Once online, the machine is listed as "Switch cloud" among your
+ machines. A server whose machines run the hosted worker works as before.
+- **Give a machine a provider login.** In the New agent form, a machine with
+ a provider installed but not signed in offers to give it a login: a Claude
+ setup token or API key, a Codex or Cursor API key, or this computer's own
+ Codex, OpenCode or Antigravity sign-in. Console seals it to the machine's own
+ key before it leaves, so the server only relays it, and shows whether the
+ provider signs in with it on the machine.
+
#### Fixed
- **A machine whose controller Switch refuses for good says so instead of
restarting it forever.** When the server needs a newer agents controller than
@@ -3053,6 +3095,27 @@ tags; see RELEASING.md.
### [Unreleased]
+#### Added
+- **Provider logins given to the machine, on demand.** The controller makes an
+ X25519 keypair at enrollment (or on its next run, for one enrolled before)
+ and registers the public half. When its owner gives the machine a login
+ sealed to it, a `provider.login` operation has the controller open it, check
+ the provider signs in with it, and report whether it does. The machine's own
+ login still comes first; agents of a provider it has none for get the given
+ one, run in a process of their own, and restart when it changes or is
+ withdrawn. Status reports `auth_source: "sealed"` for it.
+
+- **Each agent as a Linux user of its own.** On Linux with systemd and polkit,
+ `sudo switch-agent-controller install-service --separate-users` sets up, once,
+ a pool of agent users, a unit template that runs an agent as one of them, a
+ polkit rule that lets the controller start only those units, and the
+ controller as a system service. Each agent then runs as its own user and sees
+ only its own directory. It cannot see the home directories, the controller's
+ data, the other agents or the cloud instance metadata, and the rest of the
+ system is read-only to it. Agents get provider keys from the controller's
+ `--env-file`, not its user's own logins. `--agent-users` sets how many agents
+ can run (16 by default).
+
#### Fixed
- **A refusal that cannot pass stops the controller instead of retrying it
forever.** The server answers `protocol_unsupported` when it no longer speaks
diff --git a/console/agents/workflows/managed-service-sign-in.md b/console/agents/workflows/managed-service-sign-in.md
index 83e69f82f..59ae75261 100644
--- a/console/agents/workflows/managed-service-sign-in.md
+++ b/console/agents/workflows/managed-service-sign-in.md
@@ -9,29 +9,21 @@ and an invalid value is reported where the Cloud would be.
Add server → Connect to Switch Cloud reuses server registration, password/SSO
sign-in (or creating an account, where the server allows sign-up) and encrypted
session storage. A matching server entry is reused, and an account already
-signed in goes straight on to the provider selector. The provider selector shows
-the registered providers; Claude Code is the only enabled provider in this
-increment.
-
-## Claude Code connection
-
-The screen offers an API key or subscription setup token with official setup
-instructions. Verify and connect sends the credential over the authenticated
-HTTPS gateway connection. The server runs a fixed, tool-free Claude Code request
-before storing the credential encrypted for the current tenant and user.
-Verification consumes a small amount of API credit or subscription allowance.
-
-The status endpoint returns only the credential kind and last successful check
-time. The UI can reopen a connection, replace it, or remove it. Failed verification
-does not overwrite an existing connection. Removal deletes the server-side
-credential; it does not revoke it at Anthropic. Signing out does not remove a
-cloud connection. Legacy local drafts are never uploaded automatically and are
-still cleared on sign-out or server removal.
-
-See `deploy/hosted/README.md` for the backend verifier image. A backend without
-verification enabled returns a visible unavailable error. GitHub connection,
-worker credential delivery and cloud agent creation are separate steps; connecting
-Claude does not provision a worker or start a room agent.
+signed in goes straight on to the connections step, then to a new agent whose
+run location is Switch cloud.
+
+## Switch cloud agents
+
+Signing in to Switch Cloud claims and starts the user's cloud machine
+(`/gateway/hosted-machines/ensure`). The machine runs the agents controller, so
+a Switch cloud agent is a managed agent placed on that machine's controller
+once it has enrolled and is online, created like an agent on any other machine.
+Provider logins reach the machine sealed to its controller ("Give login"); the
+server stores no provider credentials of its own. Your Agents shows the machine
+with its state, disk, and start, stop and retry.
+
+Legacy local Claude drafts are never uploaded and are still cleared on sign-out
+or server removal.
## GitHub connection
@@ -55,5 +47,4 @@ URL is `<origin>/gateway/provider-connections/github/callback`. Keep expiring us
tokens enabled in the GitHub App. No app secret belongs in Console build settings.
The handoff store is bounded and process-local, matching the singleton backend.
-This step does not issue worker installation tokens or provision agents. The app
-signing key and worker credential renewal belong to cloud-agent setup.
+This step does not provision agents.
diff --git a/console/apps/switch-console-desktop/src/main/core/agent-migration/new-managed-agent-service.test.ts b/console/apps/switch-console-desktop/src/main/core/agent-migration/new-managed-agent-service.test.ts
index db1d5e9c2..3cc972109 100644
--- a/console/apps/switch-console-desktop/src/main/core/agent-migration/new-managed-agent-service.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/agent-migration/new-managed-agent-service.test.ts
@@ -20,6 +20,7 @@ const PARAMS: AddManagedAgentParams = {
instructions: 'Be brief.',
model: 'opus',
advancedConfig: { effort: 'high', tools: ['Read'] },
+ isolation: 'isolated',
entryPoint: 'sidebar',
};
@@ -101,6 +102,7 @@ describe('NewManagedAgentService.add', () => {
instructions: 'Be brief.',
auto_approve: true,
directory: '/work/pm',
+ isolation: 'isolated',
},
},
]);
diff --git a/console/apps/switch-console-desktop/src/main/core/agent-migration/new-managed-agent-service.ts b/console/apps/switch-console-desktop/src/main/core/agent-migration/new-managed-agent-service.ts
index 552b7da5a..472dc2faa 100644
--- a/console/apps/switch-console-desktop/src/main/core/agent-migration/new-managed-agent-service.ts
+++ b/console/apps/switch-console-desktop/src/main/core/agent-migration/new-managed-agent-service.ts
@@ -25,6 +25,8 @@ export type AddManagedAgentParams = {
model: string | null;
/** The provider's advanced configuration, keyed by its field keys; unset fields are absent. */
advancedConfig: AdvancedConfig;
+ /** In the machine's controller, or in a process of its own. */
+ isolation: 'shared' | 'isolated';
entryPoint: UiEntryPoint;
};
@@ -54,7 +56,7 @@ export type NewManagedAgentDeps = {
icon_url: string | null;
controller_id: string;
desired_state: 'running' | 'stopped';
- definition: ManagedDefinition;
+ definition: ManagedDefinition & { isolation: 'shared' | 'isolated' };
}
): Promise<ManagedCreateOutcome>;
log: MigrationLog;
@@ -116,7 +118,7 @@ export class NewManagedAgentService {
icon_url: input.iconUrl,
controller_id: input.machineId,
desired_state: 'running',
- definition,
+ definition: { ...definition, isolation: input.isolation },
});
if (created.kind === 'name-conflict') return { kind: 'name-conflict' };
if (created.kind === 'refused') return { kind: 'error', message: created.message };
diff --git a/console/apps/switch-console-desktop/src/main/core/embedded-controller/management-port.test.ts b/console/apps/switch-console-desktop/src/main/core/embedded-controller/management-port.test.ts
index 2565ef36b..9498bf913 100644
--- a/console/apps/switch-console-desktop/src/main/core/embedded-controller/management-port.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/embedded-controller/management-port.test.ts
@@ -67,6 +67,7 @@ describe('placedOn', () => {
revokedAt: null,
providers: [],
workspacesDir: null,
+ sealingKey: null,
},
],
[
diff --git a/console/apps/switch-console-desktop/src/main/core/managed-agents/controller.ts b/console/apps/switch-console-desktop/src/main/core/managed-agents/controller.ts
index 1e7fac80d..c1bdb3c35 100644
--- a/console/apps/switch-console-desktop/src/main/core/managed-agents/controller.ts
+++ b/console/apps/switch-console-desktop/src/main/core/managed-agents/controller.ts
@@ -1,26 +1,39 @@
+import {
+ type ProviderLogin,
+ sealingKeyId,
+ sealProviderLogin,
+} from '@switch-console/agent-providers';
import { embeddedControllerService } from '@main/core/embedded-controller/embedded-controllers';
import { hostControllerService } from '@main/core/host-controllers/host-controllers';
import {
AgentManagementUnavailableError,
deleteAgent,
- deleteManagedAgent,
fetchAdvancedConfigSchema,
fetchManagedAgents,
+ fetchMachineOperation,
fetchManagementControllers,
GatewayError,
+ giveMachineLogin,
managementErrorMessage,
setManagedAgentDesiredState,
updateManagedAgent,
} from '@main/core/switch-servers/gateway-client';
+import {
+ localProviderAuthPath,
+ readLocalProviderSignIn,
+} from '@main/core/switch-servers/local-provider-sign-in';
import { withReachableServerWorkspaceSession } from '@main/core/workspaces/workspace-session';
import { requireWorkspaceForServer } from '@main/core/workspaces/workspaces-store';
import type {
AdvancedConfigField,
ManagedAgentChanges,
ManagedAgentView,
+ MachineLoginInput,
+ MachineLoginOutcome,
ManagedMachine,
OwnedMachine,
} from '@shared/core/managed-agents/managed-agents';
+import type { AgentProviderId } from '@shared/core/providers/agent-provider-registry';
import { createRPCController } from '@shared/lib/ipc/rpc';
import { ownedMachines } from './owned-machines';
@@ -108,6 +121,67 @@ export const managedAgentsController = createRPCController({
});
},
+ /**
+ * Gives one of the owner's machines a provider login, sealed here to the
+ * machine's own key so the server only ever holds ciphertext. Answers the
+ * operation the machine takes it up with; `machineLoginOutcome` says how
+ * that went.
+ */
+ giveMachineLogin: (params: {
+ serverId: string;
+ machineId: string;
+ provider: AgentProviderId;
+ login: MachineLoginInput;
+ }): Promise<{ operationId: string }> =>
+ withReachableServerWorkspaceSession(params.serverId, async (server) => {
+ const controller = (await fetchManagementControllers(server)).find(
+ (candidate) => candidate.id === params.machineId
+ );
+ if (!controller) throw new Error('That machine is not one of yours on this server.');
+ if (!controller.sealingKey)
+ throw new Error(
+ `${controller.name} cannot be given a login: its controller has no key for one. Update the controller on it.`
+ );
+ if (sealingKeyId(controller.sealingKey.key) !== controller.sealingKey.keyId)
+ throw new Error(`The server sent a key for ${controller.name} that does not match its id.`);
+ const login = await loginToGive(params.provider, params.login);
+ const { operationId } = await giveMachineLogin(
+ server,
+ controller.id,
+ params.provider,
+ sealProviderLogin({
+ publicKey: controller.sealingKey.key,
+ controllerId: controller.id,
+ provider: params.provider,
+ login,
+ })
+ );
+ return { operationId };
+ }),
+
+ /** How the machine took up a login given to it, by the operation `giveMachineLogin` answered. */
+ machineLoginOutcome: (params: {
+ serverId: string;
+ machineId: string;
+ operationId: string;
+ }): Promise<MachineLoginOutcome> =>
+ withReachableServerWorkspaceSession(params.serverId, async (server) => {
+ const operation = await fetchMachineOperation(server, params.machineId, params.operationId);
+ if (!operation) return { state: 'pending' };
+ if (operation.state === 'succeeded') return { state: 'succeeded' };
+ if (operation.state === 'pending' || operation.state === 'claimed')
+ return { state: 'pending' };
+ return {
+ state: 'failed',
+ code: operation.error?.code ?? operation.state,
+ message:
+ operation.error?.message ??
+ (operation.state === 'expired'
+ ? 'The machine did not take the login up in time; is it online?'
+ : `The machine did not take the login up (${operation.state}).`),
+ };
+ }),
+
/** Each provider's advanced configuration fields, keyed by provider, as the server checks a definition against them. */
advancedConfigSchema: (serverId: string): Promise<Record<string, AdvancedConfigField[]>> =>
withReachableServerWorkspaceSession(serverId, (server) => fetchAdvancedConfigSchema(server)),
@@ -144,12 +218,13 @@ export const managedAgentsController = createRPCController({
setManagedAgentDesiredState(server, params.agentId, params.desiredState)
),
- /** Deletes the agent: its machine stops it, and it is gone from Switch. */
+ /** Deletes the agent: its machine stops it, and it is gone from Switch.
+ * One call: Switch stops managing an agent it deletes, while stopping
+ * managing it first would leave the agent behind if the delete then failed. */
remove: (params: { serverId: string; agentId: string }): Promise<void> =>
- withReachableServerWorkspaceSession(params.serverId, async (server) => {
- await deleteManagedAgent(server, params.agentId);
- await deleteAgent(server, params.agentId);
- }),
+ withReachableServerWorkspaceSession(params.serverId, (server) =>
+ deleteAgent(server, params.agentId)
+ ),
});
function definitionBody(changes: ManagedAgentChanges['definition']): Record<string, unknown> {
@@ -163,3 +238,25 @@ function definitionBody(changes: ManagedAgentChanges['definition']): Record<stri
if (changes.isolation !== undefined) body.isolation = changes.isolation;
return body;
}
+
+/** The login a machine is given, from what was typed or this computer's own sign-in. */
+async function loginToGive(
+ provider: AgentProviderId,
+ input: MachineLoginInput
+): Promise<ProviderLogin> {
+ if (input.source === 'typed') {
+ const credential = input.credential.trim();
+ if (!credential) throw new Error('Enter the key or token to give the machine.');
+ if (input.kind === 'setup-token' && provider !== 'claude')
+ throw new Error('Only Claude takes a setup token.');
+ return { kind: input.kind, credential };
+ }
+ if (provider !== 'codex' && provider !== 'opencode' && provider !== 'antigravity')
+ throw new Error(
+ `This computer's ${provider} sign-in cannot be given to another machine; give it an API key or token.`
+ );
+ const credential = await readLocalProviderSignIn(provider, localProviderAuthPath(provider));
+ if (!credential)
+ throw new Error(`This computer is not signed in to ${provider}. Sign in here first.`);
+ return { kind: 'auth-json', credential };
+}
diff --git a/console/apps/switch-console-desktop/src/main/core/managed-agents/give-machine-login.test.ts b/console/apps/switch-console-desktop/src/main/core/managed-agents/give-machine-login.test.ts
new file mode 100644
index 000000000..44cd89522
--- /dev/null
+++ b/console/apps/switch-console-desktop/src/main/core/managed-agents/give-machine-login.test.ts
@@ -0,0 +1,140 @@
+import {
+ generateSealingKeyPair,
+ openProviderLogin,
+ sealingKeyId,
+} from '@switch-console/agent-providers';
+import { beforeEach, describe, expect, it, vi } from 'vitest';
+
+const mocks = vi.hoisted(() => ({
+ fetchManagementControllers: vi.fn(),
+ giveMachineLogin: vi.fn(),
+ fetchMachineOperation: vi.fn(),
+ readLocalProviderSignIn: vi.fn(),
+}));
+
+vi.mock('@main/core/embedded-controller/embedded-controllers', () => ({
+ embeddedControllerService: {},
+}));
+vi.mock('@main/core/host-controllers/host-controllers', () => ({ hostControllerService: {} }));
+vi.mock('@main/core/workspaces/workspaces-store', () => ({ requireWorkspaceForServer: vi.fn() }));
+vi.mock('@main/core/workspaces/workspace-session', () => ({
+ withReachableServerWorkspaceSession: (_serverId: string, run: (server: unknown) => unknown) =>
+ run({ id: 'server-1', gatewayUrl: 'https://switch.example.com' }),
+}));
+vi.mock('@main/core/switch-servers/local-provider-sign-in', () => ({
+ localProviderAuthPath: () => '/home/me/.codex/auth.json',
+ readLocalProviderSignIn: mocks.readLocalProviderSignIn,
+}));
+vi.mock('@main/core/switch-servers/gateway-client', () => ({
+ AgentManagementUnavailableError: class extends Error {},
+ GatewayError: class extends Error {},
+ deleteAgent: vi.fn(),
+ deleteManagedAgent: vi.fn(),
+ fetchAdvancedConfigSchema: vi.fn(),
+ fetchManagedAgents: vi.fn(),
+ managementErrorMessage: vi.fn(),
+ setManagedAgentDesiredState: vi.fn(),
+ updateManagedAgent: vi.fn(),
+ fetchManagementControllers: mocks.fetchManagementControllers,
+ giveMachineLogin: mocks.giveMachineLogin,
+ fetchMachineOperation: mocks.fetchMachineOperation,
+}));
+
+const { managedAgentsController } = await import('./controller');
+
+const keys = generateSealingKeyPair();
+
+function machine(sealingKey: { key: string; keyId: string } | null) {
+ return { id: 'controller-1', name: 'build-box', sealingKey };
+}
+
+beforeEach(() => {
+ vi.clearAllMocks();
+ mocks.giveMachineLogin.mockResolvedValue({ operationId: 'op-1', revision: 1 });
+});
+
+describe('giving a machine a provider login', () => {
+ it('seals it to the machine’s own key before it leaves this computer', async () => {
+ mocks.fetchManagementControllers.mockResolvedValue([
+ machine({ key: keys.publicKey, keyId: sealingKeyId(keys.publicKey) }),
+ ]);
+ expect(
+ await managedAgentsController.giveMachineLogin({
+ serverId: 'server-1',
+ machineId: 'controller-1',
+ provider: 'claude',
+ login: { source: 'typed', kind: 'setup-token', credential: ' sk-ant-oat-given ' },
+ })
+ ).toEqual({ operationId: 'op-1' });
+ const [, controllerId, provider, sealed] = mocks.giveMachineLogin.mock.calls[0]!;
+ expect([controllerId, provider]).toEqual(['controller-1', 'claude']);
+ expect(JSON.stringify(sealed)).not.toContain('sk-ant');
+ expect(
+ openProviderLogin({ keys, controllerId: 'controller-1', provider: 'claude', sealed })
+ ).toEqual({ kind: 'setup-token', credential: 'sk-ant-oat-given' });
+ });
+
+ it("sends this computer's own sign-in, for a provider that has one", async () => {
+ mocks.fetchManagementControllers.mockResolvedValue([
+ machine({ key: keys.publicKey, keyId: sealingKeyId(keys.publicKey) }),
+ ]);
+ mocks.readLocalProviderSignIn.mockResolvedValue('{"tokens":{}}');
+ await managedAgentsController.giveMachineLogin({
+ serverId: 'server-1',
+ machineId: 'controller-1',
+ provider: 'codex',
+ login: { source: 'this-computer' },
+ });
+ const [, , , sealed] = mocks.giveMachineLogin.mock.calls[0]!;
+ expect(
+ openProviderLogin({ keys, controllerId: 'controller-1', provider: 'codex', sealed })
+ ).toEqual({ kind: 'auth-json', credential: '{"tokens":{}}' });
+ await expect(
+ managedAgentsController.giveMachineLogin({
+ serverId: 'server-1',
+ machineId: 'controller-1',
+ provider: 'claude',
+ login: { source: 'this-computer' },
+ })
+ ).rejects.toThrow(/cannot be given/);
+ });
+
+ it('refuses a machine with no key, or a key that does not match its id', async () => {
+ mocks.fetchManagementControllers.mockResolvedValue([machine(null)]);
+ const give = () =>
+ managedAgentsController.giveMachineLogin({
+ serverId: 'server-1',
+ machineId: 'controller-1',
+ provider: 'claude',
+ login: { source: 'typed', kind: 'api-key', credential: 'sk-ant-api-x' },
+ });
+ await expect(give()).rejects.toThrow(/no key/);
+ mocks.fetchManagementControllers.mockResolvedValue([
+ machine({ key: keys.publicKey, keyId: 'ffffffffffffffff' }),
+ ]);
+ await expect(give()).rejects.toThrow(/does not match/);
+ expect(mocks.giveMachineLogin).not.toHaveBeenCalled();
+ });
+
+ it('says how the machine took the login up', async () => {
+ const outcome = () =>
+ managedAgentsController.machineLoginOutcome({
+ serverId: 'server-1',
+ machineId: 'controller-1',
+ operationId: 'op-1',
+ });
+ mocks.fetchMachineOperation.mockResolvedValue({ state: 'claimed', error: null });
+ expect(await outcome()).toEqual({ state: 'pending' });
+ mocks.fetchMachineOperation.mockResolvedValue({ state: 'succeeded', error: null });
+ expect(await outcome()).toEqual({ state: 'succeeded' });
+ mocks.fetchMachineOperation.mockResolvedValue({
+ state: 'failed',
+ error: { code: 'provider_login_expired', message: 'It does not sign in.' },
+ });
+ expect(await outcome()).toEqual({
+ state: 'failed',
+ code: 'provider_login_expired',
+ message: 'It does not sign in.',
+ });
+ });
+});
diff --git a/console/apps/switch-console-desktop/src/main/core/managed-agents/owned-machines.test.ts b/console/apps/switch-console-desktop/src/main/core/managed-agents/owned-machines.test.ts
index 362b7ce3d..344fc57d5 100644
--- a/console/apps/switch-console-desktop/src/main/core/managed-agents/owned-machines.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/managed-agents/owned-machines.test.ts
@@ -12,6 +12,7 @@ function controller(patch: Partial<ManagementController> & { id: string }): Mana
revokedAt: null,
providers: [],
workspacesDir: null,
+ sealingKey: null,
...patch,
};
}
@@ -54,7 +55,7 @@ describe('ownedMachines', () => {
workspacesDir: '/data/workspaces',
}),
controller({ id: 'box', state: 'unknown' }),
- controller({ id: 'cloud-vm' }),
+ controller({ id: 'cloud-vm', sealingKey: { key: 'a2V5', keyId: 'id' } }),
controller({ id: 'gone', state: 'revoked' }),
],
{ thisComputer: 'laptop', sshHosts: [{ controllerId: 'box', sshHost: 'devbox' }] }
@@ -68,6 +69,8 @@ describe('ownedMachines', () => {
providers: [{ provider: 'claude', ready: true, problem: null }],
workspacesDir: '/data/workspaces',
local: { kind: 'this-computer' },
+ acceptsLogins: false,
+ cloud: false,
},
{
id: 'box',
@@ -77,6 +80,8 @@ describe('ownedMachines', () => {
providers: [],
workspacesDir: null,
local: { kind: 'ssh-host', sshHost: 'devbox' },
+ acceptsLogins: false,
+ cloud: false,
},
{
id: 'cloud-vm',
@@ -86,7 +91,17 @@ describe('ownedMachines', () => {
providers: [],
workspacesDir: null,
local: null,
+ acceptsLogins: true,
+ cloud: false,
},
]);
});
+
+ it('marks the owner’s Switch cloud machine', () => {
+ const [cloud] = ownedMachines([controller({ id: 'ec2', kind: 'ec2' })], {
+ thisComputer: null,
+ sshHosts: [],
+ });
+ expect(cloud?.cloud).toBe(true);
+ });
});
diff --git a/console/apps/switch-console-desktop/src/main/core/managed-agents/owned-machines.ts b/console/apps/switch-console-desktop/src/main/core/managed-agents/owned-machines.ts
index 2a3495b62..3b98dde32 100644
--- a/console/apps/switch-console-desktop/src/main/core/managed-agents/owned-machines.ts
+++ b/console/apps/switch-console-desktop/src/main/core/managed-agents/owned-machines.ts
@@ -2,7 +2,11 @@ import type {
ControllerProviderReport,
ManagementController,
} from '@main/core/switch-servers/gateway-client';
-import type { MachineProvider, OwnedMachine } from '@shared/core/managed-agents/managed-agents';
+import {
+ isCloudMachine,
+ type MachineProvider,
+ type OwnedMachine,
+} from '@shared/core/managed-agents/managed-agents';
/** This Console's own controllers on a server: this computer's, and its SSH hosts'. */
export type LocalControllers = {
@@ -39,6 +43,8 @@ export function ownedMachines(
kind: controller.kind,
state: controller.state,
providers: controller.providers.map(machineProvider),
+ acceptsLogins: controller.sealingKey !== null,
+ cloud: isCloudMachine(controller),
workspacesDir: controller.workspacesDir,
local:
controller.id === local.thisComputer
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control-launches.test.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control-launches.test.ts
deleted file mode 100644
index 6ecc5eca6..000000000
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control-launches.test.ts
+++ /dev/null
@@ -1,79 +0,0 @@
-import { beforeEach, expect, it, vi } from 'vitest';
-
-const { FakeGatewayError } = vi.hoisted(() => ({
- FakeGatewayError: class extends Error {
- constructor(
- readonly kind: 'unauthorized' | 'http' | 'network',
- message: string,
- readonly status?: number
- ) {
- super(message);
- }
- },
-}));
-
-const gateway = vi.hoisted(() => ({ answer: null as (() => unknown) | null }));
-const kvRows = vi.hoisted(() => new Map<string, unknown>());
-
-vi.mock('@main/db/kv', () => ({
- KV: class {
- get = async (key: string) => kvRows.get(key) ?? null;
- set = async (key: string, value: unknown) => void kvRows.set(key, structuredClone(value));
- del = async (key: string) => void kvRows.delete(key);
- getAll = async () => Object.fromEntries(kvRows);
- },
-}));
-
-vi.mock('@switch-console/agent-providers', () => ({
- CloudRelayClient: class {},
- CloudRelayError: class extends Error {},
- RELAY_TIMEOUT_MS: 1000,
-}));
-
-vi.mock('@main/core/switch-servers/servers-store', () => ({
- getServer: async () => ({ id: 'server' }),
-}));
-
-vi.mock('@main/core/workspaces/workspace-session', () => ({
- withServerWorkspaceSession: async (serverId: string, fn: (server: unknown) => unknown) =>
- fn({ id: serverId }),
-}));
-
-vi.mock('@main/core/switch-servers/gateway-client', () => ({
- GatewayError: FakeGatewayError,
- gatewayRequest: vi.fn(),
- gatewayFetch: vi.fn(async (_server: unknown, path: string) => {
- expect(path).toBe('/hosted-launches');
- return { json: async () => gateway.answer!() };
- }),
-}));
-
-const { gatewayFetch } = await import('@main/core/switch-servers/gateway-client');
-const { listCloudAgents } = await import('./cloud-control');
-
-function refuse(error: InstanceType<typeof FakeGatewayError>) {
- vi.mocked(gatewayFetch).mockRejectedValueOnce(error);
-}
-
-beforeEach(() => {
- gateway.answer = () => [];
-});
-
-it('reads a 404 from the launch list as a server without cloud agents', async () => {
- refuse(new FakeGatewayError('http', 'Switch gateway returned 404', 404));
- await expect(listCloudAgents('server')).resolves.toBeNull();
-});
-
-it('keeps an empty launch list distinct from a server without cloud agents', async () => {
- await expect(listCloudAgents('server')).resolves.toEqual([]);
-});
-
-it.each([
- ['an expired session', new FakeGatewayError('unauthorized', 'Switch session expired', 401)],
- ['a refusal', new FakeGatewayError('http', 'Switch gateway returned 403', 403)],
- ['a server fault', new FakeGatewayError('http', 'Switch gateway returned 500', 500)],
- ['an unreachable server', new FakeGatewayError('network', 'Could not reach the gateway')],
-])('still raises %s', async (_name, error) => {
- refuse(error);
- await expect(listCloudAgents('server')).rejects.toBe(error);
-});
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control.test.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control.test.ts
deleted file mode 100644
index bc09016b1..000000000
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control.test.ts
+++ /dev/null
@@ -1,471 +0,0 @@
-import { beforeEach, describe, expect, it, vi } from 'vitest';
-
-const server = vi.hoisted(() => ({
- operations: new Map<string, { id: string; session_id: string; action: string }>(),
- sessions: new Set<string>(),
- restarts: 0,
- loseNextResponse: false,
- refuseNext: null as { status: number; detail: string; code?: string } | null,
- launches: [] as { request_id: string }[],
- machines: [] as { machine_id: string; revision: number }[],
- machineActions: [] as unknown[],
- wakeRace: false,
- relayClients: 0,
- relayList: (async () => []) as () => Promise<unknown[]>,
-}));
-const kvRows = vi.hoisted(() => new Map<string, unknown>());
-
-vi.mock('@main/db/kv', () => ({
- KV: class {
- get = async (key: string) => kvRows.get(key) ?? null;
- set = async (key: string, value: unknown) => void kvRows.set(key, structuredClone(value));
- del = async (key: string) => void kvRows.delete(key);
- getAll = async () => Object.fromEntries(kvRows);
- },
-}));
-
-const { FakeGatewayError } = vi.hoisted(() => ({
- FakeGatewayError: class extends Error {
- constructor(
- readonly kind: 'unauthorized' | 'http' | 'network',
- message: string,
- readonly status?: number,
- readonly detail?: string,
- readonly code?: string
- ) {
- super(message);
- }
- },
-}));
-
-vi.mock('@switch-console/agent-providers', () => ({
- CloudRelayClient: class {
- constructor() {
- server.relayClients += 1;
- }
- isClosed = false;
- onClose() {}
- list() {
- return server.relayList();
- }
- },
- CloudRelayError: class extends Error {
- constructor(
- readonly relayCode: string,
- message: string,
- readonly status: number,
- readonly wakeAvailable: boolean
- ) {
- super(message);
- }
- },
- RELAY_TIMEOUT_MS: 1000,
-}));
-
-vi.mock('@main/core/switch-servers/servers-store', () => ({
- getServer: async () => ({ id: 'server' }),
-}));
-
-vi.mock('@main/core/workspaces/workspace-session', () => ({
- withServerWorkspaceSession: async (serverId: string, fn: (server: unknown) => unknown) =>
- fn({ id: serverId }),
-}));
-
-vi.mock('@main/core/switch-servers/gateway-client', () => ({
- GatewayError: FakeGatewayError,
- gatewayRequest: vi.fn(),
- gatewayFetch: vi.fn(
- async (_server: unknown, path: string, init: { method?: string; body?: unknown }) => {
- if (path === '/hosted-launches') return { json: async () => server.launches };
- if (path === '/hosted-machines') return { json: async () => ({ machines: server.machines }) };
- const [, kind, id, rest] = path.split('/');
- if (kind === 'hosted-launches' && rest === undefined)
- return { json: async () => server.launches.find((each) => each.request_id === id) };
- if (kind === 'hosted-machines') {
- const machine = server.machines.find((each) => each.machine_id === id)!;
- if (rest === undefined) return { json: async () => machine };
- server.machineActions.push(init.body);
- if (server.wakeRace) {
- server.wakeRace = false;
- Object.assign(machine, {
- state: 'provisioning',
- desired_state: 'running',
- stop_reason: null,
- sleeping: false,
- revision: machine.revision + 1,
- });
- throw new FakeGatewayError(
- 'http',
- 'Switch gateway returned 409',
- 409,
- 'revision mismatch'
- );
- }
- const started = {
- ...machine,
- desired_state: 'running',
- stop_reason: null,
- sleeping: false,
- revision: machine.revision + 1,
- };
- return { json: async () => ({ machine: started }) };
- }
- if (server.refuseNext) {
- const { status, detail, code } = server.refuseNext;
- server.refuseNext = null;
- throw new FakeGatewayError(
- 'http',
- `Switch gateway returned ${status}`,
- status,
- detail,
- code
- );
- }
- const body = init.body as { id: string; session_id: string; action: string };
- let operation = server.operations.get(body.id);
- if (!operation) {
- operation = body;
- server.operations.set(body.id, operation);
- if (body.action === 'start') server.sessions.add(body.session_id);
- else server.restarts += 1;
- }
- if (server.loseNextResponse) {
- server.loseNextResponse = false;
- throw new FakeGatewayError('network', 'Could not reach the gateway: socket hang up');
- }
- return { json: async () => ({ ...operation, state: 'applied', error: null }) };
- }
- ),
-}));
-
-const { CloudRelayError } = await import('@switch-console/agent-providers');
-const { listCloudAgents, listCloudSessions, runCloudSessionOperation, wakeCloudAgent } =
- await import('./cloud-control');
-
-const agent = 'cloud:server:00000000-0000-4000-8000-000000000001';
-const sessionId = '00000000-0000-4000-8000-0000000000aa';
-const restartId = '00000000-0000-4000-8000-0000000000bb';
-
-beforeEach(() => {
- server.operations.clear();
- server.sessions.clear();
- server.restarts = 0;
- server.loseNextResponse = false;
- server.refuseNext = null;
- server.launches = [];
- server.machines = [];
- server.machineActions = [];
- server.wakeRace = false;
- server.relayClients = 0;
-});
-
-const machineId = '3f1c2b4a-0000-4000-8000-000000000001';
-
-function launch(requestId: string, overrides: Record<string, unknown>) {
- return {
- request_id: requestId,
- name: 'reviewer',
- provider: 'claude',
- state: 'ready',
- desired_state: 'running',
- revision: 1,
- agent_id: 'agent',
- error: null,
- error_code: null,
- sleeping: false,
- machine_id: null,
- process_state: null,
- process_restarts: 0,
- oom_kills: 0,
- ...overrides,
- };
-}
-
-function machine(overrides: Record<string, unknown>) {
- return {
- machine_id: machineId,
- state: 'ready',
- desired_state: 'running',
- stop_reason: null,
- sleeping: false,
- revision: 4,
- instance_type: null,
- error: null,
- error_code: null,
- retain_until: null,
- heartbeat_at: null,
- disk: null,
- memory: null,
- agents: [],
- ...overrides,
- };
-}
-
-it('lists cloud agents from the launch list without asking any worker', async () => {
- server.launches = [
- launch('00000000-0000-4000-8000-000000000001', {}),
- launch('00000000-0000-4000-8000-000000000002', { sleeping: true, state: 'stopped' }),
- ];
- const agents = await listCloudAgents('server');
- expect(server.relayClients).toBe(0);
- expect(agents?.map((each) => [each.sessions, each.problem?.code ?? null])).toEqual([
- [null, null],
- [null, 'worker_sleeping'],
- ]);
-});
-
-it('attaches each launch’s machine and reads it first', async () => {
- server.machines = [machine({})];
- server.launches = [launch('00000000-0000-4000-8000-000000000001', { machine_id: machineId })];
- const [listed] = (await listCloudAgents('server'))!;
- expect(listed?.machine?.machine_id).toBe(machineId);
- expect(listed?.problem).toBeNull();
-});
-
-it.each([
- [
- 'worker_sleeping',
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'idle', sleeping: true }),
- {},
- true,
- ],
- [
- 'machine_stopped',
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'owner' }),
- {},
- false,
- ],
- ['worker_waking', machine({ state: 'provisioning' }), {}, false],
- ['worker_waking', machine({ state: 'retained' }), {}, false],
- ['agent_stopped', machine({}), { desired_state: 'stopped', state: 'stopped' }, false],
- [
- 'agent_stopped',
- machine({ state: 'provisioning' }),
- { desired_state: 'stopped', state: 'stopped' },
- false,
- ],
- [
- 'agent_crashed',
- machine({ state: 'provisioning' }),
- { state: 'error', error_code: 'agent_crashed', error: 'The agent crashed 5 times.' },
- false,
- ],
- [
- 'worker_sleeping',
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'idle', sleeping: true }),
- { desired_state: 'stopped', state: 'stopped' },
- false,
- ],
- [
- 'worker_sleeping',
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'idle', sleeping: true }),
- { state: 'error', error_code: 'agent_crashed', error: 'The agent crashed 5 times.' },
- false,
- ],
- ['machine_error', machine({ state: 'error', error: 'The machine did not connect.' }), {}, false],
- [
- 'agent_crashed',
- machine({}),
- { state: 'error', error_code: 'agent_crashed', error: 'The agent crashed 5 times.' },
- false,
- ],
-])('reports %s', async (code, onMachine, overrides, wakeAvailable) => {
- server.machines = [onMachine];
- server.launches = [
- launch('00000000-0000-4000-8000-000000000001', { machine_id: machineId, ...overrides }),
- ];
- const [listed] = (await listCloudAgents('server'))!;
- expect(listed?.problem).toMatchObject({ code, wakeAvailable });
-});
-
-it('lists a launch whose identity registration failed, so it can be retried or removed', async () => {
- server.machines = [machine({})];
- server.launches = [
- launch('00000000-0000-4000-8000-000000000001', {
- machine_id: machineId,
- agent_id: null,
- state: 'error',
- error: 'registration failed',
- error_code: 'identity_failed',
- }),
- ];
- const [listed] = (await listCloudAgents('server'))!;
- expect(listed?.launch).toMatchObject({ agent_id: null, error_code: 'identity_failed' });
- expect(listed?.problem?.code).toBe('worker_not_attached');
-});
-
-it('lists a removal left halfway so it can be finished, and hides a finished one', async () => {
- server.machines = [machine({})];
- server.launches = [
- launch('00000000-0000-4000-8000-000000000001', {
- machine_id: machineId,
- state: 'deleting',
- desired_state: 'deleted',
- }),
- launch('00000000-0000-4000-8000-000000000002', {
- machine_id: machineId,
- state: 'deleted',
- desired_state: 'deleted',
- }),
- ];
- const agents = (await listCloudAgents('server'))!;
- expect(agents.map((each) => [each.launch.request_id, each.problem?.code])).toEqual([
- ['00000000-0000-4000-8000-000000000001', 'worker_not_attached'],
- ]);
-});
-
-it('wakes an agent by starting its machine at the machine’s revision', async () => {
- server.machines = [
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'idle', sleeping: true }),
- ];
- server.launches = [launch('00000000-0000-4000-8000-000000000001', { machine_id: machineId })];
- const woken = await wakeCloudAgent(agent);
- expect(server.machineActions).toEqual([{ action: 'start', revision: 4 }]);
- expect(woken).toMatchObject({ desired_state: 'running', revision: 5 });
-});
-
-it('does not start a machine already asked to run', async () => {
- server.machines = [machine({ state: 'provisioning' })];
- server.launches = [launch('00000000-0000-4000-8000-000000000001', { machine_id: machineId })];
- expect(await wakeCloudAgent(agent)).toMatchObject({ state: 'provisioning', revision: 4 });
- expect(server.machineActions).toEqual([]);
-});
-
-it('reports a start whose response was lost as unknown, and the same id again starts one session', async () => {
- server.loseNextResponse = true;
- const first = await runCloudSessionOperation(agent, sessionId, sessionId, 'start');
- expect(first.state).toBe('unknown');
- expect(await runCloudSessionOperation(agent, sessionId, sessionId, 'start')).toEqual({
- state: 'applied',
- });
- expect([...server.sessions]).toEqual([sessionId]);
-});
-
-it('reports a restart whose response was lost as unknown, and the same id again restarts once', async () => {
- server.loseNextResponse = true;
- expect((await runCloudSessionOperation(agent, sessionId, restartId, 'restart')).state).toBe(
- 'unknown'
- );
- expect(await runCloudSessionOperation(agent, sessionId, restartId, 'restart')).toEqual({
- state: 'applied',
- });
- expect(server.restarts).toBe(1);
-});
-
-it('reports a refusal the server answered as a definite failure', async () => {
- server.refuseNext = { status: 409, detail: 'Start the cloud worker and wait until it is ready.' };
- expect(await runCloudSessionOperation(agent, sessionId, sessionId, 'start')).toEqual({
- state: 'failed',
- message: 'Start the cloud worker and wait until it is ready.',
- code: null,
- });
-});
-
-it.each([
- ['worker_waking', 'The cloud machine is starting. Try again in a moment.'],
- ['machine_stopped', 'The owner stopped the cloud machine. Start it in Switch Console.'],
- ['machine_error', 'The cloud machine needs attention. Retry it in Switch Console.'],
-])('reports a %s refusal with its code', async (code, detail) => {
- server.refuseNext = { status: 409, detail, code };
- expect(await runCloudSessionOperation(agent, sessionId, restartId, 'restart')).toEqual({
- state: 'failed',
- message: detail,
- code,
- });
- expect(server.restarts).toBe(0);
-});
-
-it('refuses a start whose operation id is not its session id', async () => {
- await expect(runCloudSessionOperation(agent, sessionId, restartId, 'start')).rejects.toThrow(
- /identified by its session id/
- );
-});
-
-it('does not wake a machine its owner stopped', async () => {
- server.machines = [machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'owner' })];
- server.launches = [launch('00000000-0000-4000-8000-000000000001', { machine_id: machineId })];
- await expect(wakeCloudAgent(agent)).rejects.toThrow('The owner stopped the cloud machine');
- expect(server.machineActions).toEqual([]);
-});
-
-it('reads a wake that lost the revision race to another wake as the machine waking', async () => {
- server.machines = [
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'idle', sleeping: true }),
- ];
- server.launches = [launch('00000000-0000-4000-8000-000000000001', { machine_id: machineId })];
- server.wakeRace = true;
- expect(await wakeCloudAgent(agent)).toMatchObject({ desired_state: 'running', revision: 5 });
-});
-
-describe('the sessions last read from a worker', () => {
- const session = { sessionId: 'b4105d35-0000', status: 'ready', connectivity: 'online' };
- const asleep = machine({
- state: 'stopped',
- desired_state: 'stopped',
- stop_reason: 'idle',
- sleeping: true,
- });
- const onMachine = launch('00000000-0000-4000-8000-000000000001', { machine_id: machineId });
-
- function relayRefuses(relayCode: string) {
- server.relayList = async () => {
- throw new CloudRelayError(relayCode, 'Refused.', 409, false);
- };
- }
-
- beforeEach(async () => {
- kvRows.clear();
- server.machines = [machine({})];
- server.launches = [onMachine];
- server.relayList = async () => [session];
- await expect(listCloudSessions(agent)).resolves.toEqual({
- sessions: [session],
- problem: null,
- });
- });
-
- it('are listed after a restart while the machine is asleep or stopped', async () => {
- vi.resetModules();
- const restarted = await import('./cloud-control');
- server.machines = [asleep];
- expect((await restarted.listCloudAgents('server'))?.[0]).toMatchObject({
- sessions: [session],
- problem: { code: 'worker_sleeping' },
- });
- server.machines = [
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'owner' }),
- ];
- expect((await restarted.listCloudAgents('server'))?.[0]).toMatchObject({
- sessions: [session],
- problem: { code: 'machine_stopped' },
- });
- });
-
- it('stay beside a relay that answers the machine is asleep', async () => {
- relayRefuses('worker_sleeping');
- await expect(listCloudSessions(agent)).resolves.toMatchObject({
- sessions: [session],
- problem: { code: 'worker_sleeping' },
- });
- });
-
- it('are not offered for a worker that is down for another reason', async () => {
- relayRefuses('agent_crashed');
- expect((await listCloudSessions(agent)).sessions).toBeNull();
- server.launches = [
- launch('00000000-0000-4000-8000-000000000001', {
- machine_id: machineId,
- state: 'error',
- error_code: 'agent_crashed',
- error: 'crashed',
- }),
- ];
- expect((await listCloudAgents('server'))?.[0]?.sessions).toBeNull();
- });
-
- it('are forgotten once the agent is gone', async () => {
- server.launches = [];
- await listCloudAgents('server');
- expect(kvRows.size).toBe(0);
- });
-});
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control.ts
deleted file mode 100644
index 204b8d6cd..000000000
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/cloud-control.ts
+++ /dev/null
@@ -1,430 +0,0 @@
-import { setTimeout as delay } from 'node:timers/promises';
-import {
- CloudRelayClient,
- CloudRelayError,
- RELAY_TIMEOUT_MS,
-} from '@switch-console/agent-providers';
-import type { Attachment, Session } from '@switch-console/shared/session-v1';
-import { z } from 'zod';
-import {
- GatewayError,
- gatewayFetch,
- gatewayRequest,
-} from '@main/core/switch-servers/gateway-client';
-import { getServer } from '@main/core/switch-servers/servers-store';
-import { withServerWorkspaceSession } from '@main/core/workspaces/workspace-session';
-import { KV } from '@main/db/kv';
-import {
- type CloudAgent,
- cloudAgentKey,
- cloudAgentPhase,
- type CloudLaunch,
- cloudLaunchSchema,
- type CloudMachine,
- cloudMachineSchema,
- type CloudOperation,
- type CloudOperationOutcome,
- cloudOperationSchema,
- type CloudRelayProblem,
- type CloudSessions,
- parseCloudAgentKey,
-} from '@shared/core/cloud-agents/cloud-agents';
-import type { SwitchServer } from '@shared/core/switch-servers/switch-servers';
-
-/**
- * Console's reach into a cloud agent's worker, through its Switch server.
- *
- * A sidecar is reached over SSH on its control port; a cloud worker accepts no
- * connection, so the same control messages go through the server's relay for
- * the launch. One relay client per launch, made on first use and again after
- * it closes. Transcripts stay on the worker: every snapshot, list and event
- * is asked of it through the relay.
- */
-
-export function isCloudAgent(agentId: string): boolean {
- return parseCloudAgentKey(agentId) !== null;
-}
-
-function launchOf(agentId: string): { serverId: string; requestId: string } {
- const key = parseCloudAgentKey(agentId);
- if (!key) throw new Error(`${agentId} is not a cloud agent.`);
- return key;
-}
-
-async function requireCloudServer(serverId: string): Promise<void> {
- if (!(await getServer(serverId)))
- throw new Error('The Switch server for this cloud agent was removed.');
-}
-
-/**
- * Run `fn` against the cloud agent's server with its workspace's tenant
- * selected: launches, machines and their operations belong to a tenant, so a
- * call that went out under another one would answer for that one instead.
- * One lease per request, never around a wait, so a workspace switch is not
- * held up by a long poll.
- */
-async function onCloudServer<T>(
- serverId: string,
- fn: (server: SwitchServer) => Promise<T>
-): Promise<T> {
- await requireCloudServer(serverId);
- return withServerWorkspaceSession(serverId, fn);
-}
-
-function launchPath(requestId: string, rest = ''): string {
- return `/hosted-launches/${encodeURIComponent(requestId)}${rest}`;
-}
-
-function machinePath(machineId: string, rest = ''): string {
- return `/hosted-machines/${encodeURIComponent(machineId)}${rest}`;
-}
-
-const clients = new Map<string, CloudRelayClient>();
-
-/**
- * The sessions last read from each cloud agent's worker, by agent key, kept
- * across restarts: while its machine is stopped, asleep or waking they stay
- * listed beside why, since opening one is how its user wakes it.
- */
-const lastSessions = new KV<Record<string, Session[]>>('cloud-sessions');
-const KEEPS_LAST_SESSIONS = new Set(['machine_stopped', 'worker_sleeping', 'worker_waking']);
-
-/** The relay client for this cloud agent's worker, made if there is none. */
-export async function cloudControl(agentId: string): Promise<CloudRelayClient> {
- const existing = clients.get(agentId);
- if (existing && !existing.isClosed) return existing;
- const { serverId, requestId } = launchOf(agentId);
- await requireCloudServer(serverId);
- const client = new CloudRelayClient(
- (path, init) =>
- withServerWorkspaceSession(serverId, (server) =>
- gatewayRequest(server, launchPath(requestId, path), {
- authenticated: true,
- method: init.method,
- body: init.body,
- signal: init.signal,
- })
- ),
- { retryMs: 20_000, timeoutMs: RELAY_TIMEOUT_MS }
- );
- client.onClose(() => {
- if (clients.get(agentId) === client) clients.delete(agentId);
- });
- clients.set(agentId, client);
- return client;
-}
-
-/**
- * The server's launches, or null when it has no launch list at all: a Core
- * without cloud agents answers the route with 404.
- */
-export async function listCloudLaunches(server: SwitchServer): Promise<CloudLaunch[] | null> {
- let response: Response;
- try {
- response = await gatewayFetch(server, '/hosted-launches', { authenticated: true });
- } catch (error) {
- if (error instanceof GatewayError && error.kind === 'http' && error.status === 404) return null;
- throw error;
- }
- return z.array(cloudLaunchSchema).parse(await response.json());
-}
-
-/**
- * The caller's cloud machines. Unlike the launch list, a 404 here is not read
- * as "no cloud agents": a server that lists launches but not machines is one
- * this Console does not match.
- */
-export async function listCloudMachines(server: SwitchServer): Promise<CloudMachine[]> {
- const response = await gatewayFetch(server, '/hosted-machines', { authenticated: true });
- return z.object({ machines: z.array(cloudMachineSchema) }).parse(await response.json()).machines;
-}
-
-/** The caller's cloud machines, or null when the server has no cloud agents. */
-export async function listServerCloudMachines(serverId: string): Promise<CloudMachine[] | null> {
- return onCloudServer(serverId, async (server) => {
- if ((await listCloudLaunches(server)) === null) return null;
- return listCloudMachines(server);
- });
-}
-
-/**
- * Why a launch's worker cannot be asked, machine first, or null when it can.
- * Read the way the server answers a read-only relay, which reports a sleeping
- * machine whatever the launch's own state.
- */
-function launchProblem(
- launch: CloudLaunch,
- machine: CloudMachine | null
-): CloudRelayProblem | null {
- if (launch.desired_state === 'deleted')
- return {
- code: 'worker_not_attached',
- message: 'The cloud agent is being removed.',
- wakeAvailable: false,
- };
- const phase = cloudAgentPhase(launch, machine);
- if (phase === 'machine_stopped')
- return {
- code: 'machine_stopped',
- message: 'The owner stopped the cloud machine.',
- wakeAvailable: false,
- };
- if (phase === 'machine_error')
- return {
- code: 'machine_error',
- message: machine?.error ?? 'The cloud machine is in error.',
- wakeAvailable: false,
- };
- if (machine ? machine.sleeping : launch.sleeping)
- return {
- code: 'worker_sleeping',
- message: 'The cloud machine is asleep.',
- wakeAvailable: phase === 'sleeping',
- };
- if (phase === 'waking')
- return {
- code: 'worker_waking',
- message: 'The cloud machine is starting.',
- wakeAvailable: false,
- };
- if (launch.desired_state === 'stopped')
- return { code: 'agent_stopped', message: 'The agent is stopped.', wakeAvailable: false };
- if (launch.state === 'error' && launch.error_code === 'agent_crashed')
- return {
- code: 'agent_crashed',
- message: launch.error ?? 'The agent crashed.',
- wakeAvailable: false,
- };
- if (launch.state !== 'ready' && launch.state !== 'running')
- return {
- code: 'worker_not_attached',
- message: `The cloud worker is ${launch.state}${launch.error ? `: ${launch.error}` : '.'}`,
- wakeAvailable: false,
- };
- return null;
-}
-
-/**
- * The server's cloud agents, read from the launch and machine lists, or null
- * when the server has no cloud agents. A launch whose worker cannot be asked
- * says why; the sessions of one that can are asked of its worker by
- * `listCloudSessions`, only for the agents being looked at.
- */
-export async function listCloudAgents(serverId: string): Promise<CloudAgent[] | null> {
- const listed = await onCloudServer(serverId, listCloudLaunches);
- if (listed === null) return null;
- const launches = listed.filter(
- (launch) =>
- launch.state !== 'deleted' &&
- (launch.desired_state !== 'deleted' || launch.state === 'deleting')
- );
- const stored = await lastSessions.getAll();
- const keys = new Set(launches.map((launch) => cloudAgentKey(serverId, launch.request_id)));
- for (const key of Object.keys(stored))
- if (parseCloudAgentKey(key)?.serverId === serverId && !keys.has(key))
- await lastSessions.del(key);
- if (launches.length === 0) return [];
- const machines = new Map(
- (await onCloudServer(serverId, listCloudMachines)).map((machine) => [
- machine.machine_id,
- machine,
- ])
- );
- return launches.map((launch): CloudAgent => {
- const machine = launch.machine_id === null ? null : (machines.get(launch.machine_id) ?? null);
- const key = cloudAgentKey(serverId, launch.request_id);
- const problem = launchProblem(launch, machine);
- return {
- key,
- launch,
- machine,
- sessions: problem && KEEPS_LAST_SESSIONS.has(problem.code) ? (stored[key] ?? null) : null,
- problem,
- };
- });
-}
-
-/**
- * A cloud agent's sessions, asked of its worker over the relay. A worker that
- * cannot be asked is reported with the relay's code rather than as no sessions,
- * beside the sessions last read while its machine is down.
- */
-export async function listCloudSessions(agentId: string): Promise<CloudSessions> {
- let sessions: Session[];
- try {
- sessions = await (await cloudControl(agentId)).list();
- } catch (error) {
- const problem: CloudRelayProblem =
- error instanceof CloudRelayError
- ? {
- code: error.relayCode,
- message: error.message,
- wakeAvailable: error.wakeAvailable,
- }
- : {
- code: 'failed',
- message: error instanceof Error ? error.message : String(error),
- wakeAvailable: false,
- };
- return {
- sessions: KEEPS_LAST_SESSIONS.has(problem.code) ? await lastSessions.get(agentId) : null,
- problem,
- };
- }
- await lastSessions.set(agentId, sessions);
- return { sessions, problem: null };
-}
-
-/**
- * Start the machine a sleeping cloud agent runs on. Only a machine that went to
- * sleep idle is woken: one its owner stopped is started from its card. A
- * machine already asked to run is returned as it is, so a second wake does not
- * bump its revision, and a wake that loses the revision race to another one is
- * the machine waking.
- */
-export async function wakeCloudAgent(agentId: string): Promise<CloudMachine> {
- const { serverId, requestId } = launchOf(agentId);
- const launch = cloudLaunchSchema.parse(
- await onCloudServer(serverId, async (server) =>
- (await gatewayFetch(server, launchPath(requestId), { authenticated: true })).json()
- )
- );
- if (launch.machine_id === null)
- throw new Error(`The cloud agent ${launch.name} has no machine to start.`);
- const machineId = launch.machine_id;
- const read = async () =>
- cloudMachineSchema.parse(
- await onCloudServer(serverId, async (server) =>
- (await gatewayFetch(server, machinePath(machineId), { authenticated: true })).json()
- )
- );
- const machine = await read();
- if (machine.desired_state === 'running') return machine;
- if (!machine.sleeping)
- throw new Error(
- machine.desired_state === 'stopped' && machine.stop_reason === 'owner'
- ? 'The owner stopped the cloud machine, so a message does not wake it. Start the machine in Your Agents.'
- : `The cloud machine is ${machine.desired_state}, so it cannot be woken.`
- );
- try {
- return z.object({ machine: cloudMachineSchema }).parse(
- await onCloudServer(serverId, async (server) =>
- (
- await gatewayFetch(server, machinePath(machineId, '/lifecycle'), {
- authenticated: true,
- method: 'POST',
- body: { action: 'start', revision: machine.revision },
- })
- ).json()
- )
- ).machine;
- } catch (error) {
- if (!(error instanceof GatewayError && error.kind === 'http' && error.status === 409))
- throw error;
- const now = await read();
- if (now.desired_state === 'running') return now;
- throw error;
- }
-}
-
-const OPERATION_WAIT_MS = 180_000;
-
-function errorMessage(error: unknown): string {
- return error instanceof Error ? error.message : String(error);
-}
-
-/** A refusal the server answered, so it holds no operation for this request. */
-function isDefiniteRefusal(error: unknown): error is GatewayError {
- return (
- error instanceof GatewayError &&
- (error.kind === 'unauthorized' ||
- (error.kind === 'http' && error.status !== undefined && error.status < 500))
- );
-}
-
-/**
- * Ask the worker to start a new session (`start`) or run an existing one
- * again (`restart`), and wait until it says it has. `operationId` is the
- * attempt's identity: after an `unknown` outcome, ask again with the same id
- * and the server returns the operation it already holds instead of queueing
- * another. A start's id is its session id.
- */
-export async function runCloudSessionOperation(
- agentId: string,
- sessionId: string,
- operationId: string,
- action: 'start' | 'restart'
-): Promise<CloudOperationOutcome> {
- if (action === 'start' && operationId !== sessionId)
- throw new Error('A cloud session start is identified by its session id.');
- const { serverId, requestId } = launchOf(agentId);
- let operation: CloudOperation;
- try {
- operation = cloudOperationSchema.parse(
- await onCloudServer(serverId, async (server) =>
- (
- await gatewayFetch(server, launchPath(requestId, '/sessions'), {
- authenticated: true,
- method: 'POST',
- body: { id: operationId, session_id: sessionId, action },
- })
- ).json()
- )
- );
- } catch (error) {
- if (isDefiniteRefusal(error))
- return { state: 'failed', message: error.detail ?? error.message, code: error.code ?? null };
- return {
- state: 'unknown',
- message: `The server did not confirm the session ${action}: ${errorMessage(error)}`,
- };
- }
- const deadline = Date.now() + OPERATION_WAIT_MS;
- try {
- while (operation.state === 'queued' || operation.state === 'claimed') {
- if (Date.now() >= deadline)
- return {
- state: 'unknown',
- message: `The cloud worker has not confirmed the session ${action} yet.`,
- };
- await delay(1000);
- operation = cloudOperationSchema.parse(
- await onCloudServer(serverId, async (server) =>
- (
- await gatewayFetch(
- server,
- launchPath(requestId, `/sessions/${encodeURIComponent(operationId)}`),
- { authenticated: true }
- )
- ).json()
- )
- );
- }
- } catch (error) {
- return {
- state: 'unknown',
- message: `The session ${action} could not be followed: ${errorMessage(error)}`,
- };
- }
- if (operation.state === 'applied') return { state: 'applied' };
- if (operation.state === 'failed')
- return {
- state: 'failed',
- message: operation.error ?? `The session ${action} failed.`,
- code: null,
- };
- return { state: 'unknown', message: `The outcome of the session ${action} is unknown.` };
-}
-
-/** Stage a file on the session's worker for the next message to name. */
-export async function uploadCloudAttachment(
- agentId: string,
- sessionId: string,
- file: { name: string; mimeType: string; data: string }
-): Promise<Attachment> {
- return (await cloudControl(agentId)).uploadAttachment(sessionId, {
- name: file.name,
- mimeType: file.mimeType,
- data: Buffer.from(file.data, 'base64'),
- });
-}
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/controller.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/controller.ts
index d8905815e..a6df2edc4 100644
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/controller.ts
+++ b/console/apps/switch-console-desktop/src/main/core/sdk-host/controller.ts
@@ -3,14 +3,6 @@ import { z } from 'zod';
import { getAgentById } from '@main/core/agents/getAgentById';
import { remoteSessionReconciler } from '@main/core/agents/remote-session-reconciler';
import { createRPCController } from '@shared/lib/ipc/rpc';
-import {
- listCloudAgents,
- listCloudSessions,
- listServerCloudMachines,
- runCloudSessionOperation,
- uploadCloudAttachment,
- wakeCloudAgent,
-} from './cloud-control';
import { connectionHealth } from './connection-health';
import { sharedAgentDiagnostics, sharedAgentLogs } from './diagnostics';
import { sessionIssue, sessionStartupStatus } from './host-failures';
@@ -51,25 +43,4 @@ export const sdkHostController = createRPCController({
reconcileSessionCommand(agentId, command),
sessionCommandStatus: (agentId: string, sessionId: string, commandId: string) =>
sessionCommandStatus(agentId, sessionId, commandId),
- cloudAgents: (serverId: string) => listCloudAgents(serverId),
- cloudMachines: (serverId: string) => listServerCloudMachines(serverId),
- cloudSessions: (agentId: string) => listCloudSessions(agentId),
- cloudWake: (agentId: string) => wakeCloudAgent(agentId),
- cloudSessionOperation: async (
- agentId: string,
- sessionId: string,
- operationId: string,
- action: 'start' | 'restart'
- ) =>
- runCloudSessionOperation(
- agentId,
- sessionId,
- z.string().uuid().parse(operationId),
- z.enum(['start', 'restart']).parse(action)
- ),
- cloudUploadAttachment: (
- agentId: string,
- sessionId: string,
- file: { name: string; mimeType: string; data: string }
- ) => uploadCloudAttachment(agentId, sessionId, file),
});
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/local-host.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/local-host.ts
index 7a8026c56..b8aec85e3 100644
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/local-host.ts
+++ b/console/apps/switch-console-desktop/src/main/core/sdk-host/local-host.ts
@@ -205,7 +205,6 @@ export const consoleSupervision: Supervision = {
signal,
build: bundle,
links: localSessionLinks,
- logRedactions: [],
}),
'Local SDK host supervisor stopped',
// The supervisor records a worker's own failure under this root already.
@@ -321,7 +320,6 @@ export async function startLocalWatcher(
signal,
consoleSupervision,
localWatcherControl(written.session.agentId),
- null,
openSwitchStream
),
'Local room watcher stopped',
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/place-session.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/place-session.ts
index e19ce978d..bea6dd605 100644
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/place-session.ts
+++ b/console/apps/switch-console-desktop/src/main/core/sdk-host/place-session.ts
@@ -1,7 +1,6 @@
import type { PlaceOutcome } from '@switch-console/agent-providers';
import { getAgentLocation } from '@main/core/agents/agent-location';
import { getAgentById } from '@main/core/agents/getAgentById';
-import { cloudControl, isCloudAgent } from './cloud-control';
import { localWatcherControl } from './local-host';
import { withSidecar } from './sidecar-control';
@@ -17,7 +16,6 @@ export async function placeSession(
sessionId: string,
roomId: string
): Promise<PlaceOutcome> {
- if (isCloudAgent(agentId)) return (await cloudControl(agentId)).place(sessionId, roomId);
const agent = await getAgentById(agentId);
if (!agent?.switchAgentId) throw new Error('This agent is not linked to Switch.');
if (!(await getAgentLocation(agent)).sshHost)
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/session-commands.test.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/session-commands.test.ts
index c605dcb5f..c0f2c8830 100644
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/session-commands.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/sdk-host/session-commands.test.ts
@@ -6,25 +6,12 @@ const mocks = vi.hoisted(() => ({
sshHost: null as string | null,
running: true,
hydrate: vi.fn(async () => {}),
- cloud: { request: vi.fn() },
- cloudOperation: vi.fn(async () => ({ state: 'applied' })),
}));
-const { Unavailable, Failed, RelayError } = vi.hoisted(() => ({
+const { Unavailable, Failed } = vi.hoisted(() => ({
Unavailable: class extends Error {},
Failed: class extends Error {},
- RelayError: class extends Error {
- constructor(
- readonly relayCode: string,
- message: string,
- readonly status: number,
- readonly wakeAvailable: boolean
- ) {
- super(message);
- }
- },
}));
vi.mock('@switch-console/agent-providers', () => ({
- CloudRelayError: RelayError,
SessionUnavailableError: Unavailable,
SessionHostFailedError: Failed,
sharedSessionRoot: (id: string) => `/roots/${id}`,
@@ -37,11 +24,6 @@ vi.mock('@main/core/agents/agent-location', () => ({
getAgentLocation: async () => ({ sshHost: mocks.sshHost }),
}));
vi.mock('./local-host', () => ({ localSessionLinks: mocks.local }));
-vi.mock('./cloud-control', () => ({
- isCloudAgent: (agentId: string) => agentId.startsWith('cloud:'),
- cloudControl: async () => mocks.cloud,
- runCloudSessionOperation: mocks.cloudOperation,
-}));
vi.mock('./sidecar-control', () => ({
withSidecar: async (_agentId: string, call: (client: unknown) => unknown) => call(mocks.sidecar),
}));
@@ -157,44 +139,3 @@ it('starts a session whose host failed again when the user sends it something',
expect(await submitSessionCommand('agent', command)).toEqual(applied);
expect(mocks.hydrate).toHaveBeenCalledWith('session');
});
-
-it('sends a cloud command through the relay and restarts a parked cloud session there', async () => {
- mocks.cloud.request
- .mockRejectedValueOnce(new Unavailable('The session host is not running.'))
- .mockResolvedValueOnce(applied);
- expect(await submitSessionCommand('cloud:server:launch', command)).toEqual(applied);
- expect(mocks.cloud.request).toHaveBeenCalledWith(
- 'session',
- expect.objectContaining({ type: 'command' })
- );
- expect(mocks.cloudOperation).toHaveBeenCalledWith(
- 'cloud:server:launch',
- 'session',
- expect.any(String),
- 'restart'
- );
- expect(mocks.hydrate).not.toHaveBeenCalled();
- expect(mocks.local.request).not.toHaveBeenCalled();
-});
-
-it.each(['worker_waking', 'machine_stopped', 'machine_error'])(
- 'raises a %s refusal of the cloud restart as that relay code, not sending the command',
- async (code) => {
- mocks.cloud.request.mockRejectedValueOnce(new Unavailable('The session host is not running.'));
- mocks.cloudOperation.mockResolvedValueOnce({
- state: 'failed',
- message: 'The cloud machine is starting. Try again in a moment.',
- code,
- } as never);
- const error = await submitSessionCommand('cloud:server:launch', command).catch(
- (caught: unknown) => caught
- );
- expect(error).toBeInstanceOf(RelayError);
- expect(error).toMatchObject({
- relayCode: code,
- message:
- 'The session is not running and could not be started again: The cloud machine is starting. Try again in a moment.',
- });
- expect(mocks.cloud.request).toHaveBeenCalledTimes(1);
- }
-);
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/session-commands.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/session-commands.ts
index 050f7105c..49f4cf01b 100644
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/session-commands.ts
+++ b/console/apps/switch-console-desktop/src/main/core/sdk-host/session-commands.ts
@@ -1,5 +1,4 @@
import {
- CloudRelayError,
liveSupervisor,
SessionHostFailedError,
SessionUnavailableError,
@@ -15,7 +14,6 @@ import {
import { getAgentLocation } from '@main/core/agents/agent-location';
import { getAgentById } from '@main/core/agents/getAgentById';
import { hydrateSession } from '@main/core/sessions/operations/hydrateSession';
-import { cloudControl, isCloudAgent, runCloudSessionOperation } from './cloud-control';
import { localSessionLinks } from './local-host';
import { withSidecar } from './sidecar-control';
@@ -23,10 +21,10 @@ import { withSidecar } from './sidecar-control';
* Commands for a shared session, sent to its host directly.
*
* A local session's host is Console's child; a remote one's is a child of
- * the agent's sidecar, which Console reaches over SSH, or of a cloud worker,
- * reached through its Switch server's relay. Either way the command goes down
- * the host's IPC pipe and the host answers with what it recorded. A host that parked itself after sitting idle is
- * started again and the command sent once it is back.
+ * the agent's sidecar, which Console reaches over SSH. Either way the command
+ * goes down the host's IPC pipe and the host answers with what it recorded. A
+ * host that parked itself after sitting idle is started again and the command
+ * sent once it is back.
*/
export class CommandNotRecordedError extends Error {
@@ -51,7 +49,6 @@ export async function askHost(
sessionId: string,
request: SessionRequest
): Promise<unknown> {
- if (isCloudAgent(agentId)) return (await cloudControl(agentId)).request(sessionId, request);
if (await isLocal(agentId)) {
const root = sharedSessionRoot(sessionId);
// Waits for a host that is starting, not for one nothing is running.
@@ -89,21 +86,8 @@ export async function submitSessionCommand(
}
const notRestarted = 'The session is not running and could not be started again: ';
try {
- if (isCloudAgent(agentId)) {
- const outcome = await runCloudSessionOperation(
- agentId,
- command.sessionId,
- crypto.randomUUID(),
- 'restart'
- );
- // Coded the way the relay codes the same refusal, so the composer holds
- // the message while the machine wakes or says what the user must do.
- if (outcome.state === 'failed' && outcome.code !== null)
- throw new CloudRelayError(outcome.code, `${notRestarted}${outcome.message}`, 409, false);
- if (outcome.state !== 'applied') throw new Error(outcome.message);
- } else await hydrateSession(command.sessionId);
+ await hydrateSession(command.sessionId);
} catch (error) {
- if (error instanceof CloudRelayError) throw error;
throw new Error(`${notRestarted}${error instanceof Error ? error.message : String(error)}`);
}
return commandStatusSchema.parse(await askHost(agentId, command.sessionId, request));
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/stop-shared-session.test.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/stop-shared-session.test.ts
index f4a111029..21d4d2087 100644
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/stop-shared-session.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/sdk-host/stop-shared-session.test.ts
@@ -35,11 +35,6 @@ vi.mock('@main/core/sessions/operations/hydrateSession', () => ({
}));
vi.mock('@main/lib/logger', () => ({ log: { warn: vi.fn() } }));
vi.mock('./local-host', () => ({ localSessionLinks: mocks.local }));
-vi.mock('./cloud-control', () => ({
- isCloudAgent: () => false,
- cloudControl: vi.fn(),
- runCloudSessionOperation: vi.fn(),
-}));
vi.mock('./sidecar-control', () => ({ withSidecar: mocks.withSidecar }));
vi.mock('./session-activity', () => ({ syncSdkSessionActivity: vi.fn(async () => {}) }));
vi.mock('./host-journal', () => ({
diff --git a/console/apps/switch-console-desktop/src/main/core/sdk-host/transcripts.ts b/console/apps/switch-console-desktop/src/main/core/sdk-host/transcripts.ts
index 3c364be0f..95c606802 100644
--- a/console/apps/switch-console-desktop/src/main/core/sdk-host/transcripts.ts
+++ b/console/apps/switch-console-desktop/src/main/core/sdk-host/transcripts.ts
@@ -14,7 +14,6 @@ import {
sessionTranscriptEventChannel,
sessionTranscriptResetChannel,
} from '@shared/core/sessions/sessionEvents';
-import { cloudControl, isCloudAgent } from './cloud-control';
import { recordRemoteHostFailure } from './host-failures';
import { hostJournals, JournalTail, JournalUnavailableError } from './host-journal';
import { localSessionLinks } from './local-host';
@@ -25,8 +24,8 @@ import { withSidecar } from './sidecar-control';
/**
* A shared session's transcript, pushed to the windows showing it.
*
- * The session's host is a child of Console (local), of the agent's sidecar
- * (remote) or of a cloud worker (reached through its Switch server). Its snapshot is asked for over the host's IPC pipe, and every
+ * The session's host is a child of Console (local) or of the agent's sidecar
+ * (remote). Its snapshot is asked for over the host's IPC pipe, and every
* event it records arrives on that pipe and is forwarded as it happens. A
* session that is not running is read from its journal, and picks up live
* again when its host starts, since the host goes on numbering the same
@@ -91,7 +90,7 @@ export async function currentSnapshot(agentId: string, sessionId: string): Promi
}
}
-type Place = 'local' | 'remote' | 'cloud';
+type Place = 'local' | 'remote';
/** The session as last recorded: from its host if it runs, else its journal. */
async function recordedSnapshot(agentId: string, sessionId: string, place: Place) {
@@ -101,7 +100,6 @@ async function recordedSnapshot(agentId: string, sessionId: string, place: Place
if (!(error instanceof JournalUnavailableError || error instanceof SessionHostFailedError))
throw error;
}
- if (place === 'cloud') return (await cloudControl(agentId)).journal(sessionId);
return place === 'local'
? localJournalSnapshot(sharedSessionRoot(sessionId))
: (await hostJournals.tail(agentId, sessionId)).snapshot();
@@ -112,61 +110,38 @@ async function recordedSnapshot(agentId: string, sessionId: string, place: Place
* return its snapshot. Events after the snapshot's `throughSequence` follow on
* `sessionTranscriptEventChannel`.
*/
-/** A cloud session's live events, relayed by its Switch server from the worker. */
-async function cloudFeed(agentId: string, sessionId: string): Promise<() => void> {
- const client = await cloudControl(agentId);
- const unsubscribe = await client.subscribe(
- sessionId,
- (event) => forward(sessionId, event),
- (failure) => recordRemoteHostFailure(sessionId, failure),
- (reason) => {
- if (open.get(sessionId)?.close !== close) return;
- open.delete(sessionId);
- events.emit(sessionTranscriptResetChannel, { sessionId, reason }, sessionId);
- }
- );
- const close = () => unsubscribe();
- return close;
-}
-
export async function openTranscript(agentId: string, sessionId: string): Promise<Snapshot> {
- const place: Place = isCloudAgent(agentId)
- ? 'cloud'
- : (await isLocal(agentId))
- ? 'local'
- : 'remote';
+ const place: Place = (await isLocal(agentId)) ? 'local' : 'remote';
let entry = open.get(sessionId);
if (!entry) {
const close =
- place === 'cloud'
- ? await cloudFeed(agentId, sessionId)
- : place === 'local'
- ? localSessionLinks.subscribe(sharedSessionRoot(sessionId), (event) =>
- forward(sessionId, event)
- )
- : await withSidecar(agentId, async (client) => {
- const unsubscribe = await client.subscribe(
- sessionId,
- (event) => forward(sessionId, event),
- (failure) => recordRemoteHostFailure(sessionId, failure)
+ place === 'local'
+ ? localSessionLinks.subscribe(sharedSessionRoot(sessionId), (event) =>
+ forward(sessionId, event)
+ )
+ : await withSidecar(agentId, async (client) => {
+ const unsubscribe = await client.subscribe(
+ sessionId,
+ (event) => forward(sessionId, event),
+ (failure) => recordRemoteHostFailure(sessionId, failure)
+ );
+ // Events recorded while the connection is down never arrive, so the
+ // windows showing the session reload it rather than carry on with a gap.
+ const offClose = client.onClose((error) => {
+ if (open.get(sessionId)?.close !== close) return;
+ open.delete(sessionId);
+ events.emit(
+ sessionTranscriptResetChannel,
+ { sessionId, reason: error.message },
+ sessionId
);
- // Events recorded while the connection is down never arrive, so the
- // windows showing the session reload it rather than carry on with a gap.
- const offClose = client.onClose((error) => {
- if (open.get(sessionId)?.close !== close) return;
- open.delete(sessionId);
- events.emit(
- sessionTranscriptResetChannel,
- { sessionId, reason: error.message },
- sessionId
- );
- });
- const close = () => {
- offClose();
- unsubscribe();
- };
- return close;
});
+ const close = () => {
+ offClose();
+ unsubscribe();
+ };
+ return close;
+ });
entry = { viewers: 0, close };
open.set(sessionId, entry);
}
diff --git a/console/apps/switch-console-desktop/src/main/core/switch-servers/controller.test.ts b/console/apps/switch-console-desktop/src/main/core/switch-servers/controller.test.ts
index 663932f01..69f0939fb 100644
--- a/console/apps/switch-console-desktop/src/main/core/switch-servers/controller.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/switch-servers/controller.test.ts
@@ -52,11 +52,6 @@ vi.mock('@main/core/workspaces/workspaces-store', () => ({ listWorkspacesForServ
vi.mock('@main/lib/logger', () => ({
log: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() },
}));
-vi.mock('./local-provider-sign-in', () => ({
- getLocalProviderSignIn: vi.fn(),
- localProviderAuthPath: vi.fn(),
- readLocalProviderSignIn: vi.fn(),
-}));
vi.mock('./auth', () => ({ oidcLogin: vi.fn(), passwordLogin, signup }));
// Reads this install's own agent rows, and through them the database client.
vi.mock('./backfill-agent-icons', () => ({ backfillAgentIcons: vi.fn() }));
diff --git a/console/apps/switch-console-desktop/src/main/core/switch-servers/controller.ts b/console/apps/switch-console-desktop/src/main/core/switch-servers/controller.ts
index 853c9855e..5326e4594 100644
--- a/console/apps/switch-console-desktop/src/main/core/switch-servers/controller.ts
+++ b/console/apps/switch-console-desktop/src/main/core/switch-servers/controller.ts
@@ -4,7 +4,6 @@ import { propagateServerApiUrl } from '@main/core/agents/propagate-server-api-ur
import { appService } from '@main/core/app/service';
import { embeddedControllerService } from '@main/core/embedded-controller/embedded-controllers';
import { isManagedServerRunning } from '@main/core/managed-switch-server/managed-server-status';
-import { getPlugin } from '@main/core/providers/plugin-registry';
import type { TelemetryAuthMethod, TelemetrySignInFailure } from '@main/core/telemetry/events';
import { trackEvent } from '@main/core/telemetry/telemetry-service';
import { reconcileServerWorkspaces } from '@main/core/workspaces/reconcile-workspaces';
@@ -14,15 +13,6 @@ import {
} from '@main/core/workspaces/workspace-session';
import { listWorkspacesForServer } from '@main/core/workspaces/workspaces-store';
import { log } from '@main/lib/logger';
-import type { AgentProviderId } from '@shared/core/providers/agent-provider-registry';
-import {
- validateClaudeCredential,
- type ClaudeCredentialKind,
-} from '@shared/core/switch-servers/claude-credential';
-import type {
- CloudConfigurationInput,
- CloudLaunchInput,
-} from '@shared/core/switch-servers/cloud-launch';
import {
type InviteServer,
SWITCH_CLOUD_NAME,
@@ -49,20 +39,11 @@ import { bundledChatSignInFor } from './bundled-chat-sign-in';
import {
getConnectionCatalog,
getGitHubConnection,
- createCloudLaunch,
fetchAgentIconChoices,
- cloudLifecycle,
- getCloudLaunchConfiguration,
- updateCloudLaunchConfiguration,
cloudMachineLifecycle,
ensureCloudMachine,
- getCloudProviderConnection,
- connectCloudProvider,
- disconnectCloudProvider,
+ listCloudMachines,
disconnectGitHub,
- getClaudeConnection,
- connectClaude,
- disconnectClaude,
acceptInvitation,
acceptPendingInvitation,
joinWorkspaceByDomain,
@@ -81,12 +62,6 @@ import {
confirmGitHubBrowserFlow,
cancelGitHubBrowserFlow,
} from './github-browser-flow';
-import {
- getLocalProviderSignIn,
- localProviderAuthPath,
- readLocalProviderSignIn,
- type LocalSignInProvider,
-} from './local-provider-sign-in';
import { deleteManagedClaudeCredential } from './managed-claude-credential';
import { hostUnreachable, requireReachableServer, requireServer } from './require-server';
import {
@@ -204,79 +179,15 @@ async function registerSwitchCloud({ url }: SwitchCloudEndpoint): Promise<Switch
return server;
}
-/** The agent definition a cloud launch carries, rendered by its provider like a local one. */
-function renderCloudDefinition(input: CloudConfigurationInput): string {
- const definitions = getPlugin(input.provider).behavior.repoAgents;
- return definitions
- ? definitions.renderDefinition({
- ...input.definition_attributes,
- name: input.name,
- description: input.description,
- instructions: input.instructions,
- })
- : '';
-}
-
export const switchServersController = createRPCController({
- getLocalProviderSignIn,
- connectLocalProviderSignIn: async (serverId: string, provider: LocalSignInProvider) => {
- const credential = await readLocalProviderSignIn(provider, localProviderAuthPath(provider));
- if (!credential) throw new Error('Local sign-in file is missing. Sign in locally first.');
- return withReachableServerWorkspaceSession(serverId, (server) =>
- connectCloudProvider(server, provider, 'auth-json', credential)
- );
- },
- getCloudProviderConnection: (serverId: string, provider: AgentProviderId) =>
- withReachableServerWorkspaceSession(serverId, (server) =>
- getCloudProviderConnection(server, provider)
- ),
- connectCloudProvider: (
- serverId: string,
- provider: Exclude<AgentProviderId, 'claude'>,
- kind: 'api-key' | 'auth-json',
- credential: string
- ) =>
- withReachableServerWorkspaceSession(serverId, (server) =>
- connectCloudProvider(server, provider, kind, credential)
- ),
- disconnectCloudProvider: (serverId: string, provider: Exclude<AgentProviderId, 'claude'>) =>
- withReachableServerWorkspaceSession(serverId, (server) =>
- disconnectCloudProvider(server, provider)
- ),
/** The icons the server generates for an agent called `name`, one page at a time. */
agentIconChoices: (params: { serverId: string; name: string; page: number }) =>
withReachableServerWorkspaceSession(params.serverId, (server) =>
fetchAgentIconChoices(server, params.name, params.page)
),
- createCloudLaunch: (serverId: string, input: CloudLaunchInput) =>
- withReachableServerWorkspaceSession(serverId, (server) =>
- createCloudLaunch(server, { ...input, definition: renderCloudDefinition(input) })
- ),
- getCloudLaunchConfiguration: (serverId: string, requestId: string) =>
- withReachableServerWorkspaceSession(serverId, (server) =>
- getCloudLaunchConfiguration(server, requestId)
- ),
- updateCloudLaunchConfiguration: (
- serverId: string,
- requestId: string,
- input: CloudConfigurationInput
- ) =>
- withReachableServerWorkspaceSession(serverId, (server) =>
- updateCloudLaunchConfiguration(server, requestId, {
- instructions: input.instructions,
- definition_attributes: input.definition_attributes,
- definition: renderCloudDefinition(input),
- })
- ),
- cloudLifecycle: (
- serverId: string,
- requestId: string,
- action: 'stop' | 'start' | 'restart' | 'remove' | 'retry',
- revision: number
- ) =>
- withReachableServerWorkspaceSession(serverId, (server) =>
- cloudLifecycle(server, requestId, action, revision)
- ),
+ /** The caller's cloud machines, or null when the server offers none. */
+ cloudMachines: (serverId: string) =>
+ withServerWorkspaceSession(serverId, (server) => listCloudMachines(server)),
cloudMachineLifecycle: (
serverId: string,
machineId: string,
@@ -291,14 +202,6 @@ export const switchServersController = createRPCController({
revision
)
),
- getClaudeConnection: (serverId: string) =>
- withServerWorkspaceSession(serverId, (server) => getClaudeConnection(server)),
- connectClaude: (serverId: string, kind: ClaudeCredentialKind, credential: string) => {
- const value = validateClaudeCredential(kind, credential);
- return withServerWorkspaceSession(serverId, (server) => connectClaude(server, kind, value));
- },
- disconnectClaude: (serverId: string) =>
- withServerWorkspaceSession(serverId, (server) => disconnectClaude(server)),
listServers: (): Promise<SwitchServer[]> => listServers(),
diff --git a/console/apps/switch-console-desktop/src/main/core/switch-servers/gateway-client.test.ts b/console/apps/switch-console-desktop/src/main/core/switch-servers/gateway-client.test.ts
index ced56d932..d6123561a 100644
--- a/console/apps/switch-console-desktop/src/main/core/switch-servers/gateway-client.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/switch-servers/gateway-client.test.ts
@@ -47,16 +47,13 @@ vi.mock('./console-identity', () => ({
}));
const {
- cloudLifecycle,
- getCloudLaunchConfiguration,
+ cloudMachineLifecycle,
+ listCloudMachines,
getConnectionCatalog,
getGitHubConnection,
startGitHubConnection,
completeGitHubConnection,
confirmGitHubConnection,
- getClaudeConnection,
- connectClaude,
- disconnectClaude,
acceptInvitation,
acceptPendingInvitation,
addBridgeTeam,
@@ -89,7 +86,6 @@ const {
registerKnownAgent,
updateAgentDisplayName,
updateBridge,
- updateCloudLaunchConfiguration,
AgentManagementUnavailableError,
enrollConsoleController,
fetchAdvancedConfigSchema,
@@ -100,6 +96,8 @@ const {
revokeManagementController,
updateManagementController,
fetchManagementControllers,
+ giveMachineLogin,
+ fetchMachineOperation,
fetchAgentManagementAccess,
updateCanManageAgents,
updateManagedAgent,
@@ -640,7 +638,7 @@ describe('room creation', () => {
) as never
);
- await expect(cloudLifecycle(SERVER, 'launch', 'retry', 3)).rejects.toMatchObject({
+ await expect(cloudMachineLifecycle(SERVER, 'machine', 'start', 3)).rejects.toMatchObject({
status: 409,
detail: 'The owner stopped the cloud machine. Start it in Switch Console.',
code: 'machine_stopped',
@@ -832,7 +830,7 @@ describe('ownsOwnerAddressedAgent', () => {
});
});
-describe('cloud agent edits', () => {
+describe('agent display name', () => {
beforeEach(() => {
vi.clearAllMocks();
vi.stubGlobal('fetch', fetchMock);
@@ -843,56 +841,6 @@ describe('cloud agent edits', () => {
vi.unstubAllGlobals();
});
- const configuration = {
- description: 'Reviews pull requests',
- instructions: 'Be brief.',
- definition_attributes: { model: 'opus' },
- };
-
- it('reads the configuration a launch carries', async () => {
- fetchMock.mockResolvedValue(jsonResponse(configuration) as never);
-
- await expect(getCloudLaunchConfiguration(SERVER, 'launch-1')).resolves.toEqual(configuration);
- const [url] = fetchMock.mock.calls[0] as unknown as [string];
- expect(url).toBe('https://switch.example.com/gateway/hosted-launches/launch-1/configuration');
- });
-
- it('PUTs the new instructions with the rendered definition', async () => {
- fetchMock.mockResolvedValue(jsonResponse(configuration) as never);
-
- await updateCloudLaunchConfiguration(SERVER, 'launch-1', {
- instructions: 'Be brief.',
- definition_attributes: { model: 'opus' },
- definition: '---\nname: helper\n---\nBe brief.',
- });
-
- const [url, init] = fetchMock.mock.calls[0] as unknown as [
- string,
- { method: string; body: string },
- ];
- expect(url).toBe('https://switch.example.com/gateway/hosted-launches/launch-1/configuration');
- expect(init.method).toBe('PUT');
- expect(JSON.parse(init.body)).toEqual({
- instructions: 'Be brief.',
- definition_attributes: { model: 'opus' },
- definition: '---\nname: helper\n---\nBe brief.',
- });
- });
-
- it('surfaces a refused configuration rather than reporting it saved', async () => {
- fetchMock.mockResolvedValue(
- errorResponse(409, '{"detail":"This worker has been removed."}') as never
- );
-
- await expect(
- updateCloudLaunchConfiguration(SERVER, 'launch-1', {
- instructions: '',
- definition_attributes: {},
- definition: 'x',
- })
- ).rejects.toMatchObject({ status: 409, detail: 'This worker has been removed.' });
- });
-
it('PUTs a display name, or null to clear it', async () => {
fetchMock.mockResolvedValue(
jsonResponse({
@@ -1054,73 +1002,6 @@ describe('deleteBridge', () => {
});
});
-describe('Claude cloud connection transport', () => {
- beforeEach(() => {
- vi.clearAllMocks();
- vi.stubGlobal('fetch', fetchMock);
- getSessionCookie.mockResolvedValue(makeJwt(7200));
- });
- afterEach(() => vi.unstubAllGlobals());
- it('refuses to send provider credentials over HTTP', async () => {
- await expect(
- connectClaude(
- {
- id: 'insecure',
- name: 'Insecure',
- gatewayUrl: 'http://switch.example.com',
- managed: false,
- } as never,
- 'api-key',
- 'SYNTHETIC'
- )
- ).rejects.toThrow('HTTPS');
- expect(fetchMock).not.toHaveBeenCalled();
- });
- it('sends the credential only in an authenticated PUT body', async () => {
- fetchMock.mockResolvedValueOnce(
- new Response(
- JSON.stringify({
- status: 'connected',
- kind: 'api-key',
- verified_at: '2026-01-01T00:00:00Z',
- })
- )
- );
- await connectClaude(SERVER, 'api-key', 'SYNTHETIC-CREDENTIAL');
- const [url, options] = fetchMock.mock.calls[0] as unknown as [string, RequestInit];
- expect(url).toBe('https://switch.example.com/gateway/provider-connections/claude');
- expect(options.method).toBe('PUT');
- expect(JSON.parse(options.body as string)).toEqual({
- kind: 'api-key',
- credential: 'SYNTHETIC-CREDENTIAL',
- });
- expect(cookieHeaderOf(fetchMock.mock.calls[0])).toContain('switch_auth=');
- });
- it('reads metadata and deletes without a credential body', async () => {
- fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({ status: 'not_connected' })));
- expect(await getClaudeConnection(SERVER)).toEqual({ status: 'not_connected' });
- fetchMock.mockResolvedValueOnce(new Response(null, { status: 204 }));
- await disconnectClaude(SERVER);
- const [, options] = fetchMock.mock.calls[1] as unknown as [string, RequestInit];
- expect(options.method).toBe('DELETE');
- expect(options.body).toBeUndefined();
- });
- it('rejects malformed connection status', async () => {
- fetchMock.mockResolvedValueOnce(
- new Response(JSON.stringify({ status: 'connected', kind: 'unknown' }))
- );
- await expect(getClaudeConnection(SERVER)).rejects.toThrow('invalid Claude connection status');
- });
- it('surfaces a failed verification instead of reporting a connection', async () => {
- fetchMock.mockResolvedValueOnce(
- new Response(JSON.stringify({ detail: 'Claude could not complete the check.' }), {
- status: 422,
- })
- );
- await expect(connectClaude(SERVER, 'api-key', 'SYNTHETIC-CREDENTIAL')).rejects.toThrow();
- });
-});
-
describe('GitHub connection transport', () => {
beforeEach(() => {
vi.clearAllMocks();
@@ -1269,6 +1150,7 @@ describe('sign-up support', () => {
disk: null,
memory: null,
agents: [],
+ controller_id: null,
};
fetchMock.mockResolvedValueOnce(jsonResponse(machine));
await expect(ensureCloudMachine(SERVER)).resolves.toEqual(machine);
@@ -1278,6 +1160,57 @@ describe('sign-up support', () => {
expect(cookieHeaderOf(fetchMock.mock.calls[0])).toContain('switch_auth=');
});
+ it('reads the controller a machine enrolled as', async () => {
+ fetchMock.mockResolvedValueOnce(
+ jsonResponse({
+ machine_id: 'm-1',
+ state: 'ready',
+ desired_state: 'running',
+ stop_reason: null,
+ sleeping: false,
+ revision: 2,
+ instance_type: 'c7i.large',
+ error: null,
+ error_code: null,
+ retain_until: null,
+ heartbeat_at: null,
+ disk: null,
+ memory: null,
+ agents: ['agent-1'],
+ controller_id: 'controller-7',
+ })
+ );
+ await expect(ensureCloudMachine(SERVER)).resolves.toMatchObject({
+ controller_id: 'controller-7',
+ });
+ });
+
+ it('lists the caller’s cloud machines, and none on a server without them', async () => {
+ const machine = {
+ machine_id: 'm-1',
+ state: 'ready',
+ desired_state: 'running',
+ stop_reason: null,
+ sleeping: false,
+ revision: 2,
+ instance_type: null,
+ error: null,
+ error_code: null,
+ retain_until: null,
+ heartbeat_at: null,
+ disk: null,
+ memory: null,
+ agents: [],
+ controller_id: null,
+ };
+ fetchMock.mockResolvedValueOnce(jsonResponse({ machines: [machine] }));
+ await expect(listCloudMachines(SERVER)).resolves.toEqual([machine]);
+ const [url] = fetchMock.mock.calls[0] as unknown as [string];
+ expect(url).toBe('https://switch.example.com/gateway/hosted-machines');
+ fetchMock.mockResolvedValueOnce(errorResponse(404, '{"detail":"Not Found"}'));
+ await expect(listCloudMachines(SERVER)).resolves.toBeNull();
+ });
+
it('raises the server’s explanation when no machine can be had', async () => {
fetchMock.mockResolvedValueOnce(
errorResponse(503, JSON.stringify({ detail: 'Cloud machines are not offered here.' }))
@@ -2180,6 +2113,91 @@ describe('agent management calls', () => {
expect(JSON.parse(String(init.body))).toEqual({ name: 'laptop', description: null });
});
+ it('reads the key a machine takes logins sealed to, and gives it one', async () => {
+ fetchMock.mockImplementation(async () =>
+ respond(200, [
+ {
+ id: 'controller-1',
+ name: 'box',
+ description: null,
+ kind: 'daemon',
+ state: 'online',
+ last_seen_at: null,
+ revoked_at: null,
+ public_key: { alg: 'X25519', key: 'a2V5', key_id: '0123456789abcdef' },
+ },
+ {
+ id: 'controller-2',
+ name: 'old',
+ description: null,
+ kind: 'daemon',
+ state: 'online',
+ last_seen_at: null,
+ revoked_at: null,
+ },
+ ])
+ );
+ const [keyed, keyless] = await fetchManagementControllers(SERVER);
+ expect(keyed?.sealingKey).toEqual({ key: 'a2V5', keyId: '0123456789abcdef' });
+ expect(keyless?.sealingKey).toBeNull();
+
+ const sealed = {
+ alg: 'X25519-HKDF-SHA256-A256GCM' as const,
+ key_id: '0123456789abcdef',
+ ephemeral_key: 'ZQ==',
+ nonce: 'bg==',
+ ciphertext: 'Yw==',
+ };
+ fetchMock.mockImplementation(async () =>
+ respond(200, { login: { provider: 'claude', revision: 2 }, operation: { id: 'op-1' } })
+ );
+ expect(await giveMachineLogin(SERVER, 'controller-1', 'claude', sealed)).toEqual({
+ operationId: 'op-1',
+ revision: 2,
+ });
+ const [url, init] = fetchMock.mock.calls.at(-1) as unknown as [string, RequestInit];
+ expect(url).toBe(
+ 'https://switch.example.com/gateway/management/controllers/controller-1/provider-logins/claude'
+ );
+ expect(init.method).toBe('PUT');
+ expect(JSON.parse(String(init.body))).toEqual({ sealed });
+ await expect(
+ giveMachineLogin(
+ { ...(SERVER as object), gatewayUrl: 'http://switch.example.com' } as never,
+ 'c',
+ 'claude',
+ sealed
+ )
+ ).rejects.toThrow(/HTTPS/);
+
+ fetchMock.mockImplementation(async () =>
+ respond(200, [
+ {
+ id: 'op-1',
+ state: 'failed',
+ result: {
+ outcome: 'failed',
+ error: { code: 'provider_login_expired', message: 'No.' },
+ },
+ },
+ { id: 'op-2', state: 'succeeded', result: { outcome: 'succeeded' } },
+ ])
+ );
+ expect(await fetchMachineOperation(SERVER, 'controller-1', 'op-1')).toEqual({
+ state: 'failed',
+ error: { code: 'provider_login_expired', message: 'No.' },
+ });
+ expect(await fetchMachineOperation(SERVER, 'controller-1', 'op-2')).toEqual({
+ state: 'succeeded',
+ error: null,
+ });
+ expect(await fetchMachineOperation(SERVER, 'controller-1', 'op-9')).toBeNull();
+ fetchMock.mockImplementation(async () => respond(200, { operations: [] }));
+ await expect(fetchMachineOperation(SERVER, 'controller-1', 'op-1')).rejects.toThrow(
+ /not a list/
+ );
+ });
+
it("reads each machine's providers from its last status report, and invents none", async () => {
const controller = {
name: 'box',
diff --git a/console/apps/switch-console-desktop/src/main/core/switch-servers/gateway-client.ts b/console/apps/switch-console-desktop/src/main/core/switch-servers/gateway-client.ts
index 775e5d37d..ff59275fd 100644
--- a/console/apps/switch-console-desktop/src/main/core/switch-servers/gateway-client.ts
+++ b/console/apps/switch-console-desktop/src/main/core/switch-servers/gateway-client.ts
@@ -1,3 +1,4 @@
+import type { SealedLogin } from '@switch-console/agent-providers';
import type { AdvancedConfigValue } from '@switch-console/plugins/agents';
import { z } from 'zod';
import type { KnownAgentType } from '@main/core/agents/known-agent-type';
@@ -7,29 +8,19 @@ import {
noteManagedServerUnanswered,
} from '@main/core/managed-switch-server/managed-server-status';
import { assertedTenant } from '@main/core/workspaces/asserted-tenant';
-import { cloudLaunchSchema, cloudMachineSchema } from '@shared/core/cloud-agents/cloud-agents';
+import { type CloudMachine, cloudMachineSchema } from '@shared/core/cloud-agents/cloud-agents';
import type {
AdvancedConfigField,
ManagedMachine,
} from '@shared/core/managed-agents/managed-agents';
import { ManagedServerStoppedError } from '@shared/core/managed-switch-server/managed-switch-server';
-import type { AgentProviderId } from '@shared/core/providers/agent-provider-registry';
import { HostUnreachableError } from '@shared/core/remote-hosts/reachability';
-import type {
- ClaudeCredentialKind,
- ClaudeConnection,
-} from '@shared/core/switch-servers/claude-credential';
-import type {
- CloudLaunchConfiguration,
- CloudLaunchInput,
-} from '@shared/core/switch-servers/cloud-launch';
import { connectionCatalogSchema } from '@shared/core/switch-servers/connection-catalog';
import {
gitHubConnectionSchema,
gitHubFlowSchema,
} from '@shared/core/switch-servers/github-connection';
import { policyNamesOwner } from '@shared/core/switch-servers/owner-policy';
-import { cloudProviderConnectionSchema } from '@shared/core/switch-servers/provider-credential';
import type {
AddressingPolicy,
AddTeamsTeamResult,
@@ -163,7 +154,7 @@ export class GatewayError extends Error {
* the raw status line, which reads as noise in a form. */
readonly detail?: string,
/** The refusal's machine-readable name, from a body such as
- * `{"detail": …, "code": "worker_waking"}`. Present only when the body
+ * `{"detail": …, "code": "machine_stopped"}`. Present only when the body
* carried one. */
readonly code?: string,
/** The response body as it came, for a caller that reads an envelope other
@@ -2572,101 +2563,19 @@ export async function createRoom(
return mapRoomSummary((await res.json()) as RoomSummaryJson);
}
-async function readClaudeConnection(response: Response): Promise<ClaudeConnection> {
- const value: unknown = await response.json();
- if (typeof value === 'object' && value !== null && 'status' in value) {
- if (value.status === 'not_connected') return { status: 'not_connected' };
- if (
- value.status === 'connected' &&
- 'kind' in value &&
- (value.kind === 'api-key' || value.kind === 'setup-token') &&
- 'verified_at' in value &&
- typeof value.verified_at === 'string' &&
- Number.isFinite(Date.parse(value.verified_at))
- ) {
- return { status: 'connected', kind: value.kind, verified_at: value.verified_at };
- }
+/**
+ * The caller's cloud machines, or null when the server offers none: a server
+ * without cloud machines answers the route with 404.
+ */
+export async function listCloudMachines(server: SwitchServer): Promise<CloudMachine[] | null> {
+ let response: Response;
+ try {
+ response = await gatewayFetch(server, '/hosted-machines', { authenticated: true });
+ } catch (error) {
+ if (error instanceof GatewayError && error.kind === 'http' && error.status === 404) return null;
+ throw error;
}
- throw new GatewayError('http', 'The server returned an invalid Claude connection status.');
-}
-
-export async function getClaudeConnection(server: SwitchServer): Promise<ClaudeConnection> {
- const response = await gatewayFetch(server, '/provider-connections/claude', {
- authenticated: true,
- });
- return readClaudeConnection(response);
-}
-
-export async function createCloudLaunch(
- server: SwitchServer,
- input: CloudLaunchInput & { definition: string }
-) {
- if (new URL(server.gatewayUrl).protocol !== 'https:')
- throw new Error('Cloud agents require an HTTPS Switch server.');
- return cloudLaunchSchema.parse(
- await (
- await gatewayFetch(server, '/hosted-launches', {
- authenticated: true,
- method: 'POST',
- body: input,
- })
- ).json()
- );
-}
-
-const cloudConfigurationSchema = z.object({
- description: z.string(),
- instructions: z.string(),
- definition_attributes: z.record(z.string(), z.unknown()),
-});
-
-export async function getCloudLaunchConfiguration(
- server: SwitchServer,
- requestId: string
-): Promise<CloudLaunchConfiguration> {
- return cloudConfigurationSchema.parse(
- await (
- await gatewayFetch(
- server,
- `/hosted-launches/${encodeURIComponent(requestId)}/configuration`,
- { authenticated: true }
- )
- ).json()
- ) as CloudLaunchConfiguration;
-}
-
-/** Replace a launch's instructions and definition; Core applies them at the agent's next start. */
-export async function updateCloudLaunchConfiguration(
- server: SwitchServer,
- requestId: string,
- body: Omit<CloudLaunchConfiguration, 'description'> & { definition: string }
-): Promise<CloudLaunchConfiguration> {
- return cloudConfigurationSchema.parse(
- await (
- await gatewayFetch(
- server,
- `/hosted-launches/${encodeURIComponent(requestId)}/configuration`,
- { authenticated: true, method: 'PUT', body }
- )
- ).json()
- ) as CloudLaunchConfiguration;
-}
-
-export async function cloudLifecycle(
- server: SwitchServer,
- requestId: string,
- action: 'stop' | 'start' | 'restart' | 'remove' | 'retry',
- revision: number
-) {
- return cloudLaunchSchema.extend({ access_warning: z.string().nullable().optional() }).parse(
- await (
- await gatewayFetch(server, `/hosted-launches/${encodeURIComponent(requestId)}/lifecycle`, {
- authenticated: true,
- method: 'POST',
- body: { action, revision },
- })
- ).json()
- );
+ return z.object({ machines: z.array(cloudMachineSchema) }).parse(await response.json()).machines;
}
export async function cloudMachineLifecycle(
@@ -2705,29 +2614,6 @@ export async function ensureCloudMachine(server: SwitchServer) {
return cloudMachineSchema.parse(await response.json());
}
-export async function connectClaude(
- server: SwitchServer,
- kind: ClaudeCredentialKind,
- credential: string
-): Promise<ClaudeConnection> {
- if (new URL(server.gatewayUrl).protocol !== 'https:') {
- throw new GatewayError('http', 'Claude credentials require an HTTPS Switch server.');
- }
- const response = await gatewayFetch(server, '/provider-connections/claude', {
- authenticated: true,
- method: 'PUT',
- body: { kind, credential },
- });
- return readClaudeConnection(response);
-}
-
-export async function disconnectClaude(server: SwitchServer): Promise<void> {
- await gatewayFetch(server, '/provider-connections/claude', {
- authenticated: true,
- method: 'DELETE',
- });
-}
-
export async function getConnectionCatalog(server: SwitchServer) {
return connectionCatalogSchema.parse(
await (
@@ -2825,43 +2711,6 @@ export async function disconnectGitHub(server: SwitchServer) {
return z.object({ warning: z.string().nullable() }).parse(await response.json());
}
-export async function getCloudProviderConnection(server: SwitchServer, provider: AgentProviderId) {
- return cloudProviderConnectionSchema.parse(
- await (
- await gatewayFetch(server, `/provider-connections/${encodeURIComponent(provider)}`, {
- authenticated: true,
- })
- ).json()
- );
-}
-export async function connectCloudProvider(
- server: SwitchServer,
- provider: Exclude<AgentProviderId, 'claude'>,
- kind: 'api-key' | 'auth-json',
- credential: string
-) {
- if (new URL(server.gatewayUrl).protocol !== 'https:')
- throw new Error('Provider credentials require HTTPS.');
- return cloudProviderConnectionSchema.parse(
- await (
- await gatewayFetch(server, `/provider-connections/${encodeURIComponent(provider)}`, {
- authenticated: true,
- method: 'PUT',
- body: { kind, credential },
- })
- ).json()
- );
-}
-export async function disconnectCloudProvider(
- server: SwitchServer,
- provider: Exclude<AgentProviderId, 'claude'>
-) {
- await gatewayFetch(server, `/provider-connections/${encodeURIComponent(provider)}`, {
- authenticated: true,
- method: 'DELETE',
- });
-}
-
// ── Agent management: controllers and the agents placed on them ─────────────
/**
@@ -2942,6 +2791,11 @@ export type ManagementController = {
* last reported; null before it has, or when it or the server predates it.
*/
workspacesDir: string | null;
+ /**
+ * The key provider logins given to this machine are sealed to; null when
+ * its controller has none (it keeps no key, or predates sealed logins).
+ */
+ sealingKey: { key: string; keyId: string } | null;
};
/** A provider as a controller reports it: installed, and whether its login works. */
@@ -3000,6 +2854,7 @@ type ManagementControllerJson = {
revoked_at: string | null;
status?: unknown;
workspaces_dir?: string | null;
+ public_key?: { alg: string; key: string; key_id: string } | null;
};
type ManagedAgentJson = {
@@ -3079,9 +2934,80 @@ export async function fetchManagementControllers(
revokedAt: json.revoked_at,
providers: providerReports(json.status),
workspacesDir: json.workspaces_dir ?? null,
+ sealingKey:
+ json.public_key?.alg === 'X25519'
+ ? { key: json.public_key.key, keyId: json.public_key.key_id }
+ : null,
}));
}
+/**
+ * Gives a machine a provider login sealed to its key
+ * (`PUT /controllers/{id}/provider-logins/{provider}`): the server keeps only
+ * the ciphertext. Answers the `provider.login` operation whose result says
+ * whether the provider signs in with it on the machine.
+ */
+export async function giveMachineLogin(
+ server: SwitchServer,
+ controllerId: string,
+ provider: string,
+ sealed: SealedLogin
+): Promise<{ operationId: string; revision: number }> {
+ // The login is sealed, but the key it was sealed to came from this server:
+ // over plain HTTP anyone in between could have handed over their own.
+ const gateway = new URL(server.gatewayUrl);
+ if (
+ gateway.protocol !== 'https:' &&
+ !['localhost', '127.0.0.1', '[::1]'].includes(gateway.hostname)
+ )
+ throw new Error('Giving a machine a provider login requires HTTPS.');
+ const res = await managementFetch(
+ server,
+ `/controllers/${encodeURIComponent(controllerId)}/provider-logins/${encodeURIComponent(provider)}`,
+ { method: 'PUT', authenticated: true, body: { sealed } }
+ );
+ const body = (await res.json()) as { login: { revision: number }; operation: { id: string } };
+ return { operationId: body.operation.id, revision: body.login.revision };
+}
+
+/** Takes back the provider login given to a machine. */
+export async function withdrawMachineLogin(
+ server: SwitchServer,
+ controllerId: string,
+ provider: string
+): Promise<void> {
+ await managementFetch(
+ server,
+ `/controllers/${encodeURIComponent(controllerId)}/provider-logins/${encodeURIComponent(provider)}`,
+ { method: 'DELETE', authenticated: true }
+ );
+}
+
+/** One of a machine's operations as its owner sees it, by id; null when the server lists none by it. */
+export async function fetchMachineOperation(
+ server: SwitchServer,
+ controllerId: string,
+ operationId: string
+): Promise<{ state: string; error: { code: string; message: string } | null } | null> {
+ const res = await managementFetch(
+ server,
+ `/operations?controller_id=${encodeURIComponent(controllerId)}`,
+ { authenticated: true }
+ );
+ const listed = (await res.json()) as unknown;
+ if (!Array.isArray(listed))
+ throw new Error("The server's list of the machine's operations is not a list.");
+ const found = (
+ listed as {
+ id: string;
+ state: string;
+ result?: { outcome?: string; error?: { code: string; message: string } } | null;
+ }[]
+ ).find((operation) => operation.id === operationId);
+ if (!found) return null;
+ return { state: found.state, error: found.result?.error ?? null };
+}
+
const PROVIDER_AUTH_STATES: readonly ControllerProviderReport['auth'][] = [
'ok',
'expired',
@@ -3321,7 +3247,7 @@ export async function createManagedAgent(
icon_url: string | null;
controller_id: string;
desired_state: 'running' | 'stopped';
- definition: ManagedAgentDefinitionBody;
+ definition: ManagedAgentDefinitionBody & { isolation: 'shared' | 'isolated' };
}
): Promise<string> {
const res = await managementFetch(server, '/agents', {
diff --git a/console/apps/switch-console-desktop/src/main/core/switch-servers/local-opencode-sign-in.test.ts b/console/apps/switch-console-desktop/src/main/core/switch-servers/local-opencode-sign-in.test.ts
deleted file mode 100644
index f2228cac2..000000000
--- a/console/apps/switch-console-desktop/src/main/core/switch-servers/local-opencode-sign-in.test.ts
+++ /dev/null
@@ -1,17 +0,0 @@
-import { describe, expect, it, vi } from 'vitest';
-import { openCodeLoginCommand } from './local-opencode-sign-in';
-vi.mock('@main/core/agent-runtime/impl/resolve-agent-executable', () => ({}));
-vi.mock('@main/core/dependencies/dependency-managers', () => ({}));
-vi.mock('@main/core/dependencies/host-dependency-store', () => ({}));
-vi.mock('@main/core/execution-context/local-execution-context', () => ({}));
-
-describe('OpenCode login instructions', () => {
- it.each([
- ['1.18.32', 'opencode console login [url]', 'opencode console login'],
- ['1.2.0', 'opencode [project]', 'opencode auth login'],
- ['2.0.0', '', 'opencode auth login opencode'],
- ['2.0.0-beta.1', 'opencode console login [url]', 'opencode auth login opencode'],
- ])('chooses the supported command for %s', (version, help, expected) => {
- expect(openCodeLoginCommand(version, help)).toBe(expected);
- });
-});
diff --git a/console/apps/switch-console-desktop/src/main/core/switch-servers/local-opencode-sign-in.ts b/console/apps/switch-console-desktop/src/main/core/switch-servers/local-opencode-sign-in.ts
index 2b1185d11..108f1cbee 100644
--- a/console/apps/switch-console-desktop/src/main/core/switch-servers/local-opencode-sign-in.ts
+++ b/console/apps/switch-console-desktop/src/main/core/switch-servers/local-opencode-sign-in.ts
@@ -3,10 +3,6 @@ import { homedir } from 'node:os';
import { join } from 'node:path';
import { openCodeConsoleCredentialSchema } from '@switch-console/agent-providers';
import Database from 'better-sqlite3';
-import { resolveAgentExecutable } from '@main/core/agent-runtime/impl/resolve-agent-executable';
-import { localDependencyManager } from '@main/core/dependencies/dependency-managers';
-import { hostDependencyStore } from '@main/core/dependencies/host-dependency-store';
-import { LocalExecutionContext } from '@main/core/execution-context/local-execution-context';
export function localOpenCodeDatabasePath(): string {
return join(
@@ -50,43 +46,3 @@ export function readOpenCodeConsole(path: string): string | null {
db?.close();
}
}
-
-let cliInfo: { expires: number; value: { version: string; command: string } } | undefined;
-let cliFailure: { expires: number; error: Error } | undefined;
-export async function getOpenCodeLoginCommand() {
- if (cliFailure && cliFailure.expires > Date.now()) throw cliFailure.error;
- if (cliInfo && cliInfo.expires > Date.now()) return cliInfo.value;
- const ctx = new LocalExecutionContext();
- try {
- const binary = await resolveAgentExecutable({
- providerId: 'opencode',
- binaryName: 'opencode',
- ctx,
- hostDependencyStore,
- cachedStatePath: localDependencyManager.get('opencode')?.path,
- });
- const version = (await ctx.exec(binary, ['--version'], { timeout: 10000 })).stdout.trim();
- const result = /^2\./.test(version)
- ? { stdout: '', stderr: '' }
- : await ctx.exec(binary, ['console', '--help'], { timeout: 10000 });
- const help = result.stdout + result.stderr;
- const command = openCodeLoginCommand(version, help);
- const value = { version, command };
- cliInfo = { expires: Date.now() + 30000, value };
- return value;
- } catch {
- const error = new Error(
- 'Could not detect the installed OpenCode CLI. Check its installation in Settings.'
- );
- cliFailure = { expires: Date.now() + 30000, error };
- throw error;
- } finally {
- ctx.dispose();
- }
-}
-
-export function openCodeLoginCommand(version: string, consoleHelp: string): string {
- if (/^2\./.test(version)) return 'opencode auth login opencode';
- if (/opencode console login/.test(consoleHelp)) return 'opencode console login';
- return 'opencode auth login';
-}
diff --git a/console/apps/switch-console-desktop/src/main/core/switch-servers/local-provider-sign-in.test.ts b/console/apps/switch-console-desktop/src/main/core/switch-servers/local-provider-sign-in.test.ts
index 9a5c95be0..e0e573544 100644
--- a/console/apps/switch-console-desktop/src/main/core/switch-servers/local-provider-sign-in.test.ts
+++ b/console/apps/switch-console-desktop/src/main/core/switch-servers/local-provider-sign-in.test.ts
@@ -3,22 +3,18 @@ import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
-import { getOpenCodeLoginCommand } from './local-opencode-sign-in';
-import {
- getLocalProviderSignIn,
- localProviderAuthPath,
- readLocalProviderSignIn,
-} from './local-provider-sign-in';
+import { localProviderAuthPath, readLocalProviderSignIn } from './local-provider-sign-in';
vi.mock('./local-opencode-sign-in', () => ({
localOpenCodeDatabasePath: () => '/fixture/opencode.db',
readOpenCodeConsole: () => null,
- getOpenCodeLoginCommand: vi.fn(async () => ({
- version: '2.0.0',
- command: 'opencode auth login opencode',
- })),
}));
+/** The local sign-in for `provider` at its usual path, or null when there is none. */
+function localSignIn(provider: 'codex' | 'opencode' | 'antigravity') {
+ return readLocalProviderSignIn(provider, localProviderAuthPath(provider));
+}
+
let home: string;
const subscription = {
auth_mode: 'chatgpt',
@@ -43,18 +39,17 @@ describe('local Codex subscription detection', () => {
it('uses CODEX_HOME and detects later writes without exposing credentials', async () => {
const path = join(home, 'auth.json');
expect(localProviderAuthPath('codex')).toBe(path);
- expect(await getLocalProviderSignIn('codex')).toEqual({ path, status: 'missing' });
+ expect(await localSignIn('codex')).toBeNull();
await writeFile(path, JSON.stringify(subscription));
- expect(await getLocalProviderSignIn('codex')).toEqual({ path, status: 'ready' });
- expect(await readLocalProviderSignIn('codex', path)).toBe(JSON.stringify(subscription));
+ expect(await localSignIn('codex')).toBe(JSON.stringify(subscription));
await rm(path);
- expect(await getLocalProviderSignIn('codex')).toEqual({ path, status: 'missing' });
+ expect(await localSignIn('codex')).toBeNull();
});
it('keeps partial contents out of errors and detects the completed write', async () => {
await writeFile(localProviderAuthPath('codex'), '{"placeholder-private-content":');
- await expect(getLocalProviderSignIn('codex')).rejects.toThrow('Waiting for Codex');
+ await expect(localSignIn('codex')).rejects.toThrow('Waiting for Codex');
await writeFile(localProviderAuthPath('codex'), JSON.stringify(subscription));
- expect((await getLocalProviderSignIn('codex')).status).toBe('ready');
+ expect(await localSignIn('codex')).toBe(JSON.stringify(subscription));
});
it.each([
{ OPENAI_API_KEY: 'placeholder-api-key' },
@@ -63,11 +58,11 @@ describe('local Codex subscription detection', () => {
[],
])('rejects API-only or incomplete subscription data', async (value) => {
await writeFile(localProviderAuthPath('codex'), JSON.stringify(value));
- await expect(getLocalProviderSignIn('codex')).rejects.toThrow('Sign in to Codex with ChatGPT');
+ await expect(localSignIn('codex')).rejects.toThrow('Sign in to Codex with ChatGPT');
});
it('rejects oversized files before reading their contents', async () => {
await writeFile(localProviderAuthPath('codex'), 'x'.repeat(16385));
- await expect(getLocalProviderSignIn('codex')).rejects.toThrow('smaller than 16 KiB');
+ await expect(localSignIn('codex')).rejects.toThrow('smaller than 16 KiB');
});
});
@@ -81,7 +76,7 @@ describe.each(['opencode', 'antigravity'] as const)('local %s sign-in detection'
expect(path).toBe(
join(home, provider === 'opencode' ? 'opencode/auth.json' : 'antigravity-acp/acp_token.json')
);
- expect(await getLocalProviderSignIn(provider)).toMatchObject({ path, status: 'missing' });
+ expect(await localSignIn(provider)).toBeNull();
await mkdir(dirname(path), { recursive: true });
const credential = JSON.stringify(
provider === 'opencode'
@@ -89,10 +84,9 @@ describe.each(['opencode', 'antigravity'] as const)('local %s sign-in detection'
: { placeholder: 'fixture-only' }
);
await writeFile(path, credential);
- expect(await getLocalProviderSignIn(provider)).toMatchObject({ path, status: 'ready' });
- expect(await readLocalProviderSignIn(provider, path)).toBe(credential);
+ expect(await localSignIn(provider)).toBe(credential);
await rm(path);
- expect((await getLocalProviderSignIn(provider)).status).toBe('missing');
+ expect(await localSignIn(provider)).toBeNull();
});
it.each(['{}', '[]', 'null', '"placeholder"', '{"private-content":', 'x'.repeat(16385)])(
'rejects invalid data without exposing contents (case %#)',
@@ -100,9 +94,9 @@ describe.each(['opencode', 'antigravity'] as const)('local %s sign-in detection'
const path = localProviderAuthPath(provider);
await mkdir(dirname(path), { recursive: true });
await writeFile(path, value);
- await expect(getLocalProviderSignIn(provider)).rejects.toThrow();
+ await expect(localSignIn(provider)).rejects.toThrow();
try {
- await getLocalProviderSignIn(provider);
+ await localSignIn(provider);
} catch (error) {
expect(String(error)).not.toContain(value);
}
@@ -114,7 +108,7 @@ describe.each(['opencode', 'antigravity'] as const)('local %s sign-in detection'
: { placeholder: 'fixture-only' }
)
);
- expect((await getLocalProviderSignIn(provider)).status).toBe('ready');
+ expect(await localSignIn(provider)).not.toBeNull();
}
);
});
@@ -141,16 +135,3 @@ it('retries a sign-in file that is still being written', async () => {
await writeFile(path, JSON.stringify(subscription));
expect(await reading).toBe(JSON.stringify(subscription));
});
-
-it('still detects an OpenCode login when CLI detection fails, with a visible warning', async () => {
- vi.stubEnv('XDG_DATA_HOME', home);
- const path = localProviderAuthPath('opencode');
- await mkdir(dirname(path), { recursive: true });
- await writeFile(path, JSON.stringify({ opencode: { type: 'api', key: 'fixture-only' } }));
- vi.mocked(getOpenCodeLoginCommand).mockRejectedValueOnce(new Error('CLI detection unavailable'));
- expect(await getLocalProviderSignIn('opencode')).toMatchObject({
- status: 'ready',
- path,
- detectionWarning: expect.stringContaining('CLI detection unavailable'),
- });
-});
diff --git a/console/apps/switch-console-desktop/src/main/core/switch-servers/local-provider-sign-in.ts b/console/apps/switch-console-desktop/src/main/core/switch-servers/local-provider-sign-in.ts
index 54a2f2273..a6e98e800 100644
--- a/console/apps/switch-console-desktop/src/main/core/switch-servers/local-provider-sign-in.ts
+++ b/console/apps/switch-console-desktop/src/main/core/switch-servers/local-provider-sign-in.ts
@@ -4,11 +4,7 @@ import { join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
import { z } from 'zod';
import { providerDisplayName } from '@shared/core/providers/agent-provider-registry';
-import {
- getOpenCodeLoginCommand,
- localOpenCodeDatabasePath,
- readOpenCodeConsole,
-} from './local-opencode-sign-in';
+import { localOpenCodeDatabasePath, readOpenCodeConsole } from './local-opencode-sign-in';
const subscriptionSchema = z.object({
auth_mode: z.literal('chatgpt').optional(),
@@ -116,32 +112,3 @@ async function readLocalProviderSignInOnce(provider: LocalSignInProvider, path:
await file.close();
}
}
-
-export async function getLocalProviderSignIn(provider: LocalSignInProvider) {
- if (provider === 'opencode') {
- const account = readOpenCodeConsole(localOpenCodeDatabasePath());
- const credential =
- account ?? (await readLocalProviderSignIn(provider, localProviderAuthPath(provider)));
- let info;
- try {
- info = await getOpenCodeLoginCommand();
- } catch (error) {
- if (!credential) throw error;
- return {
- path: account ? localOpenCodeDatabasePath() : localProviderAuthPath(provider),
- status: 'ready' as const,
- version: 'unknown',
- command: account ? 'opencode console login' : 'opencode auth login opencode',
- detectionWarning: String(error),
- };
- }
- const path =
- account || (!credential && info.command === 'opencode console login')
- ? localOpenCodeDatabasePath()
- : localProviderAuthPath(provider);
- return { path, status: credential ? ('ready' as const) : ('missing' as const), ...info };
- }
- const path = localProviderAuthPath(provider);
- const credential = await readLocalProviderSignIn(provider, path);
- return { path, status: credential ? ('ready' as const) : ('missing' as const) };
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/app/modal-registry.ts b/console/apps/switch-console-desktop/src/renderer/app/modal-registry.ts
index 756210f42..fa4ea16d4 100644
--- a/console/apps/switch-console-desktop/src/renderer/app/modal-registry.ts
+++ b/console/apps/switch-console-desktop/src/renderer/app/modal-registry.ts
@@ -1,4 +1,3 @@
-import { EditCloudAgentModal } from '@renderer/features/cloud-agents/edit-cloud-agent-modal';
import { CommandPaletteModal } from '@renderer/features/command-palette/command-palette-modal';
import { AddAgentModal } from '@renderer/features/locations/components/add-agent-modal/add-agent-modal';
import { DeleteAgentModal } from '@renderer/features/locations/components/delete-agent-modal';
@@ -86,10 +85,6 @@ export const modalRegistry = {
dismissOnOutsideClick: false,
}),
deleteServerModal: createModal(DeleteServerModal, { size: 'sm' }),
- editCloudAgentModal: createModal(EditCloudAgentModal, {
- size: 'md',
- dismissOnOutsideClick: false,
- }),
createRoomModal: createModal(CreateRoomModal, { size: 'lg', dismissOnOutsideClick: false }),
connectMessagingAppModal: createModal(ConnectMessagingAppModal, {
size: 'md',
diff --git a/console/apps/switch-console-desktop/src/renderer/app/view-registry.ts b/console/apps/switch-console-desktop/src/renderer/app/view-registry.ts
index a3e799fbc..a5bbdc4a9 100644
--- a/console/apps/switch-console-desktop/src/renderer/app/view-registry.ts
+++ b/console/apps/switch-console-desktop/src/renderer/app/view-registry.ts
@@ -1,6 +1,5 @@
import type { ComponentType, ReactNode } from 'react';
import { homeView } from '@renderer/app/home-view';
-import { cloudSessionView } from '@renderer/features/cloud-agents/cloud-session-view';
import { locationView } from '@renderer/features/locations/view';
import { managedAgentView } from '@renderer/features/managed-agents/managed-agent-view';
import { remoteHostView } from '@renderer/features/remote-hosts/views/remote-host-view';
@@ -27,7 +26,6 @@ export const views = {
home: homeView,
location: locationView,
session: sessionView,
- cloudSession: cloudSessionView,
managedAgent: managedAgentView,
room: roomView,
settings: settingsView,
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-list.tsx b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-list.tsx
deleted file mode 100644
index 04f2ba4ec..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-list.tsx
+++ /dev/null
@@ -1,219 +0,0 @@
-import type { Session } from '@switch-console/shared/session-v1';
-import { useQueryClient } from '@tanstack/react-query';
-import { ChevronRight, Cloud, MessageSquare, Plus } from 'lucide-react';
-import { observer } from 'mobx-react-lite';
-import { useEffect, useState } from 'react';
-import { restartsOnSend } from '@renderer/features/sessions/components/transcript/session-state';
-import { switchRoomsStore } from '@renderer/features/switch-servers/switch-rooms-store';
-import { switchServersStore } from '@renderer/features/switch-servers/switch-servers-store';
-import { AgentIcon } from '@renderer/lib/components/agent-icon';
-import { useNavigate, useParams } from '@renderer/lib/layout/navigation-provider';
-import { useWorkspaceSlots } from '@renderer/lib/layout/workspace-slots';
-import { sidebarStore } from '@renderer/lib/stores/app-state';
-import { type CloudAgent, cloudMachineReady } from '@shared/core/cloud-agents/cloud-agents';
-import { SidebarMenuButton } from '../sidebar/sidebar-primitives';
-import { cloudAgentState } from './cloud-agent-state';
-import { cloudOperationAttempts, startAttemptKey } from './cloud-operation-attempts';
-import { CloudProblem } from './cloud-problem';
-import { CloudStartAttemptStatus } from './cloud-start-attempt-status';
-import {
- serverNotSignedIn,
- useCloudAgentSessions,
- useCloudAgents,
- useCloudProblemAction,
-} from './use-cloud-agents';
-
-export function cloudSessionName(session: Session): string {
- const room = session.roomIds?.[0];
- const roomName = room ? switchRoomsStore.roomNameById(room) : null;
- return roomName ?? `Session ${session.sessionId.slice(0, 8)}`;
-}
-
-/**
- * Whether an agent's worker cannot be asked because of its machine rather than
- * itself: every agent on that machine says the same, so it is said once.
- */
-function isMachineProblem(agent: CloudAgent): boolean {
- const code = agent.problem?.code;
- if (code === 'worker_waking') return !cloudMachineReady(agent.machine);
- return code === 'worker_sleeping' || code === 'machine_stopped' || code === 'machine_error';
-}
-
-/** The machines whose state keeps their agents' workers from being asked, one agent each. */
-function machineProblems(agents: CloudAgent[]): CloudAgent[] {
- const byMachine = new Map<string, CloudAgent>();
- for (const agent of agents.filter(isMachineProblem)) {
- const machineId = agent.machine?.machine_id ?? agent.launch.machine_id ?? '';
- const shown = byMachine.get(machineId);
- if (!shown || (agent.problem!.wakeAvailable && !shown.problem!.wakeAvailable))
- byMachine.set(machineId, agent);
- }
- return [...byMachine.values()];
-}
-
-/**
- * The active server's cloud agents under the local and SSH ones: each launch,
- * and beneath it the sessions its worker reports. A machine whose state keeps
- * its agents' workers from being asked says so once, above them; a launch
- * whose worker cannot be asked for its own reason says why, above the sessions
- * last read from it. A worker is asked for its sessions only while its row is
- * expanded or one of its sessions is open.
- */
-export const CloudAgentList = observer(function CloudAgentList() {
- const serverId = switchServersStore.activeServerId;
- const agents = useCloudAgents(serverId);
- const queryClient = useQueryClient();
- useEffect(() => {
- const onFocus = () => void queryClient.invalidateQueries({ queryKey: ['cloud-agents'] });
- window.addEventListener('focus', onFocus);
- return () => window.removeEventListener('focus', onFocus);
- }, [queryClient]);
- if (serverNotSignedIn(serverId)) return null;
- if (agents.error)
- return (
- <div role="alert" className="px-3 py-2 text-xs text-foreground-destructive">
- Cloud agents could not be listed: {String(agents.error)}
- </div>
- );
- if (!agents.data?.length) return null;
- return (
- <div className="mt-2 flex flex-col gap-[2px]" aria-label="Cloud agents">
- {machineProblems(agents.data).map((agent) => (
- <CloudMachineProblem key={agent.key} agent={agent} />
- ))}
- {agents.data.map((agent) => (
- <CloudAgentRow key={agent.key} listed={agent} />
- ))}
- </div>
- );
-});
-
-function CloudMachineProblem({ agent }: { agent: CloudAgent }) {
- const action = useCloudProblemAction(agent, true);
- return (
- <CloudProblem
- problem={agent.problem!}
- machineReady={cloudMachineReady(agent.machine)}
- compact
- action={action}
- />
- );
-}
-
-const CloudAgentRow = observer(function CloudAgentRow({ listed }: { listed: CloudAgent }) {
- const { navigate } = useNavigate();
- const { currentView } = useWorkspaceSlots();
- const { params } = useParams('cloudSession');
- const groupKey = `cloud:${listed.key}`;
- const expanded = sidebarStore.isCloudGroupExpanded(groupKey);
- const attemptKey = startAttemptKey(listed.key);
- const attempt = cloudOperationAttempts.get(attemptKey);
- const agent = useCloudAgentSessions(
- listed,
- expanded ||
- (currentView === 'cloudSession' && params.agentKey === listed.key) ||
- attempt?.status === 'unknown'
- );
- const label = cloudAgentState(agent)?.label;
- const queryClient = useQueryClient();
- const [startError, setStartError] = useState<string | null>(null);
- const openSession = (sessionId: string) =>
- navigate('cloudSession', {
- agentKey: agent.key,
- sessionId,
- name: `${agent.launch.name} · Session ${sessionId.slice(0, 8)}`,
- });
- const start = async () => {
- setStartError(null);
- const result = await cloudOperationAttempts.run(attemptKey, agent.key, 'start', null);
- if (!result) return;
- if (result.outcome.state === 'applied') openSession(result.sessionId);
- else if (result.outcome.state === 'failed') setStartError(result.outcome.message);
- else void queryClient.invalidateQueries({ queryKey: ['cloud-agents'] });
- };
- const sessions = (agent.sessions ?? []).filter((session) => !session.retired);
- return (
- <div>
- <div className="group/row flex items-center">
- <SidebarMenuButton
- aria-expanded={expanded}
- onClick={() => sidebarStore.toggleCloudGroupExpanded(groupKey)}
- >
- <ChevronRight className={`size-3 shrink-0 ${expanded ? 'rotate-90' : ''}`} />
- <Cloud className="size-3.5 shrink-0" />
- <span className="flex min-w-0 items-center gap-1.5">
- <span className="truncate">{agent.launch.name}</span>
- {!sidebarStore.hideProviderMark && (
- <AgentIcon id={agent.launch.provider} size={12} className="h-3 w-3 shrink-0" />
- )}
- </span>
- {label && (
- <span className="ml-auto text-xs text-foreground-muted capitalize">{label}</span>
- )}
- </SidebarMenuButton>
- {!agent.problem && (
- <button
- type="button"
- aria-label={`New session on ${agent.launch.name}`}
- title="New session"
- className="rounded p-1 text-foreground-muted hover:text-foreground disabled:opacity-50"
- disabled={attempt?.status === 'pending'}
- onClick={() => void start()}
- >
- <Plus className="size-3.5" />
- </button>
- )}
- </div>
- {startError && (
- <div role="alert" className="px-7 py-1 text-xs text-foreground-destructive">
- The session could not be started: {startError}
- </div>
- )}
- <CloudStartAttemptStatus
- agentKey={agent.key}
- sessions={sessions}
- onOpen={openSession}
- onCheckAgain={() => void start()}
- className="px-7 py-1"
- />
- {expanded && (
- <div className="flex flex-col gap-[2px] pl-5">
- {agent.problem && !isMachineProblem(agent) && (
- <CloudProblem
- problem={agent.problem}
- machineReady={cloudMachineReady(agent.machine)}
- compact
- action={null}
- />
- )}
- {agent.sessions && sessions.length === 0 && (
- <p className="px-2 py-1 text-xs text-foreground-muted">No sessions on this worker.</p>
- )}
- {sessions.map((session) => (
- <SidebarMenuButton
- key={session.sessionId}
- isActive={
- currentView === 'cloudSession' &&
- params.agentKey === agent.key &&
- params.sessionId === session.sessionId
- }
- onClick={() =>
- navigate('cloudSession', {
- agentKey: agent.key,
- sessionId: session.sessionId,
- name: `${agent.launch.name} · ${cloudSessionName(session)}`,
- })
- }
- >
- <MessageSquare className="size-3.5 shrink-0" />
- <span className="truncate">{cloudSessionName(session)}</span>
- <span className="ml-auto text-xs text-foreground-muted">
- {restartsOnSend(session) ? null : session.status}
- </span>
- </SidebarMenuButton>
- ))}
- </div>
- )}
- </div>
- );
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-state.test.ts b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-state.test.ts
deleted file mode 100644
index 39c111527..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-state.test.ts
+++ /dev/null
@@ -1,143 +0,0 @@
-import { describe, expect, it } from 'vitest';
-import type { CloudAgent, CloudMachine } from '@shared/core/cloud-agents/cloud-agents';
-import { cloudAgentState, cloudHoldBlocker } from './cloud-agent-state';
-
-function machine(overrides: Partial<CloudMachine>): CloudMachine {
- return {
- machine_id: '3f1c2b4a-0000-4000-8000-000000000001',
- state: 'ready',
- desired_state: 'running',
- stop_reason: null,
- sleeping: false,
- revision: 4,
- instance_type: null,
- error: null,
- error_code: null,
- retain_until: null,
- heartbeat_at: null,
- disk: null,
- memory: null,
- agents: [],
- ...overrides,
- };
-}
-
-function agent(
- launch: Partial<CloudAgent['launch']>,
- onMachine: CloudMachine | null = machine({})
-): CloudAgent {
- return {
- key: 'cloud:server:launch',
- launch: {
- request_id: '00000000-0000-4000-8000-000000000001',
- name: 'reviewer',
- provider: 'claude',
- state: 'ready',
- desired_state: 'running',
- revision: 3,
- agent_id: 'agent',
- error: null,
- error_code: null,
- sleeping: false,
- machine_id: onMachine?.machine_id ?? null,
- process_state: 'running',
- process_restarts: 0,
- oom_kills: 0,
- ...launch,
- },
- machine: onMachine,
- sessions: null,
- problem: null,
- };
-}
-
-describe('a cloud agent on a machine in error', () => {
- it('reads as a machine error, not as ready', () => {
- expect(cloudAgentState(agent({}, machine({ state: 'error' })))).toEqual({
- label: 'machine error',
- tone: 'bad',
- });
- });
-});
-
-describe('a cloud agent on its way up', () => {
- it('reads as starting when only the agent starts on a ready machine', () => {
- expect(cloudAgentState(agent({ state: 'provisioning', process_state: 'starting' }))).toEqual({
- label: 'starting…',
- tone: 'busy',
- });
- });
-
- it('reads as waking while its machine starts', () => {
- expect(cloudAgentState(agent({}, machine({ state: 'provisioning' })))).toEqual({
- label: 'waking…',
- tone: 'busy',
- });
- });
-});
-
-describe('why a held message will not be delivered', () => {
- it.each([
- [
- 'a sleeping machine',
- agent(
- {},
- machine({ desired_state: 'stopped', stop_reason: 'idle', sleeping: true, state: 'stopped' })
- ),
- ],
- ['a machine on its way up', agent({}, machine({ state: 'provisioning' }))],
- ['a ready machine', agent({})],
- ])('is nothing on %s', (_name, onAgent) => {
- expect(cloudHoldBlocker(onAgent)).toBeNull();
- });
-
- it.each([
- [
- 'the owner stopped the machine',
- agent({}, machine({ desired_state: 'stopped', stop_reason: 'owner', state: 'stopped' })),
- 'The owner stopped the cloud machine.',
- ],
- [
- 'the machine is in error',
- agent({}, machine({ state: 'error' })),
- 'The cloud machine is in error.',
- ],
- [
- 'the machine needs attention',
- agent({}, machine({ state: 'error', error_code: 'machine_needs_attention' })),
- 'Contact your server administrator.',
- ],
- [
- 'the agent was stopped elsewhere',
- agent({ desired_state: 'stopped', state: 'stopping' }, machine({ state: 'provisioning' })),
- 'This agent is stopped.',
- ],
- [
- 'the agent crashed',
- agent({ state: 'error', error_code: 'agent_crashed', process_state: 'crashed' }),
- 'This agent crashed.',
- ],
- [
- 'the agent could not start',
- agent({ state: 'error', error_code: 'identity_failed' }),
- 'This agent could not start.',
- ],
- [
- 'the agent timed out connecting',
- agent({ state: 'error', error_code: 'worker_attach_timeout' }),
- 'This agent could not start.',
- ],
- [
- 'the agent lost its credential',
- agent({ state: 'error', error_code: 'agent_key_missing' }),
- 'Remove it in Your Agents and create it again.',
- ],
- [
- 'the agent is being removed',
- agent({ desired_state: 'deleted' }),
- 'This agent is being removed.',
- ],
- ])('says so when %s', (_name, onAgent, text) => {
- expect(cloudHoldBlocker(onAgent)).toContain(text);
- });
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-state.ts b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-state.ts
deleted file mode 100644
index 437a0ff58..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-agent-state.ts
+++ /dev/null
@@ -1,53 +0,0 @@
-import { relayRefusal } from '@renderer/features/sessions/components/transcript/held-message';
-import type { SessionStateTone } from '@renderer/features/sessions/components/transcript/session-state';
-import {
- type CloudAgent,
- cloudAgentPhase,
- cloudMachineReady,
-} from '@shared/core/cloud-agents/cloud-agents';
-
-/**
- * A cloud agent's state when its worker cannot be asked, read the same way in
- * the sidebar and in a session's header; null while the worker answers.
- */
-export function cloudAgentState(
- agent: CloudAgent
-): { label: string; tone: SessionStateTone } | null {
- if (agent.launch.desired_state === 'deleted') return { label: 'removing…', tone: 'idle' };
- const phase = cloudAgentPhase(agent.launch, agent.machine);
- if (phase === 'sleeping') return { label: 'sleeping', tone: 'idle' };
- if (phase === 'machine_stopped') return { label: 'machine stopped', tone: 'idle' };
- if (phase === 'machine_error') return { label: 'machine error', tone: 'bad' };
- if (phase === 'waking')
- return { label: cloudMachineReady(agent.machine) ? 'starting…' : 'waking…', tone: 'busy' };
- if (agent.launch.desired_state === 'stopped') return { label: 'stopped', tone: 'idle' };
- if (agent.launch.process_state === 'crashed' || agent.launch.error_code === 'agent_crashed')
- return { label: 'crashed', tone: 'bad' };
- if (agent.launch.state === 'error') return { label: 'error', tone: 'bad' };
- if (agent.problem) return { label: 'unreachable', tone: 'bad' };
- return null;
-}
-
-/**
- * Why a message held for a waking machine will not be delivered without the
- * user acting, or null while it still may be.
- */
-export function cloudHoldBlocker(agent: CloudAgent): string | null {
- const { launch } = agent;
- if (launch.desired_state === 'deleted') return 'This agent is being removed.';
- const phase = cloudAgentPhase(launch, agent.machine);
- if (phase === 'machine_stopped') return relayRefusal(phase);
- if (phase === 'machine_error') {
- return agent.machine?.error_code === 'machine_needs_attention'
- ? 'The cloud machine needs attention. Contact your server administrator.'
- : relayRefusal(phase);
- }
- if (launch.desired_state === 'stopped') return relayRefusal('agent_stopped');
- if (launch.process_state === 'crashed' || launch.error_code === 'agent_crashed')
- return relayRefusal('agent_crashed');
- if (launch.state === 'error' && launch.error_code === 'agent_key_missing')
- return 'Switch lost this agent’s credential. Remove it in Your Agents and create it again.';
- if (launch.state === 'error')
- return 'This agent could not start. Retry it in Your Agents, then send again.';
- return null;
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-machine-card.tsx b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-machine-card.tsx
index f389947cd..e8d4c17cb 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-machine-card.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-machine-card.tsx
@@ -6,6 +6,7 @@ import { Button } from '@renderer/lib/ui/button';
import { Progress } from '@renderer/lib/ui/progress';
import type { CloudMachine } from '@shared/core/cloud-agents/cloud-agents';
import { type MachineAction, machinePresentation } from './cloud-machine-state';
+import { CLOUD_MACHINES_KEY } from './use-cloud-agents';
function gigabytes(bytes: number): string {
return `${(bytes / 1024 ** 3).toFixed(1)} GB`;
@@ -35,11 +36,11 @@ export function CloudMachineCard({
action,
machine.revision
);
- await queryClient.invalidateQueries({ queryKey: ['cloud-agents'] });
+ await queryClient.invalidateQueries({ queryKey: [CLOUD_MACHINES_KEY] });
setConfirmStop(false);
} catch (error) {
setActionError(failureText(error, 'Machine operation failed.'));
- void queryClient.invalidateQueries({ queryKey: ['cloud-agents'] });
+ void queryClient.invalidateQueries({ queryKey: [CLOUD_MACHINES_KEY] });
} finally {
setPending(false);
}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-machine-state.test.ts b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-machine-state.test.ts
index 8ca0136f1..43a8dbf2b 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-machine-state.test.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-machine-state.test.ts
@@ -17,6 +17,7 @@ const sleepingMachine: CloudMachine = {
disk: { total_bytes: 214748364800, available_bytes: 204010946560 },
memory: { total_bytes: 17179869184, available_bytes: 12884901888 },
agents: ['req-0000000000000001'],
+ controller_id: null,
};
function machine(patch: Partial<CloudMachine>): CloudMachine {
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-operation-attempts.ts b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-operation-attempts.ts
deleted file mode 100644
index dab3cd066..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-operation-attempts.ts
+++ /dev/null
@@ -1,76 +0,0 @@
-import { observable, runInAction } from 'mobx';
-import { rpc } from '@renderer/lib/ipc';
-import type { CloudOperationOutcome } from '@shared/core/cloud-agents/cloud-agents';
-
-export type CloudOperationAttempt = {
- operationId: string;
- sessionId: string;
- status: 'pending' | 'unknown';
- message: string | null;
-};
-
-/**
- * Starts and restarts of cloud sessions that have not reached a definite
- * result, keyed by what they act on and kept outside any component so a
- * remount does not forget them. Asking again reuses the attempt's operation
- * id, which the server dedupes, so a lost response never queues a second
- * session or restart.
- */
-class CloudOperationAttempts {
- private readonly attempts = observable.map<string, CloudOperationAttempt>();
-
- get(key: string): CloudOperationAttempt | undefined {
- return this.attempts.get(key);
- }
-
- /** Forget an attempt whose outcome was settled some other way. */
- settle(key: string): void {
- runInAction(() => this.attempts.delete(key));
- }
-
- /** Resolves to null when the attempt is already in flight. */
- async run(
- key: string,
- agentKey: string,
- action: 'start' | 'restart',
- sessionId: string | null
- ): Promise<{ sessionId: string; outcome: CloudOperationOutcome } | null> {
- const existing = this.attempts.get(key);
- if (existing?.status === 'pending') return null;
- const target = existing?.sessionId ?? sessionId ?? crypto.randomUUID();
- const attempt: CloudOperationAttempt = {
- operationId: existing?.operationId ?? (action === 'start' ? target : crypto.randomUUID()),
- sessionId: target,
- status: 'pending',
- message: null,
- };
- runInAction(() => this.attempts.set(key, attempt));
- let outcome: CloudOperationOutcome;
- try {
- outcome = await rpc.sdkHost.cloudSessionOperation(
- agentKey,
- attempt.sessionId,
- attempt.operationId,
- action
- );
- } catch (error) {
- outcome = { state: 'unknown', message: String(error) };
- }
- runInAction(() => {
- if (outcome.state === 'unknown')
- this.attempts.set(key, { ...attempt, status: 'unknown', message: outcome.message });
- else this.attempts.delete(key);
- });
- return { sessionId: attempt.sessionId, outcome };
- }
-}
-
-export const cloudOperationAttempts = new CloudOperationAttempts();
-
-export function startAttemptKey(agentKey: string): string {
- return `start:${agentKey}`;
-}
-
-export function restartAttemptKey(agentKey: string, sessionId: string): string {
- return `restart:${agentKey}:${sessionId}`;
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-problem.tsx b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-problem.tsx
deleted file mode 100644
index 1739c5886..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-problem.tsx
+++ /dev/null
@@ -1,77 +0,0 @@
-import { AlertTriangle, Moon } from 'lucide-react';
-import { Button } from '@renderer/lib/ui/button';
-import type { CloudRelayProblem } from '@shared/core/cloud-agents/cloud-agents';
-import type { CloudProblemAction } from './use-cloud-agents';
-
-/** What a relay code means to the user, beside the server's own message. */
-const PROBLEM_TITLES: Record<string, string> = {
- machine_stopped: 'The owner stopped the cloud machine.',
- machine_error: 'The cloud machine is in error.',
- worker_sleeping: 'The cloud machine is asleep.',
- worker_waking: 'The cloud machine is starting.',
- agent_stopped: 'The agent is stopped.',
- agent_crashed: 'The agent crashed.',
- worker_not_attached: 'The cloud worker is not attached.',
- worker_busy: 'The cloud worker is busy. Try again shortly.',
- generation_changed: 'The cloud worker restarted.',
- relay_timeout: 'The cloud worker did not answer in time.',
- refused_message: 'Switch refused the request.',
- too_large: 'The request is too large for the relay.',
- not_found: 'Switch does not know this cloud agent.',
-};
-
-/**
- * Why a cloud agent's worker cannot be reached, said as such, with what the
- * user can do about it. A sleeping machine that can be woken and offers no
- * Wake here wakes on the next message sent to it.
- * A `worker_waking` on a machine already up is only the agent starting.
- */
-export function CloudProblem({
- problem,
- machineReady,
- compact,
- action,
-}: {
- problem: CloudRelayProblem;
- machineReady: boolean;
- compact: boolean;
- action: CloudProblemAction | null;
-}) {
- const sleeping = problem.code === 'worker_sleeping';
- const agentStarting = problem.code === 'worker_waking' && machineReady;
- const Icon = sleeping ? Moon : AlertTriangle;
- return (
- <div
- role={sleeping ? 'status' : 'alert'}
- className={`flex items-center gap-2 rounded-md bg-background-secondary text-xs ${compact ? 'px-2 py-1.5' : 'px-3 py-2'} ${sleeping ? 'text-foreground-muted' : 'text-foreground-destructive'}`}
- >
- <Icon className="size-3.5 shrink-0" />
- <span className="min-w-0 flex-1 leading-5">
- {agentStarting
- ? 'The agent is starting.'
- : (PROBLEM_TITLES[problem.code] ??
- (compact ? problem.message : 'The cloud worker could not be reached.'))}{' '}
- {!compact && !agentStarting && problem.message !== PROBLEM_TITLES[problem.code] && (
- <span className="text-foreground-muted">{problem.message} </span>
- )}
- {sleeping && problem.wakeAvailable && !action && 'Send a message to wake it. '}
- {action?.error && (
- <span role="alert" className="block text-foreground-destructive">
- {action.error}
- </span>
- )}
- </span>
- {action && (
- <Button
- size="sm"
- variant="outline"
- className="h-6 shrink-0 px-2 text-xs"
- disabled={action.pending}
- onClick={action.run}
- >
- {action.label}
- </Button>
- )}
- </div>
- );
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-session-view.tsx b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-session-view.tsx
deleted file mode 100644
index 8c810f0d9..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-session-view.tsx
+++ /dev/null
@@ -1,149 +0,0 @@
-import { SessionChatClient } from '@switch-console/shared/session-v1';
-import { Cloud } from 'lucide-react';
-import { observer } from 'mobx-react-lite';
-import { useMemo, type ReactNode } from 'react';
-import type { GuardResult, ViewDefinition } from '@renderer/app/view-registry';
-import { SessionV1Chat } from '@renderer/features/sessions/components/transcript/session-v1-chat';
-import { cloudSessionTransport } from '@renderer/features/sessions/components/transcript/shared-session-transport';
-import {
- SessionHeaderOutlet,
- SessionHeaderSlotsProvider,
-} from '@renderer/features/sessions/session-header-slots';
-import { Titlebar } from '@renderer/lib/components/titlebar/Titlebar';
-import { rpc } from '@renderer/lib/ipc';
-import { useParams } from '@renderer/lib/layout/navigation-provider';
-import {
- type CloudAgent,
- cloudAgentPhase,
- cloudMachineReady,
- parseCloudAgentKey,
-} from '@shared/core/cloud-agents/cloud-agents';
-import { cloudAgentState, cloudHoldBlocker } from './cloud-agent-state';
-import { cloudOperationAttempts, restartAttemptKey } from './cloud-operation-attempts';
-import { CloudProblem } from './cloud-problem';
-import {
- useCloudAgentSessions,
- useCloudAgents,
- useCloudProblemAction,
- useCloudWake,
-} from './use-cloud-agents';
-
-type CloudSessionParams = { agentKey: string; sessionId: string; name: string };
-
-function CloudSessionTitlebar() {
- const { params } = useParams('cloudSession');
- return (
- <Titlebar
- leftSlot={
- <div className="flex items-center gap-2">
- <Cloud className="size-4" />
- <span>{params.name}</span>
- <SessionHeaderOutlet slot="left" className="flex items-center gap-2" />
- </div>
- }
- rightSlot={<SessionHeaderOutlet slot="right" className="mr-2 flex items-center gap-2" />}
- />
- );
-}
-
-/** The launch's state over the transcript while its worker cannot be asked. */
-const CloudWorkerStatus = observer(function CloudWorkerStatus({
- agentKey,
- agent,
-}: {
- agentKey: string;
- agent: CloudAgent | undefined;
-}) {
- const agents = useCloudAgents(parseCloudAgentKey(agentKey)?.serverId ?? null);
- if (agents.error)
- return (
- <div role="alert" className="px-5 pt-3 text-xs text-foreground-destructive">
- Cloud agents could not be listed: {String(agents.error)}
- </div>
- );
- if (agents.data && !agent)
- return (
- <div role="alert" className="px-5 pt-3 text-xs text-foreground-destructive">
- This cloud agent is no longer on its Switch server.
- </div>
- );
- return agent?.problem ? <CloudAgentProblem agent={agent} /> : null;
-});
-
-function CloudAgentProblem({ agent }: { agent: CloudAgent }) {
- const action = useCloudProblemAction(agent, false);
- return agent.problem ? (
- <div className="px-5 pt-3">
- <CloudProblem
- problem={agent.problem}
- machineReady={cloudMachineReady(agent.machine)}
- compact={false}
- action={action}
- />
- </div>
- ) : null;
-}
-
-const CloudSessionPanel = observer(function CloudSessionPanel() {
- const { params } = useParams('cloudSession');
- const agents = useCloudAgents(parseCloudAgentKey(params.agentKey)?.serverId ?? null);
- const agent = useCloudAgentSessions(
- agents.data?.find((each) => each.key === params.agentKey),
- true
- );
- const client = useMemo(
- () => new SessionChatClient(params.sessionId, cloudSessionTransport(params.agentKey)),
- [params.agentKey, params.sessionId]
- );
- const wake = useCloudWake();
- return (
- <div className="flex h-full min-h-0 flex-col">
- <CloudWorkerStatus agentKey={params.agentKey} agent={agent} />
- <SessionV1Chat
- key={`${params.agentKey}:${params.sessionId}`}
- client={client}
- hostState={agent ? cloudAgentState(agent) : null}
- autoWake={{
- phase: agent ? cloudAgentPhase(agent.launch, agent.machine) : null,
- machineReady: agent ? cloudMachineReady(agent.machine) : false,
- blocked: agent ? cloudHoldBlocker(agent) : null,
- wake: () => wake.mutateAsync(params.agentKey),
- }}
- stopHost={() => rpc.sdkHost.stop(params.agentKey, params.sessionId)}
- restartHost={async () => {
- const result = await cloudOperationAttempts.run(
- restartAttemptKey(params.agentKey, params.sessionId),
- params.agentKey,
- 'restart',
- params.sessionId
- );
- if (!result) throw new Error('This session is already restarting.');
- if (result.outcome.state === 'unknown')
- throw new Error(
- `${result.outcome.message} Restart again to check: it asks for the same restart, not another.`
- );
- if (result.outcome.state === 'failed') throw new Error(result.outcome.message);
- }}
- />
- </div>
- );
-});
-
-export const cloudSessionView = {
- WrapView: ({ children }: CloudSessionParams & { children: ReactNode }) => (
- <SessionHeaderSlotsProvider>{children}</SessionHeaderSlotsProvider>
- ),
- TitlebarSlot: CloudSessionTitlebar,
- MainPanel: CloudSessionPanel,
- canActivate: (params: unknown): GuardResult => {
- const value = (params ?? {}) as Partial<Record<keyof CloudSessionParams, unknown>>;
- if (
- typeof value.agentKey !== 'string' ||
- !parseCloudAgentKey(value.agentKey) ||
- typeof value.sessionId !== 'string' ||
- typeof value.name !== 'string'
- )
- return { ok: false, redirect: 'home', discardParams: true };
- return { ok: true };
- },
-} satisfies ViewDefinition<CloudSessionParams>;
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-start-attempt-status.tsx b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-start-attempt-status.tsx
deleted file mode 100644
index 85fd0f941..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/cloud-start-attempt-status.tsx
+++ /dev/null
@@ -1,61 +0,0 @@
-import type { Session } from '@switch-console/shared/session-v1';
-import { observer } from 'mobx-react-lite';
-import { cloudOperationAttempts, startAttemptKey } from './cloud-operation-attempts';
-
-/**
- * A new session whose start was never confirmed. Once the worker lists the
- * session it offers to open it; until then it asks again for the same session,
- * which the server dedupes, rather than starting another. Renders nothing
- * unless the agent's start attempt is in that state.
- */
-export const CloudStartAttemptStatus = observer(function CloudStartAttemptStatus({
- agentKey,
- sessions,
- onOpen,
- onCheckAgain,
- className,
-}: {
- agentKey: string;
- sessions: Session[];
- onOpen: (sessionId: string) => void;
- onCheckAgain: () => void;
- className: string;
-}) {
- const attemptKey = startAttemptKey(agentKey);
- const attempt = cloudOperationAttempts.get(attemptKey);
- if (attempt?.status !== 'unknown') return null;
- const unconfirmed = sessions.find((session) => session.sessionId === attempt.sessionId);
- return (
- <div
- role="status"
- className={`flex items-center gap-2 text-xs text-foreground-muted ${className}`}
- >
- <span className="min-w-0">
- {unconfirmed
- ? 'The new session was not confirmed, but it exists.'
- : `Not yet known whether the new session started. ${attempt.message ?? ''}`}
- </span>
- {unconfirmed ? (
- <button
- type="button"
- className="shrink-0 underline hover:text-foreground"
- onClick={() => {
- cloudOperationAttempts.settle(attemptKey);
- onOpen(attempt.sessionId);
- }}
- >
- Open
- </button>
- ) : (
- <button
- type="button"
- className="shrink-0 underline hover:text-foreground"
- title="Asks again for the same session, so it cannot start a second one."
- onClick={onCheckAgain}
- >
- Check again
- </button>
- )}
- </div>
- );
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/edit-cloud-agent-modal.tsx b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/edit-cloud-agent-modal.tsx
deleted file mode 100644
index 620f55196..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/edit-cloud-agent-modal.tsx
+++ /dev/null
@@ -1,266 +0,0 @@
-import type { RepoAgentAttributes } from '@switch-console/core/agents/plugins';
-import { useQuery, useQueryClient } from '@tanstack/react-query';
-import { observer } from 'mobx-react-lite';
-import { useCallback, useId, useRef, useState } from 'react';
-import { AgentAdvancedConfig } from '@renderer/features/locations/components/add-agent-modal/agent-advanced-config';
-import { AddressingPolicyRow } from '@renderer/features/locations/components/settings-view/sections/addressing-policy-settings-section';
-import { workspacesStore } from '@renderer/features/workspaces/workspaces-store';
-import { AgentIconPicker } from '@renderer/lib/components/agent-icon-picker';
-import { failureText } from '@renderer/lib/errors/describe-failure';
-import { toast } from '@renderer/lib/hooks/use-toast';
-import { rpc } from '@renderer/lib/ipc';
-import { type BaseModalProps } from '@renderer/lib/modal/modal-provider';
-import {
- useWorkspaceAgents,
- workspaceAgentsQueryKey,
-} from '@renderer/lib/stores/use-workspace-agents';
-import { Button } from '@renderer/lib/ui/button';
-import { ConfirmButton } from '@renderer/lib/ui/confirm-button';
-import {
- DialogContentArea,
- DialogFooter,
- DialogHeader,
- DialogTitle,
-} from '@renderer/lib/ui/dialog';
-import { Field, FieldDescription, FieldGroup, FieldLabel } from '@renderer/lib/ui/field';
-import { Input } from '@renderer/lib/ui/input';
-import { Textarea } from '@renderer/lib/ui/textarea';
-import type { CloudLaunch } from '@shared/core/cloud-agents/cloud-agents';
-import { providerDisplayName } from '@shared/core/providers/agent-provider-registry';
-import type { CloudLaunchConfiguration } from '@shared/core/switch-servers/cloud-launch';
-import type { RemoteAgentSummary } from '@shared/core/switch-servers/switch-servers';
-
-export const RESTART_NOTE =
- 'Model and instructions changes apply when the agent restarts: use Restart, or Stop then Start.';
-
-type Props = BaseModalProps<void> & { serverId: string; launch: CloudLaunch };
-
-function sameAttributes(a: RepoAgentAttributes, b: RepoAgentAttributes): boolean {
- const canonical = (value: RepoAgentAttributes) =>
- JSON.stringify(Object.entries(value).sort(([x], [y]) => x.localeCompare(y)));
- return canonical(a) === canonical(b);
-}
-
-export const EditCloudAgentModal = observer(function EditCloudAgentModal({
- serverId,
- launch,
- onSuccess,
- onClose,
-}: Props) {
- const agentId = launch.agent_id;
- const workspaceId = workspacesStore.idOnServerInScope(serverId);
- const agents = useWorkspaceAgents(workspaceId);
- const configuration = useQuery({
- queryKey: ['cloud-launch-configuration', serverId, launch.request_id],
- queryFn: () => rpc.switchServers.getCloudLaunchConfiguration(serverId, launch.request_id),
- });
- const agent = agents.data?.find((candidate) => candidate.id === agentId);
- const error = agents.error ?? configuration.error;
- const missing = agents.data !== undefined && !agent;
-
- if (
- agentId === null ||
- workspaceId === null ||
- error ||
- missing ||
- !agent ||
- !configuration.data
- ) {
- return (
- <>
- <DialogHeader>
- <DialogTitle>Edit {launch.name}</DialogTitle>
- </DialogHeader>
- <DialogContentArea>
- {workspaceId === null ? (
- <p role="alert" className="text-sm text-destructive">
- This server has no workspace open, so the agent cannot be read.
- </p>
- ) : agentId === null || missing ? (
- <p role="alert" className="text-sm text-destructive">
- This cloud agent has not registered with the server yet.
- </p>
- ) : error ? (
- <p role="alert" className="text-sm text-destructive">
- {failureText(error, 'Could not load the agent’s configuration.')}
- </p>
- ) : (
- <p className="text-sm text-foreground-muted">Loading…</p>
- )}
- </DialogContentArea>
- <DialogFooter>
- <Button variant="outline" onClick={onClose}>
- Close
- </Button>
- </DialogFooter>
- </>
- );
- }
- return (
- <EditCloudAgentForm
- serverId={serverId}
- workspaceId={workspaceId}
- launch={launch}
- agent={agent}
- configuration={configuration.data}
- onSuccess={onSuccess}
- onClose={onClose}
- />
- );
-});
-
-function EditCloudAgentForm({
- serverId,
- workspaceId,
- launch,
- agent,
- configuration,
- onSuccess,
- onClose,
-}: Props & {
- workspaceId: string;
- agent: RemoteAgentSummary;
- configuration: CloudLaunchConfiguration;
-}) {
- const queryClient = useQueryClient();
- const nameId = useId();
- const instructionsId = useId();
- const [displayName, setDisplayName] = useState(agent.displayName ?? '');
- const [iconUrl, setIconUrl] = useState(agent.iconUrl);
- const [instructions, setInstructions] = useState(configuration.instructions);
- const [initialAttributes] = useState(configuration.definition_attributes);
- const attributes = useRef<RepoAgentAttributes>(configuration.definition_attributes);
- const onAttributesChange = useCallback(
- (next: RepoAgentAttributes) => {
- attributes.current = { ...initialAttributes, ...next };
- },
- [initialAttributes]
- );
- const [saving, setSaving] = useState(false);
- const [error, setError] = useState<string | null>(null);
- const editsModel = launch.provider === 'claude';
-
- const save = async () => {
- setSaving(true);
- setError(null);
- const nextName = displayName.trim() || null;
- const nextAttributes = editsModel ? attributes.current : configuration.definition_attributes;
- const configurationChanged =
- instructions.trim() !== configuration.instructions ||
- !sameAttributes(nextAttributes, configuration.definition_attributes);
- try {
- if (nextName !== agent.displayName)
- await rpc.workspaces.updateAgentDisplayName({
- workspaceId,
- agentId: agent.id,
- displayName: nextName,
- });
- if (iconUrl !== agent.iconUrl)
- await rpc.workspaces.updateAgentIcon({ workspaceId, agentId: agent.id, iconUrl });
- if (configurationChanged)
- await rpc.switchServers.updateCloudLaunchConfiguration(serverId, launch.request_id, {
- provider: launch.provider,
- name: launch.name,
- description: configuration.description,
- instructions,
- definition_attributes: nextAttributes,
- });
- } catch (failure) {
- setError(failureText(failure, 'Could not save the agent.'));
- setSaving(false);
- return;
- } finally {
- void queryClient.invalidateQueries({ queryKey: workspaceAgentsQueryKey(workspaceId) });
- void queryClient.invalidateQueries({
- queryKey: ['cloud-launch-configuration', serverId, launch.request_id],
- });
- }
- if (configurationChanged) toast({ title: 'Agent saved', description: RESTART_NOTE });
- onSuccess();
- };
-
- return (
- <>
- <DialogHeader showCloseButton={false}>
- <DialogTitle>Edit {launch.name}</DialogTitle>
- </DialogHeader>
- <DialogContentArea className="pt-0">
- <FieldGroup>
- <div className="flex justify-center">
- <AgentIconPicker
- serverId={serverId}
- name={launch.name}
- iconUrl={iconUrl}
- onChange={setIconUrl}
- size={66}
- disabled={saving}
- />
- </div>
- <Field>
- <FieldLabel htmlFor={nameId}>
- Display name <span className="text-foreground-muted">(optional)</span>
- </FieldLabel>
- <Input
- id={nameId}
- value={displayName}
- placeholder={launch.name}
- disabled={saving}
- onChange={(e) => setDisplayName(e.target.value)}
- />
- </Field>
- <Field>
- <FieldLabel>Provider</FieldLabel>
- <FieldDescription>
- {providerDisplayName(launch.provider)}. The provider and repository cannot be changed;
- create a new agent instead.
- </FieldDescription>
- </Field>
- <AddressingPolicyRow
- workspaceId={workspaceId}
- serverId={serverId}
- agentId={agent.id}
- agentName={launch.name}
- showName={false}
- />
- <Field>
- <FieldLabel htmlFor={instructionsId}>
- Agent instructions <span className="text-foreground-muted">(optional)</span>
- </FieldLabel>
- <Textarea
- id={instructionsId}
- rows={4}
- placeholder="How this agent should work"
- value={instructions}
- disabled={saving}
- onChange={(e) => setInstructions(e.target.value)}
- />
- </Field>
- {editsModel && (
- <AgentAdvancedConfig
- providerId={launch.provider}
- cloud
- sshHost={null}
- dir=""
- initial={initialAttributes}
- onChange={onAttributesChange}
- />
- )}
- <FieldDescription>{RESTART_NOTE}</FieldDescription>
- {error && (
- <p role="alert" className="text-xs text-destructive">
- {error}
- </p>
- )}
- </FieldGroup>
- </DialogContentArea>
- <DialogFooter>
- <Button variant="outline" onClick={onClose} disabled={saving}>
- Cancel
- </Button>
- <ConfirmButton onClick={() => void save()} disabled={saving}>
- {saving ? 'Saving…' : 'Save'}
- </ConfirmButton>
- </DialogFooter>
- </>
- );
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/use-cloud-agents.ts b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/use-cloud-agents.ts
index 811f9d2f4..1d6d78905 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/use-cloud-agents.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/cloud-agents/use-cloud-agents.ts
@@ -1,162 +1,35 @@
-import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
+import { useQuery } from '@tanstack/react-query';
import { switchRoomsStore } from '@renderer/features/switch-servers/switch-rooms-store';
import { switchServersStore } from '@renderer/features/switch-servers/switch-servers-store';
import { workspacesStore } from '@renderer/features/workspaces/workspaces-store';
import { rpc } from '@renderer/lib/ipc';
-import { type CloudAgent, parseCloudAgentKey } from '@shared/core/cloud-agents/cloud-agents';
+
+export const CLOUD_MACHINES_KEY = 'cloud-machines';
/**
* Whether the server's workspace on screen went unasked because Switch Console
* is not signed in to the server: the sidebar already says to sign in.
*/
-export function serverNotSignedIn(serverId: string | null): boolean {
+function serverNotSignedIn(serverId: string | null): boolean {
const workspaceId = workspacesStore.idOnServerInScope(serverId);
return switchRoomsStore.workspacesNotSignedIn.some((workspace) => workspace.id === workspaceId);
}
/**
- * The server's cloud agents, from its launch list; no worker is asked. Not
- * asked while signed out: the sidebar already says to sign in.
+ * The caller's cloud machines on the server. Not asked while signed out: the
+ * sidebar already says to sign in.
*
- * `null` means the server has no cloud agents. It is not asked again until its
- * session or declared version changes, which is when it may have gained them.
- */
-export function useCloudAgents(serverId: string | null) {
- const signedOut = serverNotSignedIn(serverId);
- const user = serverId === null ? null : (switchServersStore.statusFor(serverId)?.user ?? null);
- return useQuery({
- queryKey: ['cloud-agents', serverId, user?.id ?? null, user?.server?.version ?? null],
- queryFn: () => rpc.sdkHost.cloudAgents(serverId!),
- enabled: serverId !== null && !signedOut,
- refetchInterval: (query) => (query.state.data === null ? false : 5000),
- retry: false,
- });
-}
-
-/**
- * The caller's cloud machines on the server, asked the way `useCloudAgents`
- * asks for launches. `null` means the server has no cloud agents. Under
- * `['cloud-agents']`, so every refresh of the list refreshes this too.
+ * `null` means the server has no cloud machines. It is not asked again until
+ * its session changes, which is when it may have gained them.
*/
export function useCloudMachines(serverId: string | null) {
const signedOut = serverNotSignedIn(serverId);
const user = serverId === null ? null : (switchServersStore.statusFor(serverId)?.user ?? null);
return useQuery({
- queryKey: ['cloud-agents', serverId, 'machines', user?.id ?? null],
- queryFn: () => rpc.sdkHost.cloudMachines(serverId!),
+ queryKey: [CLOUD_MACHINES_KEY, serverId, user?.id ?? null],
+ queryFn: () => rpc.switchServers.cloudMachines(serverId!),
enabled: serverId !== null && !signedOut,
refetchInterval: (query) => (query.state.data === null ? false : 5000),
retry: false,
});
}
-
-/**
- * The agent with its worker's sessions, asked over the relay only while
- * `watched` and while the window is visible, since each ask is a round trip
- * through the server. A launch that says its worker cannot be asked is not,
- * and carries the sessions last read from it while its machine is down.
- * Under `['cloud-agents']`, so every refresh of the list refreshes this too.
- */
-export function useCloudAgentSessions(agent: CloudAgent, watched: boolean): CloudAgent;
-export function useCloudAgentSessions(
- agent: CloudAgent | undefined,
- watched: boolean
-): CloudAgent | undefined;
-export function useCloudAgentSessions(
- agent: CloudAgent | undefined,
- watched: boolean
-): CloudAgent | undefined {
- const listed = useQuery({
- queryKey: [
- 'cloud-agents',
- parseCloudAgentKey(agent?.key ?? '')?.serverId,
- 'sessions',
- agent?.key,
- ],
- queryFn: () => rpc.sdkHost.cloudSessions(agent!.key),
- enabled: watched && agent !== undefined && !agent.problem,
- refetchInterval: 5000,
- retry: false,
- });
- if (!agent || agent.problem) return agent;
- if (listed.error)
- return {
- ...agent,
- sessions: null,
- problem: { code: 'failed', message: String(listed.error), wakeAvailable: false },
- };
- return listed.data ? { ...agent, ...listed.data } : agent;
-}
-
-/** Start the machine a sleeping agent runs on, then look again. */
-export function useCloudWake() {
- const queryClient = useQueryClient();
- return useMutation({
- mutationFn: (agentKey: string) => rpc.sdkHost.cloudWake(agentKey),
- onSettled: () => queryClient.invalidateQueries({ queryKey: ['cloud-agents'] }),
- });
-}
-
-/** Start or retry the machine an agent runs on, at the revision it was read at. */
-function useCloudMachineAction() {
- const queryClient = useQueryClient();
- return useMutation({
- mutationFn: ({ agent, action }: { agent: CloudAgent; action: 'start' | 'retry' }) => {
- const key = parseCloudAgentKey(agent.key);
- if (!key || !agent.machine) throw new Error(`${agent.launch.name} has no cloud machine.`);
- return rpc.switchServers.cloudMachineLifecycle(
- key.serverId,
- agent.machine.machine_id,
- action,
- agent.machine.revision
- );
- },
- onSettled: () => queryClient.invalidateQueries({ queryKey: ['cloud-agents'] }),
- });
-}
-
-export type CloudProblemAction = {
- label: string;
- pending: boolean;
- error: string | null;
- run: () => void;
-};
-
-/**
- * What the user can do about why an agent's worker cannot be asked: wake a
- * sleeping machine (only where `wake`, since a composer wakes it by sending),
- * start one its owner stopped, or retry one in error.
- */
-export function useCloudProblemAction(agent: CloudAgent, wake: boolean): CloudProblemAction | null {
- const waking = useCloudWake();
- const machineAction = useCloudMachineAction();
- const failed = (error: Error | null, what: string) =>
- error ? `Could not ${what}: ${String(error)}` : null;
- const problem = agent.problem;
- if (wake && problem?.code === 'worker_sleeping' && problem.wakeAvailable)
- return {
- label: waking.isPending ? 'Waking…' : 'Wake',
- pending: waking.isPending,
- error: failed(waking.error, 'wake the machine'),
- run: () => waking.mutate(agent.key),
- };
- if (problem?.code === 'machine_stopped' && agent.machine)
- return {
- label: machineAction.isPending ? 'Starting…' : 'Start machine',
- pending: machineAction.isPending,
- error: failed(machineAction.error, 'start the machine'),
- run: () => machineAction.mutate({ agent, action: 'start' }),
- };
- if (
- problem?.code === 'machine_error' &&
- agent.machine &&
- agent.machine.error_code !== 'machine_needs_attention'
- )
- return {
- label: machineAction.isPending ? 'Retrying…' : 'Retry machine',
- pending: machineAction.isPending,
- error: failed(machineAction.error, 'retry the machine'),
- run: () => machineAction.mutate({ agent, action: 'retry' }),
- };
- return null;
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/agent-advanced-config.tsx b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/agent-advanced-config.tsx
index b6930c45e..805d20999 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/agent-advanced-config.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/agent-advanced-config.tsx
@@ -25,14 +25,12 @@ import { fieldCatalogueState, type ModelCatalogueResult } from '../agent-model-c
*/
export function AgentAdvancedConfig({
providerId,
- cloud,
sshHost,
dir,
initial,
onChange,
}: {
providerId: AgentProviderId | null;
- cloud: boolean;
sshHost: string | null;
dir: string;
/** The attributes the form starts from; pass a stable value, a new one resets the form. */
@@ -49,9 +47,7 @@ export function AgentAdvancedConfig({
? catalogue
: {
kind: 'unavailable' as const,
- reason: cloud
- ? 'Model suggestions are unavailable before the cloud worker starts. You can enter a model alias or ID.'
- : 'No execution directory is configured yet. You can enter a model alias or ID.',
+ reason: 'No execution directory is configured yet. You can enter a model alias or ID.',
};
const { data: allFields } = useQuery({
queryKey: ['agentDefinitionFields', providerId],
diff --git a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/cloud-agent-repository.tsx b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/cloud-agent-repository.tsx
deleted file mode 100644
index 960d3627d..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/cloud-agent-repository.tsx
+++ /dev/null
@@ -1,187 +0,0 @@
-import { useQuery } from '@tanstack/react-query';
-import { useEffect, useState } from 'react';
-import { AgentIcon } from '@renderer/lib/components/agent-icon';
-import { failureText } from '@renderer/lib/errors/describe-failure';
-import { rpc } from '@renderer/lib/ipc';
-import { Button } from '@renderer/lib/ui/button';
-import { Field, FieldDescription, FieldLabel } from '@renderer/lib/ui/field';
-import {
- Select,
- SelectContent,
- SelectItem,
- SelectTrigger,
- SelectValue,
-} from '@renderer/lib/ui/select';
-import { Spinner } from '@renderer/lib/ui/spinner';
-import {
- AGENT_PROVIDERS,
- providerDisplayName,
- type AgentProviderId,
-} from '@shared/core/providers/agent-provider-registry';
-import type { CloudRepositorySelection } from '@shared/core/switch-servers/cloud-launch';
-
-export function CloudAgentRepository({
- serverId,
- providerId,
- onProviderChange,
- onSelection,
- onConnectProvider,
- onConnectGitHub,
-}: {
- serverId: string;
- providerId: AgentProviderId;
- onProviderChange: (provider: AgentProviderId) => void;
- onSelection: (value: CloudRepositorySelection | null) => void;
- onConnectProvider: () => void;
- onConnectGitHub: () => void;
-}) {
- const [repository, setRepository] = useState<string | null>(null);
- const provider = useQuery({
- queryKey: ['cloud-agent-connections', serverId, providerId],
- queryFn: () => rpc.switchServers.getCloudProviderConnection(serverId, providerId),
- staleTime: 0,
- retry: false,
- refetchInterval: (query) => (query.state.data?.status === 'verifying' ? 2000 : false),
- });
- const github = useQuery({
- queryKey: ['cloud-agent-github', serverId],
- queryFn: () => rpc.switchServers.getGitHubConnection(serverId),
- staleTime: 0,
- retry: false,
- });
- const repositories =
- github.data?.status === 'connected'
- ? github.data.installations.flatMap((installation) =>
- installation.repositories.map((repo) => ({
- value: `${installation.id}:${repo.id}`,
- label: repo.name,
- }))
- )
- : [];
- const selected = repositories.some((repo) => repo.value === repository)
- ? repository
- : repositories.length === 1
- ? repositories[0].value
- : null;
- const connected = provider.data?.status === 'connected' || provider.data?.status === 'configured';
- useEffect(() => {
- const ids = selected?.split(':').map(Number);
- onSelection(connected && ids ? { installationId: ids[0], repositoryId: ids[1] } : null);
- return () => onSelection(null);
- }, [selected, connected, onSelection]);
- return (
- <>
- <Field>
- <FieldLabel>Agent provider</FieldLabel>
- <div className="flex items-center gap-2 rounded-md border p-3 text-sm">
- <AgentIcon id={providerId} className="size-5" />
- <Select
- value={providerId}
- onValueChange={(value) => {
- if (value) onProviderChange(value as AgentProviderId);
- }}
- >
- <SelectTrigger aria-label="Cloud provider">
- <SelectValue>{providerDisplayName(providerId)}</SelectValue>
- </SelectTrigger>
- <SelectContent>
- {AGENT_PROVIDERS.map((provider) => (
- <SelectItem key={provider.id} value={provider.id}>
- {provider.name}
- </SelectItem>
- ))}
- </SelectContent>
- </Select>
- <span className="ml-auto text-xs text-foreground-muted">
- {provider.data
- ? provider.data.status === 'connected'
- ? 'Verified'
- : provider.data.status === 'configured'
- ? 'Credential saved'
- : provider.data.status === 'verifying'
- ? 'Checking connection…'
- : provider.data.status === 'failed'
- ? 'Connection failed'
- : 'Not connected'
- : 'Checking connection'}
- </span>
- {provider.error && (
- <Button variant="outline" size="sm" onClick={onConnectProvider}>
- Retry connection
- </Button>
- )}
- {provider.data &&
- ['not_connected', 'failed', 'verifying'].includes(provider.data.status) && (
- <Button
- variant="outline"
- size="sm"
- aria-label={`Connect ${providerDisplayName(providerId)}`}
- onClick={onConnectProvider}
- >
- {provider.data.status === 'verifying'
- ? 'View'
- : provider.data.status === 'failed'
- ? 'Retry'
- : 'Connect'}
- </Button>
- )}
- </div>
- </Field>
- {(provider.isPending || github.isPending) && (
- <p role="status" className="flex items-center gap-2 text-sm">
- <Spinner /> Loading cloud connections…
- </p>
- )}
- {(provider.error || github.error) && (
- <div role="alert" className="space-y-2 text-sm text-destructive">
- <p>{failureText(provider.error || github.error, 'Could not load cloud connections.')}</p>
- <Button
- variant="outline"
- onClick={() => {
- void provider.refetch();
- void github.refetch();
- }}
- >
- Retry
- </Button>
- </div>
- )}
- <Button variant="outline" onClick={onConnectGitHub}>
- {github.data?.status === 'connected'
- ? 'Manage GitHub access'
- : github.error
- ? 'Reconnect GitHub'
- : 'Connect GitHub'}
- </Button>
- {github.data && (
- <Field>
- <FieldLabel htmlFor="cloud-agent-repository">Repository</FieldLabel>
- <Select
- items={repositories}
- value={selected}
- onValueChange={setRepository}
- disabled={!repositories.length}
- >
- <SelectTrigger id="cloud-agent-repository" className="w-full">
- <SelectValue placeholder="Choose a repository" />
- </SelectTrigger>
- <SelectContent>
- {repositories.map((repo) => (
- <SelectItem key={repo.value} value={repo.value}>
- {repo.label}
- </SelectItem>
- ))}
- </SelectContent>
- </Select>
- <FieldDescription>
- {github.data.status !== 'connected'
- ? 'Connect GitHub to choose a repository.'
- : !repositories.length
- ? 'Grant Switch access to a repository in GitHub to continue.'
- : 'Only repositories shared with Switch on GitHub appear here.'}
- </FieldDescription>
- </Field>
- )}
- </>
- );
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/configure-agent-panel.tsx b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/configure-agent-panel.tsx
index d44486d05..d031deb78 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/configure-agent-panel.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/configure-agent-panel.tsx
@@ -215,10 +215,13 @@ export const AgentSettingsSection = observer(function AgentSettingsSection({
export function AgentIdentityFields({
form,
serverId,
+ nameTaken,
}: {
form: ConfigureAgentFormState;
/** The Switch server whose generated icons are offered. */
serverId: string | null;
+ /** Another agent on the server already has this name. */
+ nameTaken: boolean;
}) {
const nameId = useId();
const descriptionId = useId();
@@ -250,7 +253,7 @@ export function AgentIdentityFields({
placeholder="Name this agent"
value={form.agentName}
onChange={(e) => form.setAgentName(e.target.value)}
- aria-invalid={form.nameIsRejected}
+ aria-invalid={form.nameIsRejected || nameTaken}
/>
{form.nameIsRejected ? (
<Alert
@@ -281,6 +284,17 @@ export function AgentIdentityFields({
</AlertAction>
)}
</Alert>
+ ) : nameTaken ? (
+ <Alert
+ variant="destructive"
+ className="border-border-destructive bg-background-destructive"
+ >
+ <TriangleAlert />
+ <AlertDescription className="text-foreground-destructive">
+ An agent called <span className="font-mono">{form.agentName}</span> already exists on
+ this server. Pick another name.
+ </AlertDescription>
+ </Alert>
) : form.agentName.length > 0 ? (
// Once there is a name, show the handle it produces rather than
// repeating the advice: the advice is about choosing a name, and it
diff --git a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/give-machine-login.tsx b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/give-machine-login.tsx
new file mode 100644
index 000000000..bf4e0f934
--- /dev/null
+++ b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/give-machine-login.tsx
@@ -0,0 +1,186 @@
+import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
+import { CircleAlert, CircleCheck, KeyRound } from 'lucide-react';
+import { useEffect, useState } from 'react';
+import { MANAGED_AGENTS_KEY } from '@renderer/features/managed-agents/use-managed-agents';
+import { rpc } from '@renderer/lib/ipc';
+import { Button } from '@renderer/lib/ui/button';
+import { Input } from '@renderer/lib/ui/input';
+import { SegmentedControl } from '@renderer/lib/ui/segmented-control';
+import { Spinner } from '@renderer/lib/ui/spinner';
+import type { MachineLoginInput, OwnedMachine } from '@shared/core/managed-agents/managed-agents';
+import {
+ type AgentProviderId,
+ providerDisplayName,
+} from '@shared/core/providers/agent-provider-registry';
+
+type LoginKind = 'setup-token' | 'api-key' | 'this-computer';
+
+/** The ways a machine can be given each provider's login. */
+export function loginKindsFor(provider: AgentProviderId): { value: LoginKind; label: string }[] {
+ switch (provider) {
+ case 'claude':
+ return [
+ { value: 'setup-token', label: 'Setup token' },
+ { value: 'api-key', label: 'API key' },
+ ];
+ case 'codex':
+ return [
+ { value: 'api-key', label: 'API key' },
+ { value: 'this-computer', label: "This computer's sign-in" },
+ ];
+ case 'cursor':
+ return [{ value: 'api-key', label: 'API key' }];
+ default:
+ return [{ value: 'this-computer', label: "This computer's sign-in" }];
+ }
+}
+
+function displayName(provider: AgentProviderId): string {
+ return providerDisplayName(provider) ?? provider;
+}
+
+function hint(provider: AgentProviderId, kind: LoginKind): string {
+ if (kind === 'setup-token')
+ return 'Run `claude setup-token` on any computer signed in to Claude, and paste the token it prints.';
+ if (kind === 'this-computer')
+ return `The ${displayName(provider)} sign-in on this computer is sent, sealed for the machine.`;
+ return `Paste a ${displayName(provider)} API key.`;
+}
+
+/**
+ * Gives a machine a provider login, on demand: sealed in Console to the
+ * machine's own key, so the server only relays it, and taken up by the machine
+ * at once. Says whether the provider signs in with it there.
+ */
+export function GiveMachineLogin({
+ serverId,
+ machine,
+ provider,
+ onClose,
+}: {
+ serverId: string;
+ machine: OwnedMachine;
+ provider: AgentProviderId;
+ onClose: () => void;
+}) {
+ const queryClient = useQueryClient();
+ const kinds = loginKindsFor(provider);
+ const [kind, setKind] = useState<LoginKind>(kinds[0]!.value);
+ const [credential, setCredential] = useState('');
+ const [operationId, setOperationId] = useState<string | null>(null);
+ const name = displayName(provider);
+
+ const give = useMutation({
+ mutationFn: () => {
+ const login: MachineLoginInput =
+ kind === 'this-computer'
+ ? { source: 'this-computer' }
+ : { source: 'typed', kind, credential };
+ return rpc.managedAgents.giveMachineLogin({
+ serverId,
+ machineId: machine.id,
+ provider,
+ login,
+ });
+ },
+ onSuccess: ({ operationId: id }) => {
+ setCredential('');
+ setOperationId(id);
+ },
+ });
+
+ const outcome = useQuery({
+ queryKey: [MANAGED_AGENTS_KEY, serverId, 'machine-login', operationId],
+ queryFn: () =>
+ rpc.managedAgents.machineLoginOutcome({
+ serverId,
+ machineId: machine.id,
+ operationId: operationId!,
+ }),
+ enabled: operationId !== null,
+ refetchInterval: (query) => (query.state.data?.state === 'pending' ? 1500 : false),
+ });
+
+ const result = outcome.data;
+ const succeeded = result?.state === 'succeeded';
+ // The machine now reports the provider ready: show it without waiting for the next poll.
+ useEffect(() => {
+ if (succeeded)
+ void queryClient.invalidateQueries({ queryKey: [MANAGED_AGENTS_KEY, serverId, 'machines'] });
+ }, [succeeded, queryClient, serverId]);
+ const waiting =
+ give.isPending || (operationId !== null && (!result || result.state === 'pending'));
+
+ return (
+ <div className="space-y-3 rounded-lg border bg-background-1 p-3 text-sm" role="group">
+ <div className="flex items-center gap-2">
+ <KeyRound className="size-4 text-foreground-muted" />
+ <span>
+ Give {machine.name} a {name} login
+ </span>
+ </div>
+ {kinds.length > 1 && (
+ <SegmentedControl
+ value={kind}
+ onChange={(next) => {
+ setKind(next);
+ setCredential('');
+ give.reset();
+ }}
+ options={kinds}
+ ariaLabel={`${name} login`}
+ />
+ )}
+ <p className="text-xs text-foreground-muted">{hint(provider, kind)}</p>
+ {kind !== 'this-computer' && (
+ <Input
+ type="password"
+ autoComplete="off"
+ aria-label={kind === 'setup-token' ? 'Setup token' : 'API key'}
+ value={credential}
+ onChange={(event) => setCredential(event.target.value)}
+ disabled={waiting}
+ />
+ )}
+ {waiting && operationId !== null && (
+ <p className="flex items-center gap-2 text-xs text-foreground-muted" role="status">
+ <Spinner className="size-3.5" />
+ Waiting for {machine.name} to check the login…
+ </p>
+ )}
+ {result?.state === 'succeeded' && (
+ <p className="flex items-center gap-2 text-xs" role="status">
+ <CircleCheck className="size-3.5 text-emerald-500" />
+ {name} signs in on {machine.name}.
+ </p>
+ )}
+ {(give.error || result?.state === 'failed') && (
+ <p className="flex items-start gap-2 text-xs" role="alert">
+ <CircleAlert className="mt-0.5 size-3.5 shrink-0 text-amber-500" />
+ {give.error
+ ? String(give.error instanceof Error ? give.error.message : give.error)
+ : result?.state === 'failed'
+ ? result.message
+ : null}
+ </p>
+ )}
+ <div className="flex justify-end gap-2">
+ <Button variant="ghost" size="sm" onClick={onClose}>
+ {result?.state === 'succeeded' ? 'Done' : 'Cancel'}
+ </Button>
+ {result?.state !== 'succeeded' && (
+ <Button
+ size="sm"
+ disabled={waiting || (kind !== 'this-computer' && !credential.trim())}
+ onClick={() => {
+ setOperationId(null);
+ give.mutate();
+ }}
+ >
+ Give login
+ </Button>
+ )}
+ </div>
+ </div>
+ );
+}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/machine-provider-picker.tsx b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/machine-provider-picker.tsx
index 43e4a0bcb..ed896691d 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/machine-provider-picker.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/machine-provider-picker.tsx
@@ -1,5 +1,5 @@
import { CircleAlert } from 'lucide-react';
-import { useEffect, useMemo } from 'react';
+import { useEffect, useMemo, useState } from 'react';
import { AgentIcon } from '@renderer/lib/components/agent-icon';
import { Field, FieldLabel } from '@renderer/lib/ui/field';
import { cn } from '@renderer/utils/utils';
@@ -9,6 +9,7 @@ import {
type AgentProviderId,
} from '@shared/core/providers/agent-provider-registry';
import { autoSelectedAgentType } from './agent-type-auto-selection';
+import { GiveMachineLogin } from './give-machine-login';
/** Each provider, and whether the machine last reported it installed and logged in. */
export function machineProviderOptions(
@@ -30,11 +31,13 @@ export function machineProviderOptions(
* server: only the providers installed and logged in there can be picked.
*/
export function MachineProviderPicker({
+ serverId,
machine,
value,
onChange,
defaultAgent,
}: {
+ serverId: string;
machine: OwnedMachine;
value: AgentProviderId | null;
onChange: (providerId: AgentProviderId) => void;
@@ -46,6 +49,11 @@ export function MachineProviderPicker({
() => options.filter((option) => option.ready).map((option) => option.id),
[options]
);
+ // Installed there but not signed in: a login can be given to the machine.
+ const givable = machine.acceptsLogins
+ ? options.filter((option) => !option.ready && option.problem !== 'not installed')
+ : [];
+ const [giving, setGiving] = useState<AgentProviderId | null>(null);
useEffect(() => {
if (value) return;
@@ -94,6 +102,30 @@ export function MachineProviderPicker({
<p className="text-xs text-foreground-muted">
As {machine.name} last reported to the server.
</p>
+ {giving ? (
+ <GiveMachineLogin
+ serverId={serverId}
+ machine={machine}
+ provider={giving}
+ onClose={() => setGiving(null)}
+ />
+ ) : (
+ givable.length > 0 && (
+ <div className="flex flex-wrap items-center gap-2 text-xs text-foreground-muted">
+ <span>Give {machine.name} a login for</span>
+ {givable.map((option) => (
+ <button
+ key={option.id}
+ type="button"
+ className="cursor-pointer rounded-md border border-border px-2 py-0.5 text-foreground hover:bg-[var(--sel-soft)]"
+ onClick={() => setGiving(option.id)}
+ >
+ {option.name}
+ </button>
+ ))}
+ </div>
+ )
+ )}
</Field>
);
}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/new-agent-form.tsx b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/new-agent-form.tsx
index 3f65b1ff8..ad9f975de 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/new-agent-form.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/new-agent-form.tsx
@@ -14,10 +14,8 @@ import {
} from '@renderer/features/remote-hosts/host-readiness-notice';
import { useAppSettingsKey } from '@renderer/features/settings/use-app-settings-key';
import { policyHasDeadRule } from '@renderer/features/switch-servers/addressing-policy-editor';
-import { ManagedGitHubStep } from '@renderer/features/switch-servers/managed-github-step';
-import { ManagedProviderConnectionStep } from '@renderer/features/switch-servers/managed-provider-connection-step';
+import { isSwitchCloudServer } from '@renderer/features/switch-servers/switch-cloud-origin';
import { switchServersStore } from '@renderer/features/switch-servers/switch-servers-store';
-import { isSwitchCloudServer } from '@renderer/features/switch-servers/use-cloud-launches';
import { workspacesStore } from '@renderer/features/workspaces/workspaces-store';
import { ProviderConnectionStatus } from '@renderer/lib/components/provider-connection-status';
import { describeFailure, failureText } from '@renderer/lib/errors/describe-failure';
@@ -47,15 +45,14 @@ import {
import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from '@renderer/lib/ui/tooltip';
import { log } from '@renderer/utils/logger';
import type { AgentProviderConfig } from '@shared/core/agents/agent-provider-config';
+import type { CloudMachine } from '@shared/core/cloud-agents/cloud-agents';
import {
describeRemoteDirRefusal,
isAbsoluteRemoteDir,
} from '@shared/core/remote-hosts/remote-dir';
-import type { CloudRepositorySelection } from '@shared/core/switch-servers/cloud-launch';
import type { UiEntryPoint } from '@shared/core/telemetry/reporting';
import { AgentAdvancedConfig } from './agent-advanced-config';
import { AgentTypePicker } from './agent-type-picker';
-import { CloudAgentRepository } from './cloud-agent-repository';
import { AgentIdentityFields, AgentSettingsSection } from './configure-agent-panel';
import { LaunchProfileConfig } from './launch-profile-config';
import { LocalDirectorySelector } from './local-directory-selector';
@@ -108,11 +105,7 @@ export const NewAgentForm = observer(function NewAgentForm({
initialRunLocation,
}: NewAgentFormProps) {
const queryClient = useQueryClient();
- const [connectingGitHub, setConnectingGitHub] = useState(false);
- const [connectingProvider, setConnectingProvider] = useState(false);
const [submitState, setSubmitState] = useState<'idle' | 'creating'>('idle');
- const [cloudRepository, setCloudRepository] = useState<CloudRepositorySelection | null>(null);
- const cloudRequestId = useRef(crypto.randomUUID());
const { navigate } = useNavigate();
const { setCloseGuard } = useModalContext();
const showAddServerModal = useShowModal('addServerModal');
@@ -156,6 +149,26 @@ export const NewAgentForm = observer(function NewAgentForm({
});
const serverMachines = askForMachines ? (machinesQuery.data ?? null) : null;
const management = serverMachines !== null;
+ // Choosing Switch cloud claims the owner's cloud machine and starts it. Once
+ // its agents controller has enrolled and is online, the agent is placed on it
+ // like on any machine.
+ const cloudMachineQuery = useQuery({
+ queryKey: ['cloud-machine-ensure', selectedServerId],
+ queryFn: () => rpc.switchServers.ensureCloudMachine(selectedServerId!),
+ enabled: isCloudRun && !!selectedServerId,
+ retry: false,
+ refetchInterval: (query) => (query.state.data ? 5000 : false),
+ });
+ const cloudController = isCloudRun
+ ? (serverMachines?.find(
+ (candidate) => candidate.id === cloudMachineQuery.data?.controller_id
+ ) ?? null)
+ : null;
+ // An online cloud machine is listed as a machine of its own; the Switch cloud
+ // entry is what claims or starts one.
+ const cloudMachineListed = !!serverMachines?.some(
+ (candidate) => candidate.cloud && candidate.state === 'online'
+ );
const machineId = isCloudRun ? null : machineIdOf(runHost);
const isMachineRun = machineId !== null;
const serverMachine = machineId
@@ -233,6 +246,11 @@ export const NewAgentForm = observer(function NewAgentForm({
}
}, [allowedHosts, runHost]);
+ // The cloud machine's controller is online: the agent goes on it, as on any machine.
+ useEffect(() => {
+ if (cloudController?.state === 'online') setRunHost(machineRunLocation(cloudController.id));
+ }, [cloudController]);
+
// This computer or an SSH host that is a machine on the server is picked as
// that machine; a machine the server no longer lists falls back to this computer.
useEffect(() => {
@@ -267,8 +285,8 @@ export const NewAgentForm = observer(function NewAgentForm({
useEffect(() => {
setRemoteRepoDir('');
setEditedMachineDir(null);
- setProviderId(isCloudRun ? 'claude' : null);
- }, [runHost, isCloudRun, setProviderId]);
+ setProviderId(null);
+ }, [runHost, setProviderId]);
const { suggestAutoApprove } = form;
const runsElsewhere =
@@ -313,7 +331,11 @@ export const NewAgentForm = observer(function NewAgentForm({
const machine = isCloudRun || isMachineRun ? undefined : machineQuery.data;
const managedRun = isMachineRun || machine?.management === true;
const machineReason = isCloudRun
- ? null
+ ? cloudMachineQuery.error
+ ? failureText(cloudMachineQuery.error, 'Your cloud machine could not be started.')
+ : cloudMachineQuery.data
+ ? cloudStartingText(cloudMachineQuery.data)
+ : null
: askForMachines && machinesQuery.isPending
? 'Checking whether this server runs managed agents…'
: askForMachines && machinesQuery.error
@@ -387,22 +409,21 @@ export const NewAgentForm = observer(function NewAgentForm({
!policyHasDeadRule(form.addressingPolicy) &&
!!pickState.serverId &&
!!pickState.providerId &&
- (isCloudRun ? cloudRepository !== null : isMachineRun || dir.trim().length > 0) &&
+ !isCloudRun &&
+ (isMachineRun || dir.trim().length > 0) &&
remoteDirIsAbsolute &&
machineProviderReady &&
runHostReachable &&
runHostReady &&
machineReason === null &&
- submitState === 'idle' &&
- !connectingProvider &&
- !connectingGitHub;
+ submitState === 'idle';
// Why "Add agent" is greyed out, in one line, shown on hover over the button.
const disabledReason: string | null =
submitState !== 'idle'
? null
- : isCloudRun && !cloudRepository
- ? 'Connect the provider and choose a GitHub repository.'
+ : isCloudRun && cloudMachineQuery.isPending
+ ? 'Starting your cloud machine…'
: !pickState.serverId
? 'Add a Switch server to register this agent on.'
: form.agentName.trim().length === 0
@@ -517,35 +538,6 @@ export const NewAgentForm = observer(function NewAgentForm({
setCloseGuard(true);
let registered = false;
try {
- if (isCloudRun && cloudRepository) {
- await rpc.switchServers.createCloudLaunch(pickState.serverId, {
- provider: pickState.providerId ?? 'claude',
- request_id: cloudRequestId.current,
- name: form.agentName,
- description: form.description.trim(),
- display_name: form.displayName.trim() || null,
- icon_url: form.iconUrl,
- instructions: form.instructions,
- installation_id: cloudRepository.installationId,
- repository_id: cloudRepository.repositoryId,
- definition_attributes:
- pickState.providerId === 'claude' ? advancedAttributesRef.current : {},
- // Every agent starts a session when addressed; there is no setting for it.
- auto_session: true,
- auto_approve: form.autoApprove,
- addressing_policy: form.addressingPolicy,
- });
- void queryClient.invalidateQueries({ queryKey: ['cloud-agents'] });
- setCloseGuard(false);
- setSubmitState('idle');
- onClose();
- navigate('serverAgents', { serverId: pickState.serverId });
- toast({
- title: 'Cloud agent is starting',
- description: 'Its progress appears in Your Agents.',
- });
- return;
- }
const identity = {
name: form.agentName,
providerId: pickState.providerId,
@@ -565,6 +557,8 @@ export const NewAgentForm = observer(function NewAgentForm({
dir: trimmedRemoteDir || null,
model: managedSettingsRef.current.model,
advancedConfig: managedSettingsRef.current.advancedConfig,
+ // A Switch cloud machine runs every agent as a user of its own.
+ isolation: serverMachine.cloud ? 'isolated' : 'shared',
});
if (created.kind !== 'created') {
reportProvisionError(created);
@@ -573,13 +567,13 @@ export const NewAgentForm = observer(function NewAgentForm({
return;
}
registered = true;
- if (form.addressingPolicy !== null) {
- await rpc.workspaces.updateAddressingPolicy({
- workspaceId: created.workspaceId,
- agentId: created.switchAgentId,
- policy: form.addressingPolicy,
- });
- }
+ // Always set: "Anyone" is no policy, and Switch creates an agent
+ // owner-only.
+ await rpc.workspaces.updateAddressingPolicy({
+ workspaceId: created.workspaceId,
+ agentId: created.switchAgentId,
+ policy: form.addressingPolicy,
+ });
if (canManageAgents) {
await rpc.workspaces.updateCanManageAgents({
workspaceId: created.workspaceId,
@@ -612,11 +606,7 @@ export const NewAgentForm = observer(function NewAgentForm({
return;
}
registered = true;
- if (
- form.addressingPolicy !== null &&
- result.agent.switchAgentId &&
- result.agent.workspaceId
- ) {
+ if (result.agent.switchAgentId && result.agent.workspaceId) {
await rpc.workspaces.updateAddressingPolicy({
workspaceId: result.agent.workspaceId,
agentId: result.agent.switchAgentId,
@@ -629,25 +619,6 @@ export const NewAgentForm = observer(function NewAgentForm({
log.error(error);
setCloseGuard(false);
setSubmitState('idle');
- if (isCloudRun) {
- try {
- const agents = await rpc.sdkHost.cloudAgents(pickState.serverId);
- const existing = (agents ?? [])
- .map((agent) => agent.launch)
- .find((launch) => launch.request_id === cloudRequestId.current);
- if (existing) {
- onClose();
- navigate('serverAgents', { serverId: pickState.serverId });
- toast({
- title: 'Cloud agent already created',
- description: `Check ${existing.name} in Your Agents for its current state.`,
- });
- return;
- }
- } catch (lookupError) {
- log.warn('Could not check the cloud creation request', lookupError);
- }
- }
if (registered) {
onClose();
navigate('serverAgents', { serverId: pickState.serverId });
@@ -661,9 +632,7 @@ export const NewAgentForm = observer(function NewAgentForm({
}
const { headline, detail } = describeFailure(
error,
- isCloudRun
- ? 'Could not confirm cloud agent creation. Retry with the same details to check the request.'
- : 'Could not add the agent. Nothing was created — check the directory is reachable and writable, then try again.'
+ 'Could not add the agent. Nothing was created — check the directory is reachable and writable, then try again.'
);
toast({ title: headline, description: detail ?? undefined, variant: 'destructive' });
}
@@ -671,370 +640,347 @@ export const NewAgentForm = observer(function NewAgentForm({
const handleCreate = () => createNewAgent();
- const finishConnection = () => {
- void queryClient.invalidateQueries({
- queryKey: ['cloud-agent-connections', pickState.serverId],
- });
- void queryClient.invalidateQueries({ queryKey: ['cloud-agent-github', pickState.serverId] });
- setConnectingProvider(false);
- setConnectingGitHub(false);
- };
-
return (
- <>
- {connectingProvider && pickState.serverId && (
- <ManagedProviderConnectionStep
- serverId={pickState.serverId}
- provider={pickState.providerId ?? 'claude'}
- continueLabel="Done"
- onBack={finishConnection}
- onDone={finishConnection}
- />
- )}
- {connectingGitHub && pickState.serverId && (
- <ManagedGitHubStep
- serverId={pickState.serverId}
- onBack={finishConnection}
- onSkip={finishConnection}
- onContinue={finishConnection}
- />
- )}
- <div hidden={connectingProvider || connectingGitHub}>
- <ModalLayout
- header={
- <DialogHeader showCloseButton={submitState === 'idle'}>
- <DialogTitle>New agent</DialogTitle>
- {targetServerId && (
- <button
- type="button"
- disabled={submitState !== 'idle'}
- onClick={() => {
- onClose();
- navigate('templates', { serverId: targetServerId, kind: 'agent' });
- }}
- className="w-fit cursor-pointer text-xs text-foreground-muted underline underline-offset-2 hover:text-foreground disabled:cursor-default disabled:opacity-50"
- >
- Or start from a template
- </button>
- )}
- </DialogHeader>
- }
- footer={
- <DialogFooter>
- {isRemoteRun && hostReadiness.checking && (
- <span className="mr-auto self-center text-xs text-foreground-muted">
- Waiting for {runLocationLabel}…
- </span>
- )}
- {onBack && (
- <Button variant="outline" onClick={onBack} disabled={submitState !== 'idle'}>
- Back
- </Button>
- )}
- <Button
- type="button"
- variant="outline"
- onClick={onClose}
- disabled={submitState !== 'idle'}
- >
- Cancel
- </Button>
- <TooltipProvider delay={150}>
- <Tooltip>
- {/* Span, not button, carries the tooltip: a disabled button emits no pointer events. */}
- <TooltipTrigger
- render={
- <span className="inline-flex">
- <ConfirmButton
- type="button"
- onClick={() => void handleCreate()}
- disabled={!canSubmit}
- >
- {submitState === 'creating' ? 'Adding…' : 'Add agent'}
- </ConfirmButton>
- </span>
- }
- />
- {disabledReason !== null && (
- <TooltipContent side="top">{disabledReason}</TooltipContent>
- )}
- </Tooltip>
- </TooltipProvider>
- </DialogFooter>
- }
- >
- <DialogContentArea
- data-autofocus
- tabIndex={-1}
- className="max-h-[calc(100dvh-2rem-var(--modal-chrome,8.5rem))] gap-4"
+ <ModalLayout
+ header={
+ <DialogHeader showCloseButton={submitState === 'idle'}>
+ <DialogTitle>New agent</DialogTitle>
+ {targetServerId && (
+ <button
+ type="button"
+ disabled={submitState !== 'idle'}
+ onClick={() => {
+ onClose();
+ navigate('templates', { serverId: targetServerId, kind: 'agent' });
+ }}
+ className="w-fit cursor-pointer text-xs text-foreground-muted underline underline-offset-2 hover:text-foreground disabled:cursor-default disabled:opacity-50"
+ >
+ Or start from a template
+ </button>
+ )}
+ </DialogHeader>
+ }
+ footer={
+ <DialogFooter>
+ {isRemoteRun && hostReadiness.checking && (
+ <span className="mr-auto self-center text-xs text-foreground-muted">
+ Waiting for {runLocationLabel}…
+ </span>
+ )}
+ {onBack && (
+ <Button variant="outline" onClick={onBack} disabled={submitState !== 'idle'}>
+ Back
+ </Button>
+ )}
+ <Button
+ type="button"
+ variant="outline"
+ onClick={onClose}
+ disabled={submitState !== 'idle'}
>
- <AgentIdentityFields form={form} serverId={pickState.serverId} />
-
- <Field>
- <FieldLabel>Run location</FieldLabel>
- {/* Icons and the right-hand kind, because the list mixes two sorts of
+ Cancel
+ </Button>
+ <TooltipProvider delay={150}>
+ <Tooltip>
+ {/* Span, not button, carries the tooltip: a disabled button emits no pointer events. */}
+ <TooltipTrigger
+ render={
+ <span className="inline-flex">
+ <ConfirmButton
+ type="button"
+ onClick={() => void handleCreate()}
+ disabled={!canSubmit}
+ >
+ {submitState === 'creating' ? 'Adding…' : 'Add agent'}
+ </ConfirmButton>
+ </span>
+ }
+ />
+ {disabledReason !== null && (
+ <TooltipContent side="top">{disabledReason}</TooltipContent>
+ )}
+ </Tooltip>
+ </TooltipProvider>
+ </DialogFooter>
+ }
+ >
+ <DialogContentArea
+ data-autofocus
+ tabIndex={-1}
+ className="max-h-[calc(100dvh-2rem-var(--modal-chrome,8.5rem))] gap-4"
+ >
+ <AgentIdentityFields form={form} serverId={pickState.serverId} nameTaken={nameTaken} />
+
+ <Field>
+ <FieldLabel>Run location</FieldLabel>
+ {/* Icons and the right-hand kind, because the list mixes two sorts of
thing: this machine, and hosts reached over SSH. The names alone
do not say which is which. */}
- <Select value={runHost} onValueChange={(v) => setRunHost(v ?? LOCAL_RUN_LOCATION)}>
- <SelectTrigger className="w-full">
- <SelectValue>
- {isCloudRun ? (
+ <Select value={runHost} onValueChange={(v) => setRunHost(v ?? LOCAL_RUN_LOCATION)}>
+ <SelectTrigger className="w-full">
+ <SelectValue>
+ {isCloudRun ? (
+ <Cloud className="size-4 text-foreground-muted" />
+ ) : isMachineRun ? (
+ serverMachine?.local?.kind === 'this-computer' ? (
+ <Monitor className="size-4 text-foreground-muted" />
+ ) : serverMachine?.cloud ? (
+ <Cloud className="size-4 text-foreground-muted" />
+ ) : (
+ <Server className="size-4 text-foreground-muted" />
+ )
+ ) : isRemoteRun ? (
+ <Server className="size-4 text-foreground-muted" />
+ ) : (
+ <Monitor className="size-4 text-foreground-muted" />
+ )}
+ <span className="truncate">{runLocationLabel}</span>
+ </SelectValue>
+ </SelectTrigger>
+ <SelectContent>
+ {serverMachines &&
+ machineRunLocations(serverMachines).map((option) => (
+ <SelectItem key={option.value} value={option.value} disabled={option.disabled}>
+ {option.icon === 'monitor' ? (
+ <Monitor className="size-4 text-foreground-muted" />
+ ) : option.icon === 'cloud' ? (
<Cloud className="size-4 text-foreground-muted" />
- ) : isMachineRun ? (
- serverMachine?.local?.kind === 'this-computer' ? (
- <Monitor className="size-4 text-foreground-muted" />
- ) : (
- <Server className="size-4 text-foreground-muted" />
- )
- ) : isRemoteRun ? (
- <Server className="size-4 text-foreground-muted" />
) : (
- <Monitor className="size-4 text-foreground-muted" />
+ <Server className="size-4 text-foreground-muted" />
)}
- <span className="truncate">{runLocationLabel}</span>
- </SelectValue>
- </SelectTrigger>
- <SelectContent>
- {serverMachines &&
- machineRunLocations(serverMachines).map((option) => (
- <SelectItem
- key={option.value}
- value={option.value}
- disabled={option.disabled}
- >
- {option.icon === 'monitor' ? (
- <Monitor className="size-4 text-foreground-muted" />
- ) : (
- <Server className="size-4 text-foreground-muted" />
- )}
- <span className="flex-1 truncate">{option.label}</span>
- <span className="text-xs text-foreground-muted">{option.tag}</span>
- </SelectItem>
- ))}
- {!(serverMachines && thisComputerIsMachine(serverMachines)) && (
- <SelectItem value={LOCAL_RUN_LOCATION}>
- <Monitor className="size-4 text-foreground-muted" />
- <span className="flex-1">This computer</span>
- <span className="text-xs text-foreground-muted">local</span>
- </SelectItem>
- )}
- {cloudAvailable && (
- <SelectItem value="cloud">
- <Cloud className="size-4 text-foreground-muted" />
- <span className="flex-1">Switch cloud</span>
- <span className="text-xs text-foreground-muted">preview</span>
- </SelectItem>
- )}
- {allowedHosts
- .filter(
- (host) => !(serverMachines && sshHostIsMachine(serverMachines, host.sshHost))
- )
- .map((host) => (
- <SelectItem key={host.sshHost} value={host.sshHost}>
- <Server className="size-4 text-foreground-muted" />
- <span className="flex-1 truncate">{host.name}</span>
- <span className="text-xs text-foreground-muted">ssh</span>
- </SelectItem>
- ))}
- </SelectContent>
- </Select>
- {!isCloudRun && !management && runLocationConstrained && (
- <p className="text-xs text-foreground-muted">
- {targetServer?.managementKind === 'remote'
- ? `This server runs on ${targetServer.sshHost}, so its agents run on this computer or on ${targetServer.sshHost}.`
- : 'This server runs on this computer, so its agents run here too.'}
- </p>
- )}
- {machine && pickState.serverId && workspaceId && (
- <ManagedRunLocationNotice
- machine={machine}
- label={runLocationLabel}
- sshHost={isRemoteRun ? runHost : null}
- serverId={pickState.serverId}
- workspaceId={workspaceId}
- onEnabled={() => {
- void machineQuery.refetch();
- void machinesQuery.refetch();
- }}
- />
+ <span className="flex-1 truncate">{option.label}</span>
+ <span className="text-xs text-foreground-muted">{option.tag}</span>
+ </SelectItem>
+ ))}
+ {!(serverMachines && thisComputerIsMachine(serverMachines)) && (
+ <SelectItem value={LOCAL_RUN_LOCATION}>
+ <Monitor className="size-4 text-foreground-muted" />
+ <span className="flex-1">This computer</span>
+ <span className="text-xs text-foreground-muted">local</span>
+ </SelectItem>
)}
- {isMachineRun && serverMachine && (
- <p className="flex items-start gap-1.5 text-xs text-foreground-muted">
- <Server className="mt-0.5 size-3.5 shrink-0" />
- <span>
- {serverMachine.state === 'online'
- ? `Runs as a managed agent on ${serverMachine.name}.`
- : machineReason}
- </span>
- </p>
+ {cloudAvailable && !cloudMachineListed && (
+ <SelectItem value="cloud">
+ <Cloud className="size-4 text-foreground-muted" />
+ <span className="flex-1">Switch cloud</span>
+ <span className="text-xs text-foreground-muted">preview</span>
+ </SelectItem>
)}
- {!isCloudRun && (machineQuery.error || (askForMachines && machinesQuery.error)) && (
- <p className="text-xs text-destructive">{machineReason}</p>
+ {allowedHosts
+ .filter(
+ (host) => !(serverMachines && sshHostIsMachine(serverMachines, host.sshHost))
+ )
+ .map((host) => (
+ <SelectItem key={host.sshHost} value={host.sshHost}>
+ <Server className="size-4 text-foreground-muted" />
+ <span className="flex-1 truncate">{host.name}</span>
+ <span className="text-xs text-foreground-muted">ssh</span>
+ </SelectItem>
+ ))}
+ </SelectContent>
+ </Select>
+ {!isCloudRun && !management && runLocationConstrained && (
+ <p className="text-xs text-foreground-muted">
+ {targetServer?.managementKind === 'remote'
+ ? `This server runs on ${targetServer.sshHost}, so its agents run on this computer or on ${targetServer.sshHost}.`
+ : 'This server runs on this computer, so its agents run here too.'}
+ </p>
+ )}
+ {machine && pickState.serverId && workspaceId && (
+ <ManagedRunLocationNotice
+ machine={machine}
+ label={runLocationLabel}
+ sshHost={isRemoteRun ? runHost : null}
+ serverId={pickState.serverId}
+ workspaceId={workspaceId}
+ onEnabled={() => {
+ void machineQuery.refetch();
+ void machinesQuery.refetch();
+ }}
+ />
+ )}
+ {isMachineRun && serverMachine && (
+ <p className="flex items-start gap-1.5 text-xs text-foreground-muted">
+ {serverMachine.cloud ? (
+ <Cloud className="mt-0.5 size-3.5 shrink-0" />
+ ) : (
+ <Server className="mt-0.5 size-3.5 shrink-0" />
)}
- {isRemoteRun && <HostReachabilityNotice sshHost={runHost} />}
- {/* Not while it is still checking: the provider picker below is
+ <span>
+ {serverMachine.state === 'online'
+ ? `Runs as a managed agent on ${serverMachine.name}.`
+ : machineReason}
+ </span>
+ </p>
+ )}
+ {!isCloudRun && (machineQuery.error || (askForMachines && machinesQuery.error)) && (
+ <p className="text-xs text-destructive">{machineReason}</p>
+ )}
+ {isRemoteRun && <HostReachabilityNotice sshHost={runHost} />}
+ {/* Not while it is still checking: the provider picker below is
already saying so, and two spinners for one question read as two
questions. Its verdict — the part only it can report — still
lands here. */}
- {isRemoteRun && runHostReachable && !hostReadiness.checking && (
- <HostReadinessNotice
- sshHost={runHost}
- readiness={hostReadiness}
- onNavigateAway={onClose}
- />
- )}
- </Field>
-
- {/* The host still gates what comes below it: with it unreachable, or
+ {isRemoteRun && runHostReachable && !hostReadiness.checking && (
+ <HostReadinessNotice
+ sshHost={runHost}
+ readiness={hostReadiness}
+ onNavigateAway={onClose}
+ />
+ )}
+ </Field>
+
+ {/* The host still gates what comes below it: with it unreachable, or
missing its own prerequisites, we cannot know which providers it
has, so offering the tiles would be guessing. What no longer gates
anything is the directory — nothing is scanned in it, so it can be
filled in while the host is still being surveyed. */}
- {canChooseAgentType && !isCloudRun && !isMachineRun && (
- <Field>
- <FieldLabel>Directory</FieldLabel>
- {isRemoteRun ? (
- // No file picker for a host: the directory is on the other end of
- // an SSH connection, so it is typed rather than browsed.
- <Input
- value={remoteRepoDir}
- placeholder="/home/agent/repo"
- onChange={(e) => setRemoteRepoDir(e.target.value)}
- />
- ) : (
- <LocalDirectorySelector
- title="Choose the agent's working directory"
- message="The agent runs its sessions here."
- path={pickState.path}
- onPathChange={pickState.handlePathChange}
- />
- )}
- </Field>
- )}
-
- {isMachineRun && serverMachine && (
- <MachineProviderPicker
- machine={serverMachine}
- value={pickState.providerId}
- onChange={pickState.setProviderId}
- defaultAgent={defaultAgent}
+ {canChooseAgentType && !isCloudRun && !isMachineRun && (
+ <Field>
+ <FieldLabel>Directory</FieldLabel>
+ {isRemoteRun ? (
+ // No file picker for a host: the directory is on the other end of
+ // an SSH connection, so it is typed rather than browsed.
+ <Input
+ value={remoteRepoDir}
+ placeholder="/home/agent/repo"
+ onChange={(e) => setRemoteRepoDir(e.target.value)}
/>
- )}
-
- {canChooseAgentType && !isCloudRun && !isMachineRun && (
- <AgentTypePicker
- value={pickState.providerId}
- onChange={pickState.setProviderId}
- sshHost={isRemoteRun ? runHost : undefined}
- onNavigateAway={onClose}
+ ) : (
+ <LocalDirectorySelector
+ title="Choose the agent's working directory"
+ message="The agent runs its sessions here."
+ path={pickState.path}
+ onPathChange={pickState.handlePathChange}
/>
)}
-
- {/* No `dir`: signing in is a property of the machine, not of the folder
+ </Field>
+ )}
+
+ {isMachineRun && serverMachine && (
+ <MachineProviderPicker
+ serverId={selectedServerId!}
+ machine={serverMachine}
+ value={pickState.providerId}
+ onChange={pickState.setProviderId}
+ defaultAgent={defaultAgent}
+ />
+ )}
+
+ {canChooseAgentType && !isCloudRun && !isMachineRun && (
+ <AgentTypePicker
+ value={pickState.providerId}
+ onChange={pickState.setProviderId}
+ sshHost={isRemoteRun ? runHost : undefined}
+ onNavigateAway={onClose}
+ />
+ )}
+
+ {/* No `dir`: signing in is a property of the machine, not of the folder
an agent will run in — which is what the provider tiles above
already assume. */}
- {canChooseAgentType && !isCloudRun && !isMachineRun && pickState.providerId && (
- <ProviderConnectionStatus
- providerId={pickState.providerId}
- sshHost={isRemoteRun ? runHost : null}
- dir=""
- />
- )}
-
- {isCloudRun && pickState.serverId && (
- <CloudAgentRepository
- serverId={pickState.serverId}
- providerId={pickState.providerId ?? 'claude'}
- onProviderChange={setProviderId}
- onSelection={setCloudRepository}
- onConnectProvider={() => setConnectingProvider(true)}
- onConnectGitHub={() => setConnectingGitHub(true)}
- />
- )}
-
- {canConfigureAgent && !!pickState.providerId && managedRun && pickState.serverId && (
- <ManagedAdvancedConfig
- serverId={pickState.serverId}
- providerId={pickState.providerId}
- host={
- isMachineRun && !serverMachine?.local
- ? {
- kind: 'unavailable',
- reason: `${runLocationLabel} is not this computer or one of its SSH hosts, so Console cannot ask it for its models. You can enter a model alias or ID.`,
- }
- : {
- kind: 'host',
- sshHost: isRemoteRun
- ? runHost
- : serverMachine?.local?.kind === 'ssh-host'
- ? serverMachine.local.sshHost
- : null,
- // Not the suggested directory, which changes with every
- // keystroke of the name and does not exist yet: the
- // folder it will be made in answers the same.
- dir:
- isMachineRun && !editedMachineDir?.trim()
- ? (serverMachine?.workspacesDir ?? '')
- : dir,
- }
- }
- onChange={onManagedSettingsChange}
- />
- )}
-
- {canConfigureAgent && !!pickState.providerId && !managedRun && (
- <>
- {(!isCloudRun || pickState.providerId === 'claude') && (
- <AgentAdvancedConfig
- cloud={isCloudRun}
- providerId={pickState.providerId}
- sshHost={isRemoteRun ? runHost : null}
- dir={dir}
- initial={NO_ATTRIBUTES}
- onChange={onAdvancedChange}
- />
- )}
- {!isCloudRun && (
- <LaunchProfileConfig
- providerId={pickState.providerId}
- sshHost={isRemoteRun ? runHost : null}
- dir={dir}
- onChange={onLaunchProfileConfigChange}
- />
- )}
- </>
- )}
-
- {/* Last, below Advanced configuration. Everything above it is a choice
+ {canChooseAgentType && !isCloudRun && !isMachineRun && pickState.providerId && (
+ <ProviderConnectionStatus
+ providerId={pickState.providerId}
+ sshHost={isRemoteRun ? runHost : null}
+ dir=""
+ />
+ )}
+
+ {isCloudRun && machineReason && (
+ <p className="flex items-start gap-1.5 text-xs text-foreground-muted" role="status">
+ <Cloud className="mt-0.5 size-3.5 shrink-0" />
+ <span>{machineReason}</span>
+ </p>
+ )}
+
+ {canConfigureAgent && !!pickState.providerId && managedRun && pickState.serverId && (
+ <ManagedAdvancedConfig
+ serverId={pickState.serverId}
+ providerId={pickState.providerId}
+ host={
+ isMachineRun && !serverMachine?.local
+ ? {
+ kind: 'unavailable',
+ reason: `${runLocationLabel} is not this computer or one of its SSH hosts, so Console cannot ask it for its models. You can enter a model alias or ID.`,
+ }
+ : {
+ kind: 'host',
+ sshHost: isRemoteRun
+ ? runHost
+ : serverMachine?.local?.kind === 'ssh-host'
+ ? serverMachine.local.sshHost
+ : null,
+ // Not the suggested directory, which changes with every
+ // keystroke of the name and does not exist yet: the
+ // folder it will be made in answers the same.
+ dir:
+ isMachineRun && !editedMachineDir?.trim()
+ ? (serverMachine?.workspacesDir ?? '')
+ : dir,
+ }
+ }
+ onChange={onManagedSettingsChange}
+ />
+ )}
+
+ {canConfigureAgent && !!pickState.providerId && !managedRun && !isCloudRun && (
+ <>
+ <AgentAdvancedConfig
+ providerId={pickState.providerId}
+ sshHost={isRemoteRun ? runHost : null}
+ dir={dir}
+ initial={NO_ATTRIBUTES}
+ onChange={onAdvancedChange}
+ />
+ <LaunchProfileConfig
+ providerId={pickState.providerId}
+ sshHost={isRemoteRun ? runHost : null}
+ dir={dir}
+ onChange={onLaunchProfileConfigChange}
+ />
+ </>
+ )}
+
+ {/* Last, below Advanced configuration. Everything above it is a choice
the agent cannot exist without; these have working defaults and are
changeable afterwards from the agent's own settings. */}
- {canConfigureAgent && (
- <AgentSettingsSection
- form={form}
- workspaceId={workspacesStore.idOnServerInScope(pickState.serverId)}
- onAddServer={() => showAddServerModal({})}
- onOpenMessagingApps={() => {
- onClose();
- if (pickState.serverId) navigate('server', { serverId: pickState.serverId });
- }}
- >
- {isMachineRun && serverMachine && (
- <ManagedDirectoryField
- machine={serverMachine}
- machineLabel={runLocationLabel}
- value={editedMachineDir ?? suggestedMachineDir.path ?? ''}
- suggested={suggestedMachineDir}
- onChange={setEditedMachineDir}
- />
- )}
- {managedRun && (
- <CanManageAgentsField checked={canManageAgents} onChange={setCanManageAgents} />
- )}
- </AgentSettingsSection>
+ {canConfigureAgent && (
+ <AgentSettingsSection
+ form={form}
+ workspaceId={workspacesStore.idOnServerInScope(pickState.serverId)}
+ onAddServer={() => showAddServerModal({})}
+ onOpenMessagingApps={() => {
+ onClose();
+ if (pickState.serverId) navigate('server', { serverId: pickState.serverId });
+ }}
+ >
+ {isMachineRun && serverMachine && (
+ <ManagedDirectoryField
+ machine={serverMachine}
+ machineLabel={runLocationLabel}
+ value={editedMachineDir ?? suggestedMachineDir.path ?? ''}
+ suggested={suggestedMachineDir}
+ onChange={setEditedMachineDir}
+ />
+ )}
+ {managedRun && (
+ <CanManageAgentsField checked={canManageAgents} onChange={setCanManageAgents} />
)}
- </DialogContentArea>
- </ModalLayout>
- </div>
- </>
+ </AgentSettingsSection>
+ )}
+ </DialogContentArea>
+ </ModalLayout>
);
});
+
+/** What the form says while the owner's cloud machine is not ready to take an agent yet. */
+function cloudStartingText(machine: CloudMachine): string {
+ if (machine.state === 'error')
+ return machine.error ?? 'Your cloud machine needs attention. Retry it from Your Agents.';
+ if (machine.desired_state === 'stopped')
+ return 'Your cloud machine is stopped. Start it from Your Agents, then add the agent.';
+ return machine.controller_id === null
+ ? 'Starting your cloud machine. Its first start takes a few minutes…'
+ : 'Waiting for your cloud machine to come online…';
+}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/server-run-locations.test.ts b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/server-run-locations.test.ts
index 2e5e42829..4143f89a4 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/server-run-locations.test.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/server-run-locations.test.ts
@@ -17,6 +17,8 @@ function machine(patch: Partial<OwnedMachine> & { id: string }): OwnedMachine {
providers: [],
local: null,
workspacesDir: null,
+ acceptsLogins: false,
+ cloud: false,
...patch,
};
}
@@ -50,6 +52,20 @@ describe('the run locations a server with agent management offers', () => {
]);
});
+ it('lists the Switch cloud machine only while it is online, as the cloud', () => {
+ const cloud = machine({ id: 'ec2', name: 'Switch cloud', kind: 'ec2', cloud: true });
+ expect(machineRunLocations([cloud])).toEqual([
+ {
+ value: 'machine:ec2',
+ label: 'Switch cloud',
+ tag: 'cloud',
+ icon: 'cloud',
+ disabled: false,
+ },
+ ]);
+ expect(machineRunLocations([{ ...cloud, state: 'offline' }])).toEqual([]);
+ });
+
it('round-trips a machine through its run location value', () => {
expect(machineIdOf(machineRunLocation('vm'))).toBe('vm');
expect(machineIdOf('local')).toBeNull();
diff --git a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/server-run-locations.ts b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/server-run-locations.ts
index 756f190d2..9c948ee14 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/server-run-locations.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/locations/components/add-agent-modal/server-run-locations.ts
@@ -18,13 +18,18 @@ export type RunLocationOption = {
label: string;
/** The right-hand tag: what sort of machine it is. */
tag: string;
- icon: 'monitor' | 'server';
+ icon: 'monitor' | 'server' | 'cloud';
disabled: boolean;
};
-/** The server's machines as Run location entries. Offline machines are listed, but cannot be picked. */
+/**
+ * The server's machines as Run location entries. Offline machines are listed,
+ * but cannot be picked; but the owner's Switch cloud machine, which the Switch
+ * cloud entry starts, is listed only while it is online.
+ */
export function machineRunLocations(machines: OwnedMachine[]): RunLocationOption[] {
- return [...machines]
+ return machines
+ .filter((machine) => !machine.cloud || machine.state === 'online')
.sort(
(a, b) =>
Number(b.local?.kind === 'this-computer') - Number(a.local?.kind === 'this-computer')
@@ -40,8 +45,11 @@ export function machineRunLocations(machines: OwnedMachine[]): RunLocationOption
? 'this Console'
: machine.local?.kind === 'ssh-host'
? 'ssh'
- : machine.kind,
- icon: machine.local?.kind === 'this-computer' ? 'monitor' : 'server',
+ : machine.cloud
+ ? 'cloud'
+ : machine.kind,
+ icon:
+ machine.local?.kind === 'this-computer' ? 'monitor' : machine.cloud ? 'cloud' : 'server',
disabled: offline,
};
});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-changes.test.ts b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-changes.test.ts
index c88430063..71b8d4356 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-changes.test.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-changes.test.ts
@@ -125,6 +125,8 @@ describe('shownDirectory', () => {
const MACHINE: OwnedMachine = {
...AGENT.machine!,
providers: [],
+ acceptsLogins: false,
+ cloud: false,
local: null,
workspacesDir: '/srv/workspaces/',
};
diff --git a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-list.tsx b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-list.tsx
index 5f86cf6e8..e3d76491f 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-list.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-list.tsx
@@ -1,11 +1,11 @@
-import { Server, ServerOff } from 'lucide-react';
+import { Cloud, CloudOff, Server, ServerOff } from 'lucide-react';
import { observer } from 'mobx-react-lite';
import { switchServersStore } from '@renderer/features/switch-servers/switch-servers-store';
import { failureText } from '@renderer/lib/errors/describe-failure';
import { useNavigate, useParams } from '@renderer/lib/layout/navigation-provider';
import { useWorkspaceSlots } from '@renderer/lib/layout/workspace-slots';
import { Tooltip, TooltipContent, TooltipTrigger } from '@renderer/lib/ui/tooltip';
-import type { ManagedAgentView } from '@shared/core/managed-agents/managed-agents';
+import { isCloudMachine, type ManagedAgentView } from '@shared/core/managed-agents/managed-agents';
import { isValidProviderId } from '@shared/core/providers/agent-provider-registry';
import { type AgentPresence, SidebarAgentRow } from '../sidebar/agent-row';
import {
@@ -71,7 +71,13 @@ const ManagedAgentRow = observer(function ManagedAgentRow({ agent }: { agent: Ma
!thisComputer && (
<Tooltip>
<TooltipTrigger>
- {machineDown || !agent.machine ? (
+ {agent.machine && isCloudMachine(agent.machine) ? (
+ machineDown ? (
+ <CloudOff className="h-3.5 w-3.5 shrink-0 text-foreground-destructive" />
+ ) : (
+ <Cloud className="h-3.5 w-3.5 shrink-0 text-foreground-muted" />
+ )
+ ) : machineDown || !agent.machine ? (
<ServerOff className="h-3.5 w-3.5 shrink-0 text-foreground-destructive" />
) : (
<Server className="h-3.5 w-3.5 shrink-0 text-foreground-muted" />
diff --git a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-page.tsx b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-page.tsx
index 968c28c8a..02008aeb7 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-page.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-page.tsx
@@ -300,22 +300,31 @@ function ManagedAgentPageContent({ agent }: { agent: ManagedAgentView }) {
/>
)}
</SettingRow>
- <SettingRow
- title="Run in its own process"
- info={{
- label: 'More info about running in its own process',
- content:
- 'Isolated from the other agents on its machine, instead of inside the machine’s controller.',
- }}
- description="Keep it apart from the other agents on its machine."
- control={
- <Switch
- aria-label="Run in its own process"
- checked={draft.ownProcess}
- onCheckedChange={(checked) => setValue('ownProcess', checked)}
- />
- }
- />
+ {machine?.cloud ? (
+ <SettingRow
+ title="Runs in its own process"
+ info={null}
+ description="Switch cloud runs every agent as a user of its own, apart from the other agents and the machine's controller."
+ control={null}
+ />
+ ) : (
+ <SettingRow
+ title="Run in its own process"
+ info={{
+ label: 'More info about running in its own process',
+ content:
+ 'Isolated from the other agents on its machine, instead of inside the machine’s controller.',
+ }}
+ description="Keep it apart from the other agents on its machine."
+ control={
+ <Switch
+ aria-label="Run in its own process"
+ checked={draft.ownProcess}
+ onCheckedChange={(checked) => setValue('ownProcess', checked)}
+ />
+ }
+ />
+ )}
</section>
<div className="flex flex-col gap-2">
<AdvancedConfigDisclosure
diff --git a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-state.test.ts b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-state.test.ts
index 393ebde7e..a0869957c 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-state.test.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-state.test.ts
@@ -82,6 +82,8 @@ const LAPTOP: OwnedMachine = {
...AGENT.machine!,
local: { kind: 'this-computer' },
providers: [{ provider: 'claude', ready: true, problem: null }],
+ acceptsLogins: false,
+ cloud: false,
workspacesDir: null,
};
diff --git a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-view.tsx b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-view.tsx
index 60baecffb..76cfaf67c 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-view.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-agent-view.tsx
@@ -105,6 +105,8 @@ function ManagedAgentActionsMenu({ agent }: { agent: ManagedAgentView }) {
onSuccess: () =>
void act(`${label} could not be deleted.`, async () => {
await rpc.managedAgents.remove({ serverId: agent.serverId, agentId: agent.agentId });
+ // Its name is free again: the New agent form checks names against this list.
+ await queryClient.invalidateQueries({ queryKey: ['workspace-agents'] });
navigate('serverAgents', { serverId: agent.serverId });
}),
});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-machine-card.tsx b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-machine-card.tsx
index 5243a1cac..aa79776cc 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-machine-card.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/managed-agents/managed-machine-card.tsx
@@ -1,8 +1,12 @@
-import { ChevronDown, CircleAlert, Laptop, Server } from 'lucide-react';
+import { ChevronDown, CircleAlert, Cloud, Laptop, Server } from 'lucide-react';
import { failureText } from '@renderer/lib/errors/describe-failure';
import { Popover, PopoverContent, PopoverTrigger } from '@renderer/lib/ui/popover';
import { cn } from '@renderer/utils/utils';
-import type { ManagedAgentView, OwnedMachine } from '@shared/core/managed-agents/managed-agents';
+import {
+ isCloudMachine,
+ type ManagedAgentView,
+ type OwnedMachine,
+} from '@shared/core/managed-agents/managed-agents';
import { machineProblem, machineTone, managedAgentState } from './managed-agent-state';
const DOT: Record<ReturnType<typeof machineTone>, string> = {
@@ -64,7 +68,12 @@ function MachineCard({
}) {
const problem = machineProblem(agent, machine);
const state = managedAgentState(agent);
- const Icon = machine?.local?.kind === 'this-computer' ? Laptop : Server;
+ const Icon =
+ machine?.local?.kind === 'this-computer'
+ ? Laptop
+ : agent.machine && isCloudMachine(agent.machine)
+ ? Cloud
+ : Server;
const subtitle =
machine?.local?.kind === 'this-computer'
? 'This computer'
diff --git a/console/apps/switch-console-desktop/src/renderer/features/onboarding/use-onboarding-checklist.ts b/console/apps/switch-console-desktop/src/renderer/features/onboarding/use-onboarding-checklist.ts
index ff9163d10..b808d4669 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/onboarding/use-onboarding-checklist.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/onboarding/use-onboarding-checklist.ts
@@ -2,10 +2,6 @@ import { useCallback, useEffect } from 'react';
import { useAppSettingsKey } from '@renderer/features/settings/use-app-settings-key';
import { switchRoomsStore } from '@renderer/features/switch-servers/switch-rooms-store';
import { switchServersStore } from '@renderer/features/switch-servers/switch-servers-store';
-import {
- managedCloudServerId,
- useCloudLaunches,
-} from '@renderer/features/switch-servers/use-cloud-launches';
import { useNavigate } from '@renderer/lib/layout/navigation-provider';
import { useShowModal } from '@renderer/lib/modal/modal-provider';
import { appState } from '@renderer/lib/stores/app-state';
@@ -32,13 +28,11 @@ import {
*/
export function useOnboardingProgress(): OnboardingProgress {
const { data: agentTypes } = useAgentTypeAvailability();
- const cloud = useCloudLaunches(managedCloudServerId());
- const hasCloudAgent = cloud.data?.some((agent) => agent.state === 'ready') ?? false;
return {
addServer: switchServersStore.servers.length > 0,
- agentProviders: hasCloudAgent || (agentTypes ?? []).some((type) => type.available),
- onboardAgents: hasCloudAgent || appState.locations.locations.size > 0,
+ agentProviders: (agentTypes ?? []).some((type) => type.available),
+ onboardAgents: appState.locations.locations.size > 0,
createRoom: switchRoomsStore.listedRoomsInAllWorkspaces.length > 0,
};
}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/held-message.test.ts b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/held-message.test.ts
deleted file mode 100644
index 09d750137..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/held-message.test.ts
+++ /dev/null
@@ -1,230 +0,0 @@
-import {
- type ClientCommand,
- type CommandStatus,
- SessionChatClient,
- type SessionTransport,
-} from '@switch-console/shared/session-v1';
-import { describe, expect, it } from 'vitest';
-import { RpcError, serializeRpcError } from '@shared/lib/ipc/rpc-error';
-import {
- deliverHeld,
- heldMessages,
- heldStatusText,
- isWakingError,
- relayRefusalText,
-} from './held-message';
-
-function relayError(relayCode: string): RpcError {
- const error = Object.assign(new Error(`Refused: ${relayCode}`), {
- name: 'CloudRelayError',
- relayCode,
- status: 409,
- wakeAvailable: false,
- });
- return new RpcError(serializeRpcError(error));
-}
-
-const snapshot = {
- contractVersion: 1,
- throughSequence: 1,
- session: {
- sessionId: 'session-demo',
- agentId: 'agent-demo',
- provider: 'claude',
- hostId: 'host-demo',
- epoch: 'epoch-demo',
- status: 'ready',
- connectivity: 'online',
- capabilities: {
- input: 'queue',
- approvals: true,
- questions: true,
- interrupt: true,
- reset: false,
- compact: false,
- modelChange: false,
- attachmentMimeTypes: [],
- },
- pendingRequestIds: [],
- },
- turns: [],
- items: [],
- requests: [],
- commandStatuses: [],
- nextPageToken: null,
-};
-
-/**
- * A session host behind the relay: it records each command id once, and a
- * reconcile answers from that record, submitting only a command it never got.
- * `refusals` are what the next calls meet before they reach it.
- */
-function host() {
- const records = new Map<string, ClientCommand>();
- const arrivals: string[] = [];
- const refusals: Array<'waking' | 'reply-lost'> = [];
- const status = (commandId: string): CommandStatus => ({
- type: 'command.status',
- commandId,
- status: 'accepted',
- code: null,
- message: null,
- });
- const submit = async (command: ClientCommand): Promise<CommandStatus> => {
- const refusal = refusals.shift();
- if (refusal === 'waking') throw relayError('worker_waking');
- arrivals.push(command.commandId);
- if (!records.has(command.commandId)) records.set(command.commandId, command);
- if (refusal === 'reply-lost') throw new Error('The relay reply was lost.');
- return status(command.commandId);
- };
- const transport: SessionTransport = {
- snapshot: async () => structuredClone(snapshot),
- subscribe: () => () => {},
- submit,
- reconcile: async (command) => {
- if (refusals[0] === 'waking') {
- refusals.shift();
- throw relayError('worker_waking');
- }
- return records.has(command.commandId) ? status(command.commandId) : submit(command);
- },
- commandStatus: async (_sessionId, commandId) => {
- if (!records.has(commandId)) throw new Error('Not recorded.');
- return status(commandId);
- },
- };
- return { transport, records, arrivals, refusals };
-}
-
-async function connected(transport: SessionTransport): Promise<SessionChatClient> {
- const client = new SessionChatClient('session-demo', transport);
- await client.connect();
- expect(client.getSnapshot().connected).toBe(true);
- return client;
-}
-
-describe('a message held while the cloud machine wakes', () => {
- it('keeps a send refused as waking, then delivers it once under its id', async () => {
- const worker = host();
- const client = await connected(worker.transport);
- worker.refusals.push('waking');
- const refused = await client.send('hello', 'held-1', []).catch((error: unknown) => error);
- expect(isWakingError(refused)).toBe(true);
- expect(client.hasPendingCommand()).toBe(true);
- expect(worker.records.size).toBe(0);
-
- const held = { commandId: 'held-1', text: 'hello', attachments: [] };
- worker.refusals.push('waking');
- const again = await deliverHeld(client, held).catch((error: unknown) => error);
- expect(isWakingError(again)).toBe(true);
- expect(client.hasPendingCommand()).toBe(true);
-
- const delivered = await deliverHeld(client, held);
- expect(delivered.commandId).toBe('held-1');
- expect([...worker.records.keys()]).toEqual(['held-1']);
- expect(worker.arrivals).toEqual(['held-1']);
- expect(client.hasPendingCommand()).toBe(false);
- });
-
- it('reconciles a delivery whose reply was lost after the host recorded it', async () => {
- const worker = host();
- const client = await connected(worker.transport);
- worker.refusals.push('reply-lost');
- const lost = await client.send('hello', 'held-2', []).catch((error: unknown) => error);
- expect(isWakingError(lost)).toBe(false);
- expect(client.hasPendingCommand()).toBe(true);
-
- await deliverHeld(client, { commandId: 'held-2', text: 'hello', attachments: [] });
- expect([...worker.records.keys()]).toEqual(['held-2']);
- expect(worker.arrivals).toEqual(['held-2']);
- expect(client.hasPendingCommand()).toBe(false);
- });
-
- it('sends a message held before it was ever submitted, and not twice', async () => {
- const worker = host();
- const client = await connected(worker.transport);
- const held = { commandId: 'held-3', text: 'wake up', attachments: [] };
-
- await deliverHeld(client, held);
- expect([...worker.records.keys()]).toEqual(['held-3']);
- expect(worker.records.get('held-3')?.body).toMatchObject({
- type: 'message.send',
- text: 'wake up',
- });
- expect(client.hasPendingCommand()).toBe(false);
-
- await deliverHeld(client, held);
- expect(worker.records.size).toBe(1);
- });
-});
-
-describe('relay refusals the user has to act on', () => {
- it('says what to do for a stopped machine, a stopped agent and a crashed agent', () => {
- expect(relayRefusalText(relayError('machine_stopped'))).toBe(
- 'The owner stopped the cloud machine. Start it in Your Agents, then send again.'
- );
- expect(relayRefusalText(relayError('agent_stopped'))).toBe(
- 'This agent is stopped. Start it in Your Agents, then send again.'
- );
- expect(relayRefusalText(relayError('agent_crashed'))).toBe(
- 'This agent crashed. Retry it in Your Agents, then send again.'
- );
- });
-
- it('says what to do for a machine in error', () => {
- expect(relayRefusalText(relayError('machine_error'))).toBe(
- 'The cloud machine is in error. Retry it in Your Agents, then send again.'
- );
- });
-
- it('prefers the server detail for a machine that needs admin attention', () => {
- const error = Object.assign(
- new Error('The cloud machine needs attention. Contact your server administrator.'),
- {
- name: 'CloudRelayError',
- relayCode: 'machine_error',
- status: 409,
- wakeAvailable: false,
- }
- );
- const rpcError = new RpcError(serializeRpcError(error));
- expect(relayRefusalText(rpcError)).toBe(
- 'The cloud machine needs attention. Contact your server administrator.'
- );
- });
-
- it('has nothing to add for any other failure', () => {
- expect(relayRefusalText(relayError('worker_waking'))).toBeNull();
- expect(relayRefusalText(relayError('worker_busy'))).toBeNull();
- expect(relayRefusalText(new Error('machine_stopped'))).toBeNull();
- expect(relayRefusalText(relayError('toString'))).toBeNull();
- expect(isWakingError(new Error('worker_waking'))).toBe(false);
- });
-});
-
-describe('what the composer says while it holds a message', () => {
- it('says waking only while the machine is not yet awake', () => {
- expect(heldStatusText('machine')).toMatch(/^Waking…/);
- expect(heldStatusText('machine')).toContain('Keep Switch Console open');
- expect(heldStatusText('session')).not.toContain('Waking');
- expect(heldStatusText('session')).toContain('The machine is awake. Connecting to the session');
- });
-
- it('says the agent is starting when only the agent starts on a machine already up', () => {
- expect(heldStatusText('agent')).toBe(
- 'Starting the agent… Your message is sent when it is ready.'
- );
- });
-});
-
-describe('held messages by session', () => {
- it('keeps a held message for its session until it is cleared', () => {
- const held = { commandId: 'held-4', text: 'later', attachments: [] };
- heldMessages.set('session-a', held);
- expect(heldMessages.get('session-a')).toEqual(held);
- expect(heldMessages.get('session-b')).toBeNull();
- heldMessages.set('session-a', null);
- expect(heldMessages.get('session-a')).toBeNull();
- });
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/held-message.ts b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/held-message.ts
deleted file mode 100644
index 5e7113763..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/held-message.ts
+++ /dev/null
@@ -1,84 +0,0 @@
-import type {
- Attachment,
- CommandStatus,
- SessionChatClient,
-} from '@switch-console/shared/session-v1';
-import { RpcError } from '@shared/lib/ipc/rpc-error';
-
-/** A message the composer keeps while the cloud machine it goes to wakes up. */
-export type HeldMessage = { commandId: string; text: string; attachments: Attachment[] };
-
-function relayCode(error: unknown): string | undefined {
- return error instanceof RpcError && error.code === 'CloudRelayError'
- ? error.stringField('relayCode')
- : undefined;
-}
-
-/** Switch refused the message before it reached the worker, and is starting the machine. */
-export function isWakingError(error: unknown): boolean {
- return relayCode(error) === 'worker_waking';
-}
-
-const REFUSALS = {
- machine_stopped: 'The owner stopped the cloud machine. Start it in Your Agents, then send again.',
- machine_error: 'The cloud machine is in error. Retry it in Your Agents, then send again.',
- agent_stopped: 'This agent is stopped. Start it in Your Agents, then send again.',
- agent_crashed: 'This agent crashed. Retry it in Your Agents, then send again.',
-};
-
-/** What to tell the user about a refusal they have to act on before sending again. */
-export function relayRefusal(code: keyof typeof REFUSALS): string {
- return REFUSALS[code];
-}
-
-/** What to tell the user when Switch will not relay a message until they act. */
-export function relayRefusalText(error: unknown): string | null {
- const code = relayCode(error);
- if (code === undefined || !Object.hasOwn(REFUSALS, code)) return null;
- if (
- code === 'machine_error' &&
- error instanceof RpcError &&
- error.message.includes('Contact your server administrator')
- ) {
- return error.message;
- }
- return REFUSALS[code as keyof typeof REFUSALS];
-}
-
-/**
- * What a held message waits on: the machine waking, only the agent starting on
- * a machine already up, or the session connecting once both are.
- */
-export type HeldWait = 'machine' | 'agent' | 'session';
-
-/** What the composer says while it holds a message, by what it waits on. */
-export function heldStatusText(wait: HeldWait): string {
- if (wait === 'session')
- return 'The machine is awake. Connecting to the session, then your message is sent.';
- if (wait === 'agent') return 'Starting the agent… Your message is sent when it is ready.';
- return 'Waking… about 1–2 min. Keep Switch Console open until the machine is awake.';
-}
-
-const heldBySession = new Map<string, HeldMessage>();
-
-/**
- * Held messages by session, so one outlives its composer: leaving a session
- * while its machine wakes keeps the message, and it is held again on return.
- */
-export const heldMessages = {
- get: (sessionId: string): HeldMessage | null => heldBySession.get(sessionId) ?? null,
- set(sessionId: string, held: HeldMessage | null): void {
- if (held) heldBySession.set(sessionId, held);
- else heldBySession.delete(sessionId);
- },
-};
-
-/**
- * Deliver a held message once. A command the client still holds is reconciled,
- * so a host that already recorded it is not sent it a second time.
- */
-export function deliverHeld(client: SessionChatClient, held: HeldMessage): Promise<CommandStatus> {
- return client.hasPendingCommand()
- ? client.reconcile()
- : client.send(held.text, held.commandId, held.attachments);
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-state.test.ts b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-state.test.ts
index c91819c32..9a5639484 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-state.test.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-state.test.ts
@@ -4,13 +4,11 @@ import { sessionStatePill } from './session-state';
const base = {
action: null,
elapsedSeconds: 0,
- host: null,
failed: false,
retired: false,
status: 'ready',
connectivity: 'online' as const,
reachable: true,
- startable: false,
};
it('shows the session status when the host is reachable', () => {
@@ -31,12 +29,6 @@ it('reports unreachability rather than a status it cannot stand behind', () => {
});
});
-it('says nothing when the next message restarts the session', () => {
- expect(
- sessionStatePill({ ...base, reachable: false, connectivity: 'offline', startable: true })
- ).toBeNull();
-});
-
it('counts up while Console is acting on the session', () => {
expect(sessionStatePill({ ...base, action: 'restart', elapsedSeconds: 7 })).toEqual({
label: 'restarting 7s',
@@ -63,12 +55,3 @@ it('says it is connecting while the host comes up', () => {
tone: 'busy',
});
});
-
-it("shows the host's state over a stale session status, but not over an action", () => {
- const host = { label: 'sleeping', tone: 'idle' as const };
- expect(sessionStatePill({ ...base, host })).toEqual(host);
- expect(sessionStatePill({ ...base, host, action: 'restart', elapsedSeconds: 2 })).toEqual({
- label: 'restarting 2s',
- tone: 'busy',
- });
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-state.ts b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-state.ts
index b6f52728c..ed7cf5276 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-state.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-state.ts
@@ -1,23 +1,5 @@
-import type { Session } from '@switch-console/shared/session-v1';
-
export type SessionStateTone = 'ready' | 'busy' | 'bad' | 'idle';
-/**
- * Whether a cloud session is down only in a way the next message repairs:
- * sending restarts it and delivers the message, so its being offline is not
- * the user's concern and is not shown.
- */
-export function restartsOnSend(
- session: Pick<Session, 'connectivity' | 'retired' | 'status'>
-): boolean {
- return (
- session.connectivity === 'offline' &&
- !session.retired &&
- session.status !== 'stopped' &&
- session.status !== 'error'
- );
-}
-
/**
* The one reading of a session's state, for the pill beside its name.
*
@@ -31,12 +13,6 @@ export function sessionStatePill(input: {
/** An action Console is running now: stop, resume, restart or start. */
action: 'stop' | 'resume' | 'restart' | 'start' | null;
elapsedSeconds: number;
- /**
- * The host's own state when it cannot be asked, such as a sleeping cloud
- * worker. The session's last reported status is stale then, so this is shown
- * instead.
- */
- host: { label: string; tone: SessionStateTone } | null;
failed: boolean;
retired: boolean;
/** The session's own status, as the server last reported it. */
@@ -44,21 +20,18 @@ export function sessionStatePill(input: {
connectivity: 'online' | 'offline' | null;
/** Whether Console currently holds a live view of the session. */
reachable: boolean;
- /** Whether the next message restarts the session, so its being unreachable goes unsaid. */
- startable: boolean;
}): { label: string; tone: SessionStateTone } | null {
if (input.action)
return {
label: `${{ stop: 'stopping', resume: 'resuming', restart: 'restarting', start: 'starting' }[input.action]} ${input.elapsedSeconds}s`,
tone: 'busy',
};
- if (input.host) return input.host;
if (input.failed) return { label: 'connection failed', tone: 'bad' };
if (input.retired) return { label: 'retired', tone: 'idle' };
if (input.status === 'stopped') return { label: 'stopped', tone: 'idle' };
if (input.status === 'starting' && input.connectivity === 'online')
return { label: 'connecting…', tone: 'busy' };
- if (!input.reachable) return input.startable ? null : { label: 'offline', tone: 'bad' };
+ if (!input.reachable) return { label: 'offline', tone: 'bad' };
if (input.status === null) return { label: 'loading', tone: 'idle' };
return { label: input.status, tone: input.status === 'error' ? 'bad' : 'ready' };
}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-v1-chat.tsx b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-v1-chat.tsx
index ef48da92f..2fc546a73 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-v1-chat.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/session-v1-chat.tsx
@@ -5,19 +5,9 @@ import { SessionHeaderContent } from '@renderer/features/sessions/session-header
import { Button } from '@renderer/lib/ui/button';
import { MarkdownRenderer } from '@renderer/lib/ui/markdown-renderer';
import { Textarea } from '@renderer/lib/ui/textarea';
-import type { CloudAgentPhase } from '@shared/core/cloud-agents/cloud-agents';
import type { InitialPromptDelivery } from '@shared/core/sessions/session-config';
-import {
- deliverHeld,
- type HeldMessage,
- heldMessages,
- type HeldWait,
- heldStatusText,
- isWakingError,
- relayRefusalText,
-} from './held-message';
import { SessionAttachmentList, useSessionAttachments } from './session-attachments';
-import { restartsOnSend, sessionStatePill, type SessionStateTone } from './session-state';
+import { sessionStatePill } from './session-state';
import { SessionStatePill } from './session-state-pill';
import { SessionV1Controls } from './session-v1-controls';
import { SessionV1Request } from './session-v1-request';
@@ -30,28 +20,13 @@ export function SessionV1Chat({
startup,
retireHost,
initialPromptDelivery,
- hostState,
- autoWake,
}: {
client: SessionChatClient;
- /** Overrides the session's status while its host cannot be asked. */
- hostState: { label: string; tone: SessionStateTone } | null;
initialPromptDelivery?: InitialPromptDelivery;
restartHost?: () => Promise<void>;
stopHost?: () => Promise<void>;
startup?: { status: 'starting' | 'ready' | 'error'; message: string | null } | null;
retireHost?: (epoch: string) => Promise<void>;
- /**
- * A cloud agent's machine, woken by the first message sent while it sleeps.
- * `blocked` says why a held message will not be delivered without the user.
- * `machineReady` says a waking agent waits only on its own process.
- */
- autoWake?: {
- phase: CloudAgentPhase | null;
- machineReady: boolean;
- blocked: string | null;
- wake: () => Promise<unknown>;
- };
}) {
const view = useSyncExternalStore(client.subscribe, client.getSnapshot);
const transcript = useRef<HTMLDivElement>(null);
@@ -113,15 +88,11 @@ export function SessionV1Chat({
setAction(null);
}
};
- const [held, setHeld] = useState<HeldMessage | null>(() => heldMessages.get(client.sessionId));
- useEffect(() => heldMessages.set(client.sessionId, held), [client, held]);
- const [draft, setDraft] = useState(() => held?.text ?? '');
- const [pendingId, setPendingId] = useState<string | null>(() => held?.commandId ?? null);
+ const [draft, setDraft] = useState('');
+ const [pendingId, setPendingId] = useState<string | null>(null);
const [sending, setSending] = useState(false);
const [retireConfirm, setRetireConfirm] = useState(false);
const [sendError, setSendError] = useState<string | null>(null);
- const [heldRetryWait, setHeldRetryWait] = useState(false);
- const delivering = useRef(false);
useEffect(() => {
void client.connect();
return () => client.dispose();
@@ -134,21 +105,6 @@ export function SessionV1Chat({
view.connected &&
session?.connectivity === 'online' &&
(session.status === 'ready' || session.status === 'running');
- const wakeable = autoWake?.phase === 'sleeping' || autoWake?.phase === 'waking';
- const machineAwake = !wakeable;
- const heldWait: HeldWait = machineAwake
- ? 'session'
- : autoWake?.phase === 'waking' && autoWake.machineReady
- ? 'agent'
- : 'machine';
- const startable =
- Boolean(restartHost) &&
- autoWake?.phase === null &&
- !autoWake.blocked &&
- view.connected &&
- session !== undefined &&
- restartsOnSend(session);
- const restartedFor = useRef<string | null>(null);
const runningTurn = view.snapshot?.turns.find((turn) => turn.status === 'running');
const lastItems = new Map(view.snapshot?.items.map((item) => [item.turnId, item.itemId]));
const stoppedTurns = new Map(
@@ -187,105 +143,24 @@ export function SessionV1Chat({
}
};
const send = async () => {
- if (
- (!available && !wakeable && !startable) ||
- held ||
- sending ||
- uploads.blocked ||
- (!draft.trim() && !uploads.attachments.length)
- )
+ if (!available || sending || uploads.blocked || (!draft.trim() && !uploads.attachments.length))
return;
const commandId = pendingId ?? crypto.randomUUID();
- const message = { commandId, text: draft.trim(), attachments: uploads.attachments };
setPendingId(commandId);
setSendError(null);
- if (!available && autoWake) {
- setHeld(message);
- if (autoWake.phase !== 'sleeping') return;
- try {
- await autoWake.wake();
- } catch (error) {
- setHeld(null);
- if (!client.hasPendingCommand()) setPendingId(null);
- setSendError(`Could not wake the machine: ${String(error)}`);
- }
- return;
- }
setSending(true);
try {
- await client.send(message.text, commandId, message.attachments);
+ await client.send(draft.trim(), commandId, uploads.attachments);
uploads.clear();
setDraft('');
setPendingId(null);
} catch (error) {
- if (isWakingError(error)) setHeld(message);
- else {
- setSendError(relayRefusalText(error) ?? String(error));
- if (!client.hasPendingCommand()) setPendingId(null);
- }
+ setSendError(String(error));
+ if (!client.hasPendingCommand()) setPendingId(null);
} finally {
setSending(false);
}
};
- const heldStuck = !held
- ? null
- : actionError && restartedFor.current === held.commandId
- ? `${actionError} Your message was not sent.`
- : autoWake?.blocked
- ? `${autoWake.blocked} Your message was not sent.`
- : autoWake?.phase === null &&
- view.connected &&
- (session?.status === 'stopped' || session?.status === 'error')
- ? 'The machine is awake, but the session is not ready for messages. Your message was not sent.'
- : null;
- useEffect(() => {
- if (!heldStuck || sending) return;
- restartedFor.current = null;
- setHeld(null);
- setHeldRetryWait(false);
- if (!client.hasPendingCommand()) setPendingId(null);
- setSendError(heldStuck);
- }, [heldStuck, sending, client]);
- useEffect(() => {
- if (!held || heldStuck || !startable || busy || !restartHost) return;
- if (restartedFor.current === held.commandId) return;
- restartedFor.current = held.commandId;
- void runAction('restart', restartHost);
- });
- const cancelHeld = () => {
- setHeld(null);
- setHeldRetryWait(false);
- if (!client.hasPendingCommand()) setPendingId(null);
- };
- useEffect(() => {
- if (!held || heldStuck || !available || heldRetryWait || delivering.current) return;
- delivering.current = true;
- setSending(true);
- void (async () => {
- try {
- await deliverHeld(client, held);
- uploads.clear();
- setDraft('');
- setPendingId(null);
- setHeld(null);
- } catch (error) {
- if (isWakingError(error)) setHeldRetryWait(true);
- else {
- setHeld(null);
- setSendError(relayRefusalText(error) ?? String(error));
- if (!client.hasPendingCommand()) setPendingId(null);
- }
- } finally {
- delivering.current = false;
- setSending(false);
- }
- })();
- }, [held, heldStuck, available, heldRetryWait, client, uploads]);
- useEffect(() => {
- if (!heldRetryWait) return;
- const timer = setTimeout(() => setHeldRetryWait(false), 5000);
- return () => clearTimeout(timer);
- }, [heldRetryWait]);
const reconcile = async () => {
setSending(true);
try {
@@ -307,13 +182,11 @@ export function SessionV1Chat({
const pill = sessionStatePill({
action: busy ? (action ?? 'start') : null,
elapsedSeconds: elapsed,
- host: hostState,
failed: Boolean(actionError) || startup?.status === 'error',
retired: Boolean(session?.retired),
status: session?.status ?? null,
connectivity: session?.connectivity ?? null,
reachable: available || (view.connected && session?.connectivity === 'online'),
- startable,
});
return (
@@ -561,7 +434,7 @@ export function SessionV1Chat({
)}
</details>
)}
- {!busy && (view.error || (!view.connected && !held)) && (
+ {!busy && (view.error || !view.connected) && (
<div
role="status"
className="flex items-center justify-between gap-3 bg-background-1 px-5 py-2 text-sm"
@@ -689,15 +562,6 @@ export function SessionV1Chat({
</div>
</div>
<div className="mx-auto w-full max-w-[720px] px-5 pb-5">
- {held && (
- <div role="status" className="mb-2 flex items-center gap-2 text-sm text-foreground-muted">
- <Loader2 className="size-3 shrink-0 animate-spin" />
- <span className="min-w-0 flex-1">{heldStatusText(heldWait)}</span>
- <Button size="sm" variant="ghost" disabled={sending} onClick={cancelHeld}>
- Cancel
- </Button>
- </div>
- )}
{sendError && (
<div role="alert" className="mb-2 text-sm text-foreground-destructive">
{sendError}
@@ -816,21 +680,14 @@ export function SessionV1Chat({
size="sm"
className="ml-auto shrink-0"
disabled={
- (!available && !wakeable && !startable) ||
- held !== null ||
+ !available ||
sending ||
uploads.blocked ||
(!draft.trim() && !uploads.attachments.length)
}
onClick={() => void send()}
>
- {sending
- ? 'Sending…'
- : held
- ? { session: 'Connecting…', agent: 'Starting…', machine: 'Waking…' }[heldWait]
- : pendingId
- ? 'Retry message'
- : 'Send'}
+ {sending ? 'Sending…' : pendingId ? 'Retry message' : 'Send'}
</Button>
</div>
</div>
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-panel.tsx b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-panel.tsx
index 6c0411314..5f1c6b316 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-panel.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-panel.tsx
@@ -91,7 +91,6 @@ export function SharedSessionPanel({
<SessionV1Chat
client={client}
startup={startup}
- hostState={null}
stopHost={() => rpc.sdkHost.stop(agentId, sessionId)}
initialPromptDelivery={initialPromptDelivery}
restartHost={() => rpc.sessions.restartAgent(sessionId)}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-transport.test.ts b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-transport.test.ts
index 0f854d222..60b711eaf 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-transport.test.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-transport.test.ts
@@ -1,11 +1,10 @@
import { expect, it, vi } from 'vitest';
-import { cloudSessionTransport, hostJournalTransport } from './shared-session-transport';
+import { hostJournalTransport } from './shared-session-transport';
const ipc = vi.hoisted(() => ({
transcriptOpen: vi.fn(),
transcriptClose: vi.fn(),
sessionSubmit: vi.fn(),
- cloudUploadAttachment: vi.fn(),
}));
const bus = vi.hoisted(() => ({
listeners: new Map<string, (payload: never) => void>(),
@@ -100,25 +99,3 @@ it('reports a broken live feed so the view reloads the session', () => {
);
stop();
});
-
-it('uploads a cloud session’s attachment to its worker, and reads it as any host journal', async () => {
- const transport = cloudSessionTransport('cloud:server:launch');
- const staged = { attachmentId: 'ref', name: 'a.txt', mimeType: 'text/plain', bytes: 2 };
- ipc.cloudUploadAttachment.mockResolvedValueOnce(staged);
- await expect(
- transport.uploadAttachment!('session', {
- attachmentId: 'local',
- name: 'a.txt',
- mimeType: 'text/plain',
- data: 'aGk=',
- })
- ).resolves.toBe(staged);
- expect(ipc.cloudUploadAttachment).toHaveBeenCalledWith('cloud:server:launch', 'session', {
- name: 'a.txt',
- mimeType: 'text/plain',
- data: 'aGk=',
- });
- ipc.transcriptOpen.mockResolvedValueOnce('snapshot');
- await transport.snapshot('session', null);
- expect(ipc.transcriptOpen).toHaveBeenLastCalledWith('cloud:server:launch', 'session');
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-transport.ts b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-transport.ts
index be6023d83..9ccffbd2e 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-transport.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/sessions/components/transcript/shared-session-transport.ts
@@ -60,20 +60,3 @@ export function hostJournalTransport(agentId: string): SessionTransport {
commandStatusSchema.parse(await rpc.sdkHost.sessionCommandStatus(agentId, id, commandId)),
};
}
-
-/**
- * A cloud agent's session: the same host journal as `hostJournalTransport`,
- * reached through the Switch server's relay to the launch's worker. Files go
- * up to the worker in chunks and are staged there for the next message.
- */
-export function cloudSessionTransport(agentKey: string): SessionTransport {
- return {
- ...hostJournalTransport(agentKey),
- uploadAttachment: (id, file) =>
- rpc.sdkHost.cloudUploadAttachment(agentKey, id, {
- name: file.name,
- mimeType: file.mimeType,
- data: file.data,
- }),
- };
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-empty-state.test.ts b/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-empty-state.test.ts
index 6aaea2176..c4d653dac 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-empty-state.test.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-empty-state.test.ts
@@ -15,7 +15,7 @@ it('says a server with nothing on it is empty', () => {
expect(sidebarEmptyState(empty)).toBe('empty');
});
-it('does not call a server empty while it lists cloud agents', () => {
+it('does not call a server empty while it lists managed agents', () => {
expect(sidebarEmptyState({ ...empty, serverListedAgentCount: 1 })).toBeNull();
});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-empty-state.ts b/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-empty-state.ts
index 9a0582a74..612f50da7 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-empty-state.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-empty-state.ts
@@ -7,8 +7,8 @@ import type { SidebarGrouping } from '@shared/view-state';
* room view lists rooms, which are still there, and reports its own filters
* being too narrow itself. A server with nothing on it yet otherwise left the
* panel blank, which reads as the list having failed to load rather than as
- * there being nothing to list; its cloud agents are listed below the tree, so
- * a server that has them is not empty.
+ * there being nothing to list; its managed agents are listed below the tree,
+ * so a server that has them is not empty.
*/
export function sidebarEmptyState(input: {
grouping: SidebarGrouping;
@@ -17,7 +17,7 @@ export function sidebarEmptyState(input: {
activeServerId: string | null;
locationCount: number;
roomCount: number;
- /** Agents listed from the server rather than Console rows: cloud and managed ones. */
+ /** Agents listed from the server rather than Console rows: the managed ones. */
serverListedAgentCount: number;
}): 'no-filter-match' | 'empty' | null {
if (input.grouping !== 'room' && input.hasActiveFilters && input.filteredLocationCount === 0)
diff --git a/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-grouped-list.tsx b/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-grouped-list.tsx
index 91a9a0ff2..f1ea2b645 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-grouped-list.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/sidebar/sidebar-grouped-list.tsx
@@ -2,8 +2,6 @@ import { AlertTriangle } from 'lucide-react';
import { reaction } from 'mobx';
import { observer } from 'mobx-react-lite';
import { useEffect, useRef } from 'react';
-import { CloudAgentList } from '@renderer/features/cloud-agents/cloud-agent-list';
-import { useCloudAgents } from '@renderer/features/cloud-agents/use-cloud-agents';
import { ManagedAgentList } from '@renderer/features/managed-agents/managed-agent-list';
import { useManagedAgents } from '@renderer/features/managed-agents/use-managed-agents';
import { hostReachabilityStore } from '@renderer/features/remote-hosts/host-reachability-store';
@@ -89,7 +87,6 @@ export const SidebarGroupedList = observer(function SidebarGroupedList() {
};
}, []);
- const cloudAgents = useCloudAgents(switchServersStore.activeServerId);
const managedAgents = useManagedAgents(switchServersStore.activeServerId);
const activeServerId = switchServersStore.activeServerId;
const managedAgentCount = activeServerId && managedAgents.data ? managedAgents.data.length : 0;
@@ -100,7 +97,7 @@ export const SidebarGroupedList = observer(function SidebarGroupedList() {
activeServerId: switchServersStore.activeServerId,
locationCount: sidebarStore.orderedLocations.length,
roomCount: switchRoomsStore.listedRoomsInActiveScope.length,
- serverListedAgentCount: (cloudAgents.data?.length ?? 0) + managedAgentCount,
+ serverListedAgentCount: managedAgentCount,
});
return (
@@ -121,7 +118,6 @@ export const SidebarGroupedList = observer(function SidebarGroupedList() {
<AgentTree />
)}
{sidebarStore.grouping !== 'room' && <ManagedAgentList />}
- {sidebarStore.grouping !== 'room' && <CloudAgentList />}
</div>
);
});
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/AddServerModal.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/AddServerModal.tsx
index 224e32be2..101535054 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/AddServerModal.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/AddServerModal.tsx
@@ -20,7 +20,6 @@ import {
lockHolderSentence,
othersRecentlySeen,
} from '@shared/core/managed-switch-server/managed-switch-server';
-import type { AgentProviderId } from '@shared/core/providers/agent-provider-registry';
import type {
AddServerChoiceName,
AddServerStepName,
@@ -33,8 +32,6 @@ import { ConnectionsStep } from './connections-step';
import { LinkAccountsStep } from './link-accounts-step';
import { localServerStore } from './local-server-store';
import { LogTail } from './log-tail';
-import { ManagedProviderConnectionSequence } from './managed-provider-connection-step';
-import { ManagedProvidersStep } from './managed-providers-step';
import { remoteServerStore } from './remote-server-store';
import { type RemoteSetupAction, remoteSetupAction } from './remote-setup-action';
import {
@@ -92,8 +89,6 @@ type Props = BaseModalProps<void> & {
type Step =
| 'managedAgent'
| 'managedGitHub'
- | 'managedClaude'
- | 'managedReady'
| 'choose'
| 'local'
| 'remoteHost'
@@ -135,8 +130,6 @@ void _stepsAreComplete;
*/
const CHOICE_FOR_STEP: Record<Step, AddServerChoiceName | null> = {
choose: 'none',
- managedReady: 'cloud',
- managedClaude: 'cloud',
managedGitHub: 'cloud',
managedAgent: 'cloud',
local: 'local',
@@ -151,8 +144,6 @@ export const AddServerModal = observer(function AddServerModal(props: Props) {
const openedAt: Step = isEdit ? 'external' : (props.mode ?? 'choose');
const openedWith = CHOICE_FOR_STEP[openedAt] ?? 'none';
const [step, setStep] = useState<Step>(openedAt);
- const [providerIndex, setProviderIndex] = useState(0);
- const [selectedProviders, setSelectedProviders] = useState<AgentProviderId[]>([]);
// Which path was taken at the chooser, carried so every later step can be
// attributed to it. `none` while still on the chooser, which is what makes a
// drop-off before choosing distinguishable from one after.
@@ -219,7 +210,7 @@ export const AddServerModal = observer(function AddServerModal(props: Props) {
* account already signed in goes on to setting up its cloud agents.
*/
const enterCloud = (server: SwitchServer) => {
- const next: Step = switchServersStore.isConnected(server.id) ? 'managedReady' : 'signIn';
+ const next: Step = switchServersStore.isConnected(server.id) ? 'managedGitHub' : 'signIn';
setConnected(server);
setChoice('cloud');
setStep(next);
@@ -267,22 +258,6 @@ export const AddServerModal = observer(function AddServerModal(props: Props) {
/>
);
}
- if (step === 'managedClaude' && connected) {
- return (
- <>
- {machineNotice}
- <ManagedProviderConnectionSequence
- serverId={connected.id}
- providers={selectedProviders}
- index={providerIndex}
- onIndexChange={setProviderIndex}
- onBack={() => goToStep('managedReady')}
- onDone={() => goToStep('managedGitHub')}
- doneStepName="GitHub"
- />
- </>
- );
- }
if (step === 'managedAgent' && connected) {
return (
<>
@@ -304,23 +279,7 @@ export const AddServerModal = observer(function AddServerModal(props: Props) {
<ConnectionsStep
onContinue={() => goToStep('managedAgent')}
serverId={connected.id}
- onBack={() => goToStep('managedClaude')}
- onSkip={() => finish(connected.id)}
- />
- </>
- );
- }
- if (step === 'managedReady' && connected) {
- return (
- <>
- {machineNotice}
- <ManagedProvidersStep
- selected={selectedProviders}
- onSelectionChange={setSelectedProviders}
- onContinue={() => {
- setProviderIndex(0);
- goToStep('managedClaude');
- }}
+ onBack={() => goToStep('choose')}
onSkip={() => finish(connected.id)}
/>
</>
@@ -334,7 +293,7 @@ export const AddServerModal = observer(function AddServerModal(props: Props) {
onClose={props.onClose}
onSignedIn={(signedIn) => {
setMachineUnavailable(machineUnavailableReason(signedIn));
- goToStep(choice === 'cloud' ? 'managedReady' : 'linkAccounts');
+ goToStep(choice === 'cloud' ? 'managedGitHub' : 'linkAccounts');
}}
/>
);
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/CreateRoomModal.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/CreateRoomModal.tsx
index c16adda19..d601a405d 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/CreateRoomModal.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/CreateRoomModal.tsx
@@ -2,9 +2,11 @@ import { useQuery } from '@tanstack/react-query';
import { observer } from 'mobx-react-lite';
import { useCallback, useState } from 'react';
import { agentsStore } from '@renderer/features/locations/stores/agents-store';
+import { useManagedAgents } from '@renderer/features/managed-agents/use-managed-agents';
import { openRoomView } from '@renderer/features/sidebar/sidebar-room-grouping';
import { refreshSidebarRoomState } from '@renderer/features/sidebar/sidebar-tree-data';
import { workspacesStore } from '@renderer/features/workspaces/workspaces-store';
+import { agentProviderLabel } from '@renderer/lib/components/agent-mark';
import {
AgentPickerRow,
ChosenAgentTile,
@@ -77,9 +79,11 @@ export const CreateRoomModal = observer(function CreateRoomModal({
// from a stranger as far as the agents are concerned — so it is said here,
// while the app is being chosen, rather than discovered later.
const { identities } = useMyIdentities(workspaceId);
+ const managedQuery = useManagedAgents(serverId || null);
/**
- * Only agents this install registered on the server.
+ * Only agents this install registered on the server, and the user's managed
+ * agents, which Console drives through the server wherever they run.
*
* The server answers with everyone registered on it, including agents
* belonging to somebody else's Switch Console. Those cannot be shown under a
@@ -87,10 +91,18 @@ export const CreateRoomModal = observer(function CreateRoomModal({
* something this app cannot deliver — the same rule the room views already
* follow.
*/
+ const managedProviders = new Map(
+ (managedQuery.data ?? []).map((managed) => [managed.agentId, managed.definition.provider])
+ );
+ const providerLabel = (agentId: string) =>
+ managedProviders.has(agentId)
+ ? agentProviderLabel(managedProviders.get(agentId))
+ : agentProviderLabelFor(agentId, workspaceId);
const invitableAgents = (agentsQuery.data ?? []).filter((remote) =>
workspaceId === null
? false
- : agentsStore
+ : managedProviders.has(remote.id) ||
+ agentsStore
.agentsInWorkspace(workspaceId)
.some((local) => local.switchAgentId === remote.id)
);
@@ -296,7 +308,7 @@ export const CreateRoomModal = observer(function CreateRoomModal({
<ChosenAgentTile
key={agent.id}
agent={agent}
- subtitle={agentProviderLabelFor(agent.id, workspaceId)}
+ subtitle={providerLabel(agent.id)}
onRemove={() =>
setAgents((current) => current.filter((a) => a.id !== agent.id))
}
@@ -310,10 +322,7 @@ export const CreateRoomModal = observer(function CreateRoomModal({
onPick={(next) => setAgents((current) => [...current, next])}
searchText={(item) => item.name}
renderItem={(item) => (
- <AgentPickerRow
- agent={item}
- subtitle={agentProviderLabelFor(item.id, workspaceId)}
- />
+ <AgentPickerRow agent={item} subtitle={providerLabel(item.id)} />
)}
disabled={agentsQuery.isLoading}
placeholder={agentsQuery.isLoading ? 'Loading agents…' : 'Search agents to add...'}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-claude-connection-step.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-claude-connection-step.tsx
deleted file mode 100644
index 985caedcb..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-claude-connection-step.tsx
+++ /dev/null
@@ -1,132 +0,0 @@
-import { CircleCheck } from 'lucide-react';
-import { useCallback, useEffect, useState } from 'react';
-import { failureText } from '@renderer/lib/errors/describe-failure';
-import { rpc } from '@renderer/lib/ipc';
-import { Button } from '@renderer/lib/ui/button';
-import {
- DialogContentArea,
- DialogFooter,
- DialogHeader,
- DialogTitle,
-} from '@renderer/lib/ui/dialog';
-import { Spinner } from '@renderer/lib/ui/spinner';
-import type { ClaudeConnection } from '@shared/core/switch-servers/claude-credential';
-import { ManagedClaudeStep } from './managed-claude-step';
-
-export function ManagedClaudeConnectionStep({
- serverId,
- onBack,
- onDone,
- continueLabel,
-}: {
- serverId: string;
- onBack: () => void;
- onDone: () => void;
- continueLabel: string;
-}) {
- const [connection, setConnection] = useState<ClaudeConnection | null>(null);
- const [editing, setEditing] = useState(false);
- const [error, setError] = useState<string | null>(null);
- const [loading, setLoading] = useState(true);
- const load = useCallback(async () => {
- setLoading(true);
- setError(null);
- try {
- setConnection(await rpc.switchServers.getClaudeConnection(serverId));
- } catch (cause) {
- setError(failureText(cause, 'Could not load your Claude connection.'));
- } finally {
- setLoading(false);
- }
- }, [serverId]);
- useEffect(() => {
- void load();
- }, [load]);
- if (!loading && connection && (connection.status === 'not_connected' || editing)) {
- return (
- <ManagedClaudeStep
- onBack={() => {
- if (editing) setEditing(false);
- else onBack();
- }}
- onSave={async (kind, credential) => {
- const result = await rpc.switchServers.connectClaude(serverId, kind, credential);
- setConnection(result);
- setEditing(false);
- setError(null);
- }}
- />
- );
- }
- return (
- <>
- <DialogHeader>
- <DialogTitle>
- {connection?.status === 'connected' ? 'Claude Code connected' : 'Connect Claude Code'}
- </DialogTitle>
- </DialogHeader>
- <DialogContentArea className="space-y-4 pt-0">
- {loading ? (
- <div className="flex items-center gap-2 text-sm">
- <Spinner /> Checking connection…
- </div>
- ) : connection?.status === 'connected' ? (
- <>
- <div className="flex items-center gap-2 text-sm">
- <CircleCheck className="size-5 text-foreground-success" /> Verified with Claude Code
- </div>
- <p className="text-sm text-foreground-muted">
- {connection.kind === 'api-key' ? 'API key' : 'Subscription setup token'} · Verified{' '}
- {new Date(connection.verified_at).toLocaleString()}
- </p>
- <p className="text-sm">Your credential is stored encrypted on Switch.</p>
- <p className="text-xs text-foreground-muted">
- Removing this connection deletes the saved credential from Switch; it does not revoke
- it with Claude.
- </p>
- </>
- ) : null}
- {error && (
- <p role="alert" className="text-sm text-destructive">
- {error}
- </p>
- )}
- {!loading && !connection && (
- <Button variant="outline" onClick={() => void load()}>
- Retry
- </Button>
- )}
- </DialogContentArea>
- <DialogFooter>
- <Button variant="outline" onClick={onBack} disabled={loading}>
- Back
- </Button>
- {!loading && connection?.status === 'connected' && (
- <>
- <Button variant="outline" onClick={() => setEditing(true)}>
- Replace credential
- </Button>
- <Button
- variant="ghost"
- onClick={async () => {
- setLoading(true);
- setError(null);
- try {
- await rpc.switchServers.disconnectClaude(serverId);
- setConnection({ status: 'not_connected' });
- } catch (cause) {
- setError(failureText(cause, 'Could not remove the Claude connection.'));
- } finally {
- setLoading(false);
- }
- }}
- >
- Remove
- </Button>
- <Button onClick={onDone}>{continueLabel}</Button>
- </>
- )}
- </DialogFooter>
- </>
- );
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-claude-step.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-claude-step.tsx
deleted file mode 100644
index cb84b4811..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-claude-step.tsx
+++ /dev/null
@@ -1,256 +0,0 @@
-import { ExternalLink } from 'lucide-react';
-import { useState } from 'react';
-import {
- CommandRow,
- CommandActionButton,
-} from '@renderer/features/settings/agents-page/install-command-row';
-import { failureText } from '@renderer/lib/errors/describe-failure';
-import { useCloseGuard } from '@renderer/lib/modal/use-close-guard';
-import { openExternalUrl } from '@renderer/lib/open-external';
-import { Button } from '@renderer/lib/ui/button';
-import {
- DialogContentArea,
- DialogFooter,
- DialogHeader,
- DialogTitle,
-} from '@renderer/lib/ui/dialog';
-import {
- Field,
- FieldLabel,
- FieldContent,
- FieldTitle,
- FieldDescription,
-} from '@renderer/lib/ui/field';
-import { Input } from '@renderer/lib/ui/input';
-import { RadioGroup, RadioGroupItem } from '@renderer/lib/ui/radio-group';
-import type { ClaudeCredentialKind } from '@shared/core/switch-servers/claude-credential';
-
-export function ManagedClaudeStep({
- onBack,
- onSave,
-}: {
- onBack: () => void;
- onSave: (kind: ClaudeCredentialKind, credential: string) => Promise<void>;
-}) {
- const [kind, setKind] = useState<ClaudeCredentialKind>('api-key');
- const [credential, setCredential] = useState('');
- const [saving, setSaving] = useState(false);
- const [error, setError] = useState<string | null>(null);
- useCloseGuard(saving);
- const subscription = kind === 'setup-token';
- const save = async () => {
- setSaving(true);
- setError(null);
- try {
- await onSave(kind, credential.trim());
- setCredential('');
- } catch (cause) {
- setError(failureText(cause, 'Could not connect Claude Code. Try again.'));
- } finally {
- setSaving(false);
- }
- };
- return (
- <>
- <DialogHeader showCloseButton={!saving}>
- <DialogTitle>Connect Claude Code</DialogTitle>
- </DialogHeader>
- <DialogContentArea className="space-y-5 pt-0">
- <p className="text-sm text-foreground-muted">
- Choose how your Claude Code agent will authenticate.
- </p>
- <RadioGroup
- aria-label="Claude Code authentication"
- value={kind}
- disabled={saving}
- onValueChange={(value) => {
- if (value !== 'api-key' && value !== 'setup-token') return;
- setKind(value);
- setCredential('');
- setError(null);
- }}
- className="grid grid-cols-2 gap-3"
- >
- <FieldLabel>
- <Field orientation="horizontal">
- <RadioGroupItem value="api-key" />
- <FieldContent>
- <FieldTitle>API key</FieldTitle>
- <FieldDescription>Pay for API usage.</FieldDescription>
- </FieldContent>
- </Field>
- </FieldLabel>
- <FieldLabel>
- <Field orientation="horizontal">
- <RadioGroupItem value="setup-token" />
- <FieldContent>
- <FieldTitle>Subscription</FieldTitle>
- <FieldDescription>Use your Claude plan.</FieldDescription>
- </FieldContent>
- </Field>
- </FieldLabel>
- </RadioGroup>
- <section
- aria-label={subscription ? 'Get a setup token' : 'Get an API key'}
- className="space-y-4"
- >
- <div className="space-y-1">
- <h3 className="text-sm font-medium">
- {subscription ? 'Get a setup token' : 'Get an API key'}
- </h3>
- <p className="text-xs text-foreground-muted">
- {subscription
- ? 'Use a Pro, Max, Team, or Enterprise plan with Claude Code access.'
- : 'API billing is separate from your Claude subscription.'}
- </p>
- </div>
- {subscription ? (
- <ol className="space-y-4 text-sm">
- <li className="flex gap-3">
- <span
- aria-hidden="true"
- className="flex size-5 shrink-0 items-center justify-center rounded-full border text-xs text-foreground-muted"
- >
- 1
- </span>
- <div className="space-y-2">
- <p>Install Claude Code on your computer.</p>
- <Button
- variant="outline"
- size="sm"
- onClick={() =>
- void openExternalUrl(
- 'https://code.claude.com/docs/en/setup',
- 'Could not open Claude Code installation instructions.'
- )
- }
- >
- Installation guide <ExternalLink className="size-3.5" />
- </Button>
- </div>
- </li>
- <li className="flex gap-3">
- <span
- aria-hidden="true"
- className="flex size-5 shrink-0 items-center justify-center rounded-full border text-xs text-foreground-muted"
- >
- 2
- </span>
- <div className="min-w-0 flex-1 space-y-2">
- <p>Run this command in your terminal.</p>
- <CommandRow
- command="claude setup-token"
- action={
- <CommandActionButton
- aria-label="Open setup token documentation"
- onClick={() =>
- void openExternalUrl(
- 'https://code.claude.com/docs/en/authentication#generate-a-long-lived-token',
- 'Could not open setup token instructions.'
- )
- }
- >
- Help <ExternalLink className="size-3.5" />
- </CommandActionButton>
- }
- />
- </div>
- </li>
- <li className="flex gap-3">
- <span
- aria-hidden="true"
- className="flex size-5 shrink-0 items-center justify-center rounded-full border text-xs text-foreground-muted"
- >
- 3
- </span>
- <div className="space-y-1">
- <p>Approve access in your browser.</p>
- <p className="text-xs leading-relaxed text-foreground-muted">
- Sign in with your Claude subscription, then paste the token printed in your
- terminal below. Your plan’s usage limits apply.
- </p>
- </div>
- </li>
- </ol>
- ) : (
- <ol className="space-y-4 text-sm">
- <li className="flex gap-3">
- <span
- aria-hidden="true"
- className="flex size-5 shrink-0 items-center justify-center rounded-full border text-xs text-foreground-muted"
- >
- 1
- </span>
- <div className="space-y-2">
- <p>Sign in and set up API billing.</p>
- <Button
- variant="outline"
- size="sm"
- onClick={() =>
- void openExternalUrl(
- 'https://platform.claude.com/settings/keys',
- 'Could not open Claude Console.'
- )
- }
- >
- Open Claude Console <ExternalLink className="size-3.5" />
- </Button>
- </div>
- </li>
- <li className="flex gap-3">
- <span
- aria-hidden="true"
- className="flex size-5 shrink-0 items-center justify-center rounded-full border text-xs text-foreground-muted"
- >
- 2
- </span>
- <div className="space-y-1">
- <p>Create an API key.</p>
- <p className="text-xs leading-relaxed text-foreground-muted">
- Go to{' '}
- <span className="font-medium text-foreground">
- Settings → API keys → Create key
- </span>
- , then copy the new key and paste it below.
- </p>
- </div>
- </li>
- </ol>
- )}
- </section>
- <Field>
- <FieldLabel htmlFor="managed-claude-credential">
- {subscription ? 'Setup token' : 'API key'}
- </FieldLabel>
- <Input
- id="managed-claude-credential"
- type="password"
- autoComplete="off"
- spellCheck={false}
- value={credential}
- disabled={saving}
- placeholder={subscription ? 'Paste your setup token' : 'Paste your API key'}
- onChange={(event) => setCredential(event.target.value)}
- />
- </Field>
- <p className="text-xs text-foreground-muted">
- Your credential is sent securely to Switch and stored encrypted after verification. The
- check makes one small Claude request and uses your API credit or subscription allowance.
- </p>
- {error && (
- <p role="alert" className="text-sm text-destructive">
- {error}
- </p>
- )}
- </DialogContentArea>
- <DialogFooter>
- <Button variant="outline" onClick={onBack} disabled={saving}>
- Back
- </Button>
- <Button onClick={() => void save()} disabled={saving || !credential.trim()}>
- {saving ? 'Verifying with Claude…' : 'Verify and connect'}
- </Button>
- </DialogFooter>
- </>
- );
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-provider-connection-step.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-provider-connection-step.tsx
deleted file mode 100644
index c32c9120d..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-provider-connection-step.tsx
+++ /dev/null
@@ -1,389 +0,0 @@
-import { useQuery, useQueryClient } from '@tanstack/react-query';
-import { useEffect, useRef, useState } from 'react';
-import { CommandRow } from '@renderer/features/settings/agents-page/install-command-row';
-import { rpc } from '@renderer/lib/ipc';
-import { openExternalUrl } from '@renderer/lib/open-external';
-import { Button } from '@renderer/lib/ui/button';
-import {
- DialogContentArea,
- DialogFooter,
- DialogHeader,
- DialogTitle,
-} from '@renderer/lib/ui/dialog';
-import { Field, FieldLabel, FieldDescription } from '@renderer/lib/ui/field';
-import { Input } from '@renderer/lib/ui/input';
-import { SegmentedControl } from '@renderer/lib/ui/segmented-control';
-import { Spinner } from '@renderer/lib/ui/spinner';
-import {
- providerDisplayName,
- type AgentProviderId,
-} from '@shared/core/providers/agent-provider-registry';
-import type { CloudProviderConnection } from '@shared/core/switch-servers/provider-credential';
-import { ManagedClaudeConnectionStep } from './managed-claude-connection-step';
-
-const instructions = {
- codex: {
- command: 'codex login',
- file: '~/.codex/auth.json',
- docs: 'https://developers.openai.com/codex/auth/',
- },
- cursor: { command: '', file: '', docs: 'https://cursor.com/dashboard/api' },
- opencode: {
- command: 'opencode auth login',
- file: '~/.local/share/opencode/auth.json',
- docs: 'https://opencode.ai/docs/providers/',
- },
- antigravity: {
- command: 'antigravity-acp --login',
- file: '~/.local/state/switch/antigravity-acp/antigravity-acp/acp_token.json',
- docs: 'https://github.com/agentclientprotocol/registry/tree/main/antigravity-acp',
- },
-};
-
-/**
- * Connect each chosen provider in turn, then hand over to the step after them.
- *
- * The continue label is read from the same list that decides where continuing
- * goes, so it always names the step it leads to.
- */
-export function ManagedProviderConnectionSequence({
- serverId,
- providers,
- index,
- onIndexChange,
- onBack,
- onDone,
- doneStepName,
-}: {
- serverId: string;
- providers: AgentProviderId[];
- index: number;
- onIndexChange: (index: number) => void;
- onBack: () => void;
- onDone: () => void;
- doneStepName: string;
-}) {
- const next = providers[index + 1];
- return (
- <ManagedProviderConnectionStep
- serverId={serverId}
- provider={providers[index] ?? 'claude'}
- continueLabel={`Continue to ${next ? providerDisplayName(next) : doneStepName}`}
- onBack={() => (index > 0 ? onIndexChange(index - 1) : onBack())}
- onDone={() => (next ? onIndexChange(index + 1) : onDone())}
- />
- );
-}
-
-export function ManagedProviderConnectionStep({
- serverId,
- provider,
- onBack,
- onDone,
- continueLabel,
-}: {
- serverId: string;
- provider: AgentProviderId;
- onBack: () => void;
- onDone: () => void;
- continueLabel: string;
-}) {
- if (provider === 'claude')
- return (
- <ManagedClaudeConnectionStep
- serverId={serverId}
- onBack={onBack}
- onDone={onDone}
- continueLabel={continueLabel}
- />
- );
- return (
- <OtherProviderConnectionStep
- key={`${serverId}:${provider}`}
- serverId={serverId}
- provider={provider}
- onBack={onBack}
- onDone={onDone}
- continueLabel={continueLabel}
- />
- );
-}
-
-function OtherProviderConnectionStep({
- serverId,
- provider,
- onBack,
- onDone,
- continueLabel,
-}: {
- serverId: string;
- provider: Exclude<AgentProviderId, 'claude'>;
- onBack: () => void;
- onDone: () => void;
- continueLabel: string;
-}) {
- const [kind, setKind] = useState<'api-key' | 'auth-json'>(
- provider === 'codex' || provider === 'cursor' ? 'api-key' : 'auth-json'
- );
- const localAuthentication = provider !== 'cursor' && kind === 'auth-json';
- const localSignIn = useQuery({
- queryKey: ['local-provider-sign-in', provider],
- queryFn: () => {
- if (provider === 'cursor') throw new Error('Cursor requires an API key.');
- return rpc.switchServers.getLocalProviderSignIn(provider);
- },
- enabled: localAuthentication,
- refetchInterval: localAuthentication ? 2000 : false,
- staleTime: 0,
- retry: false,
- });
- const [credential, setCredential] = useState('');
- const [pending, setPending] = useState(false);
- const [error, setError] = useState<string | null>(null);
- const connection = useQuery({
- queryKey: ['cloud-provider', serverId, provider],
- refetchInterval: (query) => (query.state.data?.status === 'verifying' ? 2000 : false),
- queryFn: () => rpc.switchServers.getCloudProviderConnection(serverId, provider),
- retry: false,
- });
- const queryClient = useQueryClient();
- const wasVerifying = useRef(false);
- const attemptedLocalConnection = useRef(false);
- const mounted = useRef(true);
- useEffect(() => {
- mounted.current = true;
- return () => {
- mounted.current = false;
- };
- }, []);
- useEffect(() => {
- const status = connection.data?.status;
- if (status === 'verifying') wasVerifying.current = true;
- else if (status === 'connected' && wasVerifying.current) {
- wasVerifying.current = false;
- onDone();
- } else if (status === 'failed') wasVerifying.current = false;
- }, [connection.data?.status, onDone]);
- const verifying = connection.data?.status === 'verifying';
- const busy = pending || verifying;
- const info = instructions[provider];
- const name = providerDisplayName(provider);
- const run = async (remove: boolean) => {
- attemptedLocalConnection.current = true;
- setPending(true);
- setError(null);
- try {
- let saved: CloudProviderConnection | undefined;
- if (remove) await rpc.switchServers.disconnectCloudProvider(serverId, provider);
- else if (localAuthentication)
- saved = await rpc.switchServers.connectLocalProviderSignIn(serverId, provider);
- else
- saved = await rpc.switchServers.connectCloudProvider(serverId, provider, kind, credential);
- if (!mounted.current) return;
- if (saved) queryClient.setQueryData(['cloud-provider', serverId, provider], saved);
- await connection.refetch();
- if (!mounted.current) return;
- if (remove || saved?.status !== 'verifying') {
- setCredential('');
- }
- if (remove) onBack();
- else if (saved?.status === 'connected') onDone();
- } catch (cause) {
- if (mounted.current) setError(String(cause));
- } finally {
- if (mounted.current) setPending(false);
- }
- };
- useEffect(() => {
- if (
- localAuthentication &&
- !attemptedLocalConnection.current &&
- !busy &&
- localSignIn.isFetchedAfterMount &&
- !localSignIn.isFetching &&
- !localSignIn.isError &&
- localSignIn.data?.status === 'ready' &&
- connection.isFetchedAfterMount &&
- !connection.isFetching &&
- !connection.isError &&
- connection.data?.status === 'not_connected'
- ) {
- void run(false);
- }
- });
- return (
- <>
- <DialogHeader>
- <DialogTitle>Connect {name}</DialogTitle>
- </DialogHeader>
- <DialogContentArea className="space-y-4 pt-0">
- {connection.data &&
- (connection.data.status === 'connected' || connection.data.status === 'configured') && (
- <div className="rounded-lg border p-3 text-sm">
- {connection.data.status === 'connected'
- ? 'Verified on a cloud worker.'
- : 'Credential saved. It has not passed a connection check yet.'}
- <Button variant="ghost" size="sm" disabled={pending} onClick={() => void run(true)}>
- Disconnect
- </Button>
- </div>
- )}
- {verifying && (
- <div role="status" className="space-y-2 rounded-lg border p-3 text-sm">
- <p className="flex items-center gap-2">
- <Spinner /> Checking connection…
- </p>
- <p className="text-xs text-foreground-muted">Allow about 1–2 minutes for this check.</p>
- <p className="text-xs text-foreground-muted">
- Switch is starting a temporary worker and sending a short test request. It will shut
- down automatically. You can leave this screen while the check runs.
- </p>
- </div>
- )}
- {connection.data?.status === 'failed' && (
- <p role="alert" className="text-sm text-destructive">
- {connection.data.error}
- </p>
- )}
- {!verifying && provider === 'codex' && (
- <SegmentedControl
- value={kind}
- onChange={(next) => {
- setKind(next);
- setCredential('');
- setError(null);
- }}
- options={[
- { value: 'api-key', label: 'API key' },
- { value: 'auth-json', label: 'Subscription' },
- ]}
- ariaLabel="Codex authentication"
- />
- )}
- {!verifying && (
- <div className="space-y-3 rounded-lg border bg-background-tertiary p-4 text-sm">
- {kind === 'api-key' ? (
- <p>
- {provider === 'cursor'
- ? 'Create a User API Key in your Cursor dashboard.'
- : 'Create an API key in your OpenAI project. API usage is billed separately from a ChatGPT subscription.'}
- </p>
- ) : (
- <>
- <p>
- Sign in to {name}
- {provider === 'codex' ? ' with ChatGPT' : ''} on this computer. Switch checks for
- your sign-in automatically.
- </p>
- <CommandRow
- command={
- localSignIn.data && 'command' in localSignIn.data
- ? (localSignIn.data.command ?? info.command)
- : info.command
- }
- action={null}
- />
- <p className="font-mono text-xs break-all">{localSignIn.data?.path ?? info.file}</p>
- {provider === 'codex' && (
- <div className="space-y-2">
- <p className="text-xs text-foreground-muted">
- If Codex uses your system keychain, sign in with file storage:
- </p>
- <CommandRow
- command={`codex -c cli_auth_credentials_store='"file"' login`}
- action={null}
- />
- </div>
- )}
- {provider === 'antigravity' && (
- <p>
- Use the ACP login installed by Switch. Switch checks GEMINI_HOME when set. A
- macOS keychain login must first be saved using AGY_ACP_FORCE_FILE_STORAGE=1.
- </p>
- )}
- </>
- )}
- <Button
- variant="outline"
- size="sm"
- onClick={() => openExternalUrl(info.docs, 'Could not open provider instructions')}
- >
- {provider === 'cursor' ? 'Open API keys' : 'Sign-in instructions'}
- </Button>
- </div>
- )}
- {!verifying &&
- (localAuthentication ? (
- <div className="space-y-2 rounded-lg border p-3 text-sm" role="status">
- {localSignIn.data &&
- 'detectionWarning' in localSignIn.data &&
- localSignIn.data.detectionWarning && (
- <p className="text-muted-foreground">{localSignIn.data.detectionWarning}</p>
- )}
- <p>
- {localSignIn.error
- ? String(localSignIn.error)
- : localSignIn.data?.status === 'ready'
- ? 'Local sign-in found.'
- : 'Waiting for a local sign-in…'}
- </p>
- <p className="text-xs text-foreground-muted">
- Once found, Switch automatically checks your credential on a temporary cloud worker
- and saves it encrypted for your cloud agents.
- </p>
- </div>
- ) : (
- <Field>
- <FieldLabel>API key</FieldLabel>
- <Input
- type="password"
- autoComplete="off"
- value={credential}
- onChange={(event) => setCredential(event.target.value)}
- />
- <FieldDescription>
- Credentials are never shown in chat or stored in the repository.
- </FieldDescription>
- </Field>
- ))}
- {(error || connection.error) && (
- <p role="alert" className="text-sm text-destructive">
- {error || String(connection.error)}
- </p>
- )}
- </DialogContentArea>
- <DialogFooter>
- <Button variant="outline" onClick={onBack} disabled={pending}>
- Back
- </Button>
- {(!localAuthentication || error || connection.data?.status === 'failed') && (
- <Button
- disabled={
- busy ||
- (localAuthentication
- ? localSignIn.isError || localSignIn.data?.status !== 'ready'
- : !credential.trim())
- }
- onClick={() => void run(false)}
- >
- {busy
- ? 'Checking connection…'
- : connection.data?.status === 'failed' || localAuthentication
- ? 'Retry connection'
- : 'Save credential'}
- </Button>
- )}
- <Button
- disabled={
- busy ||
- !connection.data ||
- !['connected', 'configured'].includes(connection.data.status)
- }
- onClick={onDone}
- >
- {continueLabel}
- </Button>
- </DialogFooter>
- </>
- );
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-providers-step.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-providers-step.tsx
deleted file mode 100644
index 0b3db0709..000000000
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/managed-providers-step.tsx
+++ /dev/null
@@ -1,79 +0,0 @@
-import { AgentIcon } from '@renderer/lib/components/agent-icon';
-import { Button } from '@renderer/lib/ui/button';
-import { Checkbox } from '@renderer/lib/ui/checkbox';
-import {
- DialogContentArea,
- DialogFooter,
- DialogHeader,
- DialogTitle,
-} from '@renderer/lib/ui/dialog';
-import {
- AGENT_PROVIDERS,
- type AgentProviderId,
-} from '@shared/core/providers/agent-provider-registry';
-
-export function ManagedProvidersStep({
- selected,
- onSelectionChange,
- onContinue,
- onSkip,
-}: {
- selected: AgentProviderId[];
- onSelectionChange: (providers: AgentProviderId[]) => void;
- onContinue: () => void;
- onSkip: () => void;
-}) {
- return (
- <>
- <DialogHeader>
- <DialogTitle>Choose agents to connect</DialogTitle>
- </DialogHeader>
- <DialogContentArea className="space-y-4 pt-0">
- <p className="text-sm text-foreground-muted">
- Choose the providers you want to use with Switch. You’ll add credentials next.
- </p>
- <div role="group" aria-label="Agent providers" className="space-y-2">
- {AGENT_PROVIDERS.map((provider) => {
- const available = true;
- return (
- <label
- key={provider.id}
- className={`flex items-center gap-3 rounded-lg border border-border p-3 ${available ? 'cursor-pointer hover:bg-background-tertiary-2' : 'cursor-not-allowed opacity-60'}`}
- >
- <Checkbox
- checked={selected.includes(provider.id)}
- disabled={!available}
- onCheckedChange={(checked) => {
- if (!available) return;
- onSelectionChange(
- checked
- ? [...selected, provider.id]
- : selected.filter((id) => id !== provider.id)
- );
- }}
- />
- <AgentIcon id={provider.id} size={24} />
- <span className="flex-1 text-sm font-medium">{provider.name}</span>
- <span className="text-xs text-foreground-muted">
- {available ? 'Available' : 'Coming soon'}
- </span>
- </label>
- );
- })}
- </div>
- <p className="text-xs text-foreground-muted">
- Each connection is checked before it is marked verified. Selecting a provider does not
- start an agent.
- </p>
- </DialogContentArea>
- <DialogFooter>
- <Button variant="outline" onClick={onSkip}>
- Set up later
- </Button>
- <Button disabled={selected.length === 0} onClick={onContinue}>
- Continue
- </Button>
- </DialogFooter>
- </>
- );
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-agents-view.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-agents-view.tsx
index c0f4689c0..afc5c1c84 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-agents-view.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-agents-view.tsx
@@ -1,29 +1,9 @@
-import { useQueryClient } from '@tanstack/react-query';
-import {
- Bot,
- CircleStop,
- ExternalLink,
- MoreVertical,
- Pencil,
- Plug,
- Plus,
- RotateCcw,
- Trash2,
-} from 'lucide-react';
+import { Bot, ExternalLink, MoreVertical, Plug, Plus, RotateCcw, Trash2 } from 'lucide-react';
import { observer } from 'mobx-react-lite';
-import { useEffect, useState } from 'react';
+import { useEffect } from 'react';
import type { GuardResult, ViewDefinition } from '@renderer/app/view-registry';
import { CloudMachineCard } from '@renderer/features/cloud-agents/cloud-machine-card';
-import {
- cloudOperationAttempts,
- startAttemptKey,
-} from '@renderer/features/cloud-agents/cloud-operation-attempts';
-import { CloudStartAttemptStatus } from '@renderer/features/cloud-agents/cloud-start-attempt-status';
-import {
- useCloudAgentSessions,
- useCloudAgents,
- useCloudMachines,
-} from '@renderer/features/cloud-agents/use-cloud-agents';
+import { useCloudMachines } from '@renderer/features/cloud-agents/use-cloud-agents';
import { useConfirmDeleteAgent } from '@renderer/features/locations/hooks/use-confirm-delete-agent';
import { agentsStore } from '@renderer/features/locations/stores/agents-store';
import { getLocationStore } from '@renderer/features/locations/stores/location-selectors';
@@ -37,16 +17,14 @@ import {
} from '@renderer/features/managed-agents/use-managed-agents';
import { refreshSidebarRoomState } from '@renderer/features/sidebar/sidebar-tree-data';
import { AgentConnectionIndicator } from '@renderer/features/switch-rooms/connection-health';
-import { workspacesStore } from '@renderer/features/workspaces/workspaces-store';
import { AgentAvatar } from '@renderer/lib/components/agent-avatar';
import { failureText } from '@renderer/lib/errors/describe-failure';
import { resetAgentErrorText } from '@renderer/lib/errors/reset-agent-error';
-import { toast, useToast } from '@renderer/lib/hooks/use-toast';
+import { useToast } from '@renderer/lib/hooks/use-toast';
import { rpc } from '@renderer/lib/ipc';
import { useNavigate, useParams } from '@renderer/lib/layout/navigation-provider';
import { useShowModal } from '@renderer/lib/modal/modal-provider';
import { useAgentIconUrl } from '@renderer/lib/stores/use-workspace-agents';
-import { Badge } from '@renderer/lib/ui/badge';
import { Button } from '@renderer/lib/ui/button';
import {
DropdownMenu,
@@ -56,14 +34,8 @@ import {
DropdownMenuTrigger,
} from '@renderer/lib/ui/dropdown-menu';
import type { Agent } from '@shared/core/agents/agents';
-import {
- type CloudAgent,
- cloudAgentPhase,
- cloudMachineReady,
-} from '@shared/core/cloud-agents/cloud-agents';
import type { ManagedAgentView } from '@shared/core/managed-agents/managed-agents';
import { providerDisplayName } from '@shared/core/providers/agent-provider-registry';
-import { RpcError } from '@shared/lib/ipc/rpc-error';
import { ServerPage } from './server-page';
import { ServerSectionTitlebar } from './server-section-titlebar';
import { switchRoomsStore } from './switch-rooms-store';
@@ -89,7 +61,6 @@ const ServerAgentsPanel = observer(function ServerAgentsPanel() {
void refreshSidebarRoomState(false);
}, [serverId]);
- const cloud = useCloudAgents(serverId);
const machines = useCloudMachines(serverId);
const managed = useManagedAgents(serverId);
const agents = withoutManaged(agentsStore.agentsOnServer(serverId), managed.data);
@@ -100,7 +71,7 @@ const ServerAgentsPanel = observer(function ServerAgentsPanel() {
title="Your Agents"
description={`Agents on ${server?.name ?? 'this server'}. Add one, set how it is addressed, and start sessions.`}
action={
- cloud.data && (
+ machines.data && (
<Button variant="outline" size="sm" onClick={() => showConnectionsModal({ serverId })}>
<Plug className="size-4" />
Connections
@@ -141,281 +112,16 @@ const ServerAgentsPanel = observer(function ServerAgentsPanel() {
{managed.data?.map((agent) => (
<ManagedAgentCard key={agent.agentId} agent={agent} />
))}
- {cloud.data
- ?.filter((listed) => listed.launch.state !== 'deleted')
- .map((listed) => (
- <CloudAgentCard key={listed.key} listed={listed} serverId={serverId} />
- ))}
</div>
{managed.error && (
<p role="alert" className="mt-3 text-sm text-destructive">
{failureText(managed.error, 'Could not load managed agents.')}
</p>
)}
- {cloud.error && (
- <p role="alert" className="mt-3 text-sm text-destructive">
- {failureText(cloud.error, 'Could not load cloud agents.')}
- </p>
- )}
</ServerPage>
);
});
-/** What a launch in error means to its owner, by the server's `error_code`. */
-const LAUNCH_ERRORS: Record<string, string> = {
- agent_crashed:
- 'The agent keeps crashing. Retry it. If it crashes again, contact your server administrator.',
- identity_failed: 'Switch could not register the agent. Retry.',
- worker_needs_attention: 'The cloud worker needs attention. Contact your server administrator.',
- agent_key_missing:
- 'Switch lost this agent’s credential, so its machine cannot run it. Remove the agent and create it again.',
- agent_identity_missing: 'Switch lost this agent’s identity. Retry to register it again.',
- worker_attach_timeout:
- 'The agent started but did not connect to Switch. Check your provider connection, then retry.',
- agent_stop_timeout: 'The agent did not stop in time. Retry to start it again, or remove it.',
-};
-
-function launchErrorText(code: string | null): string {
- const text =
- code !== null && Object.hasOwn(LAUNCH_ERRORS, code) ? LAUNCH_ERRORS[code] : undefined;
- return (
- text ??
- 'The cloud worker could not start. Check your provider and GitHub connections, then retry. If it still fails, contact your server administrator.'
- );
-}
-
-/** Launch errors a retry cannot clear. */
-const NOT_RETRYABLE = new Set(['worker_needs_attention', 'agent_key_missing']);
-
-function lifecycleFailureText(error: unknown): string {
- const code =
- error instanceof RpcError && error.code === 'GatewayError'
- ? error.stringField('code')
- : undefined;
- if (code === 'machine_stopped')
- return 'The owner stopped the cloud machine. Start the machine, then try again.';
- return failureText(error, 'Cloud operation failed.');
-}
-
-const CloudAgentCard = observer(function CloudAgentCard({
- listed,
- serverId,
-}: {
- listed: CloudAgent;
- serverId: string;
-}) {
- const launch = listed.launch;
- const [pending, setPending] = useState(false);
- const [actionError, setActionError] = useState<string | null>(null);
- useEffect(() => setActionError(null), [launch.revision, launch.state]);
- const [confirmRemove, setConfirmRemove] = useState(false);
- const queryClient = useQueryClient();
- const { navigate } = useNavigate();
- const agentKey = listed.key;
- const attempt = cloudOperationAttempts.get(startAttemptKey(agentKey));
- // Asked only while a start is unconfirmed, to tell whether its session exists.
- const withSessions = useCloudAgentSessions(listed, attempt?.status === 'unknown');
- const openSession = (sessionId: string) =>
- navigate('cloudSession', {
- agentKey,
- sessionId,
- name: `${launch.name} · Session ${sessionId.slice(0, 8)}`,
- });
- const newSession = async () => {
- setActionError(null);
- const result = await cloudOperationAttempts.run(
- startAttemptKey(agentKey),
- agentKey,
- 'start',
- null
- );
- if (!result) return;
- if (result.outcome.state === 'applied') openSession(result.sessionId);
- else if (result.outcome.state === 'failed') setActionError(result.outcome.message);
- else void queryClient.invalidateQueries({ queryKey: ['cloud-agents'] });
- };
- const run = async (action: 'stop' | 'start' | 'restart' | 'remove' | 'retry') => {
- setPending(true);
- setActionError(null);
- try {
- const result = await rpc.switchServers.cloudLifecycle(
- serverId,
- launch.request_id,
- action,
- launch.revision
- );
- if (result.access_warning)
- toast({ title: 'GitHub access cleanup is pending', description: result.access_warning });
- await queryClient.invalidateQueries({ queryKey: ['cloud-agents'] });
- setConfirmRemove(false);
- } catch (error) {
- setActionError(lifecycleFailureText(error));
- void queryClient.invalidateQueries({ queryKey: ['cloud-agents'] });
- } finally {
- setPending(false);
- }
- };
- const editAgent = useShowModal('editCloudAgentModal');
- const iconUrl = useAgentIconUrl(workspacesStore.idOnServerInScope(serverId), launch.agent_id);
- const phase = launch.desired_state === 'deleted' ? null : cloudAgentPhase(launch, listed.machine);
- const stateLabel =
- phase === 'sleeping'
- ? 'Sleeping'
- : phase === 'machine_stopped'
- ? 'Machine stopped'
- : phase === 'machine_error'
- ? 'Machine error'
- : phase === 'waking'
- ? cloudMachineReady(listed.machine)
- ? 'Starting…'
- : 'Waking…'
- : launch.desired_state === 'stopped'
- ? launch.state === 'stopping'
- ? 'Stopping…'
- : 'Stopped'
- : {
- queued: 'Queued',
- provisioning: 'Starting…',
- ready: 'Ready',
- error: 'Needs attention',
- stopping: 'Stopping…',
- stopped: 'Stopped',
- deleting: 'Removing…',
- deleted: 'Removed',
- }[launch.state];
- const usable = launch.agent_id !== null && launch.state === 'ready' && phase === null;
- const machineDown = listed.machine
- ? listed.machine.sleeping || listed.machine.desired_state === 'stopped'
- : launch.sleeping;
- const stoppable =
- !machineDown &&
- launch.desired_state === 'running' &&
- launch.agent_id !== null &&
- ['ready', 'provisioning', 'queued', 'error'].includes(launch.state);
- const crashed = launch.process_state === 'crashed' || launch.error_code === 'agent_crashed';
- return (
- <div className="group relative flex min-h-[184px] flex-col rounded-[11px] bg-[var(--surface-2)] p-[14px]">
- <div className="absolute top-2 right-2 flex items-center opacity-0 transition-opacity group-hover:opacity-100 focus-within:opacity-100">
- <DropdownMenu>
- <DropdownMenuTrigger
- render={
- <Button variant="ghost" size="icon-xs" aria-label={`${launch.name} actions`}>
- <MoreVertical className="size-3" />
- </Button>
- }
- />
- <DropdownMenuContent align="end">
- <DropdownMenuItem
- disabled={launch.agent_id === null || launch.desired_state === 'deleted'}
- onClick={() => editAgent({ serverId, launch })}
- >
- <Pencil className="size-4" />
- Edit agent…
- </DropdownMenuItem>
- {stoppable && (
- <DropdownMenuItem disabled={pending} onClick={() => void run('stop')}>
- <CircleStop className="size-4" />
- <span className="flex flex-col">
- <span>Stop agent</span>
- <span className="text-xs text-foreground-muted">
- Stops replies and frees the machine
- </span>
- </span>
- </DropdownMenuItem>
- )}
- </DropdownMenuContent>
- </DropdownMenu>
- </div>
- <div className="flex flex-1 items-center justify-center py-3">
- <AgentAvatar name={launch.name} iconUrl={iconUrl} size={66} />
- </div>
- <div className="truncate text-sm font-medium">{launch.name}</div>
- <div className="text-xs text-foreground-muted">
- {providerDisplayName(launch.provider)} · Cloud · {stateLabel}
- </div>
- {(launch.oom_kills > 0 || crashed) && (
- <div className="mt-1 flex flex-wrap gap-1">
- {launch.oom_kills > 0 && (
- <Badge variant="secondary">
- Restarted after running out of memory {launch.oom_kills}×
- </Badge>
- )}
- {crashed && <Badge variant="destructive">Crashed</Badge>}
- </div>
- )}
- {launch.error && (
- <p role="alert" className="mt-2 text-xs text-destructive">
- {launchErrorText(launch.error_code)}
- </p>
- )}
- {actionError && (
- <p role="alert" className="mt-2 text-xs text-destructive">
- {actionError}
- </p>
- )}
- <CloudStartAttemptStatus
- agentKey={agentKey}
- sessions={(withSessions.sessions ?? []).filter((session) => !session.retired)}
- onOpen={openSession}
- onCheckAgain={() => void newSession()}
- className="mt-2 flex-wrap"
- />
- <div className="mt-2 flex flex-wrap gap-1">
- {usable && (
- <>
- <Button
- variant="outline"
- size="sm"
- disabled={pending || attempt?.status === 'pending'}
- aria-busy={attempt?.status === 'pending'}
- onClick={() => void newSession()}
- >
- {attempt?.status === 'pending' ? 'Starting session…' : 'New session'}
- </Button>
- <Button
- variant="ghost"
- size="sm"
- disabled={pending}
- onClick={() => void run('restart')}
- >
- Restart
- </Button>
- </>
- )}
- {launch.state === 'error' && !NOT_RETRYABLE.has(launch.error_code ?? '') && (
- <Button variant="outline" size="sm" disabled={pending} onClick={() => void run('retry')}>
- Retry
- </Button>
- )}
- {launch.desired_state === 'stopped' && (
- <Button variant="outline" size="sm" disabled={pending} onClick={() => void run('start')}>
- Start agent
- </Button>
- )}
- <Button variant="ghost" size="sm" disabled={pending} onClick={() => setConfirmRemove(true)}>
- Remove
- </Button>
- </div>
- {confirmRemove && (
- <div className="mt-2 text-xs">
- <p>
- Remove this agent and its sessions from Switch? Its working copy on the cloud machine is
- deleted right away, with any uncommitted changes. If it is the last agent on your
- machine, the machine shuts down and its disk is kept until the date shown on the machine
- card.
- </p>
- <Button size="sm" disabled={pending} onClick={() => void run('remove')}>
- Remove agent
- </Button>
- <Button variant="ghost" size="sm" onClick={() => setConfirmRemove(false)}>
- Cancel
- </Button>
- </div>
- )}
- </div>
- );
-});
-
/** A managed agent Console has no row for, as its server lists it. */
function ManagedAgentCard({ agent }: { agent: ManagedAgentView }) {
const { navigate } = useNavigate();
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-sign-in.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-sign-in.tsx
index 9674aad88..dfbee658a 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-sign-in.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-sign-in.tsx
@@ -8,8 +8,8 @@ import { Button } from '@renderer/lib/ui/button';
import { Input } from '@renderer/lib/ui/input';
import { Label } from '@renderer/lib/ui/label';
import type { SignupMachine, SwitchAuthConfig } from '@shared/core/switch-servers/switch-servers';
+import { loadSwitchCloudOrigin, managedCloudServerId } from './switch-cloud-origin';
import { switchServersStore } from './switch-servers-store';
-import { loadSwitchCloudOrigin, managedCloudServerId } from './use-cloud-launches';
/**
* Signing in to a Switch server, wherever that is asked for.
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-stat-tiles.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-stat-tiles.tsx
index ab38fa41b..1f1c299f8 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-stat-tiles.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/server-stat-tiles.tsx
@@ -3,10 +3,8 @@ import { observer } from 'mobx-react-lite';
import { useEffect } from 'react';
import { agentsStore } from '@renderer/features/locations/stores/agents-store';
import { workspacesStore } from '@renderer/features/workspaces/workspaces-store';
-import { failureText } from '@renderer/lib/errors/describe-failure';
import { rpc } from '@renderer/lib/ipc';
import { switchRoomsStore } from './switch-rooms-store';
-import { useCloudLaunches } from './use-cloud-launches';
/**
* How much of this server Switch Console is holding: the agents onboarded
@@ -21,7 +19,6 @@ export const ServerStatTiles = observer(function ServerStatTiles({
}: {
serverId: string;
}) {
- const cloud = useCloudLaunches(serverId);
const workspaceId = workspacesStore.idOnServerInScope(serverId);
// Shares the key every other bridge reader uses, so the list is already in
@@ -42,15 +39,7 @@ export const ServerStatTiles = observer(function ServerStatTiles({
<div className="grid grid-cols-3 gap-3">
<StatTile
label="Your Agents"
- // A server without cloud agents answers with none, which counts as zero;
- // a failed ask leaves the total unknown rather than reporting the local
- // agents alone as all of them.
- value={
- agentsStore.loaded && cloud.isSuccess && cloud.data
- ? agentsStore.agentsOnServer(serverId).length + cloud.data.length
- : null
- }
- failure={cloud.error ? failureText(cloud.error, 'Could not count cloud agents.') : null}
+ value={agentsStore.loaded ? agentsStore.agentsOnServer(serverId).length : null}
/>
<StatTile
label="Your Rooms"
@@ -59,9 +48,8 @@ export const ServerStatTiles = observer(function ServerStatTiles({
? null
: switchRoomsStore.readableRoomsInWorkspace(workspaceId).length
}
- failure={null}
/>
- <StatTile label="Messaging apps" value={bridgesQuery.data?.length ?? null} failure={null} />
+ <StatTile label="Messaging apps" value={bridgesQuery.data?.length ?? null} />
</div>
);
});
@@ -69,24 +57,11 @@ export const ServerStatTiles = observer(function ServerStatTiles({
/** A number that is not known yet reads as an em dash rather than as zero:
* "no messaging apps" is a fact worth acting on and must not be faked while
* the list is still loading. */
-function StatTile({
- label,
- value,
- failure,
-}: {
- label: string;
- value: number | null;
- failure: string | null;
-}) {
+function StatTile({ label, value }: { label: string; value: number | null }) {
return (
<div className="bg-card rounded-lg border border-border px-4 py-3">
<p className="text-xs text-foreground-muted">{label}</p>
<p className="mt-1 text-2xl text-foreground">{value ?? '—'}</p>
- {failure && (
- <p role="alert" className="mt-1 text-xs text-destructive">
- {failure}
- </p>
- )}
</div>
);
}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/use-cloud-launches.ts b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/switch-cloud-origin.ts
similarity index 77%
rename from console/apps/switch-console-desktop/src/renderer/features/switch-servers/use-cloud-launches.ts
rename to console/apps/switch-console-desktop/src/renderer/features/switch-servers/switch-cloud-origin.ts
index c023b410b..8e0bb909d 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/use-cloud-launches.ts
+++ b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/switch-cloud-origin.ts
@@ -1,5 +1,4 @@
import { observable, runInAction } from 'mobx';
-import { useCloudAgents } from '@renderer/features/cloud-agents/use-cloud-agents';
import { rpc } from '@renderer/lib/ipc';
import { log } from '@renderer/utils/logger';
import { type SwitchServer, urlOrigin } from '@shared/core/switch-servers/switch-servers';
@@ -46,17 +45,3 @@ export function managedCloudServerId(): string | null {
if (switchCloudOrigin.get() === null) return null;
return switchServersStore.servers.find(isSwitchCloudServer)?.id ?? null;
}
-
-/**
- * The server's cloud launches, read from the same query as the sidebar's cloud
- * agents so a lifecycle change refreshes both. Empty when the server has none.
- */
-export function useCloudLaunches(serverId: string | null) {
- const agents = useCloudAgents(serverId);
- return {
- data: agents.data === undefined ? undefined : (agents.data ?? []).map((agent) => agent.launch),
- isSuccess: agents.isSuccess,
- isLoading: agents.isLoading,
- error: agents.error,
- };
-}
diff --git a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/view.tsx b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/view.tsx
index 9f256d71c..0962c06da 100644
--- a/console/apps/switch-console-desktop/src/renderer/features/switch-servers/view.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/features/switch-servers/view.tsx
@@ -60,11 +60,11 @@ import {
import { ServerStatTiles } from './server-stat-tiles';
import { useSharedActionConfirm } from './shared-action-confirm';
import { SharedConsolesSection } from './shared-consoles-section';
+import { loadSwitchCloudOrigin, managedCloudServerId } from './switch-cloud-origin';
import { isSwitchCloudServer } from './switch-cloud-store';
import { switchRoomsStore } from './switch-rooms-store';
import { switchServersStore } from './switch-servers-store';
import { TelemetryConsentNotice } from './TelemetryConsentNotice';
-import { loadSwitchCloudOrigin, managedCloudServerId } from './use-cloud-launches';
import { myIdentitiesQueryKey } from './use-my-identities';
import { VersionDriftNotice } from './VersionDriftNotice';
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/advanced-config-sections.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/advanced-config-sections.test.tsx
index d03dd74ac..babe0b186 100644
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/advanced-config-sections.test.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/tests/browser/advanced-config-sections.test.tsx
@@ -76,7 +76,6 @@ async function sectionCount(): Promise<number> {
root!.render(
<QueryClientProvider client={client}>
<AgentAdvancedConfig
- cloud={false}
sshHost={null}
dir=""
providerId={'claude' as never}
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-agent-list.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-agent-list.test.tsx
deleted file mode 100644
index 261d1fdd8..000000000
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-agent-list.test.tsx
+++ /dev/null
@@ -1,411 +0,0 @@
-/**
- * A new cloud session whose start was never confirmed is not shown as a
- * failure: once the session appears, the row offers to open it, and until
- * then it asks again for the same session rather than a new one. A worker is
- * asked for its sessions only while its row is expanded.
- */
-import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
-import { act } from 'react';
-import { createRoot, type Root } from 'react-dom/client';
-import { afterEach, beforeEach, expect, it, vi } from 'vitest';
-import type { CloudAgent, CloudLaunch, CloudMachine } from '@shared/core/cloud-agents/cloud-agents';
-
-const sdkHost = vi.hoisted(() => ({
- cloudAgents: vi.fn(),
- cloudSessions: vi.fn(),
- cloudSessionOperation: vi.fn(),
- cloudWake: vi.fn(),
-}));
-const expandedCloudGroups = await vi.hoisted(async () => {
- const { observable } = await import('mobx');
- return observable.set<string>();
-});
-const navigate = vi.hoisted(() => vi.fn());
-
-vi.hoisted(() => {
- window.electronAPI ??= {
- invoke: () => Promise.resolve(undefined),
- eventOn: () => () => {},
- eventSend: () => {},
- } as unknown as typeof window.electronAPI;
-});
-
-vi.mock('@renderer/lib/ipc', () => ({
- events: { on: () => () => {} },
- rpc: { sdkHost, switchServers: { cloudMachineLifecycle: vi.fn() } },
-}));
-
-vi.mock('@renderer/features/switch-servers/switch-rooms-store', () => ({
- switchRoomsStore: { workspacesNotSignedIn: [], roomNameById: () => null },
-}));
-
-vi.mock('@renderer/features/switch-servers/switch-servers-store', () => ({
- switchServersStore: { activeServerId: 'server', statusFor: () => null },
-}));
-
-vi.mock('@renderer/lib/layout/navigation-provider', () => ({
- useNavigate: () => ({ navigate }),
- useParams: () => ({ params: {} }),
-}));
-
-vi.mock('@renderer/lib/layout/workspace-slots', () => ({
- useWorkspaceSlots: () => ({ currentView: 'home' }),
-}));
-
-vi.mock('@renderer/lib/stores/app-state', () => ({
- sidebarStore: {
- isCloudGroupExpanded: (key: string) => expandedCloudGroups.has(key),
- toggleCloudGroupExpanded: (key: string) =>
- expandedCloudGroups.has(key) ? expandedCloudGroups.delete(key) : expandedCloudGroups.add(key),
- hideProviderMark: true,
- },
-}));
-
-import { CloudAgentList } from '@renderer/features/cloud-agents/cloud-agent-list';
-import {
- cloudOperationAttempts,
- startAttemptKey,
-} from '@renderer/features/cloud-agents/cloud-operation-attempts';
-
-const agentKey = 'cloud:server:launch';
-
-function agent(key = agentKey, name = 'reviewer'): CloudAgent {
- return {
- key,
- launch: {
- request_id: '00000000-0000-4000-8000-000000000001',
- name,
- provider: 'claude',
- state: 'ready',
- desired_state: 'running',
- revision: 4,
- agent_id: 'agent',
- error: null,
- error_code: null,
- sleeping: false,
- machine_id: null,
- process_state: null,
- process_restarts: 0,
- oom_kills: 0,
- },
- machine: null,
- sessions: null,
- problem: null,
- };
-}
-
-function sessions(sessionIds: string[]) {
- return {
- sessions: sessionIds.map(
- (sessionId) => ({ sessionId, status: 'ready', connectivity: 'online' }) as never
- ),
- problem: null,
- };
-}
-
-let container: HTMLDivElement | null = null;
-let root: Root | null = null;
-
-beforeEach(() => {
- navigate.mockReset();
- sdkHost.cloudSessionOperation.mockReset();
- sdkHost.cloudSessions.mockReset();
- sdkHost.cloudWake.mockReset();
- expandedCloudGroups.clear();
- cloudOperationAttempts.settle(startAttemptKey(agentKey));
-});
-
-afterEach(async () => {
- if (root) await act(async () => root!.unmount());
- container?.remove();
- container = null;
- root = null;
-});
-
-async function render(): Promise<HTMLDivElement> {
- container = document.createElement('div');
- document.body.appendChild(container);
- root = createRoot(container);
- const client = new QueryClient({ defaultOptions: { queries: { retry: false, gcTime: 0 } } });
- await act(async () =>
- root!.render(
- <QueryClientProvider client={client}>
- <CloudAgentList />
- </QueryClientProvider>
- )
- );
- await act(async () => await new Promise((resolve) => setTimeout(resolve, 20)));
- return container;
-}
-
-function button(el: HTMLElement, name: RegExp): HTMLButtonElement | undefined {
- return [...el.querySelectorAll('button')].find(
- (b) => name.test(b.textContent ?? '') || name.test(b.getAttribute('aria-label') ?? '')
- );
-}
-
-it('offers Open for an unconfirmed start whose session exists, not an error', async () => {
- let started = '';
- sdkHost.cloudSessionOperation.mockImplementation(async (_key: string, sessionId: string) => {
- started = sessionId;
- return { state: 'unknown', message: 'The server did not confirm the session start.' };
- });
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- sdkHost.cloudSessions.mockResolvedValue(sessions([]));
- const el = await render();
- await act(async () => button(el, /new session/i)!.click());
- expect(el.querySelector('[role="alert"]')).toBeNull();
- expect(button(el, /check again/i)).toBeDefined();
-
- sdkHost.cloudSessions.mockResolvedValue(sessions([started]));
- await act(async () => root!.unmount());
- const remounted = await render();
- const open = button(remounted, /^open$/i);
- expect(open).toBeDefined();
- await act(async () => open!.click());
- expect(navigate).toHaveBeenCalledWith(
- 'cloudSession',
- expect.objectContaining({ agentKey, sessionId: started })
- );
- expect(sdkHost.cloudSessionOperation).toHaveBeenCalledTimes(1);
-});
-
-it('asks again for the same session from Check again', async () => {
- sdkHost.cloudSessionOperation.mockResolvedValueOnce({ state: 'unknown', message: 'lost' });
- sdkHost.cloudSessionOperation.mockResolvedValueOnce({ state: 'applied' });
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- sdkHost.cloudSessions.mockResolvedValue(sessions([]));
- const el = await render();
- await act(async () => button(el, /new session/i)!.click());
- await act(async () => button(el, /check again/i)!.click());
- const [first, second] = sdkHost.cloudSessionOperation.mock.calls;
- expect(second).toEqual(first);
- expect(navigate).toHaveBeenCalledWith(
- 'cloudSession',
- expect.objectContaining({ sessionId: first[1] })
- );
-});
-
-it('asks no worker while its row is collapsed, and one when that row is expanded', async () => {
- const other = 'cloud:server:other';
- sdkHost.cloudAgents.mockResolvedValue([agent(), agent(other, 'writer')]);
- sdkHost.cloudSessions.mockResolvedValue(sessions(['s1']));
- const el = await render();
- expect(sdkHost.cloudAgents).toHaveBeenCalled();
- expect(sdkHost.cloudSessions).not.toHaveBeenCalled();
-
- const row = [...el.querySelectorAll('button[aria-expanded]')].find((b) =>
- b.textContent?.includes('writer')
- ) as HTMLButtonElement;
- await act(async () => row.click());
- await act(async () => await new Promise((resolve) => setTimeout(resolve, 20)));
- expect(sdkHost.cloudSessions).toHaveBeenCalledTimes(1);
- expect(sdkHost.cloudSessions).toHaveBeenCalledWith(other);
- expect(el.textContent).toContain('Session s1');
-});
-
-it('keeps a cloud row expanded after remount', async () => {
- const other = 'cloud:server:other';
- sdkHost.cloudAgents.mockResolvedValue([agent(), agent(other, 'writer')]);
- sdkHost.cloudSessions.mockResolvedValue(sessions(['s1']));
- const el = await render();
-
- const row = [...el.querySelectorAll('button[aria-expanded]')].find((b) =>
- b.textContent?.includes('writer')
- ) as HTMLButtonElement;
- await act(async () => row.click());
- await act(async () => await new Promise((resolve) => setTimeout(resolve, 20)));
-
- await act(async () => root!.unmount());
- const remounted = await render();
- await act(async () => await new Promise((resolve) => setTimeout(resolve, 20)));
- expect(sdkHost.cloudSessions).toHaveBeenCalled();
- expect(remounted.textContent).toContain('Session s1');
-});
-
-it('labels no session the next message restarts, but still a stopped or working one', async () => {
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- sdkHost.cloudSessions.mockResolvedValue({
- sessions: [
- { sessionId: 'down', status: 'ready', connectivity: 'offline' },
- { sessionId: 'done', status: 'stopped', connectivity: 'offline' },
- { sessionId: 'busy', status: 'running', connectivity: 'online' },
- ] as never,
- problem: null,
- });
- const el = await render();
- await act(async () => button(el, /reviewer/i)!.click());
- await act(async () => await new Promise((resolve) => setTimeout(resolve, 20)));
- expect(button(el, /Session down/)!.textContent).toBe('Session down');
- expect(button(el, /Session done/)!.textContent).toBe('Session donestopped');
- expect(button(el, /Session busy/)!.textContent).toBe('Session busyrunning');
- expect(el.textContent).not.toMatch(/offline/i);
-});
-
-function asleep(overrides: Partial<CloudAgent['launch']>, wakeAvailable: boolean): CloudAgent {
- return {
- ...agent(),
- launch: { ...agent().launch, ...overrides },
- problem: { code: 'worker_sleeping', message: 'The cloud machine is asleep.', wakeAvailable },
- };
-}
-
-it('wakes a sleeping agent’s machine from the list', async () => {
- sdkHost.cloudAgents.mockResolvedValue([asleep({}, true)]);
- sdkHost.cloudWake.mockResolvedValue(undefined);
- const el = await render();
- expect(el.textContent).not.toContain('Send a message to wake it.');
- await act(async () => button(el, /^wake$/i)!.click());
- expect(sdkHost.cloudWake).toHaveBeenCalledWith(agentKey);
-});
-
-it('says once that the machine its agents share is asleep, not under each agent', async () => {
- const other = 'cloud:server:other';
- sdkHost.cloudAgents.mockResolvedValue([
- { ...asleep({ desired_state: 'stopped', state: 'stopped' }, false), key: other },
- asleep({}, true),
- ]);
- sdkHost.cloudWake.mockResolvedValue(undefined);
- expandedCloudGroups.add(`cloud:${agentKey}`);
- expandedCloudGroups.add(`cloud:${other}`);
- const el = await render();
- expect(
- [...el.querySelectorAll('button')].filter((b) => /^wake$/i.test(b.textContent ?? ''))
- ).toHaveLength(1);
- await act(async () => button(el, /^wake$/i)!.click());
- expect(sdkHost.cloudWake).toHaveBeenCalledWith(agentKey);
-});
-
-it('still says under the agent why only that agent cannot be asked', async () => {
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...onMachine({}, { process_state: 'crashed', error_code: 'agent_crashed' }),
- problem: { code: 'agent_crashed', message: 'The agent crashed.', wakeAvailable: false },
- },
- ]);
- const el = await render();
- expect(el.textContent).not.toContain('(agent_crashed)');
- expandedCloudGroups.add(`cloud:${agentKey}`);
- await act(async () => await new Promise((resolve) => setTimeout(resolve, 20)));
- expect(el.textContent).toContain('crashed');
- expect(el.textContent).not.toContain('(agent_crashed)');
-});
-
-it('offers no wake for a stopped agent on a sleeping machine', async () => {
- sdkHost.cloudAgents.mockResolvedValue([
- asleep({ desired_state: 'stopped', state: 'stopped' }, false),
- ]);
- expandedCloudGroups.add(`cloud:${agentKey}`);
- const el = await render();
- expect(el.textContent).toContain('asleep');
- expect(el.textContent).not.toContain('wake it');
- expect(button(el, /^wake$/i)).toBeUndefined();
-});
-
-function onMachine(machine: Partial<CloudMachine>, launch: Partial<CloudLaunch> = {}): CloudAgent {
- return {
- ...agent(),
- launch: { ...agent().launch, machine_id: 'machine', ...launch },
- machine: {
- machine_id: 'machine',
- state: 'ready',
- desired_state: 'running',
- stop_reason: null,
- sleeping: false,
- revision: 2,
- instance_type: null,
- error: null,
- error_code: null,
- retain_until: null,
- heartbeat_at: null,
- disk: null,
- memory: null,
- agents: [],
- ...machine,
- },
- };
-}
-
-it.each([
- [
- 'stopped by its owner',
- {
- ...onMachine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'owner' }),
- problem: {
- code: 'machine_stopped',
- message: 'The owner stopped the cloud machine.',
- wakeAvailable: false,
- },
- },
- ],
- [
- 'asleep',
- {
- ...onMachine({
- state: 'stopped',
- desired_state: 'stopped',
- stop_reason: 'idle',
- sleeping: true,
- }),
- problem: {
- code: 'worker_sleeping',
- message: 'The cloud machine is asleep.',
- wakeAvailable: true,
- },
- },
- ],
-])(
- 'lists the stored sessions on a cold start while the machine is %s, with nothing cached',
- async (_name, down) => {
- sdkHost.cloudAgents.mockResolvedValue([
- { ...down, sessions: sessions(['b4105d35-0000']).sessions },
- ]);
- expandedCloudGroups.add(`cloud:${agentKey}`);
- const el = await render();
- expect(sdkHost.cloudSessions).not.toHaveBeenCalled();
- const session = button(el, /Session b4105d35/);
- expect(session).toBeDefined();
- await act(async () => session!.click());
- expect(navigate).toHaveBeenCalledWith(
- 'cloudSession',
- expect.objectContaining({ agentKey, sessionId: 'b4105d35-0000' })
- );
- }
-);
-
-it('says the agent is starting when only its launch starts on a running machine', async () => {
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...onMachine({}, { state: 'provisioning', process_state: 'starting' }),
- problem: {
- code: 'worker_waking',
- message: 'The cloud machine is starting.',
- wakeAvailable: false,
- },
- },
- ]);
- expandedCloudGroups.add(`cloud:${agentKey}`);
- const el = await render();
- expect(el.textContent).toContain('starting…');
- expect(el.textContent).toContain('The agent is starting.');
- expect(el.textContent).not.toContain('waking…');
- expect(el.textContent).not.toContain('The cloud machine is starting.');
-});
-
-it('says the machine is starting while the machine itself wakes', async () => {
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...onMachine({ state: 'provisioning' }),
- problem: {
- code: 'worker_waking',
- message: 'The cloud machine is starting.',
- wakeAvailable: false,
- },
- },
- ]);
- expandedCloudGroups.add(`cloud:${agentKey}`);
- const el = await render();
- expect(el.textContent).toContain('waking…');
- expect(el.textContent).toContain('The cloud machine is starting.');
- expect(el.textContent).not.toContain('The agent is starting.');
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-agents-support.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-agents-support.test.tsx
deleted file mode 100644
index b45930ee4..000000000
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-agents-support.test.tsx
+++ /dev/null
@@ -1,141 +0,0 @@
-/**
- * A server without cloud agents (its Core has no launch list) shows no error
- * and is not polled; it is asked again once its session or version changes.
- * A server with an empty list keeps being polled, and a failure still shows.
- */
-import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
-import { act } from 'react';
-import { createRoot, type Root } from 'react-dom/client';
-import { afterEach, beforeEach, expect, it, vi } from 'vitest';
-
-const sdkHost = vi.hoisted(() => ({ cloudAgents: vi.fn() }));
-const status = await vi.hoisted(async () => {
- const { observable } = await import('mobx');
- return observable.box<{ user: { id: string; server: { version: string } } | null }>({
- user: { id: 'user', server: { version: '0.28.0' } },
- });
-});
-
-vi.mock('@renderer/lib/ipc', () => ({
- events: { on: () => () => {} },
- rpc: { sdkHost },
-}));
-
-vi.mock('@renderer/features/switch-servers/switch-rooms-store', () => ({
- switchRoomsStore: { workspacesNotSignedIn: [], roomNameById: () => null },
-}));
-
-vi.mock('@renderer/features/switch-servers/switch-servers-store', () => ({
- switchServersStore: {
- activeServerId: 'server',
- statusFor: () => ({ serverId: 'server', connected: true, ...status.get() }),
- },
-}));
-
-vi.mock('@renderer/lib/layout/navigation-provider', () => ({
- useNavigate: () => ({ navigate: vi.fn() }),
- useParams: () => ({ params: {} }),
-}));
-
-vi.mock('@renderer/lib/layout/workspace-slots', () => ({
- useWorkspaceSlots: () => ({ currentView: 'home' }),
-}));
-
-vi.mock('@renderer/lib/stores/app-state', () => ({
- sidebarStore: { isGroupExpanded: () => false, toggleGroupExpanded: () => {} },
-}));
-
-import { runInAction } from 'mobx';
-import { CloudAgentList } from '@renderer/features/cloud-agents/cloud-agent-list';
-
-let container: HTMLDivElement | null = null;
-let root: Root | null = null;
-
-beforeEach(() => {
- vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'setInterval', 'clearInterval'] });
- sdkHost.cloudAgents.mockReset();
- runInAction(() => status.set({ user: { id: 'user', server: { version: '0.28.0' } } }));
-});
-
-afterEach(async () => {
- if (root) await act(async () => root!.unmount());
- container?.remove();
- container = null;
- root = null;
- vi.useRealTimers();
-});
-
-async function settle(ms = 0): Promise<void> {
- await act(async () => {
- await vi.advanceTimersByTimeAsync(ms);
- });
-}
-
-async function render(): Promise<HTMLDivElement> {
- container = document.createElement('div');
- document.body.appendChild(container);
- root = createRoot(container);
- const client = new QueryClient({ defaultOptions: { queries: { retry: false, gcTime: 0 } } });
- await act(async () =>
- root!.render(
- <QueryClientProvider client={client}>
- <CloudAgentList />
- </QueryClientProvider>
- )
- );
- await settle();
- return container;
-}
-
-it('shows nothing for a server without cloud agents, and stops asking it', async () => {
- sdkHost.cloudAgents.mockResolvedValue(null);
- const el = await render();
- expect(el.querySelector('[role="alert"]')).toBeNull();
- await settle(20_000);
- expect(sdkHost.cloudAgents).toHaveBeenCalledTimes(1);
-});
-
-it('keeps asking a server whose launch list is empty', async () => {
- sdkHost.cloudAgents.mockResolvedValue([]);
- const el = await render();
- expect(el.querySelector('[role="alert"]')).toBeNull();
- await settle(5_000);
- expect(sdkHost.cloudAgents).toHaveBeenCalledTimes(2);
-});
-
-it.each([
- ['Switch session expired — please sign in again.'],
- ['Could not reach http://switch.example.com: connection refused'],
- ['Switch gateway returned 500'],
-])('still shows a failure: %s', async (message) => {
- sdkHost.cloudAgents.mockRejectedValue(new Error(message));
- const el = await render();
- expect(el.querySelector('[role="alert"]')?.textContent).toContain(message);
-});
-
-it.each([
- ['an upgrade', { id: 'user', server: { version: '0.29.0' } }],
- ['a new sign-in', { id: 'someone-else', server: { version: '0.28.0' } }],
-])('asks a server without cloud agents again after %s', async (_name, user) => {
- sdkHost.cloudAgents.mockResolvedValue(null);
- await render();
- expect(sdkHost.cloudAgents).toHaveBeenCalledTimes(1);
-
- await act(async () => runInAction(() => status.set({ user })));
- await settle();
- expect(sdkHost.cloudAgents).toHaveBeenCalledTimes(2);
-});
-
-it('asks a server without cloud agents again once it reconnects', async () => {
- sdkHost.cloudAgents.mockResolvedValue(null);
- await render();
- await act(async () => runInAction(() => status.set({ user: null })));
- await settle();
- const whileDisconnected = sdkHost.cloudAgents.mock.calls.length;
-
- await act(async () =>
- runInAction(() => status.set({ user: { id: 'user', server: { version: '0.28.0' } } }))
- );
- await settle();
- expect(sdkHost.cloudAgents).toHaveBeenCalledTimes(whileDisconnected + 1);
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-problem.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-problem.test.tsx
deleted file mode 100644
index 7e3b779fe..000000000
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-problem.test.tsx
+++ /dev/null
@@ -1,143 +0,0 @@
-/**
- * A cloud agent whose worker cannot be asked says why: a sleeping machine reads
- * as asleep and says a message wakes it, or offers Wake where there is no
- * composer, and any other relay refusal is an alert.
- */
-import { act } from 'react';
-import { createRoot, type Root } from 'react-dom/client';
-import { afterEach, expect, it, vi } from 'vitest';
-import { CloudProblem } from '@renderer/features/cloud-agents/cloud-problem';
-import type { CloudProblemAction } from '@renderer/features/cloud-agents/use-cloud-agents';
-import type { CloudRelayProblem } from '@shared/core/cloud-agents/cloud-agents';
-
-let container: HTMLDivElement | null = null;
-let root: Root | null = null;
-
-afterEach(async () => {
- if (root) await act(async () => root!.unmount());
- container?.remove();
- container = null;
- root = null;
-});
-
-async function render(
- problem: CloudRelayProblem,
- action: CloudProblemAction | null = null,
- machineReady = false,
- compact = true
-): Promise<HTMLDivElement> {
- container = document.createElement('div');
- document.body.appendChild(container);
- root = createRoot(container);
- await act(async () =>
- root!.render(
- <CloudProblem
- problem={problem}
- machineReady={machineReady}
- compact={compact}
- action={action}
- />
- )
- );
- return container;
-}
-
-it('reads a sleeping machine as asleep and says a message wakes it', async () => {
- const el = await render({
- code: 'worker_sleeping',
- message: 'The cloud machine is asleep.',
- wakeAvailable: true,
- });
- const status = el.querySelector('[role="status"]');
- expect(status?.textContent).toContain('asleep');
- expect(status?.textContent).toContain('Send a message to wake it.');
- expect(el.querySelector('[role="alert"]')).toBeNull();
- expect(el.querySelector('button')).toBeNull();
-});
-
-it('shows any other refusal as an alert', async () => {
- const el = await render({
- code: 'worker_busy',
- message: 'The worker has too many requests in flight.',
- wakeAvailable: false,
- });
- const alert = el.querySelector('[role="alert"]');
- expect(alert?.textContent).toContain('busy');
- expect(alert?.textContent).not.toContain('wake');
- expect(el.querySelector('button')).toBeNull();
-});
-
-it('offers Wake in place of the hint where there is no composer', async () => {
- const run = vi.fn();
- const el = await render(
- { code: 'worker_sleeping', message: 'The cloud machine is asleep.', wakeAvailable: true },
- { label: 'Wake', pending: false, error: null, run }
- );
- expect(el.textContent).not.toContain('Send a message to wake it.');
- const wake = el.querySelector('button');
- expect(wake?.textContent).toBe('Wake');
- await act(async () => wake!.click());
- expect(run).toHaveBeenCalledOnce();
-});
-
-it('says nothing about waking a sleeping machine that a message would not wake', async () => {
- const el = await render({
- code: 'worker_sleeping',
- message: 'The cloud machine is asleep.',
- wakeAvailable: false,
- });
- expect(el.textContent).not.toContain('wake it');
- expect(el.querySelector('button')).toBeNull();
-});
-
-it('shows a machine in error with Retry and why a retry failed', async () => {
- const el = await render(
- { code: 'machine_error', message: 'The machine did not connect.', wakeAvailable: false },
- {
- label: 'Retry machine',
- pending: false,
- error: 'Could not retry the machine: boom',
- run: vi.fn(),
- }
- );
- expect(el.textContent).toContain('The cloud machine is in error.');
- expect(el.textContent).toContain('Could not retry the machine: boom');
- expect(el.querySelector('button')?.textContent).toBe('Retry machine');
-});
-
-const waking: CloudRelayProblem = {
- code: 'worker_waking',
- message: 'The cloud machine is starting.',
- wakeAvailable: false,
-};
-
-it('says only the agent is starting when its machine is already running', async () => {
- const el = await render(waking, null, true);
- expect(el.textContent).toContain('The agent is starting.');
- expect(el.textContent).not.toContain('cloud machine');
-});
-
-it('says the machine is starting while the machine itself wakes', async () => {
- const el = await render(waking, null, false);
- expect(el.textContent).toContain('The cloud machine is starting.');
- expect(el.textContent).not.toContain('The agent is starting.');
-});
-
-const unknownCode: CloudRelayProblem = {
- code: 'quota_exceeded',
- message: 'The account has used its cloud hours for this month.',
- wakeAvailable: false,
-};
-
-it('shows the server’s reason for a code it has no title for', async () => {
- const el = await render(unknownCode);
- expect(el.textContent).toContain('The account has used its cloud hours for this month.');
- expect(el.textContent).not.toContain('could not be reached');
-});
-
-it('shows the server’s reason beside the generic title outside the compact view', async () => {
- const el = await render(unknownCode, null, false, false);
- expect(el.textContent).toContain('The cloud worker could not be reached.');
- const detail = el.querySelector('[role="alert"] .text-foreground-muted');
- expect(detail?.textContent).toContain('The account has used its cloud hours for this month.');
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-session-header.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-session-header.test.tsx
deleted file mode 100644
index bc003b5c1..000000000
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/cloud-session-header.test.tsx
+++ /dev/null
@@ -1,237 +0,0 @@
-/**
- * A cloud session's header says what its launch is doing while the worker
- * cannot be asked: the session's last reported status is stale then, so a
- * sleeping launch must not read as ready.
- */
-import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
-import { act } from 'react';
-import { createRoot, type Root } from 'react-dom/client';
-import { afterEach, expect, it, vi } from 'vitest';
-import type { CloudAgent, CloudSessions } from '@shared/core/cloud-agents/cloud-agents';
-
-const sdkHost = vi.hoisted(() => ({
- cloudAgents: vi.fn(),
- cloudSessions: vi.fn(),
- cloudWake: vi.fn(),
- transcriptOpen: vi.fn(),
- transcriptClose: vi.fn(),
-}));
-
-vi.hoisted(() => {
- window.electronAPI ??= {
- invoke: () => Promise.resolve(undefined),
- eventOn: () => () => {},
- eventSend: () => {},
- } as unknown as typeof window.electronAPI;
-});
-
-vi.mock('@renderer/lib/ipc', () => ({
- events: { on: () => () => {} },
- rpc: { sdkHost },
-}));
-
-vi.mock('@renderer/features/switch-servers/switch-rooms-store', () => ({
- switchRoomsStore: { workspacesNotSignedIn: [], roomNameById: () => null },
-}));
-
-vi.mock('@renderer/lib/layout/navigation-provider', () => ({
- useParams: () => ({
- params: { agentKey: 'cloud:server:launch', sessionId: 'session-demo', name: 'reviewer' },
- }),
-}));
-
-import { cloudSessionView } from '@renderer/features/cloud-agents/cloud-session-view';
-import {
- SessionHeaderOutlet,
- SessionHeaderSlotsProvider,
-} from '@renderer/features/sessions/session-header-slots';
-
-const snapshot = {
- contractVersion: 1,
- throughSequence: 10,
- session: {
- sessionId: 'session-demo',
- agentId: 'agent-demo',
- provider: 'claude',
- hostId: 'host-demo',
- epoch: 'epoch-demo',
- status: 'ready',
- connectivity: 'online',
- capabilities: {
- input: 'queue',
- approvals: true,
- questions: true,
- interrupt: true,
- reset: false,
- compact: false,
- modelChange: false,
- attachmentMimeTypes: [],
- },
- pendingRequestIds: [],
- },
- turns: [],
- items: [],
- requests: [],
- commandStatuses: [],
- nextPageToken: null,
-};
-
-function agent(overrides: Partial<CloudAgent>): CloudAgent {
- return {
- key: 'cloud:server:launch',
- launch: {
- request_id: '00000000-0000-4000-8000-000000000001',
- name: 'reviewer',
- provider: 'claude',
- state: 'ready',
- desired_state: 'running',
- revision: 4,
- agent_id: 'agent',
- error: null,
- error_code: null,
- sleeping: false,
- machine_id: null,
- process_state: null,
- process_restarts: 0,
- oom_kills: 0,
- },
- machine: null,
- sessions: null,
- problem: null,
- ...overrides,
- };
-}
-
-let container: HTMLDivElement | null = null;
-let root: Root | null = null;
-
-afterEach(async () => {
- if (root) await act(async () => root!.unmount());
- container?.remove();
- container = null;
- root = null;
-});
-
-async function headerStatus(
- cloudAgent: CloudAgent,
- relayed: CloudSessions,
- opened: typeof snapshot
-): Promise<string | null | undefined> {
- sdkHost.cloudAgents.mockResolvedValue([cloudAgent]);
- sdkHost.cloudSessions.mockResolvedValue(relayed);
- sdkHost.transcriptOpen.mockResolvedValue(opened);
- container = document.createElement('div');
- document.body.appendChild(container);
- root = createRoot(container);
- const client = new QueryClient({ defaultOptions: { queries: { retry: false, gcTime: 0 } } });
- const Panel = cloudSessionView.MainPanel;
- await act(async () =>
- root!.render(
- <QueryClientProvider client={client}>
- <SessionHeaderSlotsProvider>
- <div data-testid="header">
- <SessionHeaderOutlet slot="left" className="" />
- </div>
- <Panel />
- </SessionHeaderSlotsProvider>
- </QueryClientProvider>
- )
- );
- for (let i = 0; i < 20; i++) {
- await act(async () => await new Promise((resolve) => setTimeout(resolve, 10)));
- const text = container.querySelector('[data-testid="header"] [role="status"]')?.textContent;
- if (text && text !== 'loading' && text !== 'offline') return text;
- }
- return container.querySelector('[data-testid="header"] [role="status"]')?.textContent;
-}
-
-it('reads ready while the worker answers', async () => {
- expect(await headerStatus(agent({}), { sessions: [], problem: null }, snapshot)).toBe('ready');
-});
-
-it('reads sleeping, not ready, while the launch is asleep', async () => {
- const status = await headerStatus(
- agent({
- launch: { ...agent({}).launch, sleeping: true, state: 'stopped', desired_state: 'running' },
- sessions: null,
- problem: {
- code: 'worker_sleeping',
- message: 'The cloud worker is asleep.',
- wakeAvailable: true,
- },
- }),
- { sessions: [], problem: null },
- snapshot
- );
- expect(status).toBe('sleeping');
-});
-
-it('reads unreachable when the relay refuses the worker', async () => {
- const status = await headerStatus(
- agent({}),
- {
- sessions: null,
- problem: { code: 'worker_busy', message: 'Too many requests.', wakeAvailable: false },
- },
- snapshot
- );
- expect(status).toBe('unreachable');
-});
-
-function offline(status: string) {
- return { ...snapshot, session: { ...snapshot.session, status, connectivity: 'offline' } };
-}
-
-it('says nothing about a session the next message restarts', async () => {
- const status = await headerStatus(agent({}), { sessions: [], problem: null }, offline('ready'));
- expect(status).toBeUndefined();
-});
-
-it('still says a stopped session is stopped', async () => {
- const status = await headerStatus(agent({}), { sessions: [], problem: null }, offline('stopped'));
- expect(status).toBe('stopped');
-});
-
-it('opens a session on a sleeping machine whose worker cannot be read, and wakes it on send', async () => {
- const asleep = agent({
- launch: { ...agent({}).launch, sleeping: true, state: 'stopped' },
- problem: {
- code: 'worker_sleeping',
- message: 'The cloud machine is asleep.',
- wakeAvailable: true,
- },
- });
- sdkHost.cloudWake.mockResolvedValue(undefined);
- sdkHost.transcriptOpen.mockRejectedValue(new Error('The cloud machine is asleep.'));
- sdkHost.cloudAgents.mockResolvedValue([asleep]);
- container = document.createElement('div');
- document.body.appendChild(container);
- root = createRoot(container);
- const client = new QueryClient({ defaultOptions: { queries: { retry: false, gcTime: 0 } } });
- const Panel = cloudSessionView.MainPanel;
- await act(async () =>
- root!.render(
- <QueryClientProvider client={client}>
- <SessionHeaderSlotsProvider>
- <Panel />
- </SessionHeaderSlotsProvider>
- </QueryClientProvider>
- )
- );
- await act(async () => await new Promise((resolve) => setTimeout(resolve, 50)));
- expect(container.textContent).toContain('Send a message to wake it.');
-
- const input = container.querySelector<HTMLTextAreaElement>(
- 'textarea[aria-label="Message the agent"]'
- )!;
- const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, 'value')!.set!;
- await act(async () => {
- setter.call(input, 'good morning');
- input.dispatchEvent(new Event('input', { bubbles: true }));
- });
- const send = [...container.querySelectorAll('button')].find((b) => b.textContent === 'Send')!;
- expect(send.disabled).toBe(false);
- await act(async () => send.click());
- expect(sdkHost.cloudWake).toHaveBeenCalledWith('cloud:server:launch');
- expect(container.textContent).toContain('Waking… about 1–2 min.');
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/connections-modal.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/connections-modal.test.tsx
index 9ed6e1aa1..4e0d3d670 100644
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/connections-modal.test.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/tests/browser/connections-modal.test.tsx
@@ -1,6 +1,6 @@
/**
* Connections open from a server's Your Agents page at any time, not only from
- * the setup wizard. The button is offered on a server with cloud agents; the
+ * the setup wizard. The button is offered on a server with cloud machines; the
* modal shows the server's catalog, says so when the catalog cannot be read
* (an older server has no catalog endpoint) rather than showing an empty one,
* and opens the GitHub step in place, with a way back to the grid.
@@ -12,23 +12,20 @@ import { afterEach, beforeEach, expect, it, vi } from 'vitest';
import type { ConnectionCatalogEntry } from '@shared/core/switch-servers/connection-catalog';
import { RpcError, serializeRpcError } from '@shared/lib/ipc/rpc-error';
-const sdkHost = vi.hoisted(() => ({
- cloudAgents: vi.fn(),
- cloudMachines: vi.fn(),
-}));
const modalHost = vi.hoisted(() => ({
show: (_id: string, _args: unknown): void => {
throw new Error('No modal host is mounted.');
},
}));
const switchServers = vi.hoisted(() => ({
+ cloudMachines: vi.fn(),
getConnectionCatalog: vi.fn(),
getGitHubConnection: vi.fn(),
}));
vi.mock('@renderer/lib/ipc', () => ({
events: { on: () => () => {} },
- rpc: { sdkHost, switchServers },
+ rpc: { switchServers },
}));
vi.mock('@renderer/features/locations/stores/agents-store', () => ({
@@ -113,9 +110,7 @@ let container: HTMLDivElement | null = null;
let root: Root | null = null;
beforeEach(() => {
- sdkHost.cloudAgents.mockReset();
- sdkHost.cloudMachines.mockReset();
- sdkHost.cloudMachines.mockResolvedValue(null);
+ switchServers.cloudMachines.mockReset();
switchServers.getConnectionCatalog.mockReset();
switchServers.getGitHubConnection.mockReset();
});
@@ -158,24 +153,24 @@ async function openConnections(): Promise<HTMLElement> {
return dialog()!;
}
-it('offers Connections on a server with cloud agents', async () => {
- sdkHost.cloudAgents.mockResolvedValue([]);
+it('offers Connections on a server with cloud machines', async () => {
+ switchServers.cloudMachines.mockResolvedValue([]);
await render();
await vi.waitFor(() => expect(button(/^connections$/i, container!)).toBeDefined());
});
-it('offers no Connections on a server without cloud agents', async () => {
- sdkHost.cloudAgents.mockResolvedValue(null);
+it('offers no Connections on a server without cloud machines', async () => {
+ switchServers.cloudMachines.mockResolvedValue(null);
await render();
- await vi.waitFor(() => expect(sdkHost.cloudAgents).toHaveBeenCalled());
+ await vi.waitFor(() => expect(switchServers.cloudMachines).toHaveBeenCalled());
await act(async () => {});
expect(button(/^connections$/i, container!)).toBeUndefined();
});
it('shows the server’s connection grid', async () => {
- sdkHost.cloudAgents.mockResolvedValue([]);
+ switchServers.cloudMachines.mockResolvedValue([]);
switchServers.getConnectionCatalog.mockResolvedValue(CATALOG);
await render();
const modal = await openConnections();
@@ -190,7 +185,7 @@ it('shows the server’s connection grid', async () => {
});
it('shows the real error when the catalog cannot be read', async () => {
- sdkHost.cloudAgents.mockResolvedValue([]);
+ switchServers.cloudMachines.mockResolvedValue([]);
switchServers.getConnectionCatalog.mockRejectedValue(
new RpcError(
serializeRpcError(
@@ -216,7 +211,7 @@ it('shows the real error when the catalog cannot be read', async () => {
});
it('opens the GitHub step from the grid and returns to it', async () => {
- sdkHost.cloudAgents.mockResolvedValue([]);
+ switchServers.cloudMachines.mockResolvedValue([]);
switchServers.getConnectionCatalog.mockResolvedValue(CATALOG);
switchServers.getGitHubConnection.mockResolvedValue({
status: 'not_connected',
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/edit-cloud-agent-modal.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/edit-cloud-agent-modal.test.tsx
deleted file mode 100644
index 2d4d19609..000000000
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/edit-cloud-agent-modal.test.tsx
+++ /dev/null
@@ -1,245 +0,0 @@
-/**
- * Editing a cloud agent: the name and icon go to the agent, the instructions
- * and model to the launch, which the agent runs from its next restart. Only
- * what changed is sent, and a refusal is shown rather than closing the dialog.
- */
-import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
-import { act, type ReactNode } from 'react';
-import { createRoot, type Root } from 'react-dom/client';
-import { afterEach, beforeEach, expect, it, vi } from 'vitest';
-import type { CloudLaunch } from '@shared/core/cloud-agents/cloud-agents';
-
-const switchServers = vi.hoisted(() => ({
- getCloudLaunchConfiguration: vi.fn(),
- updateCloudLaunchConfiguration: vi.fn(),
-}));
-const workspaces = vi.hoisted(() => ({
- listAgents: vi.fn(),
- updateAgentDisplayName: vi.fn(),
- updateAgentIcon: vi.fn(),
-}));
-const agents = vi.hoisted(() => ({
- definitionFields: vi.fn(),
- modelCatalogue: vi.fn(),
-}));
-const toast = vi.hoisted(() => vi.fn());
-
-vi.mock('@renderer/lib/ipc', () => ({
- events: { on: () => () => {} },
- rpc: { switchServers, workspaces, agents },
-}));
-// The server's workspace in scope: the agent and its name and icon live there.
-vi.mock('@renderer/features/workspaces/workspaces-store', () => ({
- workspacesStore: {
- idOnServerInScope: (serverId: string | null) => (serverId === 'server' ? 'workspace' : null),
- },
-}));
-vi.mock('@renderer/lib/hooks/use-toast', () => ({ toast }));
-vi.mock('@renderer/lib/ui/dialog', () => {
- const Plain = ({ children }: { children?: ReactNode }) => <div>{children}</div>;
- return {
- DialogHeader: Plain,
- DialogTitle: Plain,
- DialogContentArea: Plain,
- DialogFooter: Plain,
- };
-});
-vi.mock('@renderer/lib/ui/confirm-button', () => ({
- ConfirmButton: (props: { children: ReactNode; onClick: () => void; disabled: boolean }) => (
- <button onClick={props.onClick} disabled={props.disabled}>
- {props.children}
- </button>
- ),
-}));
-vi.mock('@renderer/lib/components/agent-icon-picker', () => ({
- AgentIconPicker: (props: { onChange: (url: string | null) => void }) => (
- <button onClick={() => props.onChange('https://icons.example.com/new.png')}>Pick icon</button>
- ),
-}));
-vi.mock(
- '@renderer/features/locations/components/settings-view/sections/addressing-policy-settings-section',
- () => ({ AddressingPolicyRow: () => <div>Who can talk to your agent</div> })
-);
-
-import { EditCloudAgentModal } from '@renderer/features/cloud-agents/edit-cloud-agent-modal';
-
-const MODEL = { key: 'model', label: 'Model', type: 'string' as const };
-
-function launch(provider: CloudLaunch['provider']): CloudLaunch {
- return {
- request_id: '00000000-0000-4000-8000-000000000001',
- name: 'reviewer',
- provider,
- state: 'ready',
- desired_state: 'running',
- revision: 4,
- agent_id: 'agent',
- error: null,
- error_code: null,
- sleeping: false,
- machine_id: null,
- process_state: null,
- process_restarts: 0,
- oom_kills: 0,
- };
-}
-
-let container: HTMLDivElement | null = null;
-let root: Root | null = null;
-const onSuccess = vi.fn();
-const onClose = vi.fn();
-
-beforeEach(() => {
- for (const fn of [
- ...Object.values(switchServers),
- ...Object.values(workspaces),
- ...Object.values(agents),
- toast,
- ])
- fn.mockReset();
- onSuccess.mockReset();
- onClose.mockReset();
- workspaces.listAgents.mockResolvedValue([
- { id: 'agent', name: 'reviewer', displayName: 'Reviewer', iconUrl: null },
- ]);
- switchServers.getCloudLaunchConfiguration.mockResolvedValue({
- description: 'Reviews pull requests',
- instructions: 'Be brief.',
- definition_attributes: { model: 'opus' },
- });
- agents.definitionFields.mockResolvedValue([MODEL]);
- agents.modelCatalogue.mockResolvedValue({ kind: 'unavailable', reason: 'not asked' });
-});
-
-afterEach(async () => {
- if (root) await act(async () => root!.unmount());
- container?.remove();
- container = null;
- root = null;
-});
-
-async function settle() {
- await act(async () => {
- await new Promise((resolve) => setTimeout(resolve, 0));
- });
-}
-
-async function render(provider: CloudLaunch['provider']) {
- container = document.createElement('div');
- document.body.appendChild(container);
- root = createRoot(container);
- const client = new QueryClient({ defaultOptions: { queries: { retry: false, gcTime: 0 } } });
- await act(async () =>
- root!.render(
- <QueryClientProvider client={client}>
- <EditCloudAgentModal
- serverId="server"
- launch={launch(provider)}
- onSuccess={onSuccess}
- onClose={onClose}
- />
- </QueryClientProvider>
- )
- );
- await settle();
- await settle();
-}
-
-async function type(field: HTMLInputElement | HTMLTextAreaElement, value: string) {
- const proto =
- field instanceof HTMLTextAreaElement
- ? HTMLTextAreaElement.prototype
- : HTMLInputElement.prototype;
- const setter = Object.getOwnPropertyDescriptor(proto, 'value')!.set!;
- await act(async () => {
- setter.call(field, value);
- field.dispatchEvent(new Event('input', { bubbles: true }));
- });
-}
-
-function button(label: string): HTMLButtonElement {
- const found = [...container!.querySelectorAll('button')].find((b) =>
- b.textContent?.trim().startsWith(label)
- );
- if (!found) throw new Error(`no ${label} button`);
- return found;
-}
-
-async function click(label: string) {
- await act(async () => button(label).click());
- await settle();
-}
-
-it('sends a changed name and instructions, keeping the model, and says when they apply', async () => {
- await render('claude');
- expect(container!.textContent).toContain('apply when the agent restarts');
-
- await type(container!.querySelector('input')!, 'Code Reviewer');
- await type(container!.querySelector('textarea')!, 'Review carefully.');
- await click('Save');
-
- expect(workspaces.updateAgentDisplayName).toHaveBeenCalledWith({
- workspaceId: 'workspace',
- agentId: 'agent',
- displayName: 'Code Reviewer',
- });
- expect(workspaces.updateAgentIcon).not.toHaveBeenCalled();
- expect(switchServers.updateCloudLaunchConfiguration).toHaveBeenCalledWith(
- 'server',
- '00000000-0000-4000-8000-000000000001',
- {
- provider: 'claude',
- name: 'reviewer',
- description: 'Reviews pull requests',
- instructions: 'Review carefully.',
- definition_attributes: { model: 'opus' },
- }
- );
- expect(toast).toHaveBeenCalled();
- expect(onSuccess).toHaveBeenCalled();
-});
-
-it('sends only the icon when only the icon changed', async () => {
- await render('claude');
-
- await click('Pick icon');
- await click('Save');
-
- expect(workspaces.updateAgentIcon).toHaveBeenCalledWith({
- workspaceId: 'workspace',
- agentId: 'agent',
- iconUrl: 'https://icons.example.com/new.png',
- });
- expect(workspaces.updateAgentDisplayName).not.toHaveBeenCalled();
- expect(switchServers.updateCloudLaunchConfiguration).not.toHaveBeenCalled();
- expect(onSuccess).toHaveBeenCalled();
-});
-
-it('shows a refusal and stays open', async () => {
- switchServers.updateCloudLaunchConfiguration.mockRejectedValue(
- new Error('This worker has been removed.')
- );
- await render('claude');
-
- await type(container!.querySelector('textarea')!, 'Review carefully.');
- await click('Save');
-
- expect(container!.querySelector('[role="alert"]')?.textContent).toContain(
- 'This worker has been removed.'
- );
- expect(onSuccess).not.toHaveBeenCalled();
-});
-
-it('offers no model for a provider whose cloud launch takes none', async () => {
- await render('codex');
-
- expect(container!.textContent).not.toContain('Advanced configuration');
- await type(container!.querySelector('textarea')!, 'Review carefully.');
- await click('Save');
-
- expect(switchServers.updateCloudLaunchConfiguration).toHaveBeenCalledWith(
- 'server',
- '00000000-0000-4000-8000-000000000001',
- expect.objectContaining({ provider: 'codex', definition_attributes: { model: 'opus' } })
- );
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/managed-agent-page.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/managed-agent-page.test.tsx
index 7e6b51b62..00ede3215 100644
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/managed-agent-page.test.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/tests/browser/managed-agent-page.test.tsx
@@ -75,6 +75,8 @@ const LAPTOP: OwnedMachine = {
...AGENT.machine!,
local: { kind: 'this-computer' },
providers: [{ provider: 'claude', ready: true, problem: null }],
+ acceptsLogins: false,
+ cloud: false,
workspacesDir: '/home/me/workspaces',
};
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/managed-provider-sequence.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/managed-provider-sequence.test.tsx
deleted file mode 100644
index 6f1027dd7..000000000
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/managed-provider-sequence.test.tsx
+++ /dev/null
@@ -1,111 +0,0 @@
-/**
- * Connecting several providers names the step each Continue leads to: the next
- * provider while any are left, and the step after them only once none are.
- */
-import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
-import { act, useState } from 'react';
-import { createRoot, type Root } from 'react-dom/client';
-import { afterEach, expect, it, vi } from 'vitest';
-
-const switchServers = vi.hoisted(() => ({
- getClaudeConnection: vi.fn(async () => ({
- status: 'connected',
- kind: 'api-key',
- verified_at: '2026-01-01T00:00:00Z',
- })),
- getCloudProviderConnection: vi.fn(async () => ({ status: 'connected' })),
-}));
-
-vi.mock('@renderer/lib/ipc', () => ({
- events: { on: () => () => {} },
- rpc: { switchServers },
-}));
-
-import { ManagedProviderConnectionSequence } from '@renderer/features/switch-servers/managed-provider-connection-step';
-import { Dialog, DialogContent } from '@renderer/lib/ui/dialog';
-import type { AgentProviderId } from '@shared/core/providers/agent-provider-registry';
-
-let container: HTMLDivElement | null = null;
-let root: Root | null = null;
-
-afterEach(async () => {
- if (root) await act(async () => root!.unmount());
- container?.remove();
- container = null;
- root = null;
-});
-
-function Harness({ providers }: { providers: AgentProviderId[] }) {
- const [index, setIndex] = useState(0);
- const [done, setDone] = useState(false);
- if (done) return <p>GitHub step</p>;
- return (
- <ManagedProviderConnectionSequence
- serverId="server"
- providers={providers}
- index={index}
- onIndexChange={setIndex}
- onBack={() => {}}
- onDone={() => setDone(true)}
- doneStepName="GitHub"
- />
- );
-}
-
-async function render(providers: AgentProviderId[]): Promise<HTMLElement> {
- container = document.createElement('div');
- document.body.appendChild(container);
- root = createRoot(container);
- const client = new QueryClient({ defaultOptions: { queries: { retry: false, gcTime: 0 } } });
- await act(async () =>
- root!.render(
- <QueryClientProvider client={client}>
- <Dialog open>
- <DialogContent>
- <Harness providers={providers} />
- </DialogContent>
- </Dialog>
- </QueryClientProvider>
- )
- );
- return document.body;
-}
-
-async function continueButton(el: HTMLElement): Promise<HTMLButtonElement> {
- await vi.waitFor(() => {
- const button = [...el.querySelectorAll('button')].find((b) =>
- b.textContent?.startsWith('Continue to')
- );
- expect(button?.disabled).toBe(false);
- });
- return [...el.querySelectorAll('button')].find((b) => b.textContent?.startsWith('Continue to'))!;
-}
-
-it('names the next provider, then GitHub, and goes where it says', async () => {
- const el = await render(['claude', 'codex']);
-
- const first = await continueButton(el);
- expect(el.textContent).toContain('Claude Code connected');
- expect(first.textContent).toBe('Continue to Codex');
- await act(async () => first.click());
-
- const second = await continueButton(el);
- expect(el.textContent).toContain('Connect Codex');
- expect(second.textContent).toBe('Continue to GitHub');
- await act(async () => second.click());
-
- expect(el.textContent).toContain('GitHub step');
-});
-
-it('names Claude Code when it is chosen after another provider', async () => {
- const el = await render(['codex', 'claude']);
-
- const first = await continueButton(el);
- expect(el.textContent).toContain('Connect Codex');
- expect(first.textContent).toBe('Continue to Claude Code');
- await act(async () => first.click());
-
- const second = await continueButton(el);
- expect(el.textContent).toContain('Claude Code connected');
- expect(second.textContent).toBe('Continue to GitHub');
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-display-name.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-display-name.test.tsx
index b3a690c53..050cb0f6a 100644
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-display-name.test.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-display-name.test.tsx
@@ -136,7 +136,7 @@ afterEach(async () => {
function Fields() {
domForm = useConfigureAgentForm();
- return <AgentIdentityFields form={domForm} serverId={null} />;
+ return <AgentIdentityFields form={domForm} serverId={null} nameTaken={false} />;
}
async function renderFields(): Promise<HTMLDivElement> {
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-managed-machine.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-managed-machine.test.tsx
index b4e052515..add0e43cb 100644
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-managed-machine.test.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-managed-machine.test.tsx
@@ -15,6 +15,8 @@ const defaultWorkspace = vi.hoisted(() => vi.fn());
const hostEnable = vi.hoisted(() => vi.fn());
const modelCatalogue = vi.hoisted(() => vi.fn());
const advancedConfigSchema = vi.hoisted(() => vi.fn());
+const giveMachineLogin = vi.hoisted(() => vi.fn());
+const machineLoginOutcome = vi.hoisted(() => vi.fn());
vi.hoisted(() => {
window.electronAPI ??= {
@@ -30,7 +32,7 @@ vi.mock('@renderer/lib/ipc', () => ({
embeddedController: { enable: embeddedEnable, defaultWorkspace },
hostControllers: { enable: hostEnable },
agents: { modelCatalogue },
- managedAgents: { advancedConfigSchema },
+ managedAgents: { advancedConfigSchema, giveMachineLogin, machineLoginOutcome },
},
}));
@@ -294,6 +296,8 @@ describe('the directory a new managed agent runs in', () => {
});
const BOX: OwnedMachine = {
+ acceptsLogins: false,
+ cloud: false,
id: 'controller-7',
name: 'build-box',
kind: 'daemon',
@@ -320,6 +324,7 @@ describe('the providers a server machine offers', () => {
const onChange = vi.fn();
const el = await render(
<MachineProviderPicker
+ serverId="server-1"
machine={BOX}
value="claude"
onChange={onChange}
@@ -341,6 +346,7 @@ describe('the providers a server machine offers', () => {
const onChange = vi.fn();
await render(
<MachineProviderPicker
+ serverId="server-1"
machine={BOX}
value={null}
onChange={onChange}
@@ -354,6 +360,7 @@ describe('the providers a server machine offers', () => {
const onChange = vi.fn();
const el = await render(
<MachineProviderPicker
+ serverId="server-1"
machine={{ ...BOX, providers: [] }}
value={null}
onChange={onChange}
@@ -364,4 +371,55 @@ describe('the providers a server machine offers', () => {
expect([...el.querySelectorAll('button')].every((button) => button.disabled)).toBe(true);
expect(onChange).not.toHaveBeenCalled();
});
+
+ it('gives the machine a login for a provider installed there but not signed in', async () => {
+ giveMachineLogin.mockReset().mockResolvedValue({ operationId: 'op-1' });
+ machineLoginOutcome.mockReset().mockResolvedValue({ state: 'succeeded' });
+ const el = await render(
+ <MachineProviderPicker
+ serverId="server-1"
+ machine={{ ...BOX, acceptsLogins: true }}
+ value="claude"
+ onChange={vi.fn()}
+ defaultAgent="claude"
+ />
+ );
+ expect(el.textContent).toMatch(/Give build-box a login for/);
+ const offered = [...el.querySelectorAll('button')].filter((button) =>
+ ['Codex', 'Cursor', 'OpenCode'].includes(button.textContent ?? '')
+ );
+ // Not OpenCode: it is not installed there, and no login would change that.
+ expect(offered.map((button) => button.textContent)).toEqual(['Codex', 'Cursor']);
+ await act(async () => offered[0]!.click());
+ const input = el.querySelector<HTMLInputElement>('input[aria-label="API key"]')!;
+ await act(async () => {
+ const setValue = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value')!.set!;
+ setValue.call(input, 'sk-proj-placeholder');
+ input.dispatchEvent(new Event('input', { bubbles: true }));
+ });
+ const giveButton = [...el.querySelectorAll('button')].find(
+ (button) => button.textContent === 'Give login'
+ )!;
+ await act(async () => giveButton.click());
+ await vi.waitFor(() => expect(el.textContent).toMatch(/Codex signs in on build-box/));
+ expect(giveMachineLogin).toHaveBeenCalledWith({
+ serverId: 'server-1',
+ machineId: 'controller-7',
+ provider: 'codex',
+ login: { source: 'typed', kind: 'api-key', credential: 'sk-proj-placeholder' },
+ });
+ });
+
+ it('offers no login to a machine that cannot take one', async () => {
+ const el = await render(
+ <MachineProviderPicker
+ serverId="server-1"
+ machine={BOX}
+ value="claude"
+ onChange={vi.fn()}
+ defaultAgent="claude"
+ />
+ );
+ expect(el.textContent).not.toMatch(/Give build-box a login/);
+ });
});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-name-suggestion.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-name-suggestion.test.tsx
index 2f8401652..7d1b764b0 100644
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-name-suggestion.test.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/tests/browser/new-agent-name-suggestion.test.tsx
@@ -41,12 +41,12 @@ afterEach(async () => {
form = null;
});
-function Fields() {
+function Fields({ nameTaken }: { nameTaken: boolean }) {
form = useConfigureAgentForm();
- return <AgentIdentityFields form={form} serverId={null} />;
+ return <AgentIdentityFields form={form} serverId={null} nameTaken={nameTaken} />;
}
-async function renderFields(): Promise<HTMLDivElement> {
+async function renderFields(nameTaken = false): Promise<HTMLDivElement> {
container = document.createElement('div');
document.body.appendChild(container);
root = createRoot(container);
@@ -54,7 +54,7 @@ async function renderFields(): Promise<HTMLDivElement> {
await act(async () =>
root!.render(
<QueryClientProvider client={client}>
- <Fields />
+ <Fields nameTaken={nameTaken} />
</QueryClientProvider>
)
);
@@ -157,3 +157,16 @@ describe('a rejected agent name', () => {
expect(suggestionButton(el)).toBeUndefined();
});
});
+
+describe('a name another agent on the server has', () => {
+ it('says so under the name, not only on the greyed-out button', async () => {
+ const el = await renderFields(true);
+ await typeName(el, 'cloud-codex-test');
+ expect(el.textContent).toContain(
+ 'An agent called cloud-codex-test already exists on this server. Pick another name.'
+ );
+ expect(
+ el.querySelector('input[placeholder="Name this agent"]')?.getAttribute('aria-invalid')
+ ).toBe('true');
+ });
+});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/server-agents-cloud-card.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/server-agents-cloud-card.test.tsx
index 86e9aa4ac..ff55fac28 100644
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/server-agents-cloud-card.test.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/tests/browser/server-agents-cloud-card.test.tsx
@@ -1,34 +1,22 @@
/**
- * A cloud agent's card on Your Agents reads the same start attempt as the
- * sidebar: while a new session is being started the card says so, and a start
- * whose reply was lost is shown as not yet known, with Check again (the same
- * session, never a second one) or, once the session exists, Open.
- *
- * The card reads the machine before the launch, and the machine card above the
- * grid stops the machine only once the owner confirms.
+ * The owner's cloud machine is shown above the grid on Your Agents, and its
+ * card stops the machine only once the owner confirms.
*/
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import { act } from 'react';
import { createRoot, type Root } from 'react-dom/client';
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
-import type { CloudAgent, CloudMachine } from '@shared/core/cloud-agents/cloud-agents';
-import { RpcError, serializeRpcError } from '@shared/lib/ipc/rpc-error';
+import type { CloudMachine } from '@shared/core/cloud-agents/cloud-agents';
-const sdkHost = vi.hoisted(() => ({
- cloudAgents: vi.fn(),
- cloudMachines: vi.fn(),
- cloudSessions: vi.fn(),
- cloudSessionOperation: vi.fn(),
-}));
const switchServers = vi.hoisted(() => ({
- cloudLifecycle: vi.fn(),
+ cloudMachines: vi.fn(),
cloudMachineLifecycle: vi.fn(),
}));
const navigate = vi.hoisted(() => vi.fn());
vi.mock('@renderer/lib/ipc', () => ({
events: { on: () => () => {} },
- rpc: { sdkHost, switchServers },
+ rpc: { switchServers },
}));
vi.mock('@renderer/features/locations/stores/agents-store', () => ({
@@ -61,39 +49,8 @@ vi.mock('@renderer/lib/stores/use-remote-agents', () => ({
useAgentIconUrl: () => null,
}));
-import {
- cloudOperationAttempts,
- startAttemptKey,
-} from '@renderer/features/cloud-agents/cloud-operation-attempts';
import { serverAgentsView } from '@renderer/features/switch-servers/server-agents-view';
-const agentKey = 'cloud:server:00000000-0000-4000-8000-000000000001';
-
-function agent(): CloudAgent {
- return {
- key: agentKey,
- launch: {
- request_id: '00000000-0000-4000-8000-000000000001',
- name: 'reviewer',
- provider: 'claude',
- state: 'ready',
- desired_state: 'running',
- revision: 4,
- agent_id: 'agent',
- error: null,
- error_code: null,
- sleeping: false,
- machine_id: null,
- process_state: null,
- process_restarts: 0,
- oom_kills: 0,
- },
- machine: null,
- sessions: null,
- problem: null,
- };
-}
-
function sleepingMachine(): CloudMachine {
return {
machine_id: '3f1c2b4a-0000-4000-8000-000000000001',
@@ -110,15 +67,7 @@ function sleepingMachine(): CloudMachine {
disk: { total_bytes: 214748364800, available_bytes: 204010946560 },
memory: { total_bytes: 17179869184, available_bytes: 12884901888 },
agents: ['00000000-0000-4000-8000-000000000001'],
- };
-}
-
-function sessions(sessionIds: string[]) {
- return {
- sessions: sessionIds.map(
- (sessionId) => ({ sessionId, status: 'ready', connectivity: 'online' }) as never
- ),
- problem: null,
+ controller_id: null,
};
}
@@ -128,14 +77,8 @@ let root: Root | null = null;
beforeEach(() => {
vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'setInterval', 'clearInterval'] });
navigate.mockReset();
- sdkHost.cloudAgents.mockReset();
- sdkHost.cloudMachines.mockReset();
- sdkHost.cloudMachines.mockResolvedValue(null);
- switchServers.cloudLifecycle.mockReset();
+ switchServers.cloudMachines.mockReset();
switchServers.cloudMachineLifecycle.mockReset();
- sdkHost.cloudSessions.mockReset();
- sdkHost.cloudSessionOperation.mockReset();
- cloudOperationAttempts.settle(startAttemptKey(agentKey));
});
afterEach(async () => {
@@ -173,332 +116,12 @@ function button(el: HTMLElement, name: RegExp): HTMLButtonElement | undefined {
return [...el.querySelectorAll('button')].find((b) => name.test(b.textContent ?? ''));
}
-it('shows a start in progress and does not offer a second one', async () => {
- let reply: (outcome: unknown) => void = () => {};
- sdkHost.cloudSessionOperation.mockImplementation(
- () => new Promise((resolve) => (reply = resolve))
- );
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- const el = await render();
- await act(async () => button(el, /new session/i)!.click());
-
- const starting = button(el, /starting session/i);
- expect(starting?.disabled).toBe(true);
- expect(button(el, /new session/i)).toBeUndefined();
-
- await act(async () => reply({ state: 'applied' }));
- expect(navigate).toHaveBeenCalledWith('cloudSession', expect.objectContaining({ agentKey }));
-});
-
-it('shows a lost reply as not yet known, and asks again for the same session', async () => {
- sdkHost.cloudSessionOperation.mockResolvedValueOnce({ state: 'unknown', message: 'lost' });
- sdkHost.cloudSessionOperation.mockResolvedValueOnce({ state: 'applied' });
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- sdkHost.cloudSessions.mockResolvedValue(sessions([]));
- const el = await render();
- await act(async () => button(el, /new session/i)!.click());
- await settle();
-
- expect(el.querySelector('[role="status"]')?.textContent).toMatch(/not yet known/i);
- expect(el.querySelector('[role="alert"]')).toBeNull();
- await act(async () => button(el, /check again/i)!.click());
- const [first, second] = sdkHost.cloudSessionOperation.mock.calls;
- expect(second).toEqual(first);
- expect(navigate).toHaveBeenCalledWith(
- 'cloudSession',
- expect.objectContaining({ agentKey, sessionId: first[1] })
- );
-});
-
-it('offers Open once the unconfirmed session exists', async () => {
- let started = '';
- sdkHost.cloudSessionOperation.mockImplementation(async (_key: string, sessionId: string) => {
- started = sessionId;
- return { state: 'unknown', message: 'lost' };
- });
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- sdkHost.cloudSessions.mockResolvedValue(sessions([]));
- const el = await render();
- await act(async () => button(el, /new session/i)!.click());
- await settle();
-
- sdkHost.cloudSessions.mockResolvedValue(sessions([started]));
- await act(async () => root!.unmount());
- const remounted = await render();
- const open = button(remounted, /^open$/i);
- expect(open).toBeDefined();
- await act(async () => open!.click());
- expect(navigate).toHaveBeenCalledWith(
- 'cloudSession',
- expect.objectContaining({ agentKey, sessionId: started })
- );
- expect(sdkHost.cloudSessionOperation).toHaveBeenCalledTimes(1);
-});
-
-it('reads a sleeping machine before the launch and offers no session on it', async () => {
- const machine = sleepingMachine();
- sdkHost.cloudAgents.mockResolvedValue([
- { ...agent(), launch: { ...agent().launch, machine_id: machine.machine_id }, machine },
- ]);
- sdkHost.cloudMachines.mockResolvedValue([machine]);
- const el = await render();
-
- expect(el.textContent).toMatch(/Cloud · Sleeping/);
- expect(button(el, /new session/i)).toBeUndefined();
- expect(button(el, /^remove$/i)).toBeDefined();
-});
-
-it('shows a crashed agent with its out-of-memory restarts and offers Retry', async () => {
- const base = agent();
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...base,
- launch: {
- ...base.launch,
- state: 'error',
- error: 'crashed',
- error_code: 'agent_crashed',
- process_state: 'crashed',
- oom_kills: 2,
- },
- },
- ]);
- const el = await render();
-
- expect(el.textContent).toMatch(/Restarted after running out of memory 2×/);
- expect(el.textContent).toMatch(/Crashed/);
- expect(el.querySelector('[role="alert"]')?.textContent).toMatch(/keeps crashing/);
- expect(button(el, /^retry$/i)).toBeDefined();
- expect(button(el, /stop agent/i)).toBeUndefined();
- await openMenu(el);
- expect(menuItem('Stop agent')).toBeDefined();
-});
-
-async function openMenu(el: HTMLElement): Promise<void> {
- await act(async () => el.querySelector<HTMLElement>('[aria-label="reviewer actions"]')!.click());
-}
-
-function menuItem(label: string): HTMLElement | undefined {
- return [...document.querySelectorAll<HTMLElement>('[role="menuitem"]')].find((item) =>
- item.textContent?.startsWith(label)
- );
-}
-
-it('stops a running agent from its menu, not from the card', async () => {
- switchServers.cloudLifecycle.mockResolvedValue({});
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- const el = await render();
-
- expect(button(el, /stop agent/i)).toBeUndefined();
- await openMenu(el);
- const stop = menuItem('Stop agent');
- expect(stop?.textContent).toMatch(/Stops replies and frees the machine/);
- await act(async () => stop!.click());
- expect(switchServers.cloudLifecycle).toHaveBeenCalledWith(
- 'server',
- agent().launch.request_id,
- 'stop',
- 4
- );
-});
-
-it.each([
- ['asleep', sleepingMachine(), false],
- [
- 'stopped by its owner',
- { ...sleepingMachine(), stop_reason: 'owner', sleeping: false } satisfies CloudMachine,
- false,
- ],
- [
- 'ready',
- {
- ...sleepingMachine(),
- state: 'ready',
- desired_state: 'running',
- stop_reason: null,
- sleeping: false,
- } satisfies CloudMachine,
- true,
- ],
-])('offers Stop agent only while its machine is awake: %s', async (_, machine, offered) => {
- const base = agent();
- sdkHost.cloudAgents.mockResolvedValue([
- { ...base, launch: { ...base.launch, machine_id: machine.machine_id }, machine },
- ]);
- sdkHost.cloudMachines.mockResolvedValue([machine]);
- const el = await render();
-
- await openMenu(el);
- expect(menuItem('Stop agent') !== undefined).toBe(offered);
-});
-
-it('offers no Add to rooms on a usable agent', async () => {
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- const el = await render();
-
- expect(button(el, /new session/i)).toBeDefined();
- expect(button(el, /add to rooms/i)).toBeUndefined();
-});
-
-function actions(el: HTMLElement): string[] {
- return [...el.querySelectorAll('button')]
- .filter((b) => !b.closest('header'))
- .map((b) => b.textContent ?? '')
- .filter(Boolean);
-}
-
-it('offers only Retry and Remove for an agent Switch could not register', async () => {
- const base = agent();
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...base,
- launch: {
- ...base.launch,
- agent_id: null,
- state: 'error',
- error: 'registration failed',
- error_code: 'identity_failed',
- },
- problem: { code: 'worker_not_attached', message: 'error', wakeAvailable: false },
- },
- ]);
- const el = await render();
-
- expect(el.querySelector('[role="alert"]')?.textContent).toMatch(/could not register/);
- expect(actions(el)).toEqual(['Retry', 'Remove']);
-});
-
-it.each([
- [
- 'agent_key_missing',
- /lost this agent’s credential.*Remove the agent and create it again/,
- false,
- ],
- ['agent_identity_missing', /lost this agent’s identity\. Retry/, true],
- ['worker_attach_timeout', /did not connect to Switch/, true],
- ['agent_stop_timeout', /did not stop in time/, true],
-])('explains a launch in error with %s', async (code, text, retry) => {
- const base = agent();
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...base,
- launch: { ...base.launch, state: 'error', error: 'from the server', error_code: code },
- problem: { code: 'worker_not_attached', message: 'error', wakeAvailable: false },
- },
- ]);
- const el = await render();
-
- expect(el.querySelector('[role="alert"]')?.textContent).toMatch(text);
- expect(button(el, /^retry$/i) !== undefined).toBe(retry);
-});
-
-it('says to start the machine when a retry is refused because its owner stopped it', async () => {
- const machine = {
- ...sleepingMachine(),
- stop_reason: 'owner',
- sleeping: false,
- } satisfies CloudMachine;
- const base = agent();
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...base,
- launch: {
- ...base.launch,
- machine_id: machine.machine_id,
- state: 'error',
- error: 'from the server',
- error_code: 'worker_attach_timeout',
- },
- machine,
- },
- ]);
- sdkHost.cloudMachines.mockResolvedValue([machine]);
- switchServers.cloudLifecycle.mockRejectedValue(
- new RpcError(
- serializeRpcError(
- Object.assign(new Error('Switch gateway returned 409'), {
- name: 'GatewayError',
- kind: 'http',
- status: 409,
- detail: 'The owner stopped the cloud machine. Start it in Switch Console.',
- code: 'machine_stopped',
- })
- )
- )
- );
- const el = await render();
-
- await act(async () => button(el, /^retry$/i)!.click());
- await settle();
- expect(el.textContent).toMatch(
- /The owner stopped the cloud machine\. Start the machine, then try again\./
- );
-});
-
-it('offers only Remove for a removal left halfway', async () => {
- const base = agent();
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...base,
- launch: { ...base.launch, state: 'deleting', desired_state: 'deleted' },
- problem: { code: 'worker_not_attached', message: 'removing', wakeAvailable: false },
- },
- ]);
- const el = await render();
-
- expect(el.textContent).toMatch(/Cloud · Removing…/);
- expect(actions(el)).toEqual(['Remove']);
-});
-
-it('shows a stopped agent on a sleeping machine as stopped', async () => {
- const machine = sleepingMachine();
- const base = agent();
- sdkHost.cloudAgents.mockResolvedValue([
- {
- ...base,
- launch: {
- ...base.launch,
- machine_id: machine.machine_id,
- desired_state: 'stopped',
- state: 'stopped',
- sleeping: true,
- },
- machine,
- },
- ]);
- const el = await render();
-
- expect(el.textContent).toMatch(/Cloud · Stopped/);
- expect(button(el, /start agent/i)).toBeDefined();
- await openMenu(el);
- expect(menuItem('Stop agent')).toBeUndefined();
-});
-
-it('removes a running agent once confirmed', async () => {
- switchServers.cloudLifecycle.mockResolvedValue({});
- sdkHost.cloudAgents.mockResolvedValue([agent()]);
- const el = await render();
-
- await act(async () => button(el, /^remove$/i)!.click());
- expect(el.textContent).toMatch(/Its working copy on the cloud machine is\s+deleted right away/);
- expect(el.textContent).toMatch(/the machine shuts down and its disk is kept/);
- expect(switchServers.cloudLifecycle).not.toHaveBeenCalled();
- await act(async () => button(el, /remove agent/i)!.click());
- expect(switchServers.cloudLifecycle).toHaveBeenCalledWith(
- 'server',
- agent().launch.request_id,
- 'remove',
- 4
- );
-});
-
it.each([
['stopped', sleepingMachine()],
['stopping', { ...sleepingMachine(), state: 'stopping' } satisfies CloudMachine],
])('shows the machine card for a sleeping machine (%s)', async (_state, machine) => {
switchServers.cloudMachineLifecycle.mockResolvedValue(machine);
- sdkHost.cloudAgents.mockResolvedValue([]);
- sdkHost.cloudMachines.mockResolvedValue([machine]);
+ switchServers.cloudMachines.mockResolvedValue([machine]);
const el = await render();
expect(el.textContent).toMatch(/Cloud machine/);
@@ -517,8 +140,7 @@ it('stops a ready machine once confirmed', async () => {
sleeping: false,
} satisfies CloudMachine;
switchServers.cloudMachineLifecycle.mockResolvedValue(machine);
- sdkHost.cloudAgents.mockResolvedValue([]);
- sdkHost.cloudMachines.mockResolvedValue([machine]);
+ switchServers.cloudMachines.mockResolvedValue([machine]);
const el = await render();
expect(el.textContent).toMatch(/Ready · c7i\.2xlarge · 1 agent/);
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/server-stat-tiles.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/server-stat-tiles.test.tsx
index 857c1641c..1cd726da5 100644
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/server-stat-tiles.test.tsx
+++ b/console/apps/switch-console-desktop/src/renderer/tests/browser/server-stat-tiles.test.tsx
@@ -1,16 +1,11 @@
/**
- * "Your Agents" adds the cloud agents to the local ones. A server that has no
- * cloud agents counts them as zero; a server whose cloud list could not be
- * read leaves the total unknown and says so, rather than showing the local
- * agents alone as if they were all of them.
+ * "Your Agents" counts the agents onboarded through Switch Console on the
+ * server, the same ones the sidebar and the Your Agents page list.
*/
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import { act } from 'react';
import { createRoot, type Root } from 'react-dom/client';
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
-import type { CloudAgent } from '@shared/core/cloud-agents/cloud-agents';
-
-const sdkHost = vi.hoisted(() => ({ cloudAgents: vi.fn() }));
vi.hoisted(() => {
window.electronAPI ??= {
@@ -22,7 +17,7 @@ vi.hoisted(() => {
vi.mock('@renderer/lib/ipc', () => ({
events: { on: () => () => {} },
- rpc: { sdkHost, workspaces: { listBridges: async () => [] } },
+ rpc: { workspaces: { listBridges: async () => [] } },
}));
vi.mock('@renderer/features/locations/stores/agents-store', () => ({
@@ -45,37 +40,11 @@ vi.mock('@renderer/features/switch-servers/switch-servers-store', () => ({
import { ServerStatTiles } from '@renderer/features/switch-servers/server-stat-tiles';
-function cloudAgent(): CloudAgent {
- return {
- key: 'cloud:server:launch',
- launch: {
- request_id: '00000000-0000-4000-8000-000000000001',
- name: 'reviewer',
- provider: 'claude',
- state: 'ready',
- desired_state: 'running',
- revision: 1,
- agent_id: 'agent',
- error: null,
- error_code: null,
- sleeping: false,
- machine_id: null,
- process_state: null,
- process_restarts: 0,
- oom_kills: 0,
- },
- machine: null,
- sessions: null,
- problem: null,
- };
-}
-
let container: HTMLDivElement | null = null;
let root: Root | null = null;
beforeEach(() => {
vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'setInterval', 'clearInterval'] });
- sdkHost.cloudAgents.mockReset();
});
afterEach(async () => {
@@ -108,23 +77,7 @@ function agentsTile(el: HTMLElement): HTMLElement {
return [...el.querySelectorAll('p')].find((p) => p.textContent === 'Your Agents')!.parentElement!;
}
-it('adds the cloud agents to the local ones', async () => {
- sdkHost.cloudAgents.mockResolvedValue([cloudAgent()]);
- const tile = agentsTile(await render());
- expect(tile.querySelectorAll('p')[1].textContent).toBe('3');
- expect(tile.querySelector('[role="alert"]')).toBeNull();
-});
-
-it('counts a server without cloud agents as having none', async () => {
- sdkHost.cloudAgents.mockResolvedValue(null);
+it('counts the agents onboarded on the server', async () => {
const tile = agentsTile(await render());
expect(tile.querySelectorAll('p')[1].textContent).toBe('2');
- expect(tile.querySelector('[role="alert"]')).toBeNull();
-});
-
-it('leaves the total unknown and shows the failure when the cloud list fails', async () => {
- sdkHost.cloudAgents.mockRejectedValue(new Error('Request failed with status 500'));
- const tile = agentsTile(await render());
- expect(tile.querySelectorAll('p')[1].textContent).toBe('—');
- expect(tile.querySelector('[role="alert"]')?.textContent).toMatch(/cloud agents/i);
});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/browser/session-held-message.test.tsx b/console/apps/switch-console-desktop/src/renderer/tests/browser/session-held-message.test.tsx
deleted file mode 100644
index d3bd91006..000000000
--- a/console/apps/switch-console-desktop/src/renderer/tests/browser/session-held-message.test.tsx
+++ /dev/null
@@ -1,325 +0,0 @@
-/**
- * A message sent to a cloud agent whose machine sleeps is held and sent once
- * the session is ready. The composer says where the machine is, never leaves
- * the message stuck with Send locked, and keeps it when the user leaves the
- * session and comes back.
- */
-import {
- type ClientCommand,
- SessionChatClient,
- type SessionTransport,
-} from '@switch-console/shared/session-v1';
-import { act } from 'react';
-import { createRoot, type Root } from 'react-dom/client';
-import { afterEach, expect, it, vi } from 'vitest';
-import { heldMessages } from '@renderer/features/sessions/components/transcript/held-message';
-import { SessionV1Chat } from '@renderer/features/sessions/components/transcript/session-v1-chat';
-import { SessionHeaderSlotsProvider } from '@renderer/features/sessions/session-header-slots';
-import type { CloudAgentPhase } from '@shared/core/cloud-agents/cloud-agents';
-
-vi.hoisted(() => {
- window.electronAPI ??= {
- invoke: () => Promise.resolve(undefined),
- eventOn: () => () => {},
- eventSend: () => {},
- } as unknown as typeof window.electronAPI;
-});
-
-const SESSION = 'session-held';
-
-const snapshot = {
- contractVersion: 1,
- throughSequence: 1,
- session: {
- sessionId: SESSION,
- agentId: 'agent-demo',
- provider: 'claude',
- hostId: 'host-demo',
- epoch: 'epoch-demo',
- status: 'ready',
- connectivity: 'online',
- capabilities: {
- input: 'queue',
- approvals: true,
- questions: true,
- interrupt: true,
- reset: false,
- compact: false,
- modelChange: false,
- attachmentMimeTypes: [],
- },
- pendingRequestIds: [],
- },
- turns: [],
- items: [],
- requests: [],
- commandStatuses: [],
- nextPageToken: null,
-};
-
-/** A worker behind the relay: unreachable while its machine sleeps. */
-function worker() {
- const state = { awake: false, submitted: [] as ClientCommand[] };
- const transport: SessionTransport = {
- snapshot: async () => {
- if (!state.awake) throw new Error('The relay refused: worker_sleeping');
- return structuredClone(snapshot);
- },
- subscribe: () => () => {},
- submit: async (command) => {
- state.submitted.push(command);
- return {
- type: 'command.status',
- commandId: command.commandId,
- status: 'accepted',
- code: null,
- message: null,
- };
- },
- commandStatus: async () => {
- throw new Error('Not recorded.');
- },
- };
- return { state, transport };
-}
-
-let container: HTMLDivElement | null = null;
-let root: Root | null = null;
-
-afterEach(async () => {
- if (root) await act(async () => root!.unmount());
- container?.remove();
- container = null;
- root = null;
- heldMessages.set(SESSION, null);
-});
-
-const wake = vi.fn(async () => undefined);
-
-async function render(
- client: SessionChatClient,
- phase: CloudAgentPhase | null,
- blocked: string | null,
- restartHost?: () => Promise<void>,
- machineReady = false
-): Promise<HTMLDivElement> {
- if (!container) {
- container = document.createElement('div');
- document.body.appendChild(container);
- root = createRoot(container);
- }
- await act(async () =>
- root!.render(
- <SessionHeaderSlotsProvider>
- <SessionV1Chat
- client={client}
- hostState={null}
- autoWake={{ phase, machineReady, blocked, wake }}
- restartHost={restartHost}
- />
- </SessionHeaderSlotsProvider>
- )
- );
- await settle();
- return container;
-}
-
-async function settle(ms = 20) {
- await act(async () => await new Promise((resolve) => setTimeout(resolve, ms)));
-}
-
-async function remount() {
- await act(async () => root!.unmount());
- root = createRoot(container!);
-}
-
-function textarea(el: HTMLElement): HTMLTextAreaElement {
- return el.querySelector('textarea[aria-label="Message the agent"]')!;
-}
-
-function button(el: HTMLElement, name: RegExp): HTMLButtonElement | undefined {
- return [...el.querySelectorAll('button')].find((b) => name.test(b.textContent ?? ''));
-}
-
-async function type(el: HTMLElement, text: string) {
- const input = textarea(el);
- const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, 'value')!.set!;
- await act(async () => {
- setter.call(input, text);
- input.dispatchEvent(new Event('input', { bubbles: true }));
- });
-}
-
-async function holdMessage(text: string) {
- const { state, transport } = worker();
- const client = new SessionChatClient(SESSION, transport);
- const el = await render(client, 'sleeping', null);
- await type(el, text);
- await act(async () => button(el, /^send$/i)!.click());
- await settle();
- return { el, client, state, transport };
-}
-
-it('says waking while the machine wakes, and connecting once it is awake', async () => {
- const { el, client } = await holdMessage('hello');
- expect(wake).toHaveBeenCalled();
- expect(el.textContent).toContain('Waking… about 1–2 min.');
-
- await render(client, null, null);
- expect(el.textContent).not.toContain('Waking…');
- expect(el.textContent).toContain('The machine is awake. Connecting to the session');
- expect(button(el, /^connecting…$/i)?.disabled).toBe(true);
-});
-
-it('says the agent is starting when only the agent starts on a machine already up', async () => {
- const { transport } = worker();
- const client = new SessionChatClient(SESSION, transport);
- const el = await render(client, 'waking', null, undefined, true);
- await type(el, 'hello');
- await act(async () => button(el, /^send$/i)!.click());
- await settle();
- expect(el.textContent).toContain('Starting the agent… Your message is sent when it is ready.');
- expect(el.textContent).not.toContain('Waking…');
- expect(button(el, /^starting…$/i)?.disabled).toBe(true);
-});
-
-it('says waking while the machine itself starts', async () => {
- const { transport } = worker();
- const client = new SessionChatClient(SESSION, transport);
- const el = await render(client, 'waking', null, undefined, false);
- await type(el, 'hello');
- await act(async () => button(el, /^send$/i)!.click());
- await settle();
- expect(el.textContent).toContain('Waking… about 1–2 min.');
- expect(el.textContent).not.toContain('Starting the agent');
-});
-
-it('keeps the connection error in view while it holds a message', async () => {
- const { el } = await holdMessage('hello');
- expect(el.textContent).toContain('The relay refused: worker_sleeping');
-});
-
-it('cancels a held message back into an editable draft', async () => {
- const { el } = await holdMessage('hello');
- await act(async () => button(el, /^cancel$/i)!.click());
- expect(el.textContent).not.toContain('Waking…');
- expect(textarea(el).value).toBe('hello');
- expect(textarea(el).readOnly).toBe(false);
- expect(button(el, /^send$/i)?.disabled).toBe(false);
-});
-
-it('stops holding when the message will not be delivered, says why and unlocks the draft', async () => {
- const { el, client } = await holdMessage('hello');
- await render(
- client,
- 'machine_error',
- 'The cloud machine is in error. Retry it in Your Agents, then send again.'
- );
- expect(el.querySelector('[role="alert"]')?.textContent).toContain(
- 'The cloud machine is in error. Retry it in Your Agents, then send again. Your message was not sent.'
- );
- expect(el.textContent).not.toContain('Waking…');
- expect(textarea(el).value).toBe('hello');
- expect(textarea(el).readOnly).toBe(false);
-});
-
-it('keeps holding a message after leaving the session, and sends it once on return', async () => {
- const { transport, state } = await holdMessage('hello');
- await remount();
- const client = new SessionChatClient(SESSION, transport);
- const el = await render(client, 'waking', null);
- expect(el.textContent).toContain('Waking…');
- expect(textarea(el).value).toBe('hello');
-
- state.awake = true;
- await render(client, null, null);
- await act(async () => await client.connect());
- await settle();
- expect(state.submitted.map((command) => command.body)).toEqual([
- expect.objectContaining({ type: 'message.send', text: 'hello' }),
- ]);
- expect(el.textContent).not.toContain('The machine is awake');
- expect(textarea(el).value).toBe('');
-});
-
-/**
- * A worker whose relay reattaches after its machine restarts, but whose session
- * host stays offline until the session is restarted.
- */
-function restartable(restart: () => Promise<void>) {
- const { state, transport } = worker();
- const host = { online: false };
- const restartHost = vi.fn(async () => {
- await restart();
- host.online = true;
- });
- const client = new SessionChatClient(SESSION, {
- ...transport,
- snapshot: async (...args) => {
- const current = (await transport.snapshot(...args)) as typeof snapshot;
- return {
- ...current,
- session: { ...current.session, connectivity: host.online ? 'online' : 'offline' },
- };
- },
- });
- return { state, client, restartHost };
-}
-
-it('restarts an offline session host once for a message held across a machine restart, and sends it once', async () => {
- const { state, client, restartHost } = restartable(async () => undefined);
- const el = await render(client, 'sleeping', null, restartHost);
- await type(el, 'hello');
- await act(async () => button(el, /^send$/i)!.click());
- await settle();
- expect(el.textContent).toContain('Waking…');
-
- state.awake = true;
- await render(client, null, null, restartHost);
- await act(async () => await client.connect());
- await settle();
- await render(client, null, null, restartHost);
- expect(restartHost).toHaveBeenCalledTimes(1);
- expect(state.submitted.map((command) => command.body)).toEqual([
- expect.objectContaining({ type: 'message.send', text: 'hello' }),
- ]);
- expect(el.textContent).not.toContain('The machine is awake');
- expect(textarea(el).value).toBe('');
-});
-
-it('lets the user send to an offline session on an awake machine, restarting its host first', async () => {
- const { state, client, restartHost } = restartable(async () => undefined);
- state.awake = true;
- const el = await render(client, null, null, restartHost);
- await type(el, 'hello');
- expect(textarea(el).readOnly).toBe(false);
- expect(button(el, /^send$/i)?.disabled).toBe(false);
- expect(restartHost).not.toHaveBeenCalled();
-
- await act(async () => button(el, /^send$/i)!.click());
- await settle();
- expect(restartHost).toHaveBeenCalledTimes(1);
- expect(state.submitted.map((command) => command.body)).toEqual([
- expect.objectContaining({ type: 'message.send', text: 'hello' }),
- ]);
- expect(textarea(el).value).toBe('');
-});
-
-it('stops holding when the session host does not restart, says why and unlocks the draft', async () => {
- const { state, client, restartHost } = restartable(async () => {
- throw new Error('The worker did not start the session.');
- });
- state.awake = true;
- const el = await render(client, null, null, restartHost);
- await type(el, 'hello');
- await act(async () => button(el, /^send$/i)!.click());
- await settle();
- expect(restartHost).toHaveBeenCalledTimes(1);
- expect(state.submitted).toEqual([]);
- expect(el.textContent).not.toContain('The machine is awake');
- expect(
- [...el.querySelectorAll('[role="alert"]')].map((alert) => alert.textContent).join('\n')
- ).toContain('The worker did not start the session. Your message was not sent.');
- expect(textarea(el).value).toBe('hello');
- expect(textarea(el).readOnly).toBe(false);
-});
diff --git a/console/apps/switch-console-desktop/src/renderer/tests/cloud-operation-attempts.test.ts b/console/apps/switch-console-desktop/src/renderer/tests/cloud-operation-attempts.test.ts
deleted file mode 100644
index 87dcb15f8..000000000
--- a/console/apps/switch-console-desktop/src/renderer/tests/cloud-operation-attempts.test.ts
+++ /dev/null
@@ -1,109 +0,0 @@
-import { beforeEach, expect, it, vi } from 'vitest';
-
-const server = vi.hoisted(() => ({
- operations: new Map<string, { sessionId: string; action: string }>(),
- sessions: new Set<string>(),
- restarts: 0,
- loseNextResponse: false,
- failNext: false,
-}));
-
-const cloudSessionOperation = vi.hoisted(() =>
- vi.fn(async (_agentKey: string, sessionId: string, operationId: string, action: string) => {
- if (server.failNext) {
- server.failNext = false;
- return { state: 'failed', message: 'session limit', code: null };
- }
- if (!server.operations.has(operationId)) {
- server.operations.set(operationId, { sessionId, action });
- if (action === 'start') server.sessions.add(sessionId);
- else server.restarts += 1;
- }
- if (server.loseNextResponse) {
- server.loseNextResponse = false;
- return { state: 'unknown', message: 'The server did not confirm the session.' };
- }
- return { state: 'applied' };
- })
-);
-
-vi.mock('@renderer/lib/ipc', () => ({ rpc: { sdkHost: { cloudSessionOperation } } }));
-
-const { cloudOperationAttempts, restartAttemptKey, startAttemptKey } =
- await import('@renderer/features/cloud-agents/cloud-operation-attempts');
-
-const agentKey = 'cloud:server:launch';
-
-beforeEach(() => {
- server.operations.clear();
- server.sessions.clear();
- server.restarts = 0;
- server.loseNextResponse = false;
- server.failNext = false;
- cloudSessionOperation.mockClear();
- cloudOperationAttempts.settle(startAttemptKey(agentKey));
- cloudOperationAttempts.settle(restartAttemptKey(agentKey, 'session'));
-});
-
-it('asks for the same session again after a lost start response, so one session results', async () => {
- const key = startAttemptKey(agentKey);
- server.loseNextResponse = true;
- const first = await cloudOperationAttempts.run(key, agentKey, 'start', null);
- expect(first?.outcome.state).toBe('unknown');
- expect(cloudOperationAttempts.get(key)).toMatchObject({
- status: 'unknown',
- sessionId: first?.sessionId,
- });
- const second = await cloudOperationAttempts.run(key, agentKey, 'start', null);
- expect(second).toEqual({ sessionId: first?.sessionId, outcome: { state: 'applied' } });
- const [a, b] = cloudSessionOperation.mock.calls;
- expect(b).toEqual(a);
- expect(a[2]).toBe(a[1]);
- expect(server.sessions.size).toBe(1);
- expect(cloudOperationAttempts.get(key)).toBeUndefined();
-});
-
-it('asks for the same restart again after a lost response, so it restarts once', async () => {
- const key = restartAttemptKey(agentKey, 'session');
- server.loseNextResponse = true;
- expect(
- (await cloudOperationAttempts.run(key, agentKey, 'restart', 'session'))?.outcome.state
- ).toBe('unknown');
- expect(
- (await cloudOperationAttempts.run(key, agentKey, 'restart', 'session'))?.outcome.state
- ).toBe('applied');
- const [a, b] = cloudSessionOperation.mock.calls;
- expect(b).toEqual(a);
- expect(server.restarts).toBe(1);
- expect(cloudOperationAttempts.get(key)).toBeUndefined();
-});
-
-it('treats a call that never answered as unknown and keeps the attempt', async () => {
- const key = startAttemptKey(agentKey);
- cloudSessionOperation.mockRejectedValueOnce(new Error('IPC channel closed'));
- const first = await cloudOperationAttempts.run(key, agentKey, 'start', null);
- expect(first?.outcome.state).toBe('unknown');
- expect(cloudOperationAttempts.get(key)?.operationId).toBe(first?.sessionId);
-});
-
-it('ignores a second ask while the first is in flight', async () => {
- const key = startAttemptKey(agentKey);
- const first = cloudOperationAttempts.run(key, agentKey, 'start', null);
- expect(await cloudOperationAttempts.run(key, agentKey, 'start', null)).toBeNull();
- await first;
- expect(cloudSessionOperation).toHaveBeenCalledTimes(1);
-});
-
-it('uses a fresh id after a definite failure', async () => {
- const key = restartAttemptKey(agentKey, 'session');
- server.failNext = true;
- expect((await cloudOperationAttempts.run(key, agentKey, 'restart', 'session'))?.outcome).toEqual({
- state: 'failed',
- message: 'session limit',
- code: null,
- });
- expect(cloudOperationAttempts.get(key)).toBeUndefined();
- await cloudOperationAttempts.run(key, agentKey, 'restart', 'session');
- const [a, b] = cloudSessionOperation.mock.calls;
- expect(b[2]).not.toBe(a[2]);
-});
diff --git a/console/apps/switch-console-desktop/src/shared/core/cloud-agents/cloud-agents.test.ts b/console/apps/switch-console-desktop/src/shared/core/cloud-agents/cloud-agents.test.ts
index 86cf33828..388b05a74 100644
--- a/console/apps/switch-console-desktop/src/shared/core/cloud-agents/cloud-agents.test.ts
+++ b/console/apps/switch-console-desktop/src/shared/core/cloud-agents/cloud-agents.test.ts
@@ -1,14 +1,8 @@
-import { readdirSync, readFileSync } from 'node:fs';
+import { readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { expect, it } from 'vitest';
-import {
- cloudAgentPhase,
- type CloudLaunch,
- cloudLaunchSchema,
- type CloudMachine,
- cloudMachineSchema,
-} from './cloud-agents';
+import { cloudMachineSchema } from './cloud-agents';
const CORE_FIXTURES = join(
dirname(fileURLToPath(import.meta.url)),
@@ -21,126 +15,12 @@ function coreFixture(name: string): unknown {
const idleSleeping = coreFixture('machine_summary_sleeping.json');
-const launch: CloudLaunch = cloudLaunchSchema.parse({
- request_id: '00000000-0000-4000-8000-000000000001',
- name: 'reviewer',
- provider: 'claude',
- state: 'ready',
- desired_state: 'running',
- revision: 3,
- agent_id: '3f1c2b4a-0000-4000-8000-0000000000a1',
- error: null,
- error_code: null,
- sleeping: false,
- machine_id: '3f1c2b4a-0000-4000-8000-000000000001',
- process_state: 'running',
- process_restarts: 0,
- oom_kills: 0,
-});
-
-function machine(overrides: Partial<CloudMachine>): CloudMachine {
- return {
- ...cloudMachineSchema.parse(idleSleeping),
- state: 'ready',
- desired_state: 'running',
- stop_reason: null,
- sleeping: false,
- ...overrides,
- };
-}
-
it('parses the idle-sleeping machine summary Core serves', () => {
expect(cloudMachineSchema.parse(idleSleeping)).toEqual(idleSleeping);
});
-it('parses every launch summary Core serves', () => {
- for (const name of readdirSync(CORE_FIXTURES).filter((file) =>
- /^launch_summary.*\.json$/.test(file)
- )) {
- const summary = coreFixture(name);
- expect(cloudLaunchSchema.parse(summary), name).toEqual(summary);
- }
-});
-
-it('parses a launch summary with its machine and process', () => {
- expect(launch).toMatchObject({
- machine_id: '3f1c2b4a-0000-4000-8000-000000000001',
- process_state: 'running',
- process_restarts: 0,
- oom_kills: 0,
- });
- expect(
- cloudLaunchSchema.parse({ ...launch, machine_id: null, process_state: null })
- ).toMatchObject({ machine_id: null, process_state: null });
-});
-
it('refuses a machine in a state it does not know', () => {
expect(() =>
cloudMachineSchema.parse({ ...(idleSleeping as object), state: 'hibernating' })
).toThrow();
});
-
-it.each([
- ['an idle-sleeping machine', cloudMachineSchema.parse(idleSleeping), 'sleeping'],
- [
- 'a machine its owner stopped',
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'owner' }),
- 'machine_stopped',
- ],
- [
- 'a machine in error',
- machine({ state: 'error', error: 'boom', error_code: 'machine_connect_timeout' }),
- 'machine_error',
- ],
- ['a machine on its way up', machine({ state: 'provisioning' }), 'waking'],
- ['a retained machine being reused', machine({ state: 'retained' }), 'waking'],
- ['a ready machine', machine({}), null],
-] as const)('reads %s', (_name, onMachine, phase) => {
- expect(cloudAgentPhase(launch, onMachine)).toBe(phase);
-});
-
-const stopped: CloudLaunch = { ...launch, desired_state: 'stopped', state: 'stopped' };
-const crashed: CloudLaunch = {
- ...launch,
- state: 'error',
- error: 'crashed',
- error_code: 'agent_crashed',
- process_state: 'crashed',
-};
-
-it.each([
- ['stopped', stopped],
- ['crashed', crashed],
-] as const)('does not sleep or wake with its machine a launch that is %s', (_name, onLaunch) => {
- expect(cloudAgentPhase(onLaunch, cloudMachineSchema.parse(idleSleeping))).toBeNull();
- expect(cloudAgentPhase({ ...onLaunch, sleeping: true, machine_id: null }, null)).toBeNull();
- expect(cloudAgentPhase(onLaunch, machine({ state: 'provisioning' }))).toBeNull();
-});
-
-it('reads a machine its owner stopped whatever the launch', () => {
- expect(
- cloudAgentPhase(
- stopped,
- machine({ state: 'stopped', desired_state: 'stopped', stop_reason: 'owner' })
- )
- ).toBe('machine_stopped');
-});
-
-it('reads a machine in error whatever the launch', () => {
- expect(cloudAgentPhase(stopped, machine({ state: 'error' }))).toBe('machine_error');
-});
-
-it('reads a machine in error before its owner stopping it', () => {
- expect(
- cloudAgentPhase(
- stopped,
- machine({ state: 'error', desired_state: 'stopped', stop_reason: 'owner' })
- )
- ).toBe('machine_error');
-});
-
-it('reads a launch without a machine from the launch', () => {
- expect(
- cloudAgentPhase({ ...launch, machine_id: null, sleeping: true, state: 'stopped' }, null)
- ).toBe('sleeping');
-});
diff --git a/console/apps/switch-console-desktop/src/shared/core/cloud-agents/cloud-agents.ts b/console/apps/switch-console-desktop/src/shared/core/cloud-agents/cloud-agents.ts
index 2b41291d8..369ea97f3 100644
--- a/console/apps/switch-console-desktop/src/shared/core/cloud-agents/cloud-agents.ts
+++ b/console/apps/switch-console-desktop/src/shared/core/cloud-agents/cloud-agents.ts
@@ -1,56 +1,5 @@
-import type { Session } from '@switch-console/shared/session-v1';
import { z } from 'zod';
-/**
- * A cloud agent is a hosted launch on a Switch server. Its sessions run on
- * the launch's worker and are reached through the server's relay, so where a
- * local or SSH agent is named by its Console agent id, a cloud one is named by
- * this key and the same session calls take either.
- */
-
-const PREFIX = 'cloud:';
-
-export function cloudAgentKey(serverId: string, requestId: string): string {
- return `${PREFIX}${serverId}:${requestId}`;
-}
-
-export function parseCloudAgentKey(key: string): { serverId: string; requestId: string } | null {
- if (!key.startsWith(PREFIX)) return null;
- const at = key.lastIndexOf(':');
- const serverId = key.slice(PREFIX.length, at);
- const requestId = key.slice(at + 1);
- return serverId && requestId ? { serverId, requestId } : null;
-}
-
-export const cloudLaunchSchema = z.object({
- request_id: z.string().uuid(),
- name: z.string(),
- provider: z.enum(['claude', 'codex', 'opencode', 'cursor', 'antigravity']),
- state: z.string(),
- desired_state: z.enum(['running', 'stopped', 'restart', 'deleted']),
- revision: z.number().int().positive(),
- agent_id: z.string().nullable(),
- error: z.string().nullable(),
- error_code: z.string().nullable(),
- sleeping: z.boolean(),
- machine_id: z.string().nullable(),
- process_state: z
- .enum([
- 'pending',
- 'starting',
- 'running',
- 'stopping',
- 'stopped',
- 'restarting',
- 'crashed',
- 'failed',
- ])
- .nullable(),
- process_restarts: z.number().int().nonnegative(),
- oom_kills: z.number().int().nonnegative(),
-});
-export type CloudLaunch = z.infer<typeof cloudLaunchSchema>;
-
const machineCapacitySchema = z
.object({
total_bytes: z.number().int().nonnegative(),
@@ -59,7 +8,8 @@ const machineCapacitySchema = z
.nullable();
/**
- * The owner's cloud machine, which every one of their launches runs on.
+ * The owner's cloud machine. It runs the agents controller, and its agents are
+ * managed agents placed on `controller_id` (null until it has enrolled).
* `sleeping` is Core's reading of desired `stopped` for `idle`; `disk` and
* `memory` come from the machine's last heartbeat.
*/
@@ -88,86 +38,6 @@ export const cloudMachineSchema = z.object({
disk: machineCapacitySchema,
memory: machineCapacitySchema,
agents: z.array(z.string()),
+ controller_id: z.string().nullable(),
});
export type CloudMachine = z.infer<typeof cloudMachineSchema>;
-
-export const cloudOperationSchema = z.object({
- id: z.string().uuid(),
- session_id: z.string().uuid(),
- action: z.enum(['start', 'restart']),
- state: z.enum(['queued', 'claimed', 'applied', 'failed', 'unknown']),
- error: z.string().nullable(),
-});
-export type CloudOperation = z.infer<typeof cloudOperationSchema>;
-
-/**
- * Where a start or restart ended. `unknown` means the server may hold the
- * operation: ask again with the same id, which the server dedupes, rather
- * than a new one. A failure the server refused with a code carries it, the
- * relay's vocabulary (`worker_waking`, `machine_stopped`, `machine_error`).
- */
-export type CloudOperationOutcome =
- | { state: 'applied' }
- | { state: 'failed'; message: string; code: string | null }
- | { state: 'unknown'; message: string };
-
-/** Why a cloud agent's sessions could not be read, with the relay's code. */
-export type CloudRelayProblem = {
- code: string;
- message: string;
- /** Set on `worker_sleeping`: starting the launch's machine would wake it. */
- wakeAvailable: boolean;
-};
-
-/** A worker's sessions, or why they could not be read. */
-export type CloudSessions = {
- sessions: Session[] | null;
- problem: CloudRelayProblem | null;
-};
-
-/**
- * A launch with its worker's sessions, or why they could not be read.
- * `sessions` is null until the worker has been asked. `machine` is the
- * machine the launch runs on, null when it has none or it is not listed.
- */
-export type CloudAgent = CloudSessions & {
- key: string;
- launch: CloudLaunch;
- machine: CloudMachine | null;
-};
-
-export type CloudAgentPhase = 'sleeping' | 'waking' | 'machine_stopped' | 'machine_error';
-
-/**
- * Whether the agent's machine is in error, stopped by its owner, asleep, or on
- * its way up; an error outranks a stop. The machine is read first; a launch
- * without one says for itself.
- * Only a launch asked to run and not in error sleeps or wakes with its
- * machine: a message to a stopped or crashed one is refused rather than
- * waking it.
- */
-export function cloudAgentPhase(
- launch: CloudLaunch,
- machine: CloudMachine | null
-): CloudAgentPhase | null {
- if (machine?.state === 'error') return 'machine_error';
- if (machine?.desired_state === 'stopped' && machine.stop_reason === 'owner')
- return 'machine_stopped';
- if (launch.desired_state !== 'running' || launch.state === 'error') return null;
- if (machine ? machine.sleeping : launch.sleeping) return 'sleeping';
- if (
- machine?.desired_state === 'running' &&
- ['queued', 'provisioning', 'stopping', 'stopped', 'retained'].includes(machine.state)
- )
- return 'waking';
- if (['queued', 'provisioning'].includes(launch.state)) return 'waking';
- return null;
-}
-
-/**
- * Whether a waking agent waits only on its own process, its machine already
- * up. The relay says `worker_waking` either way.
- */
-export function cloudMachineReady(machine: CloudMachine | null): boolean {
- return machine?.state === 'ready';
-}
diff --git a/console/apps/switch-console-desktop/src/shared/core/managed-agents/managed-agents.ts b/console/apps/switch-console-desktop/src/shared/core/managed-agents/managed-agents.ts
index 53ff4fc53..c689ef301 100644
--- a/console/apps/switch-console-desktop/src/shared/core/managed-agents/managed-agents.ts
+++ b/console/apps/switch-console-desktop/src/shared/core/managed-agents/managed-agents.ts
@@ -53,6 +53,11 @@ export type ManagedMachine = {
state: 'online' | 'offline' | 'unknown' | 'revoked';
};
+/** Whether the machine is the owner's Switch cloud machine: Switch enrolls its controller as kind `ec2`. */
+export function isCloudMachine(machine: Pick<ManagedMachine, 'kind'>): boolean {
+ return machine.kind === 'ec2';
+}
+
/** A provider on a machine, as the machine last reported it. */
export type MachineProvider = {
/** The Switch definition provider id (`claude`, `codex`, …). */
@@ -63,6 +68,20 @@ export type MachineProvider = {
problem: string | null;
};
+/**
+ * A provider login to give a machine: a key or token typed in, or this
+ * computer's own sign-in for the provider, read from its file.
+ */
+export type MachineLoginInput =
+ | { source: 'typed'; kind: 'api-key' | 'setup-token'; credential: string }
+ | { source: 'this-computer' };
+
+/** How giving a machine a login went: still being taken up, taken up, or why not. */
+export type MachineLoginOutcome =
+ | { state: 'pending' }
+ | { state: 'succeeded' }
+ | { state: 'failed'; code: string; message: string };
+
/** What a machine is to this Console: this computer, one of its SSH hosts, or neither. */
export type MachineLocal = { kind: 'this-computer' } | { kind: 'ssh-host'; sshHost: string } | null;
@@ -70,6 +89,10 @@ export type MachineLocal = { kind: 'this-computer' } | { kind: 'ssh-host'; sshHo
export type OwnedMachine = ManagedMachine & {
/** Empty before the machine has reported. */
providers: MachineProvider[];
+ /** It can be given a provider login sealed to its key: its controller registered one. */
+ acceptsLogins: boolean;
+ /** The owner's Switch cloud machine, offered as "Switch cloud" rather than by name. */
+ cloud: boolean;
/**
* Where the machine makes agents' workspaces (an agent's is `<workspacesDir>/<name>`);
* null before it has said.
diff --git a/console/apps/switch-console-desktop/src/shared/core/switch-servers/add-server-steps.ts b/console/apps/switch-console-desktop/src/shared/core/switch-servers/add-server-steps.ts
index b2f6ebb76..e1f86201f 100644
--- a/console/apps/switch-console-desktop/src/shared/core/switch-servers/add-server-steps.ts
+++ b/console/apps/switch-console-desktop/src/shared/core/switch-servers/add-server-steps.ts
@@ -7,8 +7,6 @@
*/
export const ADD_SERVER_STEPS = [
'choose',
- 'managedReady',
- 'managedClaude',
'managedGitHub',
'managedAgent',
'local',
diff --git a/console/apps/switch-console-desktop/src/shared/core/switch-servers/claude-credential.ts b/console/apps/switch-console-desktop/src/shared/core/switch-servers/claude-credential.ts
deleted file mode 100644
index 5792ca152..000000000
--- a/console/apps/switch-console-desktop/src/shared/core/switch-servers/claude-credential.ts
+++ /dev/null
@@ -1,25 +0,0 @@
-export type ClaudeCredentialKind = 'api-key' | 'setup-token';
-
-export function validateClaudeCredential(kind: ClaudeCredentialKind, value: string): string {
- if (kind !== 'api-key' && kind !== 'setup-token') {
- throw new Error('Choose API key or subscription.');
- }
- const credential = value.trim();
- if (!credential || credential.length > 16 * 1024 || /[^\x21-\x7e]/.test(credential)) {
- throw new Error('Paste a single credential without spaces or line breaks.');
- }
- if (kind === 'api-key' && credential.startsWith('sk-ant-oat')) {
- throw new Error('This looks like a setup token. Choose Subscription instead.');
- }
- if (kind === 'setup-token' && !credential.startsWith('sk-ant-oat')) {
- throw new Error('Paste the setup token printed by claude setup-token.');
- }
- if (kind === 'api-key' && !credential.startsWith('sk-ant-api')) {
- throw new Error('Paste an API key created in Claude Console.');
- }
- return credential;
-}
-
-export type ClaudeConnection =
- | { status: 'not_connected' }
- | { status: 'connected'; kind: ClaudeCredentialKind; verified_at: string };
diff --git a/console/apps/switch-console-desktop/src/shared/core/switch-servers/cloud-launch.ts b/console/apps/switch-console-desktop/src/shared/core/switch-servers/cloud-launch.ts
deleted file mode 100644
index a7a299d38..000000000
--- a/console/apps/switch-console-desktop/src/shared/core/switch-servers/cloud-launch.ts
+++ /dev/null
@@ -1,40 +0,0 @@
-import type { RepoAgentAttributes } from '@switch-console/core/agents/plugins';
-import type { AgentProviderId } from '@shared/core/providers/agent-provider-registry';
-import type { AddressingPolicy } from './switch-servers';
-
-export type CloudLaunchInput = {
- provider: AgentProviderId;
- request_id: string;
- name: string;
- description: string;
- display_name: string | null;
- icon_url: string | null;
- instructions: string;
- installation_id: number;
- repository_id: number;
- definition_attributes: RepoAgentAttributes;
- auto_session: boolean;
- auto_approve: boolean;
- addressing_policy: AddressingPolicy | null;
-};
-
-/** What a cloud agent's next start runs, as Core stores it on the launch. */
-export type CloudLaunchConfiguration = {
- description: string;
- instructions: string;
- definition_attributes: RepoAgentAttributes;
-};
-
-/** An edit to a launch's configuration, with what its definition is rendered from. */
-export type CloudConfigurationInput = {
- provider: AgentProviderId;
- name: string;
- description: string;
- instructions: string;
- definition_attributes: RepoAgentAttributes;
-};
-
-export type CloudRepositorySelection = {
- installationId: number;
- repositoryId: number;
-};
diff --git a/console/apps/switch-console-desktop/src/shared/core/switch-servers/provider-credential.ts b/console/apps/switch-console-desktop/src/shared/core/switch-servers/provider-credential.ts
deleted file mode 100644
index 60deb1c6e..000000000
--- a/console/apps/switch-console-desktop/src/shared/core/switch-servers/provider-credential.ts
+++ /dev/null
@@ -1,17 +0,0 @@
-import { z } from 'zod';
-export const cloudProviderConnectionSchema = z.discriminatedUnion('status', [
- z.object({ status: z.literal('not_connected') }),
- z.object({
- status: z.enum(['verifying', 'failed']),
- kind: z.enum(['api-key', 'setup-token', 'auth-json']),
- verification_id: z.string(),
- created_at: z.string(),
- error: z.string().nullable(),
- }),
- z.object({
- status: z.enum(['connected', 'configured']),
- kind: z.enum(['api-key', 'setup-token', 'auth-json']),
- verified_at: z.string(),
- }),
-]);
-export type CloudProviderConnection = z.infer<typeof cloudProviderConnectionSchema>;
diff --git a/console/apps/switch-console-desktop/src/shared/core/views/view-ids.ts b/console/apps/switch-console-desktop/src/shared/core/views/view-ids.ts
index 55708009f..be6242260 100644
--- a/console/apps/switch-console-desktop/src/shared/core/views/view-ids.ts
+++ b/console/apps/switch-console-desktop/src/shared/core/views/view-ids.ts
@@ -11,7 +11,6 @@ export const VIEW_IDS = [
'home',
'location',
'session',
- 'cloudSession',
'managedAgent',
'room',
'settings',
diff --git a/console/packages/agent-controller/README.md b/console/packages/agent-controller/README.md
index 9a198fdad..597441ebb 100644
--- a/console/packages/agent-controller/README.md
+++ b/console/packages/agent-controller/README.md
@@ -20,6 +20,10 @@ same one in-process agent hosts use, over a WebSocket on the relay's port, and i
not stopped when the controller exits: it reconnects when the controller is back.
Changing an agent's isolation restarts it the other way.
+On Linux, a controller set up for it runs every agent, whatever its isolation, as
+a Linux user of its own under systemd, seeing only its own directory: see
+[Each agent as a Linux user of its own](#each-agent-as-a-linux-user-of-its-own).
+
The controller holds **one** connection to Switch, a WebSocket, for all of its agents and hands
each shared agent's events to its agent host directly. Each agent host makes its calls to Switch
through a **local relay** on a loopback port, which adds the controller's
@@ -176,6 +180,107 @@ model overrides, and the Vertex AI, Bedrock and Google Cloud settings), plus the
basics a CLI needs. The file is read once at start; restart the service after
changing it.
+### Each agent as a Linux user of its own
+
+By default every agent runs as the user the controller runs as, so an agent can
+read the controller's files, its credential and the other agents' files. On Linux
+with systemd and polkit, each agent can instead run as a Linux user of its own,
+which sees only its own directory. Set it up once, as root, after enrolling:
+
+```bash
+sudo npm install --global <the release's .tgz URL> # Node and the controller system-wide
+switch-agent-controller enroll --server https://switch.example.com --code <code>
+sudo switch-agent-controller install-service --separate-users [--env-file /etc/switch/agents.env]
+```
+
+`install-service --separate-users` runs as root and sets up, for the user who ran
+`sudo` (or `--user <name>`) and the data directory (`--data-dir`, by default that
+user's default one):
+
+- a pool of system users, `sa<uid>-01` … (16 by default, `--agent-users <n>` up to
+ 99), all in a group of their own, `switch-agents-<uid>`;
+- a systemd template unit, `switch-agent-<uid>@.service`, that runs an agent as
+ one of them;
+- a polkit rule that lets the controller's user start, stop, restart and reset
+ those units and nothing else, so the controller itself never needs root;
+- the agents' directories, in `/var/lib/switch-agents/<uid>` (`--agents-dir`);
+- the controller as a system service of that user,
+ `switch-agent-controller-<uid>.service`, run with
+ `--agent-runtime separate-user` and the agents' group as a supplementary group,
+ so it can read what its agents write. It replaces the user service, which must
+ be uninstalled first.
+
+The setup is written to `/etc/switch-agent-controller/separate-users-<uid>.json`.
+Running the setup again rewrites everything and adds agent users. `sudo
+switch-agent-controller uninstall-service --separate-users` stops the controller
+and its agents, then removes what the setup made except the agents' directories.
+
+Each agent claims a free user from the pool when it is first started, and runs as
+`switch-agent-<uid>@<NN>.service`. systemd restarts an agent host that crashes. The
+agent:
+
+- sees only its own directory, always at `/var/lib/switch-agents/<uid>/agent`,
+ whichever user it runs as, with its `home` and its `workspace` in it. The
+ controller reports that path as its workspaces folder, so an agent created
+ with no directory works in a folder named after it there. A definition naming
+ a directory outside it is refused (`definition_invalid`);
+- cannot see the home directories, the controller's data directory, the other
+ agents' processes, or the cloud instance metadata address, and the rest of the
+ system is read-only to it;
+- gets its relay credentials and the provider settings from the controller's
+ environment (the `--env-file`) through systemd, never the controller user's own
+ provider logins in its home. Give each provider an API key or a token, for
+ example `CLAUDE_CODE_OAUTH_TOKEN` from `claude setup-token`, or give the
+ machine a login from Console (see
+ [Provider logins given to the machine](#provider-logins-given-to-the-machine)).
+ `doctor` checks the providers the way an agent would see them.
+
+Node, the controller and the provider CLIs must be installed outside the home
+directories (`/usr`, `/usr/local` or `/opt`), where the agents can reach them; the
+setup and the controller refuse what is not. An agent removed from the machine
+frees its user, and its directory is moved to `released/<agent id>` under the
+agents' directory, where it is picked up again if the agent comes back. Before an
+agent starts, root hands it back the files in its directory that another agent
+user owns, so an agent can move from one user to another. Only files in the
+agents' group are handed back.
+
+When every user in the pool runs an agent, the next agent fails with
+`capacity_exceeded` until the setup is run again with a larger `--agent-users`.
+Agents' units keep running when the controller stops, as isolated agents do.
+`status` run in a shell without the agents' group shows each agent's unit state
+only.
+
+### Provider logins given to the machine
+
+A machine with no login of its own for a provider (a cloud machine, a fresh SSH
+host) can be given one by its owner, on demand, at any time after it is up.
+`enroll` makes an X25519 keypair, keeps the private half in the secret store
+(`sealing-key`) and sends the public half to Switch. A controller enrolled
+before this registers its key once, when it next runs. Console seals the login
+to that key before sending it, so Switch stores and relays only ciphertext it
+cannot open.
+
+Switch then queues a `provider.login` operation. The controller fetches the
+login, opens it, checks the provider signs in with it, and reports the result.
+Console shows that result: the operation fails with `provider_login_expired`
+when the provider does not sign in with the login, and `provider_login_missing`
+when there is none. From then on the provider is reported ready with
+`auth_source: "sealed"`.
+
+- The machine's own login comes first. A given login is used only for a
+ provider the machine has none for, on `PATH` or in the `--env-file`.
+- An agent using a given login gets it with its relay credentials. Its token
+ (Claude, Cursor) goes in its environment, and its file (Codex, OpenCode,
+ Antigravity) is written by the agent host in its own state. The agent runs in
+ a process of its own, as a separate user under `--separate-users`, and is
+ restarted when the login changes or is withdrawn.
+- Logins are kept in memory only, and fetched again every ten minutes and on
+ each `provider.login` and `provider.recheck`. When Switch cannot be reached,
+ the login already held is kept.
+- A controller whose credential is handed over (`--credential-stdin`) keeps no
+ key across starts, so it registers none and uses only the machine's own
+ logins.
+
### The controller credential
`enroll --secret-store` says where the controller credential is kept, and the data
diff --git a/console/packages/agent-controller/src/agent-hub.test.ts b/console/packages/agent-controller/src/agent-hub.test.ts
index 8db7dd536..d11a11ec0 100644
--- a/console/packages/agent-controller/src/agent-hub.test.ts
+++ b/console/packages/agent-controller/src/agent-hub.test.ts
@@ -61,7 +61,6 @@ function watcher(overrides: Partial<SwitchEventStreamDeps> = {}) {
const stream = hub.open(AGENT, {
creds: { agentId: AGENT, apiEndpoint: 'http://127.0.0.1:1', token: 'swlr_x' },
connectionId: 'connection-1',
- worker: null,
scope: 'all',
filter: 'addressed',
spawnCapable: true,
diff --git a/console/packages/agent-controller/src/agent-hub.ts b/console/packages/agent-controller/src/agent-hub.ts
index 0d3877880..b0270850b 100644
--- a/console/packages/agent-controller/src/agent-hub.ts
+++ b/console/packages/agent-controller/src/agent-hub.ts
@@ -423,8 +423,6 @@ export class AgentHub {
agent.placements = new Map(Object.entries(placements));
this.deps.onChange();
},
- workerCall: () =>
- Promise.reject(new Error('An agent run by an agents controller is not a hosted worker.')),
};
}
diff --git a/console/packages/agent-controller/src/api.ts b/console/packages/agent-controller/src/api.ts
index dd1ca59b1..09f5c97a8 100644
--- a/console/packages/agent-controller/src/api.ts
+++ b/console/packages/agent-controller/src/api.ts
@@ -20,6 +20,9 @@ import {
operationListSchema,
type OperationResult,
operationSchema,
+ type Provider,
+ type SealedLoginResponse,
+ sealedLoginResponseSchema,
PROTOCOL_VERSION,
type StatusReport,
type StatusResponse,
@@ -397,6 +400,33 @@ export class ControllerClient {
return parsed(response, controllerInfoResponseSchema);
}
+ /**
+ * Registers the key this machine's provider logins are sealed to. A server
+ * already holding another key for it refuses (409): the machine must be
+ * enrolled again to get a new one.
+ */
+ async registerPublicKey(publicKey: string): Promise<void> {
+ await this.request(this.controllerPath, {
+ method: 'PATCH',
+ body: { public_key: { alg: 'X25519', key: publicKey } },
+ });
+ }
+
+ /** The provider login given to this machine, still sealed; null when none was given. */
+ async sealedLogin(provider: Provider): Promise<SealedLoginResponse | null> {
+ const response = await this.request(
+ `${this.controllerPath}/provider-credentials/${encodeURIComponent(provider)}`,
+ { method: 'GET' },
+ (status) => status === 200 || status === 404
+ );
+ if (response.status === 404) {
+ const error = await failure(response);
+ if (error.code === 'provider_login_missing') return null;
+ throw error;
+ }
+ return parsed(response, sealedLoginResponseSchema);
+ }
+
async rotateCredential(): Promise<string> {
const response = await this.request(`${this.controllerPath}/credential/rotate`, {
method: 'POST',
diff --git a/console/packages/agent-controller/src/controller.test.ts b/console/packages/agent-controller/src/controller.test.ts
index 517e1e188..56e3ef156 100644
--- a/console/packages/agent-controller/src/controller.test.ts
+++ b/console/packages/agent-controller/src/controller.test.ts
@@ -5,12 +5,18 @@ import { join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
import type { AgentBridgeEvent } from '@sandboxaq/switch-agent-runtime';
import { callOperation, SESSION_SELECTOR_HEADERS } from '@sandboxaq/switch-agent-runtime/hosted';
-import { openHubStream } from '@switch-console/agent-providers';
+import { openHubStream, sealProviderLogin } from '@switch-console/agent-providers';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
-import { type ControllerDeps, type ControllerExit, runController } from './controller';
+import {
+ type ControllerDeps,
+ type ControllerExit,
+ ensureRelayCredentials,
+ runController,
+} from './controller';
import { ConfigurationError } from './errors';
import { adoptIdentity } from './handover';
import { silentLogger } from './log';
+import type { LocalRelay } from './relay';
import type { AgentAssignment, StatusReport } from './schemas';
import { CONTROLLER_CREDENTIAL, FileSecretStore, MemorySecretStore } from './secrets';
import { ControllerStore } from './store';
@@ -101,7 +107,6 @@ function watcher(agentId = 'agent-1') {
const stream = runtime.openStream!(agentId)({
creds: { agentId, apiEndpoint: credentials.endpoint, token: credentials.token },
connectionId: `controller-${agentId}`,
- worker: null,
scope: 'all',
filter: 'addressed',
spawnCapable: true,
@@ -258,7 +263,6 @@ describe('runController', () => {
openHubStream(credentials.hub)({
creds: { agentId: 'agent-1', apiEndpoint: credentials.endpoint, token: credentials.token },
connectionId: 'isolated-host',
- worker: null,
scope: 'all',
filter: 'addressed',
rooms: [],
@@ -345,6 +349,7 @@ describe('runController', () => {
endpoint: `http://127.0.0.1:${port}`,
token: 'swlr_old',
hub: '',
+ providerLogin: null,
});
runtime.calls.length = 0;
running = runController(deps(), stop.signal);
@@ -377,6 +382,7 @@ describe('runController', () => {
endpoint: `http://127.0.0.1:${port}`,
token: 'swlr_old',
hub: `ws://127.0.0.1:${port}/hub`,
+ providerLogin: null,
});
runtime.calls.length = 0;
running = runController(deps(), stop.signal);
@@ -459,6 +465,63 @@ describe('runController', () => {
expect(await running).toBe('stopped');
});
+ it('takes up a login given to the machine on demand, and moves its agent onto it', async () => {
+ runtime.readiness = { status: 'unauthenticated', message: 'Not signed in.', models: [] };
+ core.setAssignment({ revision: 1, agents: [agent(1)] });
+ running = runController(deps(), stop.signal);
+ await waitFor(() => runtime.launches('agent-1').length === 1, 'the first start');
+ await waitFor(() => core.publicKey !== null, 'the machine registering its key');
+ expect(runtime.launches('agent-1')[0]!.options.isolation).toBe('shared');
+ expect(runtime.credentials.get('agent-1')?.providerLogin).toBeNull();
+
+ core.sealedLogins.set('claude', {
+ revision: 1,
+ sealed: sealProviderLogin({
+ publicKey: core.publicKey!,
+ controllerId: core.controllerId,
+ provider: 'claude',
+ login: { kind: 'setup-token', credential: 'sk-ant-oat-given' },
+ }),
+ });
+ core.addOperation({
+ id: 'op-give',
+ kind: 'provider.login',
+ agent_id: null,
+ params: { provider: 'claude', method: 'sealed', revision: 1 },
+ created_at: '2026-01-01T00:00:00Z',
+ });
+ core.push('operation.pending', {
+ operation_id: 'op-give',
+ kind: 'provider.login',
+ agent_id: null,
+ });
+ await waitFor(() => core.results.has('op-give'), 'the login taken up');
+ expect(core.results.get('op-give')).toMatchObject({
+ outcome: 'succeeded',
+ output: { provider: { auth: 'ok', auth_source: 'sealed' } },
+ });
+ await waitFor(() => runtime.launches('agent-1').length === 2, 'the agent moved onto it');
+ const moved = runtime.launches('agent-1')[1]!;
+ expect(moved.options).toMatchObject({ isolation: 'isolated', restart: true });
+ expect(moved.template.start.input.env).toMatchObject({
+ CLAUDE_CODE_OAUTH_TOKEN: 'sk-ant-oat-given',
+ });
+ expect(runtime.credentials.get('agent-1')?.providerLogin).toMatchObject({
+ provider: 'claude',
+ revision: '1',
+ });
+ await waitFor(
+ () =>
+ core.statusReports.some(
+ (report) =>
+ report.providers.find((entry) => entry.provider === 'claude')?.auth_source === 'sealed'
+ ),
+ 'a status report saying Claude is ready through the given login'
+ );
+ stop.abort();
+ expect(await running).toBe('stopped');
+ });
+
it('resyncs after the stream drops and reconnects', async () => {
core.setAssignment({ revision: 1, agents: [agent(1)] });
running = runController(deps(), stop.signal);
@@ -477,6 +540,7 @@ describe('runController', () => {
endpoint: 'http://127.0.0.1:1',
token: 'swlr_k',
hub: 'ws://127.0.0.1:1/hub',
+ providerLogin: null,
});
runtime.calls.length = 0;
await core.stop();
@@ -567,3 +631,61 @@ describe('runController', () => {
await expect(wiped).rejects.toBeInstanceOf(ConfigurationError);
});
});
+
+describe('ensureRelayCredentials', () => {
+ const login = {
+ status: 'connected' as const,
+ provider: 'claude' as const,
+ revision: '1',
+ kind: 'setup-token' as const,
+ credential: 'sk-ant-oat-given',
+ };
+
+ function fakeRelay() {
+ const registered = new Map<string, string>();
+ let minted = 0;
+ return {
+ registered,
+ relay: {
+ endpoint: 'http://127.0.0.1:43210',
+ hubUrl: 'ws://127.0.0.1:43210/hub',
+ isRegistered: (agentId: string, token: string) => registered.get(agentId) === token,
+ register: (agentId: string, token: string) => void registered.set(agentId, token),
+ mint: (agentId: string) => {
+ const token = `swlr_${++minted}`;
+ registered.set(agentId, token);
+ return token;
+ },
+ } as unknown as LocalRelay,
+ };
+ }
+
+ it('rewrites the file, keeping its token, when the given login changes', async () => {
+ const runtime = new FakeRuntime();
+ const { relay } = fakeRelay();
+ const deps = { runtime, relay, log: silentLogger };
+ expect(await ensureRelayCredentials('agent-1', null, deps)).toBe(true);
+ expect(await ensureRelayCredentials('agent-1', null, deps)).toBe(false);
+ expect(await ensureRelayCredentials('agent-1', login, deps)).toBe(true);
+ expect(runtime.credentials.get('agent-1')).toMatchObject({
+ token: 'swlr_1',
+ providerLogin: login,
+ });
+ expect(await ensureRelayCredentials('agent-1', login, deps)).toBe(false);
+ // The same login with its fields in another order is the same login.
+ const reordered = {
+ credential: login.credential,
+ kind: login.kind,
+ revision: login.revision,
+ provider: login.provider,
+ status: login.status,
+ };
+ expect(await ensureRelayCredentials('agent-1', reordered, deps)).toBe(false);
+ expect(await ensureRelayCredentials('agent-1', { ...login, revision: '2' }, deps)).toBe(true);
+ expect(await ensureRelayCredentials('agent-1', null, deps)).toBe(true);
+ expect(runtime.credentials.get('agent-1')).toMatchObject({
+ token: 'swlr_1',
+ providerLogin: null,
+ });
+ });
+});
diff --git a/console/packages/agent-controller/src/controller.ts b/console/packages/agent-controller/src/controller.ts
index edf830ef5..27c4dbcb9 100644
--- a/console/packages/agent-controller/src/controller.ts
+++ b/console/packages/agent-controller/src/controller.ts
@@ -21,9 +21,11 @@ import type { AgentRuntime } from './runtime';
import {
type AgentCursor,
type Assignment,
+ isProvider,
type StatusReport,
statusReportSchema,
} from './schemas';
+import { type GivenLogin, registerSealingKey, SealedLogins, sealingKeys } from './sealed-logins';
import { CONTROLLER_CREDENTIAL, type SecretStore } from './secrets';
import {
type ProviderLocator,
@@ -113,28 +115,48 @@ export type ControllerExit =
* which a running agent host only reads when it starts, so it is restarted:
* that is also what moves an agent host from before the hub onto it.
*/
+function sameLogin(a: GivenLogin | null, b: GivenLogin | null): boolean {
+ if (a === null || b === null) return a === b;
+ return (
+ a.provider === b.provider &&
+ a.revision === b.revision &&
+ a.kind === b.kind &&
+ a.credential === b.credential
+ );
+}
+
export async function ensureRelayCredentials(
agentId: string,
+ providerLogin: GivenLogin | null,
deps: { runtime: AgentRuntime; relay: LocalRelay; log: Logger }
): Promise<boolean> {
const current = await deps.runtime.readCredentials(agentId);
const endpoint = deps.relay.endpoint;
const hub = deps.relay.hubUrl;
- if (
- current &&
+ const relayCurrent =
+ current !== null &&
current.endpoint === endpoint &&
current.hub === hub &&
- current.token.startsWith(RELAY_TOKEN_PREFIX)
- ) {
+ current.token.startsWith(RELAY_TOKEN_PREFIX);
+ const loginCurrent = sameLogin(current?.providerLogin ?? null, providerLogin);
+ if (current && relayCurrent && loginCurrent) {
if (!deps.relay.isRegistered(agentId, current.token))
deps.relay.register(agentId, current.token);
return false;
}
- const token = deps.relay.mint(agentId);
- await deps.runtime.writeCredentials(agentId, { endpoint, token, hub });
+ const token = current && relayCurrent ? current.token : deps.relay.mint(agentId);
+ if (current && relayCurrent && !deps.relay.isRegistered(agentId, token))
+ deps.relay.register(agentId, token);
+ await deps.runtime.writeCredentials(agentId, { endpoint, token, hub, providerLogin });
deps.log.info('Wrote the agent’s relay credentials', {
agentId,
- why: current ? 'they named another endpoint, hub or a Switch key' : 'it had none',
+ why: !current
+ ? 'it had none'
+ : !relayCurrent
+ ? 'they named another endpoint, hub or a Switch key'
+ : providerLogin
+ ? `the ${providerLogin.provider} login given to this machine changed`
+ : 'the login given to this machine is no longer used',
});
return true;
}
@@ -332,9 +354,22 @@ export async function runController(
});
};
+ const keys = await sealingKeys(deps.secrets);
+ const sealed = keys
+ ? new SealedLogins({ client, keys, controllerId: identity.controllerId })
+ : null;
+ if (!keys)
+ log.info(
+ 'This controller keeps no key across starts (its credential is handed over), so provider logins given to the machine are not used; its own are.'
+ );
+ else
+ void registerSealingKey(client, keys, log).then((registered) => {
+ if (!registered) providers.disableGiven();
+ });
const providers = new ProviderStatuses({
locator: deps.locator,
runtime,
+ sealed,
probeCwd: deps.dataDir,
now: deps.now,
log,
@@ -353,7 +388,13 @@ export async function runController(
const reconcileDeps: ReconcileDeps = {
store,
runtime,
- ensureCredentials: (agentId) => ensureRelayCredentials(agentId, { runtime, relay, log }),
+ ensureCredentials: async (agentId, provider) =>
+ ensureRelayCredentials(
+ agentId,
+ isProvider(provider) ? await providers.givenLogin(provider) : null,
+ { runtime, relay, log }
+ ),
+ givenLogin: (provider) => providers.givenLogin(provider),
forgetAgent: (agentId) => {
relay.unregister(agentId);
hub.forget(agentId);
@@ -418,8 +459,16 @@ export async function runController(
recheckProvider: async (provider) => {
const status = await providers.check(provider);
await reporter?.sendNow();
+ // Agents move onto a login given since, or off one withdrawn.
+ void reconcileLocally();
return status;
},
+ takeUpLogin: async (provider) => {
+ const status = await providers.check(provider);
+ await reporter?.sendNow();
+ void reconcileLocally();
+ return { status, problem: providers.loginProblem(provider) };
+ },
log,
});
if (ran) reporter?.request();
diff --git a/console/packages/agent-controller/src/doctor.test.ts b/console/packages/agent-controller/src/doctor.test.ts
index 9378dee21..9bbced521 100644
--- a/console/packages/agent-controller/src/doctor.test.ts
+++ b/console/packages/agent-controller/src/doctor.test.ts
@@ -24,6 +24,7 @@ function inputs(overrides: Partial<DoctorInputs> = {}): DoctorInputs {
probe: async () => ({ status: 'authenticated', message: 'Signed in.', models: [] }),
service: async () => 'running',
latest: async () => ({ version: '0.2.0', packageUrl: 'https://example.invalid/p.tgz' }),
+ separateUsers: null,
...overrides,
};
}
diff --git a/console/packages/agent-controller/src/doctor.ts b/console/packages/agent-controller/src/doctor.ts
index 36d11a2ea..e6c5da0ad 100644
--- a/console/packages/agent-controller/src/doctor.ts
+++ b/console/packages/agent-controller/src/doctor.ts
@@ -31,6 +31,8 @@ export type DoctorInputs = {
probe: (bundle: string, provider: Provider, binary: string) => Promise<ProviderReadiness>;
service: () => Promise<ServiceState>;
latest: () => Promise<ControllerRelease | null>;
+ /** The setup that runs every agent as a user of its own, or null when agents run as this user. */
+ separateUsers: { agentsDir: string; agentUsers: number; controllerUnit: string } | null;
};
function nodeCheck(nodeVersion: string): Check {
@@ -164,13 +166,26 @@ export async function runDoctor(inputs: DoctorInputs): Promise<Check[]> {
} catch (error) {
checks.push({ name: 'Shared host', status: 'fail', detail: errorMessage(error) });
}
+ if (inputs.separateUsers)
+ checks.push({
+ name: 'Agent users',
+ status: 'ok',
+ detail: `Each agent runs as one of ${inputs.separateUsers.agentUsers} users of its own, in ${inputs.separateUsers.agentsDir}; a provider is ready only through what the --env-file gives agents.`,
+ });
checks.push(...(await providerChecks(inputs, bundle)));
const service = await inputs.service().catch((error: unknown) => errorMessage(error));
+ const unit = inputs.separateUsers?.controllerUnit;
checks.push(
service === 'running'
? { name: 'Service', status: 'ok', detail: 'Installed and running' }
: service === 'stopped'
- ? { name: 'Service', status: 'warn', detail: 'Installed, but not running' }
+ ? {
+ name: 'Service',
+ status: 'warn',
+ detail: unit
+ ? `${unit} is not running: sudo systemctl start ${unit}`
+ : 'Installed, but not running',
+ }
: service === 'not-installed'
? {
name: 'Service',
diff --git a/console/packages/agent-controller/src/hub-socket.ts b/console/packages/agent-controller/src/hub-socket.ts
index 113440703..e833bd295 100644
--- a/console/packages/agent-controller/src/hub-socket.ts
+++ b/console/packages/agent-controller/src/hub-socket.ts
@@ -116,7 +116,6 @@ export class HubSocket {
filter: 'addressed',
rooms: [],
...(message.startCursor === null ? {} : { startCursor: message.startCursor }),
- worker: null,
log,
signal: lifetime.signal,
onEvent: (event) => request((id) => ({ type: 'event', id, event })),
diff --git a/console/packages/agent-controller/src/main.ts b/console/packages/agent-controller/src/main.ts
index 39125c7e1..70419e032 100644
--- a/console/packages/agent-controller/src/main.ts
+++ b/console/packages/agent-controller/src/main.ts
@@ -3,6 +3,7 @@ import { realpathSync } from 'node:fs';
import { hostname } from 'node:os';
import { resolve } from 'node:path';
import { parseArgs } from 'node:util';
+import { generateSealingKeyPair } from '@switch-console/agent-providers';
import packageJson from '../package.json' with { type: 'json' };
import {
AccessTokens,
@@ -53,9 +54,22 @@ import {
isSecretStoreKind,
MemorySecretStore,
runCommand as runSecretCommand,
+ SEALING_KEY,
SECRET_STORE_KINDS,
secretStoreFor,
} from './secrets';
+import {
+ assertControllerCanRunSeparateUsers,
+ DEFAULT_AGENT_USERS,
+ findSeparateUsersConfig,
+ installSeparateUsers,
+ loadSeparateUsersConfig,
+ MAX_AGENT_USERS,
+ type SeparateUsersConfig,
+ separateUserNames,
+ uninstallSeparateUsers,
+ unitAgentRoot,
+} from './separate-users';
import {
installService,
LAUNCHD_FINAL_EXIT_CODES,
@@ -65,6 +79,7 @@ import {
} from './service';
import { contractPlatform, mapAgentProcess, PathProviderLocator } from './status';
import { ControllerStore } from './store';
+import { SystemdRuntime, systemctl } from './systemd-runtime';
import { installPrefix, isNewer, latestRelease, releasesRepository } from './update';
export const VERSION: string = packageJson.version;
@@ -84,8 +99,10 @@ Commands:
at most 500 characters, editable later in the gateway).
run [--data-dir <dir>] [--shared-host-bundle <path>]
[--controller-id <id> --server <agent-bridge-url> [--name <name>]]
- [--credential-stdin]
+ [--credential-stdin] [--agent-runtime default|separate-user]
Run the agents assigned to this machine and report their status.
+ --agent-runtime separate-user runs every agent as a Linux user of its
+ own, as set up by install-service --separate-users.
--controller-id and --server adopt an identity enrolled elsewhere when
the data directory holds none, and move the same identity to a new
server URL when it holds that one. --credential-stdin reads the
@@ -103,6 +120,19 @@ Commands:
that may pass. It keeps the PATH of the shell that installed it.
uninstall-service [--data-dir <dir>]
Stop the service and remove it. The enrollment and data are kept.
+ install-service --separate-users [--user <name>] [--data-dir <dir>]
+ [--env-file <path>] [--agents-dir <dir>] [--agent-users <n>] [--no-block]
+ As root, once (Linux with systemd and polkit): run every agent as a
+ Linux user of its own, seeing only its own directory. Makes <n> agent
+ users (default 16), a unit template for them, a polkit rule that lets
+ the controller start only those units, and runs the controller as a
+ system service of --user (default: the user who ran sudo). Node and the
+ controller must be installed system-wide. Run it again to change it.
+ --no-block queues the controller's start rather than waiting for it,
+ for a service the controller's unit is ordered after.
+ uninstall-service --separate-users [--user <name>]
+ As root: stop the controller and its agents, and remove what the setup
+ made. The agents' directories are kept.
doctor [--data-dir <dir>] [--shared-host-bundle <path>]
Check this machine can run agents: Node, enrollment, the credential, the
server, the provider CLIs and their sign-in, the service, and updates.
@@ -191,8 +221,12 @@ async function enrollCommand(args: string[]): Promise<number> {
throw new ConfigurationError(
`${dataDir} already belongs to controller ${existing.controllerId} on ${existing.server}. Use another --data-dir, or remove that directory to enroll this machine afresh.`
);
+ // Provider logins given to this machine are sealed to this key; only the
+ // machine holds the private half.
+ const keys = generateSealingKeyPair();
const enrolled = await enroll(fetch, server, {
proof: { kind: 'enrollment_code', code: values.code },
+ public_key: { alg: 'X25519', key: keys.publicKey },
controller: {
kind: 'daemon',
name,
@@ -202,6 +236,7 @@ async function enrollCommand(args: string[]): Promise<number> {
},
});
await secrets.set(CONTROLLER_CREDENTIAL, enrolled.credential);
+ await secrets.set(SEALING_KEY, JSON.stringify(keys));
store.saveSecretStoreKind(secretStore);
store.saveIdentity({
controllerId: enrolled.controller_id,
@@ -308,10 +343,14 @@ async function runCommand(args: string[]): Promise<number> {
'credential-stdin': { type: 'boolean' },
'env-file': { type: 'string' },
launchd: { type: 'boolean' },
+ 'agent-runtime': { type: 'string' },
},
strict: true,
});
assertSupportedPlatform(process.platform);
+ const agentRuntime = values['agent-runtime'] ?? 'default';
+ if (agentRuntime !== 'default' && agentRuntime !== 'separate-user')
+ throw new UsageError('--agent-runtime must be default or separate-user.');
// Before anything reads the environment: the agents inherit it.
const fromFile = values['env-file'] ? await readEnvFile(resolve(values['env-file'])) : {};
Object.assign(process.env, fromFile);
@@ -329,6 +368,16 @@ async function runCommand(args: string[]): Promise<number> {
: null;
const sharedHostBundle = bundlePath(values['shared-host-bundle']);
const { dataDir, layout, store, secrets: fileSecrets } = await openState(values['data-dir']);
+ let separateUsers: SeparateUsersConfig | null = null;
+ if (agentRuntime === 'separate-user') {
+ try {
+ separateUsers = await loadSeparateUsersConfig(currentUid(), dataDir);
+ assertControllerCanRunSeparateUsers(separateUsers);
+ } catch (error) {
+ store.close();
+ throw error;
+ }
+ }
const secrets =
credential === null
? fileSecrets
@@ -373,23 +422,34 @@ async function runCommand(args: string[]): Promise<number> {
store,
secrets,
runtime: (openStream, workspaces) =>
- new AgentRuntimes(
- new InProcessRuntime({
- layout,
- workspaces,
- bundlePath: sharedHostBundle,
- openStream,
- log,
- crashBackoffMs: 2_000,
- }),
- new DetachedRuntime({ layout, bundlePath: sharedHostBundle })
- ),
+ separateUsers
+ ? new SystemdRuntime({
+ config: separateUsers,
+ store,
+ systemctl,
+ env: process.env,
+ log,
+ now: Date.now,
+ })
+ : new AgentRuntimes(
+ new InProcessRuntime({
+ layout,
+ workspaces,
+ bundlePath: sharedHostBundle,
+ openStream,
+ log,
+ crashBackoffMs: 2_000,
+ }),
+ new DetachedRuntime({ layout, bundlePath: sharedHostBundle })
+ ),
locator: new PathProviderLocator(process.env.PATH),
fetch,
openWebSocket: nodeWebSocket,
log,
dataDir,
- workspacesFor: serverWorkspacesDir,
+ // Each agent sees its own directory at the same path: an agent named
+ // no directory works in a folder of its own there.
+ workspacesFor: separateUsers ? () => unitAgentRoot(separateUsers) : serverWorkspacesDir,
version: VERSION,
now: Date.now,
random: Math.random,
@@ -412,7 +472,22 @@ async function statusCommand(args: string[]): Promise<number> {
});
const { dataDir, layout, store, secrets } = await openState(values['data-dir']);
try {
+ const separateUsers = await findSeparateUsersConfig(currentUid(), dataDir);
+ const separateRuntime = separateUsers
+ ? new SystemdRuntime({
+ config: separateUsers,
+ store,
+ systemctl,
+ env: process.env,
+ log: createLogger({ level: 'error', write: (line) => process.stderr.write(line) }),
+ now: Date.now,
+ })
+ : null;
const out: string[] = [`Data directory: ${dataDir}`];
+ if (separateUsers)
+ out.push(
+ `Agents: each as a user of its own, in ${separateUsers.agentsDir} (${separateUserNames(separateUsers.uid).controllerUnit})`
+ );
const identity = store.identity();
if (!identity) {
out.push('Not enrolled.');
@@ -451,9 +526,16 @@ async function statusCommand(args: string[]): Promise<number> {
);
for (const entry of cached.assignment.agents) {
const row = store.agent(entry.agent_id);
- const observation = definitionProblem(entry)
- ? emptyObservation()
- : await observeOnDisk(layout, entry.agent_id);
+ let observation = emptyObservation();
+ let unreadable: string | null = null;
+ if (!definitionProblem(entry)) {
+ if (separateRuntime)
+ observation = await separateRuntime.observe(entry.agent_id).catch((error: unknown) => {
+ unreadable = errorMessage(error);
+ return emptyObservation();
+ });
+ else observation = await observeOnDisk(layout, entry.agent_id);
+ }
// Whether events flow is the running controller's to know; this reads only disk.
const mapped = mapAgentProcess({
assignment: entry,
@@ -466,7 +548,9 @@ async function statusCommand(args: string[]): Promise<number> {
'',
` ${entry.definition.name} (${entry.agent_id})`,
` provider ${entry.definition.provider}, desired ${entry.desired_state}, revision ${entry.revision}, applied ${row?.appliedRevision ?? 'never'}`,
- ` ${mapped.process}${mapped.reason ? ` [${mapped.reason}]` : ''}${mapped.detail ? `: ${mapped.detail}` : ''}`
+ unreadable && separateUsers
+ ? ` ${await unitSummary(separateUsers, store.agentUser(entry.agent_id))}; the rest needs the agents' group (${unreadable})`
+ : ` ${mapped.process}${mapped.reason ? ` [${mapped.reason}]` : ''}${mapped.detail ? `: ${mapped.detail}` : ''}`
);
}
process.stdout.write(`${out.join('\n')}\n`);
@@ -503,10 +587,19 @@ async function installServiceCommand(args: string[]): Promise<number> {
'data-dir': { type: 'string' },
'env-file': { type: 'string' },
'shared-host-bundle': { type: 'string' },
+ 'separate-users': { type: 'boolean' },
+ user: { type: 'string' },
+ 'agents-dir': { type: 'string' },
+ 'agent-users': { type: 'string' },
+ 'no-block': { type: 'boolean' },
},
strict: true,
});
assertSupportedPlatform(process.platform);
+ if (values['separate-users']) return installSeparateUsersCommand(values);
+ for (const flag of ['user', 'agents-dir', 'agent-users', 'no-block'] as const)
+ if (values[flag] !== undefined)
+ throw new UsageError(`--${flag} is for install-service --separate-users.`);
const envFile = values['env-file'] ? resolve(values['env-file']) : null;
// Read now, so a broken file fails here rather than in the service.
if (envFile) await readEnvFile(envFile);
@@ -537,12 +630,69 @@ async function installServiceCommand(args: string[]): Promise<number> {
return EXIT_OK;
}
+async function installSeparateUsersCommand(values: {
+ 'data-dir'?: string;
+ 'env-file'?: string;
+ 'shared-host-bundle'?: string;
+ user?: string;
+ 'agents-dir'?: string;
+ 'agent-users'?: string;
+ 'no-block'?: boolean;
+}): Promise<number> {
+ const user = values.user ?? process.env.SUDO_USER;
+ if (!user)
+ throw new UsageError(
+ 'install-service --separate-users needs --user <name>, the user the controller runs as, when not run through sudo.'
+ );
+ const agentUsers =
+ values['agent-users'] === undefined ? DEFAULT_AGENT_USERS : Number(values['agent-users']);
+ if (!Number.isInteger(agentUsers) || agentUsers < 1 || agentUsers > MAX_AGENT_USERS)
+ throw new UsageError(`--agent-users must be a whole number from 1 to ${MAX_AGENT_USERS}.`);
+ const envFile = values['env-file'] ? resolve(values['env-file']) : null;
+ if (envFile) await readEnvFile(envFile);
+ const report = await installSeparateUsers(
+ {
+ user,
+ dataDir: values['data-dir'] ? resolve(values['data-dir']) : null,
+ agentsDir: values['agents-dir'] ? resolve(values['agents-dir']) : null,
+ agentUsers,
+ node: process.execPath,
+ cli: cliPath(),
+ bundle: bundlePath(values['shared-host-bundle']),
+ envFile,
+ path: process.env.PATH ?? '/usr/local/bin:/usr/bin:/bin',
+ noBlock: values['no-block'] === true,
+ },
+ runSecretCommand
+ );
+ process.stdout.write(['Agents now run as users of their own.', ...report.notes, ''].join('\n'));
+ return EXIT_OK;
+}
+
async function uninstallServiceCommand(args: string[]): Promise<number> {
const { values } = parseArgs({
args,
- options: { 'data-dir': { type: 'string' } },
+ options: {
+ 'data-dir': { type: 'string' },
+ 'separate-users': { type: 'boolean' },
+ user: { type: 'string' },
+ },
strict: true,
});
+ if (values['separate-users']) {
+ const user = values.user ?? process.env.SUDO_USER;
+ if (!user)
+ throw new UsageError(
+ 'uninstall-service --separate-users needs --user <name> when not run through sudo.'
+ );
+ const report = await uninstallSeparateUsers(user, runSecretCommand);
+ process.stdout.write(
+ ['Removed the setup for agents as users of their own.', ...report.notes, ''].join('\n')
+ );
+ return EXIT_OK;
+ }
+ if (values.user !== undefined)
+ throw new UsageError('--user is for uninstall-service --separate-users.');
const removed = await uninstallService(resolveDataDir(values['data-dir']), runSecretCommand);
process.stdout.write(
removed ? 'Stopped and removed the service.\n' : 'No service was installed.\n'
@@ -559,6 +709,17 @@ async function doctorCommand(args: string[]): Promise<number> {
const { dataDir, store, secrets } = await openState(values['data-dir']);
const locator = new PathProviderLocator(process.env.PATH);
try {
+ const separateUsers = await findSeparateUsersConfig(currentUid(), dataDir);
+ const separateRuntime = separateUsers
+ ? new SystemdRuntime({
+ config: separateUsers,
+ store,
+ systemctl,
+ env: process.env,
+ log: createLogger({ level: 'error', write: (line) => process.stderr.write(line) }),
+ now: Date.now,
+ })
+ : null;
const checks = await runDoctor({
version: VERSION,
nodeVersion: process.version,
@@ -571,8 +732,19 @@ async function doctorCommand(args: string[]): Promise<number> {
exchange: (server, controllerId, credential) =>
exchangeToken(fetch, server, controllerId, credential),
locate: (provider) => locator.locate(provider),
- probe: (bundle, provider, binary) => probeProvider(bundle, provider, binary, dataDir),
- service: () => serviceState(dataDir, runSecretCommand),
+ probe: (bundle, provider, binary) =>
+ separateRuntime
+ ? separateRuntime.probe(provider, binary, dataDir, null)
+ : probeProvider(bundle, provider, binary, dataDir, process.env),
+ service: () =>
+ separateUsers
+ ? systemServiceState(separateUserNames(separateUsers.uid).controllerUnit)
+ : serviceState(dataDir, runSecretCommand),
+ separateUsers: separateUsers && {
+ agentsDir: separateUsers.agentsDir,
+ agentUsers: separateUsers.agentUsers,
+ controllerUnit: separateUserNames(separateUsers.uid).controllerUnit,
+ },
latest: () => latestRelease(fetch, releasesRepository(process.env)),
});
process.stdout.write(`${formatChecks(checks)}\n`);
@@ -677,6 +849,28 @@ export async function main(argv: string[]): Promise<number> {
}
}
+/** What systemd says of an agent's unit, for a shell without the agents' group. */
+async function unitSummary(config: SeparateUsersConfig, slot: number | null): Promise<string> {
+ if (slot === null) return 'no agent user claimed';
+ const unit = separateUserNames(config.uid).agentUnit(slot);
+ const state = await systemctl(['is-active', unit]).catch(
+ (error: { stdout?: string }) => error.stdout ?? 'unknown'
+ );
+ return `${unit} is ${state.trim()}`;
+}
+
+function currentUid(): number {
+ const uid = process.getuid?.();
+ if (uid === undefined) throw new ConfigurationError('This platform has no user ids.');
+ return uid;
+}
+
+/** Whether a system service is running, as a user may ask. */
+async function systemServiceState(unit: string): Promise<'running' | 'stopped'> {
+ const state = await systemctl(['is-active', unit]).catch(() => 'inactive');
+ return state.trim() === 'active' ? 'running' : 'stopped';
+}
+
/** The last line is the reason: a parent that supervises this process shows it. */
function reportFailure(error: unknown): void {
if (error instanceof UsageError || isParseArgsError(error))
diff --git a/console/packages/agent-controller/src/operations.test.ts b/console/packages/agent-controller/src/operations.test.ts
index a189dd5aa..4e153b3e9 100644
--- a/console/packages/agent-controller/src/operations.test.ts
+++ b/console/packages/agent-controller/src/operations.test.ts
@@ -65,6 +65,7 @@ function deps(overrides: Partial<OperationDeps> = {}) {
rechecked.push(provider);
return claudeStatus;
},
+ takeUpLogin: async () => ({ status: claudeStatus, problem: null }),
log: silentLogger,
...overrides,
};
@@ -121,6 +122,34 @@ describe('executeOperation', () => {
).toMatchObject({ outcome: 'failed', error: { code: 'validation_error' } });
});
+ it('takes up a login given to the machine, and says why when it cannot', async () => {
+ const login = (params: Record<string, unknown>) =>
+ operation({ kind: 'provider.login', agent_id: null, params });
+ const sealedStatus = { ...claudeStatus, auth_source: 'sealed' as const };
+ const ok = deps({ takeUpLogin: async () => ({ status: sealedStatus, problem: null }) });
+ expect(
+ await executeOperation(login({ provider: 'claude', method: 'sealed', revision: 2 }), ok.deps)
+ ).toEqual({ outcome: 'succeeded', output: { provider: sealedStatus } });
+ const expired = deps({
+ takeUpLogin: async () => ({
+ status: { ...claudeStatus, auth: 'expired', auth_source: 'sealed' },
+ problem: { code: 'provider_login_expired', message: 'It does not sign in.' },
+ }),
+ });
+ expect(
+ await executeOperation(login({ provider: 'claude', method: 'sealed' }), expired.deps)
+ ).toEqual({
+ outcome: 'failed',
+ error: { code: 'provider_login_expired', message: 'It does not sign in.' },
+ });
+ expect(
+ await executeOperation(login({ provider: 'claude', method: 'device_code' }), ok.deps)
+ ).toMatchObject({ error: { code: 'operation_unsupported' } });
+ expect(await executeOperation(login({ method: 'sealed' }), ok.deps)).toMatchObject({
+ error: { code: 'validation_error' },
+ });
+ });
+
it('answers any other kind with operation_unsupported', async () => {
const { deps: d } = deps();
for (const kind of ['agent.start', 'session.send', 'something.new'])
diff --git a/console/packages/agent-controller/src/operations.ts b/console/packages/agent-controller/src/operations.ts
index b89324d39..92d74103c 100644
--- a/console/packages/agent-controller/src/operations.ts
+++ b/console/packages/agent-controller/src/operations.ts
@@ -18,6 +18,13 @@ export type OperationDeps = {
restartAgent: (entry: AgentAssignment) => Promise<{ reason: string; detail: string } | null>;
/** Checks the provider now and sends a status report carrying the result. */
recheckProvider: (provider: Provider) => Promise<ProviderStatus>;
+ /**
+ * Takes up the login Switch gave the machine for the provider now, as a
+ * recheck does, and says why it is not used when it is not.
+ */
+ takeUpLogin: (
+ provider: Provider
+ ) => Promise<{ status: ProviderStatus; problem: { code: string; message: string } | null }>;
log: Logger;
};
@@ -25,7 +32,10 @@ function failed(code: string, message: string): OperationResult {
return { outcome: 'failed', error: { code, message } };
}
-/** Runs one claimed operation. v1 runs `agent.restart` and `provider.recheck`; anything else is refused. */
+/**
+ * Runs one claimed operation: `agent.restart`, `provider.recheck`, and
+ * `provider.login` with `method: "sealed"`; anything else is refused.
+ */
export async function executeOperation(
operation: Operation,
deps: OperationDeps
@@ -60,6 +70,25 @@ export async function executeOperation(
const status = await deps.recheckProvider(provider);
return { outcome: 'succeeded', output: { provider: status } };
}
+ case 'provider.login': {
+ const { provider, method } = operation.params;
+ if (typeof provider !== 'string' || !isProvider(provider))
+ return failed(
+ 'validation_error',
+ `provider.login needs params.provider set to a provider this controller runs; got ${JSON.stringify(provider)}.`
+ );
+ if (method !== 'sealed')
+ return failed(
+ 'operation_unsupported',
+ `This controller takes up only logins sealed to it (method "sealed"), not ${JSON.stringify(method)}.`
+ );
+ const { status, problem } = await deps.takeUpLogin(provider);
+ if (status.auth === 'ok') return { outcome: 'succeeded', output: { provider: status } };
+ return failed(
+ problem?.code ?? status.reason ?? 'provider_login_missing',
+ problem?.message ?? `The ${provider} login given to this machine is not used.`
+ );
+ }
default:
return failed(
'operation_unsupported',
diff --git a/console/packages/agent-controller/src/reconcile.test.ts b/console/packages/agent-controller/src/reconcile.test.ts
index d6e33c138..94742902a 100644
--- a/console/packages/agent-controller/src/reconcile.test.ts
+++ b/console/packages/agent-controller/src/reconcile.test.ts
@@ -6,6 +6,7 @@ import { silentLogger } from './log';
import { planReconcile, reconcile, type ReconcileDeps } from './reconcile';
import { emptyObservation } from './runtime';
import type { AgentAssignment, Assignment } from './schemas';
+import type { GivenLogin } from './sealed-logins';
import { LAUNCH_GRACE_MS } from './status';
import { ControllerStore } from './store';
import { FakeRuntime } from './testing/fake-runtime';
@@ -47,6 +48,7 @@ let forgotten: string[];
let endpoint: string;
let clock: number;
let missingProvider: boolean;
+let givenLogin: GivenLogin | null;
function deps(): ReconcileDeps {
return {
@@ -59,9 +61,11 @@ function deps(): ReconcileDeps {
endpoint,
token: `swlr_${minted.length}`,
hub: `${endpoint.replace('http', 'ws')}/hub`,
+ providerLogin: null,
});
return true;
},
+ givenLogin: async () => givenLogin,
forgetAgent: (agentId) => void forgotten.push(agentId),
binaryPath: async (provider) => (missingProvider ? null : `/usr/bin/${provider}`),
now: () => clock,
@@ -70,6 +74,7 @@ function deps(): ReconcileDeps {
}
beforeEach(() => {
+ givenLogin = null;
dir = mkdtempSync(join(tmpdir(), 'controller-reconcile-'));
store = ControllerStore.open(join(dir, 'controller.db'));
runtime = new FakeRuntime();
@@ -93,6 +98,7 @@ describe('reconcile', () => {
endpoint: RELAY,
token: 'swlr_1',
hub: `${RELAY.replace('http', 'ws')}/hub`,
+ providerLogin: null,
});
const [launch] = runtime.launches();
expect(launch!.options).toEqual({
@@ -110,6 +116,22 @@ describe('reconcile', () => {
expect(store.restartsSince('agent-1', 0)).toBe(0);
});
+ it('gives an agent the login given to the machine, in a process of its own', async () => {
+ givenLogin = {
+ status: 'connected',
+ provider: 'claude',
+ revision: '2',
+ kind: 'setup-token',
+ credential: 'sk-ant-oat-given',
+ };
+ await reconcile(assignment(agent()), deps());
+ const [launch] = runtime.launches();
+ expect(launch!.options.isolation).toBe('isolated');
+ expect(launch!.template.start.input.env).toMatchObject({
+ CLAUDE_CODE_OAUTH_TOKEN: 'sk-ant-oat-given',
+ });
+ });
+
it('does nothing more once the revision is applied and running', async () => {
await reconcile(assignment(agent()), deps());
runtime.calls.length = 0;
diff --git a/console/packages/agent-controller/src/reconcile.ts b/console/packages/agent-controller/src/reconcile.ts
index 5b8707c21..5dabc2a1d 100644
--- a/console/packages/agent-controller/src/reconcile.ts
+++ b/console/packages/agent-controller/src/reconcile.ts
@@ -1,3 +1,4 @@
+import { providerLoginEnvironment } from '@switch-console/agent-providers';
import { ReasonedError } from './errors';
import { errorMessage, type Logger } from './log';
import { isSafeSegment } from './paths';
@@ -9,6 +10,7 @@ import {
type Provider,
type ReasonCode,
} from './schemas';
+import type { GivenLogin } from './sealed-logins';
import { isCredentialFailure, LAUNCH_GRACE_MS } from './status';
import type { AgentRow, ControllerStore } from './store';
import { advancedConfigDefinitionProblem, buildWatcherTemplate } from './template';
@@ -200,7 +202,9 @@ export type ReconcileDeps = {
* Makes the agent's credentials file name the relay and a token it accepts.
* Resolves true when the file had to be (re)written.
*/
- ensureCredentials: (agentId: string) => Promise<boolean>;
+ ensureCredentials: (agentId: string, provider: string) => Promise<boolean>;
+ /** The login Switch gave the machine for `provider`, when its agents use it rather than the machine's own. */
+ givenLogin: (provider: Provider) => Promise<GivenLogin | null>;
/** The agent is no longer assigned here: the relay stops accepting its token. */
forgetAgent: (agentId: string) => void;
binaryPath: (provider: Provider) => Promise<string | null>;
@@ -231,8 +235,9 @@ export async function startAgent(
const provider = definition.provider;
if (!isProvider(provider))
throw new ReasonedError('definition_invalid', `Unknown provider '${provider}'.`);
- await deps.ensureCredentials(agentId);
- const cwd = await deps.runtime.workingDirectory(definition.name, definition.directory);
+ await deps.ensureCredentials(agentId, provider);
+ const login = await deps.givenLogin(provider);
+ const cwd = await deps.runtime.workingDirectory(agentId, definition.name, definition.directory);
const binaryPath = await deps.binaryPath(provider);
if (!binaryPath)
throw new ReasonedError(
@@ -247,6 +252,14 @@ export async function startAgent(
credentialsPath: deps.runtime.credentialsPath(agentId),
binaryPath,
});
+ // A login Switch gave the machine: its environment goes in the agent's
+ // configuration, and its files are written by the agent host, which then
+ // runs in a process of its own.
+ if (login)
+ template.start.input.env = {
+ ...template.start.input.env,
+ ...providerLoginEnvironment(deps.runtime.agentStateRoot(agentId), login),
+ };
const observation = await deps.runtime.observe(agentId);
const replaceIdentity =
action.replaceIdentity ||
@@ -254,7 +267,7 @@ export async function startAgent(
(observation.configured.provider !== provider || observation.configured.cwd !== cwd));
const restart = action.restart || replaceIdentity;
await deps.runtime.launch(agentId, template, {
- isolation: definition.isolation === 'isolated' ? 'isolated' : 'shared',
+ isolation: definition.isolation === 'isolated' || login ? 'isolated' : 'shared',
restart,
replaceIdentity,
clearTakenOver: action.clearTakenOver,
@@ -332,7 +345,8 @@ export async function reconcile(assignment: Assignment, deps: ReconcileDeps): Pr
observations.set(entry.agent_id, await deps.runtime.observe(entry.agent_id));
if (entry.desired_state !== 'running') continue;
try {
- if (await deps.ensureCredentials(entry.agent_id)) credentialsChanged.add(entry.agent_id);
+ if (await deps.ensureCredentials(entry.agent_id, entry.definition.provider))
+ credentialsChanged.add(entry.agent_id);
} catch (error) {
// Starting it writes them again, and records the failure on the agent.
deps.log.error('Could not write an agent’s relay credentials', {
diff --git a/console/packages/agent-controller/src/relay.test.ts b/console/packages/agent-controller/src/relay.test.ts
index 37d724fb3..7ed86fe35 100644
--- a/console/packages/agent-controller/src/relay.test.ts
+++ b/console/packages/agent-controller/src/relay.test.ts
@@ -157,7 +157,6 @@ function host(overrides: Partial<SwitchEventStreamDeps> = {}, as = token) {
const stream: AgentEventStream = openHubStream(relay.hubUrl)({
creds: { agentId: AGENT, apiEndpoint: relay.endpoint, token: as },
connectionId: 'host-connection',
- worker: null,
scope: 'all',
filter: 'addressed',
rooms: [],
@@ -518,7 +517,6 @@ describe('forwarding to Switch', () => {
.open('agent-2', {
creds: { agentId: 'agent-2', apiEndpoint: '', token: '' },
connectionId: 'in-process',
- worker: null,
scope: 'all',
filter: 'addressed',
rooms: [],
diff --git a/console/packages/agent-controller/src/runtime.test.ts b/console/packages/agent-controller/src/runtime.test.ts
index dc14be23f..f5a935650 100644
--- a/console/packages/agent-controller/src/runtime.test.ts
+++ b/console/packages/agent-controller/src/runtime.test.ts
@@ -32,6 +32,7 @@ const RELAY = {
endpoint: 'http://127.0.0.1:43210',
token: 'swlr_relay-token-placeholder',
hub: 'ws://127.0.0.1:43210/hub',
+ providerLogin: null,
};
const LAUNCH = {
isolation: 'shared' as const,
@@ -63,7 +64,6 @@ function openStream(agentId: string): OpenAgentStream {
start: () => {},
setSpawnCapable: () => {},
replacePlacements: async () => {},
- workerCall: () => Promise.reject(new Error('not a worker')),
};
};
}
@@ -256,7 +256,7 @@ describe('InProcessRuntime', () => {
});
it('probes a provider through the bundle', async () => {
- const readiness = await runtime.probe('claude', '/usr/bin/claude', '/tmp');
+ const readiness = await runtime.probe('claude', '/usr/bin/claude', '/tmp', null);
expect(readiness).toEqual({
status: 'authenticated',
message: '--probe claude /tmp /usr/bin/claude',
@@ -265,34 +265,38 @@ describe('InProcessRuntime', () => {
});
it('resolves working directories', async () => {
- expect(await runtime.workingDirectory('scout', null)).toBe(
+ expect(await runtime.workingDirectory('agent-1', 'scout', null)).toBe(
join(dir, 'data', 'workspaces', 'scout')
);
expect(statSync(join(dir, 'data', 'workspaces', 'scout')).isDirectory()).toBe(true);
- expect(await runtime.workingDirectory('scout', dir)).toBe(dir);
- await expect(runtime.workingDirectory('scout', 'relative/dir')).rejects.toMatchObject({
+ expect(await runtime.workingDirectory('agent-1', 'scout', dir)).toBe(dir);
+ await expect(
+ runtime.workingDirectory('agent-1', 'scout', 'relative/dir')
+ ).rejects.toMatchObject({
reason: 'definition_invalid',
});
- await expect(runtime.workingDirectory('scout', join(dir, 'missing'))).rejects.toMatchObject({
+ await expect(
+ runtime.workingDirectory('agent-1', 'scout', join(dir, 'missing'))
+ ).rejects.toMatchObject({
reason: 'definition_invalid',
});
- await expect(runtime.workingDirectory('scout', bundle)).rejects.toMatchObject({
+ await expect(runtime.workingDirectory('agent-1', 'scout', bundle)).rejects.toMatchObject({
reason: 'definition_invalid',
});
});
it('makes a missing directory inside the workspaces directory, and only there', async () => {
const named = join(dir, 'data', 'workspaces', 'chosen', 'nested');
- expect(await runtime.workingDirectory('scout', named)).toBe(named);
+ expect(await runtime.workingDirectory('agent-1', 'scout', named)).toBe(named);
expect(statSync(named).isDirectory()).toBe(true);
const escaping = join(dir, 'data', 'workspaces', '..', 'outside');
- await expect(runtime.workingDirectory('scout', escaping)).rejects.toMatchObject({
+ await expect(runtime.workingDirectory('agent-1', 'scout', escaping)).rejects.toMatchObject({
reason: 'definition_invalid',
});
expect(existsSync(join(dir, 'data', 'outside'))).toBe(false);
- await expect(runtime.workingDirectory('scout', join(dir, 'data', 'workspaces'))).resolves.toBe(
- join(dir, 'data', 'workspaces')
- );
+ await expect(
+ runtime.workingDirectory('agent-1', 'scout', join(dir, 'data', 'workspaces'))
+ ).resolves.toBe(join(dir, 'data', 'workspaces'));
});
});
diff --git a/console/packages/agent-controller/src/runtime.ts b/console/packages/agent-controller/src/runtime.ts
index cfeadc33a..04bdc9e76 100644
--- a/console/packages/agent-controller/src/runtime.ts
+++ b/console/packages/agent-controller/src/runtime.ts
@@ -1,6 +1,6 @@
import { execFile } from 'node:child_process';
import { randomUUID } from 'node:crypto';
-import { mkdir, open, readFile, rename, stat, unlink } from 'node:fs/promises';
+import { mkdir, open, readFile, rename, rm, stat, unlink } from 'node:fs/promises';
import { homedir } from 'node:os';
import { isAbsolute, join, relative, resolve, sep } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
@@ -8,7 +8,9 @@ import { promisify } from 'node:util';
import {
clearTakenOver,
ensureSharedProcess,
+ hostedCredentialSchema,
inProcessSupervision,
+ materializeHostedProvider,
type OpenAgentStream,
type ProviderReadiness,
providerReadinessSchema,
@@ -33,6 +35,7 @@ import { ConfigurationError, ReasonedError } from './errors';
import { errorMessage, type Logger } from './log';
import { agentWorkspace, type DataLayout } from './paths';
import type { Isolation, Provider } from './schemas';
+import type { GivenLogin } from './sealed-logins';
const execute = promisify(execFile);
@@ -80,7 +83,13 @@ export type LaunchOptions = {
* What an agent host reads to reach Switch: the controller's relay, a token
* for it, and the hub an agent host in a process of its own hears its events on.
*/
-export type RelayCredentials = { endpoint: string; token: string; hub: string };
+export type RelayCredentials = {
+ endpoint: string;
+ token: string;
+ hub: string;
+ /** A login Switch gave the machine for the agent's provider, which it has none of its own for. */
+ providerLogin: GivenLogin | null;
+};
/** Runs agent hosts one way: in this controller's process, or each in a process of its own. */
export interface AgentRunner {
@@ -104,33 +113,98 @@ export interface AgentRuntime extends AgentRunner {
writeCredentials(agentId: string, credentials: RelayCredentials): Promise<void>;
deleteCredentials(agentId: string): Promise<void>;
/**
- * `directory` from the definition, or a workspace under the data directory.
- * A directory inside the workspaces folder is made when missing; any other
- * must already exist.
+ * `directory` from the definition, or a workspace of the agent's own when it
+ * names none. A directory inside the workspaces folder is made when
+ * missing; any other must already exist.
+ */
+ workingDirectory(agentId: string, name: string, directory: string | null): Promise<string>;
+ /**
+ * Whether the provider signs in as an agent would run it: with the
+ * machine's own login, or with `login`, one Switch gave the machine.
*/
- workingDirectory(name: string, directory: string | null): Promise<string>;
- probe(provider: Provider, binaryPath: string, cwd: string): Promise<ProviderReadiness>;
+ probe(
+ provider: Provider,
+ binaryPath: string,
+ cwd: string,
+ login: GivenLogin | null
+ ): Promise<ProviderReadiness>;
+ /** The agent host's state root as the agent host sees it: where a given login's files go. */
+ agentStateRoot(agentId: string): string;
+}
+
+/**
+ * The environment to probe `login` with: its files written under `dir`, made
+ * afresh, and the variables pointing at them or holding its token.
+ */
+export async function loginProbeEnvironment(
+ dir: string,
+ login: GivenLogin,
+ binaryPath: string
+): Promise<Record<string, string>> {
+ await rm(dir, { recursive: true, force: true });
+ await mkdir(dir, { recursive: true, mode: 0o700 });
+ const env: Record<string, string> = {};
+ await materializeHostedProvider(dir, env, login, binaryPath);
+ return env;
}
/** How long an agent host asked to stop is given; each session host is allowed 20 s of it. */
export const STOP_TIMEOUT_MS = 30_000;
const PROBE_TIMEOUT_MS = 90_000;
-/** Asks a provider's CLI, through the shared-host bundle's `--probe`, whether it is signed in. */
+/**
+ * Asks a provider's CLI, through the shared-host bundle's `--probe`, whether
+ * it is signed in, with the environment the agents would have.
+ */
export async function probeProvider(
bundlePath: string,
provider: Provider,
binaryPath: string,
- cwd: string
+ cwd: string,
+ env: NodeJS.ProcessEnv
): Promise<ProviderReadiness> {
const { stdout } = await execute(
process.execPath,
[bundlePath, '--probe', provider, cwd, binaryPath],
- { timeout: PROBE_TIMEOUT_MS, maxBuffer: 4 * 1024 * 1024, env: process.env }
+ { timeout: PROBE_TIMEOUT_MS, maxBuffer: 4 * 1024 * 1024, env }
);
const line = stdout.trim().split('\n').at(-1) ?? '';
return providerReadinessSchema.parse(JSON.parse(line));
}
+/** What an agent host's credentials file holds; see `readSharedCredentials`. */
+export function relayCredentialsBody(agentId: string, credentials: RelayCredentials): string {
+ return JSON.stringify({
+ env: {
+ SWITCH_API_ENDPOINT: credentials.endpoint,
+ SWITCH_API_TOKEN: credentials.token,
+ SWITCH_AGENT_ID: agentId,
+ SWITCH_AGENT_HUB: credentials.hub,
+ },
+ providerLogin: credentials.providerLogin,
+ });
+}
+
+/** A credentials file's relay, token and hub; null when it is not one, or is another agent's. */
+export function parseRelayCredentials(text: string, agentId: string): RelayCredentials | null {
+ try {
+ const parsed = JSON.parse(text) as { env?: Record<string, unknown>; providerLogin?: unknown };
+ const env = parsed.env ?? {};
+ const endpoint = env.SWITCH_API_ENDPOINT;
+ const token = env.SWITCH_API_TOKEN;
+ // Written before the hub: an empty one, which no relay names, so the file is written again.
+ const hub = typeof env.SWITCH_AGENT_HUB === 'string' ? env.SWITCH_AGENT_HUB : '';
+ if (env.SWITCH_AGENT_ID !== agentId) return null;
+ const login = hostedCredentialSchema.nullable().safeParse(parsed.providerLogin ?? null);
+ // A login it cannot read is none: the file is written again with the one wanted.
+ const providerLogin = login.success && login.data?.status === 'connected' ? login.data : null;
+ return typeof endpoint === 'string' && typeof token === 'string'
+ ? { endpoint, token, hub, providerLogin }
+ : null;
+ } catch {
+ return null;
+ }
+}
+
/** Failures an agent host is started again after, within `CRASH_WINDOW_MS`. */
const MAX_CRASHES = 3;
const CRASH_WINDOW_MS = 10 * 60 * 1000;
@@ -300,42 +374,23 @@ export class InProcessRuntime implements AgentRuntime {
async readCredentials(agentId: string): Promise<RelayCredentials | null> {
const text = await readOptional(this.credentialsPath(agentId));
- if (text === null) return null;
- try {
- const env = (JSON.parse(text) as { env?: Record<string, unknown> }).env ?? {};
- const endpoint = env.SWITCH_API_ENDPOINT;
- const token = env.SWITCH_API_TOKEN;
- // Written before the hub: an empty one, which no relay names, so the file is written again.
- const hub = typeof env.SWITCH_AGENT_HUB === 'string' ? env.SWITCH_AGENT_HUB : '';
- if (env.SWITCH_AGENT_ID !== agentId) return null;
- return typeof endpoint === 'string' && typeof token === 'string'
- ? { endpoint, token, hub }
- : null;
- } catch {
- return null;
- }
+ return text === null ? null : parseRelayCredentials(text, agentId);
}
async writeCredentials(agentId: string, credentials: RelayCredentials): Promise<void> {
await mkdir(this.deps.layout.agentDir(agentId), { recursive: true, mode: 0o700 });
- await writeAtomic(
- this.credentialsPath(agentId),
- JSON.stringify({
- env: {
- SWITCH_API_ENDPOINT: credentials.endpoint,
- SWITCH_API_TOKEN: credentials.token,
- SWITCH_AGENT_ID: agentId,
- SWITCH_AGENT_HUB: credentials.hub,
- },
- })
- );
+ await writeAtomic(this.credentialsPath(agentId), relayCredentialsBody(agentId, credentials));
}
async deleteCredentials(agentId: string): Promise<void> {
await removeOptional(this.credentialsPath(agentId));
}
- async workingDirectory(name: string, directory: string | null): Promise<string> {
+ async workingDirectory(
+ _agentId: string,
+ name: string,
+ directory: string | null
+ ): Promise<string> {
if (directory === null) {
const path = agentWorkspace(this.deps.workspaces, name);
await mkdir(path, { recursive: true });
@@ -459,15 +514,7 @@ export class InProcessRuntime implements AgentRuntime {
const stopRecording = recordWatcherHealth(root, control);
const done = (async () => {
try {
- await runAgentHost(
- root,
- config,
- signal,
- sessions,
- control,
- null,
- this.deps.openStream(agentId)
- );
+ await runAgentHost(root, config, signal, sessions, control, this.deps.openStream(agentId));
} finally {
stopRecording();
await sessions.close();
@@ -551,8 +598,27 @@ export class InProcessRuntime implements AgentRuntime {
}
}
- probe(provider: Provider, binaryPath: string, cwd: string): Promise<ProviderReadiness> {
- return probeProvider(this.deps.bundlePath, provider, binaryPath, cwd);
+ async probe(
+ provider: Provider,
+ binaryPath: string,
+ cwd: string,
+ login: GivenLogin | null
+ ): Promise<ProviderReadiness> {
+ const given = login
+ ? await loginProbeEnvironment(
+ join(this.deps.layout.root, 'login-check', provider),
+ login,
+ binaryPath
+ )
+ : {};
+ return probeProvider(this.deps.bundlePath, provider, binaryPath, cwd, {
+ ...process.env,
+ ...given,
+ });
+ }
+
+ agentStateRoot(agentId: string): string {
+ return this.deps.layout.watcherRoot(agentId);
}
private async writeFlags(root: string, flags: WatchFlags): Promise<void> {
diff --git a/console/packages/agent-controller/src/runtimes.ts b/console/packages/agent-controller/src/runtimes.ts
index 252bcfbef..083cbd8f3 100644
--- a/console/packages/agent-controller/src/runtimes.ts
+++ b/console/packages/agent-controller/src/runtimes.ts
@@ -8,6 +8,7 @@ import type {
RelayCredentials,
} from './runtime';
import type { Provider } from './schemas';
+import type { GivenLogin } from './sealed-logins';
/**
* Runs each agent the way its definition asks: its host in this controller's
@@ -65,11 +66,21 @@ export class AgentRuntimes implements AgentRuntime {
return this.shared.deleteCredentials(agentId);
}
- workingDirectory(name: string, directory: string | null): Promise<string> {
- return this.shared.workingDirectory(name, directory);
+ workingDirectory(agentId: string, name: string, directory: string | null): Promise<string> {
+ return this.shared.workingDirectory(agentId, name, directory);
}
- probe(provider: Provider, binaryPath: string, cwd: string): Promise<ProviderReadiness> {
- return this.shared.probe(provider, binaryPath, cwd);
+ probe(
+ provider: Provider,
+ binaryPath: string,
+ cwd: string,
+ login: GivenLogin | null
+ ): Promise<ProviderReadiness> {
+ return this.shared.probe(provider, binaryPath, cwd, login);
+ }
+
+ /** The same for both: moving an agent between them keeps its root. */
+ agentStateRoot(agentId: string): string {
+ return this.shared.agentStateRoot(agentId);
}
}
diff --git a/console/packages/agent-controller/src/schemas.ts b/console/packages/agent-controller/src/schemas.ts
index 4bd3a0b97..1a4065029 100644
--- a/console/packages/agent-controller/src/schemas.ts
+++ b/console/packages/agent-controller/src/schemas.ts
@@ -1,3 +1,4 @@
+import { sealedLoginSchema } from '@switch-console/agent-providers';
import { z } from 'zod';
/**
@@ -97,6 +98,8 @@ export const enrollRequestSchema = z.object({
platform: platformSchema,
version: z.string().min(1),
}),
+ /** The key the provider logins its owner gives this machine are sealed to. */
+ public_key: z.object({ alg: z.literal('X25519'), key: z.string().min(1) }).optional(),
});
export type EnrollRequest = z.infer<typeof enrollRequestSchema>;
@@ -114,6 +117,8 @@ export const credentialRotateResponseSchema = z.object({ credential: id });
export const controllerInfoRequestSchema = z.object({
name: z.string().min(1).max(200).optional(),
description: z.string().max(500).nullable().optional(),
+ /** Registered once, by a controller enrolled before it made a key. */
+ public_key: z.object({ alg: z.literal('X25519'), key: z.string().min(1) }).optional(),
});
export type ControllerInfoChange = z.infer<typeof controllerInfoRequestSchema>;
@@ -125,6 +130,14 @@ export const controllerInfoResponseSchema = z.object({
});
export type ControllerInfo = z.infer<typeof controllerInfoResponseSchema>;
+/** `GET .../provider-credentials/{provider}`: the login given to this machine, sealed to its key. */
+export const sealedLoginResponseSchema = z.object({
+ provider: z.string(),
+ revision: z.number().int().positive(),
+ sealed: sealedLoginSchema,
+});
+export type SealedLoginResponse = z.infer<typeof sealedLoginResponseSchema>;
+
// §2 Assignment (v1 definition)
export const agentDefinitionSchema = z.object({
diff --git a/console/packages/agent-controller/src/sealed-logins.ts b/console/packages/agent-controller/src/sealed-logins.ts
new file mode 100644
index 000000000..ced6e911c
--- /dev/null
+++ b/console/packages/agent-controller/src/sealed-logins.ts
@@ -0,0 +1,121 @@
+import {
+ generateSealingKeyPair,
+ type HostedCredential,
+ openProviderLogin,
+ type SealingKeyPair,
+} from '@switch-console/agent-providers';
+import { z } from 'zod';
+import { ControllerApiError, type ControllerClient } from './api';
+import { errorMessage, type Logger } from './log';
+import type { Provider } from './schemas';
+import { SEALING_KEY, type SecretStore } from './secrets';
+
+/** A login Switch gave this machine for a provider, opened. */
+export type GivenLogin = Extract<HostedCredential, { status: 'connected' }>;
+
+const keyPairSchema = z.strictObject({ publicKey: z.string(), privateKey: z.string() });
+
+/**
+ * This machine's sealing keypair, from the secret store, made and kept there
+ * when it has none. Null for a store that does not outlive the process (a
+ * credential handed over by a parent): a key made there would be lost at the
+ * next start, while Switch went on sealing to it.
+ */
+export async function sealingKeys(secrets: SecretStore): Promise<SealingKeyPair | null> {
+ if (!secrets.persistent) return null;
+ const saved = await secrets.get(SEALING_KEY);
+ if (saved !== null) return keyPairSchema.parse(JSON.parse(saved));
+ const keys = generateSealingKeyPair();
+ await secrets.set(SEALING_KEY, JSON.stringify(keys));
+ return keys;
+}
+
+/**
+ * Registers the key with Switch, which keeps the first one it is given. Says
+ * so and answers false when Switch holds another, or cannot take one: then no
+ * login given to this machine could be opened here.
+ */
+export async function registerSealingKey(
+ client: Pick<ControllerClient, 'registerPublicKey'>,
+ keys: SealingKeyPair,
+ log: Logger
+): Promise<boolean> {
+ try {
+ await client.registerPublicKey(keys.publicKey);
+ return true;
+ } catch (error) {
+ if (error instanceof ControllerApiError && error.status === 409) {
+ log.error(
+ 'Switch holds another key for this machine, so provider logins given to it cannot be opened here. Enroll the machine again to give it this one.',
+ { error: error.message }
+ );
+ return false;
+ }
+ if (error instanceof ControllerApiError && error.status === 422) {
+ log.warn(
+ 'This Switch server does not take a key for sealed provider logins; only logins on this machine are used.',
+ { error: error.message }
+ );
+ return false;
+ }
+ log.warn(
+ 'Could not register the key for sealed provider logins; trying again at the next start',
+ {
+ error: errorMessage(error),
+ }
+ );
+ return true;
+ }
+}
+
+/** Why a login given to this machine is not used. */
+export type LoginProblem = { code: 'provider_login_missing' | 'internal'; message: string };
+
+/**
+ * The provider logins Switch gave this machine, sealed to its key: fetched,
+ * opened with the private key, and kept in memory only. Nothing of them is
+ * written to disk here; the agent that uses one gets it with its credentials.
+ */
+export class SealedLogins {
+ constructor(
+ private readonly deps: {
+ client: Pick<ControllerClient, 'sealedLogin'>;
+ keys: SealingKeyPair;
+ controllerId: string;
+ }
+ ) {}
+
+ /**
+ * The provider's login as Switch has it now: the login, or why there is
+ * none to use. A failure to reach Switch throws.
+ */
+ async fetch(provider: Provider): Promise<{ login: GivenLogin } | { problem: LoginProblem }> {
+ const answer = await this.deps.client.sealedLogin(provider);
+ if (answer === null)
+ return {
+ problem: {
+ code: 'provider_login_missing',
+ message: `No ${provider} login has been given to this machine.`,
+ },
+ };
+ try {
+ const opened = openProviderLogin({
+ keys: this.deps.keys,
+ controllerId: this.deps.controllerId,
+ provider,
+ sealed: answer.sealed,
+ });
+ return {
+ login: {
+ status: 'connected',
+ provider,
+ revision: String(answer.revision),
+ kind: opened.kind,
+ credential: opened.credential,
+ },
+ };
+ } catch (error) {
+ return { problem: { code: 'internal', message: errorMessage(error) } };
+ }
+ }
+}
diff --git a/console/packages/agent-controller/src/secrets.ts b/console/packages/agent-controller/src/secrets.ts
index 6f3e8dca7..6f3777f6c 100644
--- a/console/packages/agent-controller/src/secrets.ts
+++ b/console/packages/agent-controller/src/secrets.ts
@@ -5,6 +5,11 @@ import { join } from 'node:path';
/** The one secret v1 keeps: the long-lived controller credential (`swcc_…`). */
export const CONTROLLER_CREDENTIAL = 'controller-credential';
+/**
+ * The keypair the provider logins given to this machine are sealed to
+ * (`SealingKeyPair` as JSON); kept only by a store that outlives the process.
+ */
+export const SEALING_KEY = 'sealing-key';
/**
* Where the controller keeps secrets: a file backend for a controller run on
@@ -15,6 +20,8 @@ export const CONTROLLER_CREDENTIAL = 'controller-credential';
export interface SecretStore {
/** Names the backend in logs and `status`. */
readonly description: string;
+ /** What is set outlives this process. */
+ readonly persistent: boolean;
/** Logged once when the controller starts, for a backend with a caveat worth stating. */
startupWarning(): string | null;
get(name: string): Promise<string | null>;
@@ -30,6 +37,7 @@ const NAME = /^[a-z0-9-]+$/;
* controller user's files can read them.
*/
export class FileSecretStore implements SecretStore {
+ readonly persistent = true;
readonly description: string;
constructor(private readonly dir: string) {
@@ -94,6 +102,7 @@ export class FileSecretStore implements SecretStore {
* the parent is told of a revocation by the exit code.
*/
export class MemorySecretStore implements SecretStore {
+ readonly persistent = false;
readonly description: string;
private readonly values: Map<string, string>;
@@ -149,6 +158,11 @@ export const runCommand: CommandRunner = (file, args, input) =>
}
resolve({ stdout });
});
+ // A command that exits without reading its input closes the pipe first;
+ // its exit status says how it went, not the write.
+ child.stdin?.on('error', (error: NodeJS.ErrnoException) => {
+ if (error.code !== 'EPIPE') reject(error);
+ });
child.stdin?.end(input ?? '');
});
@@ -180,6 +194,7 @@ function account(dataDir: string, name: string): string {
* could read it.
*/
export class KeychainSecretStore implements SecretStore {
+ readonly persistent = true;
readonly description = 'the macOS login keychain';
constructor(
@@ -232,6 +247,7 @@ export class KeychainSecretStore implements SecretStore {
* why it is chosen only when asked for.
*/
export class SecretServiceStore implements SecretStore {
+ readonly persistent = true;
readonly description = 'the desktop keyring (Secret Service)';
constructor(
diff --git a/console/packages/agent-controller/src/separate-users.test.ts b/console/packages/agent-controller/src/separate-users.test.ts
new file mode 100644
index 000000000..fcce906d7
--- /dev/null
+++ b/console/packages/agent-controller/src/separate-users.test.ts
@@ -0,0 +1,133 @@
+import { describe, expect, it } from 'vitest';
+import {
+ agentPath,
+ agentUnitTemplate,
+ controllerSystemUnit,
+ homeRootOf,
+ parsePasswd,
+ polkitRule,
+ type SeparateUsersConfig,
+ separateUserNames,
+ separateUsersConfigSchema,
+ slotName,
+ unitCredentialsPath,
+} from './separate-users';
+
+const config: SeparateUsersConfig = {
+ v: 1,
+ user: 'builder',
+ uid: 1001,
+ gid: 990,
+ dataDir: '/home/builder/.local/state/switch/agent-controller',
+ agentsDir: '/var/lib/switch-agents/1001',
+ agentUsers: 4,
+ node: '/usr/bin/node',
+ bundle: '/usr/lib/node_modules/switch-agent-controller/shared-host.mjs',
+};
+
+function lines(text: string): string[] {
+ return text.split('\n');
+}
+
+describe('separate users', () => {
+ it('names everything after the controller user’s uid', () => {
+ const names = separateUserNames(1001);
+ expect(names.group).toBe('switch-agents-1001');
+ expect(names.agentUnit(3)).toBe('[email protected]');
+ expect(names.agentUser(3)).toBe('sa1001-03');
+ expect(names.controllerUnit).toBe('switch-agent-controller-1001.service');
+ expect(unitCredentialsPath(config, 3)).toBe(
+ '/run/credentials/[email protected]/relay'
+ );
+ expect(() => slotName(0)).toThrow();
+ expect(() => slotName(100)).toThrow();
+ // A user name systemd and useradd both take: at most 31 characters, even for a 10-digit uid.
+ expect(separateUserNames(4_294_967_294).agentUser(99).length).toBeLessThanOrEqual(31);
+ });
+
+ it('writes an agent unit that runs as the agent’s user and sees only its own directory', () => {
+ const unit = lines(
+ agentUnitTemplate(config, {
+ sh: '/bin/sh',
+ find: '/usr/bin/find',
+ chown: '/usr/bin/chown',
+ path: '/usr/local/bin:/usr/bin:/bin',
+ })
+ );
+ expect(unit).toContain('User=sa1001-%i');
+ expect(unit).toContain('Group=switch-agents-1001');
+ expect(unit).toContain(
+ 'ExecStart=/usr/bin/node /usr/lib/node_modules/switch-agent-controller/shared-host.mjs /var/lib/switch-agents/1001/agent/watcher /var/lib/switch-agents/1001/agent/watcher/config.json --watch-worker'
+ );
+ expect(unit).toContain(
+ 'LoadCredential=relay:/home/builder/.local/state/switch/agent-controller/units/%i/relay.json'
+ );
+ expect(unit).toContain('TemporaryFileSystem=/var/lib/switch-agents/1001:ro');
+ // Every agent sees its own directory at the same path, whichever user it runs as.
+ expect(unit).toContain(
+ 'BindPaths=/var/lib/switch-agents/1001/%i:/var/lib/switch-agents/1001/agent'
+ );
+ expect(unit).toContain('Environment=HOME=/var/lib/switch-agents/1001/agent/home');
+ expect(unit).toContain('InaccessiblePaths=-/home/builder/.local/state/switch/agent-controller');
+ expect(unit).toContain('ProtectHome=yes');
+ expect(unit).toContain('ProtectProc=invisible');
+ expect(unit).toContain('RestrictSUIDSGID=yes');
+ expect(unit).toContain('IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128');
+ // Root hands back only what another agent user owns, never following a link.
+ expect(unit).toContain(
+ `ExecStartPre=+/bin/sh -c 'dir=/var/lib/switch-agents/1001/agent; [ -d "$$dir" ] || dir=/var/lib/switch-agents/1001/%i; exec /usr/bin/find "$$dir" -xdev -mindepth 1 -group switch-agents-1001 ! -user sa1001-%i ! -user 1001 -exec /usr/bin/chown --no-dereference sa1001-%i {} +'`
+ );
+ });
+
+ it('lets the controller’s user manage its agents’ units and nothing else', () => {
+ const rule = polkitRule(config);
+ expect(rule).toContain('subject.user !== "builder"');
+ expect(rule).toContain('/^switch-agent-1001@[0-9]{2}\\.service$/');
+ expect(rule).toContain('["start", "stop", "restart", "reset-failed"]');
+ });
+
+ it('runs the controller as a system service of its user, in the agents’ group', () => {
+ const unit = lines(
+ controllerSystemUnit(config, {
+ cli: '/usr/lib/node_modules/switch-agent-controller/cli.mjs',
+ envFile: '/etc/switch/agents.env',
+ path: '/usr/local/bin:/usr/bin:/bin',
+ })
+ );
+ expect(unit).toContain('User=builder');
+ expect(unit).toContain('SupplementaryGroups=switch-agents-1001');
+ expect(unit).toContain('RestartPreventExitStatus=2 3 4 5 6');
+ const exec = unit.find((line) => line.startsWith('ExecStart='))!;
+ expect(exec).toContain(
+ '"run" "--data-dir" "/home/builder/.local/state/switch/agent-controller"'
+ );
+ expect(exec).toContain('"--agent-runtime" "separate-user"');
+ expect(exec).toContain('"--env-file" "/etc/switch/agents.env"');
+ });
+
+ it('keeps out of the agents’ PATH what they cannot reach', () => {
+ expect(
+ agentPath('/home/builder/.local/bin:/usr/local/bin:relative:/root/bin:/usr/bin', [
+ '/srv/builder',
+ ])
+ ).toBe('/usr/local/bin:/usr/bin');
+ expect(homeRootOf('/srv/builder/bin/claude', ['/srv/builder'])).toBe('/srv/builder');
+ expect(homeRootOf('/home/builder/.local/bin/claude', [])).toBe('/home');
+ expect(homeRootOf('/usr/local/bin/claude', [])).toBeNull();
+ expect(homeRootOf('/homestead/bin', [])).toBeNull();
+ });
+
+ it('reads a passwd entry and checks a setup', () => {
+ expect(parsePasswd('builder:x:1001:1001:Builder:/home/builder:/bin/bash\n')).toEqual({
+ name: 'builder',
+ uid: 1001,
+ gid: 1001,
+ home: '/home/builder',
+ });
+ expect(separateUsersConfigSchema.safeParse(config).success).toBe(true);
+ expect(
+ separateUsersConfigSchema.safeParse({ ...config, agentsDir: '/var/lib/with space' }).success
+ ).toBe(false);
+ expect(separateUsersConfigSchema.safeParse({ ...config, agentUsers: 100 }).success).toBe(false);
+ });
+});
diff --git a/console/packages/agent-controller/src/separate-users.ts b/console/packages/agent-controller/src/separate-users.ts
new file mode 100644
index 000000000..fea21f7a9
--- /dev/null
+++ b/console/packages/agent-controller/src/separate-users.ts
@@ -0,0 +1,658 @@
+import {
+ access,
+ chmod,
+ chown,
+ mkdir,
+ readdir,
+ readFile,
+ rm,
+ stat,
+ writeFile,
+} from 'node:fs/promises';
+import { delimiter, dirname, isAbsolute, join, relative, resolve, sep } from 'node:path';
+import { z } from 'zod';
+import { ConfigurationError } from './errors';
+import { defaultDataDir } from './paths';
+import { CommandFailure, type CommandRunner } from './secrets';
+
+/**
+ * Running every agent as a Linux user of its own.
+ *
+ * Root sets it up once (`sudo switch-agent-controller install-service
+ * --separate-users`): a pool of system users for one controller's agents, all
+ * in a group of their own that the controller also gets; a systemd template
+ * unit that runs an agent as one of them; a polkit rule that lets the
+ * controller's user start, stop and restart those units and nothing else; and
+ * the controller itself as a system service of that user. After that the
+ * controller needs no root: each agent it is given claims a free user from the
+ * pool, and runs as `switch-agent-<uid>@<NN>.service`, seeing only its own
+ * directory.
+ *
+ * Everything is named after the controller user's uid, so several controllers
+ * on one machine each get their own.
+ */
+
+export const SEPARATE_USERS_CONFIG_DIR = '/etc/switch-agent-controller';
+const SYSTEMD_SYSTEM_DIR = '/etc/systemd/system';
+const POLKIT_RULES_DIR = '/etc/polkit-1/rules.d';
+const AGENTS_PARENT = '/var/lib/switch-agents';
+export const MAX_AGENT_USERS = 99;
+export const DEFAULT_AGENT_USERS = 16;
+/** Where agents cannot reach, so nothing an agent runs may live there. */
+const HOME_ROOTS = ['/home', '/root', '/run/user'];
+
+const absolutePath = z.string().refine((value) => isAbsolute(value) && isUnitSafe(value), {
+ message: 'must be an absolute path of letters, digits and ._/@+-',
+});
+
+export const separateUsersConfigSchema = z.strictObject({
+ v: z.literal(1),
+ user: z.string().regex(/^[a-z_][a-z0-9_.-]*\$?$/i),
+ uid: z.number().int().positive(),
+ gid: z.number().int().positive(),
+ dataDir: absolutePath,
+ agentsDir: absolutePath,
+ agentUsers: z.number().int().min(1).max(MAX_AGENT_USERS),
+ node: absolutePath,
+ bundle: absolutePath,
+});
+export type SeparateUsersConfig = z.infer<typeof separateUsersConfigSchema>;
+
+/** The names one controller's agents run under, all derived from its user's uid. */
+export function separateUserNames(uid: number) {
+ return {
+ group: `switch-agents-${uid}`,
+ controllerUnit: `switch-agent-controller-${uid}.service`,
+ agentTemplate: `switch-agent-${uid}@.service`,
+ agentUnitPattern: `switch-agent-${uid}@*.service`,
+ polkitRule: join(POLKIT_RULES_DIR, `50-switch-agent-controller-${uid}.rules`),
+ config: separateUsersConfigPath(uid),
+ agentUnit: (slot: number) => `switch-agent-${uid}@${slotName(slot)}.service`,
+ agentUser: (slot: number) => `sa${uid}-${slotName(slot)}`,
+ };
+}
+
+export function separateUsersConfigPath(uid: number): string {
+ return join(SEPARATE_USERS_CONFIG_DIR, `separate-users-${uid}.json`);
+}
+
+/** `1` → `01`: the instance name of an agent's unit, and the suffix of its user. */
+export function slotName(slot: number): string {
+ if (!Number.isInteger(slot) || slot < 1 || slot > MAX_AGENT_USERS)
+ throw new Error(`Agent user ${slot} is not one of 1–${MAX_AGENT_USERS}.`);
+ return String(slot).padStart(2, '0');
+}
+
+/** Where agents' directories are when setup names none. */
+export function defaultAgentsDir(uid: number): string {
+ return join(AGENTS_PARENT, String(uid));
+}
+
+/** Where the directory of the agent agent user `slot` runs is, on the machine. */
+export function agentRoot(config: SeparateUsersConfig, slot: number): string {
+ return join(config.agentsDir, slotName(slot));
+}
+
+/**
+ * Where an agent sees its own directory, whichever agent user it runs as: the
+ * same path for every agent, each in its own mount namespace, so the paths
+ * an agent records (its sessions' working directory, its provider's state)
+ * stay valid when it later runs as another user.
+ */
+export function unitAgentRoot(config: SeparateUsersConfig): string {
+ return join(config.agentsDir, 'agent');
+}
+
+/** Where the controller keeps what agent user `slot`'s unit loads: its relay credentials and environment. */
+export function unitFilesDir(config: SeparateUsersConfig, slot: number): string {
+ return join(config.dataDir, 'units', slotName(slot));
+}
+
+/** The path a unit's process reads its relay credentials from, as systemd hands them over. */
+export function unitCredentialsPath(config: SeparateUsersConfig, slot: number): string {
+ return join('/run/credentials', separateUserNames(config.uid).agentUnit(slot), 'relay');
+}
+
+/** The home directory `path` is under, where agents cannot reach; null when it is under none. */
+export function homeRootOf(path: string, homes: string[]): string | null {
+ for (const home of [...HOME_ROOTS, ...homes])
+ if (path === home || isWithin(home, path)) return home;
+ return null;
+}
+
+/** Characters a path may hold to be written into a unit file as is (no quoting, no `%`). */
+function isUnitSafe(value: string): boolean {
+ return /^[A-Za-z0-9._/@+-]+$/.test(value);
+}
+
+function isWithin(root: string, path: string): boolean {
+ const rest = relative(resolve(root), resolve(path));
+ return rest !== '' && rest !== '..' && !rest.startsWith(`..${sep}`) && !isAbsolute(rest);
+}
+
+/**
+ * The template unit an agent runs as: one of the pool's users, in the
+ * controller's agents' group, with only its own directory to see and write
+ * (at `unitAgentRoot`),
+ * no access to the controller's data, the home directories or the cloud
+ * instance metadata, and systemd restarting it after a crash.
+ *
+ * Before it starts, root hands it back anything in its directory owned by
+ * another agent user (its directory moves to whichever user it is given):
+ * only files in the agents' group, never through a link, and never what the
+ * controller wrote. Nothing of the agent runs then: its unit is stopped, so
+ * nothing can swap a path mid-walk. A file outside the agents' group is never
+ * touched, so not even a controller that pointed an agent's directory
+ * elsewhere could have root hand it a file it does not already share.
+ */
+export function agentUnitTemplate(
+ config: SeparateUsersConfig,
+ tools: { sh: string; find: string; chown: string; path: string }
+): string {
+ const names = separateUserNames(config.uid);
+ const onMachine = `${config.agentsDir}/%i`;
+ const root = unitAgentRoot(config);
+ const user = `sa${config.uid}-%i`;
+ return [
+ '# Written by switch-agent-controller install-service --separate-users;',
+ '# run that again rather than editing this file.',
+ '[Unit]',
+ `Description=Switch agent %i of the agents controller run by ${config.user}`,
+ 'StartLimitIntervalSec=600',
+ 'StartLimitBurst=5',
+ '',
+ '[Service]',
+ 'Type=simple',
+ `User=${user}`,
+ `Group=${names.group}`,
+ 'UMask=0007',
+ 'Environment=SWITCH_HOST_SHARED_GROUP=1',
+ `Environment=HOME=${root}/home`,
+ `Environment=PATH=${tools.path}`,
+ `EnvironmentFile=-${config.dataDir}/units/%i/environment`,
+ `LoadCredential=relay:${config.dataDir}/units/%i/relay.json`,
+ `WorkingDirectory=${root}`,
+ // Some systemd versions run a `+` command inside the unit's mount
+ // namespace, where the directory is at `root`; others outside it.
+ `ExecStartPre=+${tools.sh} -c 'dir=${root}; [ -d "$$dir" ] || dir=${onMachine}; exec ${tools.find} "$$dir" -xdev -mindepth 1 -group ${names.group} ! -user ${user} ! -user ${config.uid} -exec ${tools.chown} --no-dereference ${user} {} +'`,
+ `ExecStart=${config.node} ${config.bundle} ${root}/watcher ${root}/watcher/config.json --watch-worker`,
+ 'StandardInput=null',
+ 'StandardOutput=journal',
+ 'StandardError=journal',
+ 'Restart=on-failure',
+ 'RestartSec=3',
+ `TemporaryFileSystem=${config.agentsDir}:ro`,
+ `BindPaths=${onMachine}:${root}`,
+ `InaccessiblePaths=-${config.dataDir}`,
+ 'ProtectSystem=strict',
+ 'ProtectHome=yes',
+ 'PrivateTmp=yes',
+ 'PrivateDevices=yes',
+ 'NoNewPrivileges=yes',
+ 'CapabilityBoundingSet=',
+ 'AmbientCapabilities=',
+ 'ProtectProc=invisible',
+ 'ProtectKernelTunables=yes',
+ 'ProtectKernelModules=yes',
+ 'ProtectKernelLogs=yes',
+ 'ProtectControlGroups=yes',
+ 'ProtectClock=yes',
+ 'ProtectHostname=yes',
+ 'RestrictRealtime=yes',
+ 'RestrictSUIDSGID=yes',
+ 'LockPersonality=yes',
+ 'RemoveIPC=yes',
+ 'RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK',
+ 'IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128',
+ '',
+ ].join('\n');
+}
+
+/** The polkit rule: the controller's user may start, stop, restart and reset its agents' units, and nothing else. */
+export function polkitRule(config: SeparateUsersConfig): string {
+ return `// Written by switch-agent-controller install-service --separate-users.
+// ${config.user} may start, stop, restart and reset its agents' units, and
+// nothing else. Every other request falls through to the defaults.
+polkit.addRule(function (action, subject) {
+ if (action.id !== "org.freedesktop.systemd1.manage-units" || subject.user !== ${JSON.stringify(config.user)}) return;
+ var unit = action.lookup("unit"), verb = action.lookup("verb");
+ if (/^switch-agent-${config.uid}@[0-9]{2}\\.service$/.test(unit) &&
+ ["start", "stop", "restart", "reset-failed"].indexOf(verb) >= 0) return polkit.Result.YES;
+});
+`;
+}
+
+/**
+ * The controller as a system service of its user, with the agents' group as a
+ * supplementary group, so it can read what its agents write. Restarted on exit
+ * code 1 only, as the user service is.
+ */
+export function controllerSystemUnit(
+ config: SeparateUsersConfig,
+ run: { cli: string; envFile: string | null; path: string }
+): string {
+ const quote = (value: string) => `"${value.replace(/(["\\])/g, '\\$1').replace(/%/g, '%%')}"`;
+ const args = [
+ config.node,
+ run.cli,
+ 'run',
+ '--data-dir',
+ config.dataDir,
+ '--agent-runtime',
+ 'separate-user',
+ '--shared-host-bundle',
+ config.bundle,
+ ...(run.envFile ? ['--env-file', run.envFile] : []),
+ ];
+ return [
+ '# Written by switch-agent-controller install-service --separate-users;',
+ '# run that again rather than editing this file.',
+ '[Unit]',
+ `Description=Switch agents controller run by ${config.user}, each agent as a user of its own`,
+ 'After=network-online.target',
+ 'Wants=network-online.target',
+ '',
+ '[Service]',
+ `User=${config.user}`,
+ `SupplementaryGroups=${separateUserNames(config.uid).group}`,
+ `ExecStart=${args.map(quote).join(' ')}`,
+ `Environment=${quote(`PATH=${run.path}`)}`,
+ 'Restart=on-failure',
+ 'RestartSec=5',
+ 'RestartPreventExitStatus=2 3 4 5 6',
+ '',
+ '[Install]',
+ 'WantedBy=multi-user.target',
+ '',
+ ].join('\n');
+}
+
+/** `PATH` without the entries agents cannot reach. */
+export function agentPath(path: string, homes: string[]): string {
+ const kept = path
+ .split(delimiter)
+ .filter((entry) => isAbsolute(entry) && isUnitSafe(entry) && !homeRootOf(entry, homes));
+ return [...new Set(kept)].join(delimiter) || '/usr/local/bin:/usr/bin:/bin';
+}
+
+/** One line of `getent passwd`. */
+export type PasswdEntry = { name: string; uid: number; gid: number; home: string };
+
+export function parsePasswd(line: string): PasswdEntry {
+ const fields = line.trim().split(':');
+ if (fields.length < 7) throw new Error(`Not a passwd entry: ${line.trim()}`);
+ return {
+ name: fields[0]!,
+ uid: Number(fields[2]),
+ gid: Number(fields[3]),
+ home: fields[5]!,
+ };
+}
+
+/** The setup made for the controller user `uid`, or null when there is none. */
+async function readSetup(uid: number): Promise<SeparateUsersConfig | null> {
+ const path = separateUsersConfigPath(uid);
+ let text: string;
+ try {
+ text = await readFile(path, 'utf8');
+ } catch (error) {
+ if ((error as NodeJS.ErrnoException).code === 'ENOENT') return null;
+ throw new ConfigurationError(`${path} cannot be read: ${(error as Error).message}`);
+ }
+ const parsed = separateUsersConfigSchema.safeParse(JSON.parse(text));
+ if (!parsed.success)
+ throw new ConfigurationError(
+ `${path} is not a setup this controller can read (${parsed.error.issues.map((issue) => `${issue.path.join('.')}: ${issue.message}`).join('; ')}). Run the setup again.`
+ );
+ if (parsed.data.uid !== uid)
+ throw new ConfigurationError(`${path} was set up for uid ${parsed.data.uid}, not ${uid}.`);
+ return parsed.data;
+}
+
+/**
+ * The setup for the controller running as `uid` on `dataDir`, or null when
+ * agents are not set up to run as users of their own for it.
+ */
+export async function findSeparateUsersConfig(
+ uid: number,
+ dataDir: string
+): Promise<SeparateUsersConfig | null> {
+ const config = await readSetup(uid);
+ return config && resolve(config.dataDir) === resolve(dataDir) ? config : null;
+}
+
+/** The setup for the controller running as `uid` on `dataDir`; a missing or different one is a configuration error. */
+export async function loadSeparateUsersConfig(
+ uid: number,
+ dataDir: string
+): Promise<SeparateUsersConfig> {
+ const config = await readSetup(uid);
+ const path = separateUsersConfigPath(uid);
+ if (!config)
+ throw new ConfigurationError(
+ `Agents cannot run as users of their own here: ${path} does not exist. Set it up once as root: sudo switch-agent-controller install-service --separate-users --data-dir ${dataDir}`
+ );
+ if (resolve(config.dataDir) !== resolve(dataDir))
+ throw new ConfigurationError(
+ `${path} was set up for the data directory ${config.dataDir}, not ${dataDir}. Run the setup again with --data-dir ${dataDir}.`
+ );
+ return config;
+}
+
+/** What the agents' group and unit need from the running controller process. */
+export function assertControllerCanRunSeparateUsers(config: SeparateUsersConfig): void {
+ if (!process.getgroups?.().includes(config.gid))
+ throw new ConfigurationError(
+ `This controller is not in the agents' group ${separateUserNames(config.uid).group}, so it cannot read what its agents write. Run it as the service the setup installed: sudo systemctl start ${separateUserNames(config.uid).controllerUnit}`
+ );
+}
+
+export type SetupInput = {
+ user: string;
+ /** The controller's data directory; the user's default one when null. */
+ dataDir: string | null;
+ agentsDir: string | null;
+ agentUsers: number;
+ node: string;
+ cli: string;
+ bundle: string;
+ envFile: string | null;
+ path: string;
+ /**
+ * Queue the controller's restart rather than wait for it: for a caller the
+ * controller's unit is ordered after, such as a machine's boot service.
+ */
+ noBlock: boolean;
+};
+
+export type SetupReport = { notes: string[] };
+
+/**
+ * Sets everything up as root, and starts the controller as a system service.
+ * Run again, it rewrites the files from what it is given and adds agent users
+ * up to `agentUsers`; it never removes one.
+ */
+export async function installSeparateUsers(
+ input: SetupInput,
+ run: CommandRunner
+): Promise<SetupReport> {
+ await assertSetupHost();
+ run = onSystemPath(run);
+ const account = parsePasswd((await run('getent', ['passwd', input.user], null)).stdout);
+ if (account.uid === 0)
+ throw new ConfigurationError('The controller must run as a user other than root.');
+ const homes = [account.home];
+ const names = separateUserNames(account.uid);
+ for (const [what, path] of [
+ ['Node', input.node],
+ ['The controller', input.cli],
+ ['The shared-host bundle', input.bundle],
+ ] as const) {
+ const home = homeRootOf(path, homes);
+ if (home)
+ throw new ConfigurationError(
+ `${what} is at ${path}, under ${home}, which agents running as users of their own cannot reach. Install Node and the controller system-wide (sudo npm install --global <the release's .tgz URL>) and run the setup again.`
+ );
+ if (!isUnitSafe(path))
+ throw new ConfigurationError(
+ `${what} is at ${path}; a path in a unit file must hold only letters, digits and ._/@+-.`
+ );
+ }
+ if (input.envFile && !isAbsolute(input.envFile))
+ throw new ConfigurationError(`--env-file must be an absolute path, not ${input.envFile}.`);
+ const dataDir = resolve(
+ input.dataDir ?? defaultDataDir({ platform: process.platform, env: {}, home: account.home })
+ );
+ await assertEnrolled(dataDir, account);
+ await assertNoUserService(dataDir, account);
+
+ const group = await ensureGroup(names.group, run);
+ const agentsDir = input.agentsDir ?? defaultAgentsDir(account.uid);
+ if (homeRootOf(agentsDir, homes))
+ throw new ConfigurationError(
+ `--agents-dir ${agentsDir} is under a home directory, which agents cannot reach.`
+ );
+ const config = separateUsersConfigSchema.parse({
+ v: 1,
+ user: account.name,
+ uid: account.uid,
+ gid: group,
+ dataDir,
+ agentsDir: resolve(agentsDir),
+ agentUsers: input.agentUsers,
+ node: input.node,
+ bundle: input.bundle,
+ } satisfies SeparateUsersConfig);
+ for (let slot = 1; slot <= config.agentUsers; slot++)
+ await ensureAgentUser(names.agentUser(slot), names.group, run);
+
+ await mkdir(dirname(config.agentsDir), { recursive: true, mode: 0o755 });
+ await mkdir(config.agentsDir, { recursive: true, mode: 0o700 });
+ await chown(config.agentsDir, account.uid, account.gid);
+ await chmod(config.agentsDir, 0o700);
+
+ const tools = {
+ sh: await which('sh', SYSTEM_PATH),
+ find: await which('find', SYSTEM_PATH),
+ chown: await which('chown', SYSTEM_PATH),
+ path: agentPath(input.path, homes),
+ };
+ await mkdir(SEPARATE_USERS_CONFIG_DIR, { recursive: true, mode: 0o755 });
+ await writeFile(names.config, `${JSON.stringify(config, null, 2)}\n`, { mode: 0o644 });
+ await writeFile(join(SYSTEMD_SYSTEM_DIR, names.agentTemplate), agentUnitTemplate(config, tools), {
+ mode: 0o644,
+ });
+ await writeFile(names.polkitRule, polkitRule(config), { mode: 0o644 });
+ await writeFile(
+ join(SYSTEMD_SYSTEM_DIR, names.controllerUnit),
+ controllerSystemUnit(config, { cli: input.cli, envFile: input.envFile, path: input.path }),
+ { mode: 0o644 }
+ );
+ await run('systemctl', ['daemon-reload'], null);
+ await run('systemctl', ['enable', names.controllerUnit], null);
+ await run(
+ 'systemctl',
+ input.noBlock
+ ? ['restart', '--no-block', names.controllerUnit]
+ : ['restart', names.controllerUnit],
+ null
+ );
+ return {
+ notes: [
+ `${config.agentUsers} agent users (${names.agentUser(1)}…), in the group ${names.group}.`,
+ `Agents' directories: ${config.agentsDir}`,
+ `Each agent runs as ${names.agentUnitPattern}; its log: journalctl -u ${names.agentUnit(1)}`,
+ `The controller runs as ${names.controllerUnit}; its log: journalctl -u ${names.controllerUnit} -f`,
+ 'Agents see only their own directory: a definition naming a directory elsewhere is refused.',
+ "Agents do not see this user's provider logins: give each provider an API key or token in --env-file.",
+ ],
+ };
+}
+
+/**
+ * Stops the controller and every agent unit, and removes the units, the rule,
+ * the setup, the agent users and their group. The agents' directories are
+ * kept, and said so.
+ */
+export async function uninstallSeparateUsers(
+ user: string,
+ run: CommandRunner
+): Promise<SetupReport> {
+ assertRoot();
+ run = onSystemPath(run);
+ const account = parsePasswd((await run('getent', ['passwd', user], null)).stdout);
+ const names = separateUserNames(account.uid);
+ let agentsDir: string | null = null;
+ let agentUsers = MAX_AGENT_USERS;
+ try {
+ const config = separateUsersConfigSchema.parse(
+ JSON.parse(await readFile(names.config, 'utf8'))
+ );
+ agentsDir = config.agentsDir;
+ agentUsers = config.agentUsers;
+ } catch (error) {
+ if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
+ }
+ await run('systemctl', ['disable', '--now', names.controllerUnit], null).catch(ignoreFailure);
+ await run('systemctl', ['stop', names.agentUnitPattern], null).catch(ignoreFailure);
+ await run('systemctl', ['reset-failed', names.agentUnitPattern], null).catch(ignoreFailure);
+ await rm(join(SYSTEMD_SYSTEM_DIR, names.controllerUnit), { force: true });
+ await rm(join(SYSTEMD_SYSTEM_DIR, names.agentTemplate), { force: true });
+ await rm(names.polkitRule, { force: true });
+ await rm(names.config, { force: true });
+ await run('systemctl', ['daemon-reload'], null);
+ for (let slot = 1; slot <= agentUsers; slot++)
+ await run('userdel', [names.agentUser(slot)], null).catch(ignoreFailure);
+ await run('groupdel', [names.group], null).catch(ignoreFailure);
+ return {
+ notes: agentsDir
+ ? [
+ `The agents' directories are kept in ${agentsDir}; remove them when nothing in them is needed.`,
+ ]
+ : [],
+ };
+}
+
+function ignoreFailure(error: unknown): void {
+ if (!(error instanceof CommandFailure)) throw error;
+}
+
+function assertRoot(): void {
+ if (process.getuid?.() !== 0)
+ throw new ConfigurationError(
+ 'Setting agents up as users of their own needs root, once: run this with sudo.'
+ );
+}
+
+async function assertSetupHost(): Promise<void> {
+ if (process.platform !== 'linux')
+ throw new ConfigurationError(
+ 'Agents run as users of their own only on Linux with systemd. On this machine, run the controller as a service of your user: switch-agent-controller install-service'
+ );
+ assertRoot();
+ try {
+ await access('/run/systemd/system');
+ } catch {
+ throw new ConfigurationError(
+ 'systemd is not this machine’s init system, and agents run as users of their own through it.'
+ );
+ }
+ try {
+ await access(POLKIT_RULES_DIR);
+ } catch {
+ throw new ConfigurationError(
+ `${POLKIT_RULES_DIR} does not exist: install polkit (polkitd), which lets the controller start its agents without root, and run the setup again.`
+ );
+ }
+ for (const command of ['useradd', 'groupadd', 'getent', 'id', 'systemctl'])
+ await which(command, SYSTEM_PATH);
+}
+
+async function assertEnrolled(dataDir: string, account: PasswdEntry): Promise<void> {
+ const database = join(dataDir, 'controller.db');
+ let owner: number;
+ try {
+ owner = (await stat(database)).uid;
+ } catch {
+ throw new ConfigurationError(
+ `${dataDir} holds no enrolled controller. Enroll it first, as ${account.name}: switch-agent-controller enroll --server <url> --code <code> --data-dir ${dataDir}`
+ );
+ }
+ if (owner !== account.uid)
+ throw new ConfigurationError(
+ `${database} belongs to uid ${owner}, not to ${account.name}, who would run the controller.`
+ );
+}
+
+/** A user service on the same data directory would run a second controller with the same identity. */
+async function assertNoUserService(dataDir: string, account: PasswdEntry): Promise<void> {
+ const directory = join(account.home, '.config', 'systemd', 'user');
+ let entries: string[];
+ try {
+ entries = await readdir(directory);
+ } catch (error) {
+ if ((error as NodeJS.ErrnoException).code === 'ENOENT') return;
+ throw error;
+ }
+ for (const entry of entries) {
+ if (!/^switch-agent-controller.*\.service$/.test(entry)) continue;
+ const file = join(directory, entry);
+ if ((await readFile(file, 'utf8')).includes(`"${dataDir}"`))
+ throw new ConfigurationError(
+ `${account.name} already runs this controller as a user service (${file}). Remove it first, as ${account.name}: switch-agent-controller uninstall-service --data-dir ${dataDir}`
+ );
+ }
+}
+
+async function ensureGroup(group: string, run: CommandRunner): Promise<number> {
+ const existing = await run('getent', ['group', group], null).catch((error: unknown) => {
+ ignoreFailure(error);
+ return null;
+ });
+ if (!existing) await run('groupadd', ['--system', group], null);
+ const line = existing ?? (await run('getent', ['group', group], null));
+ const gid = Number(line.stdout.trim().split(':')[2]);
+ if (!Number.isInteger(gid) || gid <= 0)
+ throw new Error(`Could not read the gid of the group ${group}.`);
+ return gid;
+}
+
+async function ensureAgentUser(user: string, group: string, run: CommandRunner): Promise<void> {
+ const existing = await run('getent', ['passwd', user], null).catch((error: unknown) => {
+ ignoreFailure(error);
+ return null;
+ });
+ if (existing) {
+ const groups = (await run('id', ['-gn', user], null)).stdout.trim();
+ if (groups !== group)
+ throw new ConfigurationError(
+ `The user ${user} exists, but its group is ${groups}, not ${group}; it was not made by this setup.`
+ );
+ return;
+ }
+ await run(
+ 'useradd',
+ [
+ '--system',
+ '--gid',
+ group,
+ '--no-create-home',
+ '--home-dir',
+ '/nonexistent',
+ '--shell',
+ '/usr/sbin/nologin',
+ '--comment',
+ 'Switch agent',
+ user,
+ ],
+ null
+ );
+}
+
+/**
+ * Where the setup finds the system's own commands (`useradd`, `systemctl`,
+ * `find`…), whatever PATH it runs with: the PATH it is given is the one the
+ * controller and its agents run with, which has no business holding `sbin`.
+ */
+const SYSTEM_PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin';
+
+/** `run`, with each command found on `SYSTEM_PATH` and run by its absolute path. */
+function onSystemPath(run: CommandRunner): CommandRunner {
+ return async (file, args, input) => run(await which(file, SYSTEM_PATH), args, input);
+}
+
+async function which(command: string, path: string): Promise<string> {
+ for (const directory of path.split(delimiter)) {
+ if (!isAbsolute(directory)) continue;
+ const candidate = join(directory, command);
+ try {
+ await access(candidate);
+ if (isUnitSafe(candidate)) return candidate;
+ } catch {
+ // not in this directory
+ }
+ }
+ throw new ConfigurationError(
+ `${command} is not installed in ${path.split(delimiter).join(', ')}, and the setup needs it.`
+ );
+}
diff --git a/console/packages/agent-controller/src/status.test.ts b/console/packages/agent-controller/src/status.test.ts
index 79a04f548..1d3a268a3 100644
--- a/console/packages/agent-controller/src/status.test.ts
+++ b/console/packages/agent-controller/src/status.test.ts
@@ -1,10 +1,12 @@
import { chmodSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
+import { generateSealingKeyPair, sealProviderLogin } from '@switch-console/agent-providers';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { silentLogger } from './log';
import { type AgentObservation, emptyObservation } from './runtime';
-import { type AgentAssignment, statusReportSchema } from './schemas';
+import { type AgentAssignment, type SealedLoginResponse, statusReportSchema } from './schemas';
+import { SealedLogins } from './sealed-logins';
import {
mapAgentProcess,
PathProviderLocator,
@@ -232,6 +234,7 @@ describe('ProviderStatuses', () => {
const statuses = new ProviderStatuses({
locator,
runtime,
+ sealed: null,
probeCwd: '/tmp',
now: () => clock,
log: silentLogger,
@@ -256,6 +259,117 @@ describe('ProviderStatuses', () => {
});
});
+describe('ProviderStatuses with logins given to the machine', () => {
+ const keys = generateSealingKeyPair();
+ let answer: SealedLoginResponse | null;
+ let reachable: boolean;
+
+ function sealedAnswer(credential: string, revision = 1): SealedLoginResponse {
+ return {
+ provider: 'claude',
+ revision,
+ sealed: sealProviderLogin({
+ publicKey: keys.publicKey,
+ controllerId: 'controller-1',
+ provider: 'claude',
+ login: { kind: 'setup-token', credential },
+ }),
+ };
+ }
+
+ function build(runtime: FakeRuntime) {
+ return new ProviderStatuses({
+ locator: new FakeLocator(),
+ runtime,
+ sealed: new SealedLogins({
+ client: {
+ sealedLogin: async () => {
+ if (!reachable) throw new Error('Switch is unreachable');
+ return answer;
+ },
+ },
+ keys,
+ controllerId: 'controller-1',
+ }),
+ probeCwd: '/tmp',
+ now: () => NOW,
+ log: silentLogger,
+ onChange: () => {},
+ });
+ }
+
+ beforeEach(() => {
+ answer = null;
+ reachable = true;
+ });
+
+ it('uses the machine’s own login first, and never asks Switch then', async () => {
+ const runtime = new FakeRuntime();
+ answer = sealedAnswer('sk-ant-oat-given');
+ const statuses = build(runtime);
+ expect(await statuses.check('claude')).toMatchObject({ auth: 'ok', auth_source: 'local' });
+ expect(await statuses.givenLogin('claude')).toBeNull();
+ expect(runtime.loginProbes).toEqual([]);
+ });
+
+ it('uses a given login once the provider signs in with it', async () => {
+ const runtime = new FakeRuntime();
+ runtime.readiness = { status: 'unauthenticated', message: 'Not signed in.', models: [] };
+ answer = sealedAnswer('sk-ant-oat-given', 3);
+ const statuses = build(runtime);
+ expect(await statuses.check('claude')).toMatchObject({ auth: 'ok', auth_source: 'sealed' });
+ expect(await statuses.givenLogin('claude')).toEqual({
+ status: 'connected',
+ provider: 'claude',
+ revision: '3',
+ kind: 'setup-token',
+ credential: 'sk-ant-oat-given',
+ });
+ expect(runtime.loginProbes.map((login) => login.credential)).toEqual(['sk-ant-oat-given']);
+ });
+
+ it('says why a given login is not used', async () => {
+ const runtime = new FakeRuntime();
+ runtime.readiness = { status: 'unauthenticated', message: '', models: [] };
+ const statuses = build(runtime);
+ expect(await statuses.check('claude')).toMatchObject({
+ auth: 'missing',
+ reason: 'provider_login_missing',
+ });
+ expect(statuses.loginProblem('claude')?.code).toBe('provider_login_missing');
+
+ answer = sealedAnswer('sk-ant-oat-expired');
+ runtime.loginReadiness = { status: 'unauthenticated', message: 'Token expired.', models: [] };
+ expect(await statuses.check('claude')).toMatchObject({
+ auth: 'expired',
+ auth_source: 'sealed',
+ reason: 'provider_login_expired',
+ });
+ expect(statuses.loginProblem('claude')?.message).toContain('Token expired.');
+ expect(await statuses.givenLogin('claude')).toBeNull();
+
+ answer = {
+ ...sealedAnswer('sk-ant-oat-x'),
+ sealed: { ...sealedAnswer('y').sealed, nonce: answer.sealed.nonce },
+ };
+ expect(await statuses.check('claude')).toMatchObject({ auth: 'missing' });
+ expect(statuses.loginProblem('claude')?.code).toBe('internal');
+ });
+
+ it('keeps the login it holds while Switch cannot be asked', async () => {
+ const runtime = new FakeRuntime();
+ runtime.readiness = { status: 'unauthenticated', message: '', models: [] };
+ answer = sealedAnswer('sk-ant-oat-given');
+ const statuses = build(runtime);
+ await statuses.check('claude');
+ reachable = false;
+ expect(await statuses.check('claude')).toMatchObject({ auth: 'ok', auth_source: 'sealed' });
+ expect((await statuses.givenLogin('claude'))?.credential).toBe('sk-ant-oat-given');
+ statuses.disableGiven();
+ expect(await statuses.givenLogin('claude')).toBeNull();
+ });
+});
+
describe('PathProviderLocator', () => {
let dir: string;
beforeEach(() => {
@@ -293,6 +407,7 @@ describe('StatusCollector', () => {
const providers = new ProviderStatuses({
locator: new FakeLocator(),
runtime,
+ sealed: null,
probeCwd: dir,
now: () => clock,
log: silentLogger,
diff --git a/console/packages/agent-controller/src/status.ts b/console/packages/agent-controller/src/status.ts
index 0568d8166..b2f7ac7bf 100644
--- a/console/packages/agent-controller/src/status.ts
+++ b/console/packages/agent-controller/src/status.ts
@@ -21,6 +21,7 @@ import {
type ReasonCode,
type StatusReport,
} from './schemas';
+import type { GivenLogin, LoginProblem, SealedLogins } from './sealed-logins';
import type { AgentRow, ControllerStore } from './store';
export const PROVIDER_TTL_MS = 10 * 60 * 1000;
@@ -116,22 +117,41 @@ export function providerStatusFrom(
}
}
+/**
+ * A provider as last checked: its status, its CLI, the login Switch gave the
+ * machine when that is what agents use, and why a given login is not used.
+ */
+type ProviderEntry = {
+ status: ProviderStatus;
+ path: string | null;
+ at: number;
+ login: GivenLogin | null;
+ problem: LoginProblem | { code: 'provider_login_expired'; message: string } | null;
+ /** Which check this is, in the order they started. */
+ sequence: number;
+};
+
/**
* Each provider's installation and login, checked at most every ten minutes
* unless a recheck is forced. A check that finds something different calls
* `onChange`, so the next status report carries it.
+ *
+ * The machine's own login comes first: an agent uses a login Switch gave the
+ * machine (`sealed`) only for a provider it has none of its own for, and only
+ * once the provider signs in with it.
*/
export class ProviderStatuses {
- private readonly entries = new Map<
- Provider,
- { status: ProviderStatus; path: string | null; at: number }
- >();
+ private readonly entries = new Map<Provider, ProviderEntry>();
private refreshing: Promise<void> | null = null;
+ private givenDisabled = false;
+ private checks = 0;
constructor(
private readonly deps: {
locator: ProviderLocator;
runtime: Pick<AgentRuntime, 'probe'>;
+ /** The logins Switch gives this machine; null when it cannot open any. */
+ sealed: SealedLogins | null;
probeCwd: string;
now: () => number;
log: Logger;
@@ -139,28 +159,136 @@ export class ProviderStatuses {
}
) {}
- /** Checks one provider now. */
+ /** Checks one provider now, its own login first, then one given to the machine. */
async check(provider: Provider): Promise<ProviderStatus> {
+ const sequence = ++this.checks;
const at = this.deps.now();
+ const checkedAt = new Date(at).toISOString();
const located = await this.deps.locator.locate(provider);
- let readiness: ProviderReadiness | null = null;
- if (located)
- try {
- readiness = await this.deps.runtime.probe(provider, located.path, this.deps.probeCwd);
- } catch (error) {
- this.deps.log.warn('Could not check a provider’s login', {
- provider,
- error: errorMessage(error),
- });
+ const readiness = located ? await this.probe(provider, located.path, null) : null;
+ let status = providerStatusFrom(provider, located, readiness, checkedAt);
+ let login: GivenLogin | null = null;
+ let problem: ProviderEntry['problem'] = null;
+ if (
+ located &&
+ readiness?.status !== 'authenticated' &&
+ this.deps.sealed &&
+ !this.givenDisabled
+ ) {
+ const given = await this.fetchGiven(provider);
+ if (given && 'login' in given) {
+ const signedIn = await this.probe(provider, located.path, given.login);
+ if (signedIn?.status === 'authenticated') {
+ login = given.login;
+ status = { ...status, auth: 'ok', auth_source: 'sealed', reason: undefined };
+ } else {
+ problem = {
+ code: 'provider_login_expired',
+ message: `The ${provider} login given to this machine does not sign in${signedIn?.message ? `: ${signedIn.message}` : '.'}`,
+ };
+ status = {
+ ...status,
+ auth: 'expired',
+ auth_source: 'sealed',
+ reason: 'provider_login_expired',
+ };
+ }
+ } else if (given) problem = given.problem;
+ else {
+ const held = this.entries.get(provider);
+ login = held?.login ?? null;
+ problem = held?.problem ?? null;
+ if (held?.status.auth_source === 'sealed')
+ status = {
+ ...status,
+ auth: held.status.auth,
+ auth_source: 'sealed',
+ reason: held.status.reason,
+ };
}
- const status = providerStatusFrom(provider, located, readiness, new Date(at).toISOString());
+ }
+ if (status.reason === undefined) delete status.reason;
const previous = this.entries.get(provider);
- this.entries.set(provider, { status, path: located?.path ?? null, at });
- if (!previous || fingerprintProvider(previous.status) !== fingerprintProvider(status))
+ // A check that started before the one already recorded knows less.
+ if (previous && previous.sequence > sequence) return previous.status;
+ this.entries.set(provider, {
+ status,
+ path: located?.path ?? null,
+ at,
+ login,
+ problem,
+ sequence,
+ });
+ if (
+ !previous ||
+ fingerprintProvider(previous.status) !== fingerprintProvider(status) ||
+ previous.login?.revision !== login?.revision
+ )
this.deps.onChange();
return status;
}
+ /**
+ * The login Switch gave the machine for `provider`, when that is what its
+ * agents use: null when they use the machine's own, or there is none.
+ * Checks first when the provider never was, or the check is stale.
+ */
+ async givenLogin(provider: Provider): Promise<GivenLogin | null> {
+ const entry = this.entries.get(provider);
+ if (!entry || this.deps.now() - entry.at >= PROVIDER_TTL_MS) await this.check(provider);
+ return this.entries.get(provider)?.login ?? null;
+ }
+
+ /** Stops using logins given to the machine: Switch could not take this machine's key. */
+ disableGiven(): void {
+ this.givenDisabled = true;
+ for (const entry of this.entries.values()) entry.login = null;
+ this.deps.onChange();
+ }
+
+ /** Why the login given to the machine for `provider` is not used, as last checked. */
+ loginProblem(provider: Provider): ProviderEntry['problem'] {
+ return this.entries.get(provider)?.problem ?? null;
+ }
+
+ private async probe(
+ provider: Provider,
+ path: string,
+ login: GivenLogin | null
+ ): Promise<ProviderReadiness | null> {
+ try {
+ return await this.deps.runtime.probe(provider, path, this.deps.probeCwd, login);
+ } catch (error) {
+ this.deps.log.warn('Could not check a provider’s login', {
+ provider,
+ given: login !== null,
+ error: errorMessage(error),
+ });
+ return null;
+ }
+ }
+
+ /** The given login, or why there is none; null when Switch could not be asked, keeping what was held. */
+ private async fetchGiven(
+ provider: Provider
+ ): Promise<{ login: GivenLogin } | { problem: LoginProblem } | null> {
+ try {
+ const given = await this.deps.sealed!.fetch(provider);
+ if ('problem' in given && given.problem.code === 'internal')
+ this.deps.log.error('A login given to this machine cannot be opened', {
+ provider,
+ error: given.problem.message,
+ });
+ return given;
+ } catch (error) {
+ this.deps.log.warn('Could not fetch the login given to this machine; keeping the one held', {
+ provider,
+ error: errorMessage(error),
+ });
+ return null;
+ }
+ }
+
/** Checks every provider whose last check is older than the TTL, all at once. */
async refreshStale(): Promise<void> {
this.refreshing ??= Promise.all(
diff --git a/console/packages/agent-controller/src/store.test.ts b/console/packages/agent-controller/src/store.test.ts
index bce76aead..0a3323496 100644
--- a/console/packages/agent-controller/src/store.test.ts
+++ b/console/packages/agent-controller/src/store.test.ts
@@ -244,3 +244,18 @@ describe('ControllerStore', () => {
reopened.close();
});
});
+
+describe('agent users', () => {
+ it('claims the lowest free one, keeps it, and frees it', () => {
+ const store = ControllerStore.open(path);
+ expect(store.agentUser('agent-1')).toBeNull();
+ expect(store.claimAgentUser('agent-1', 2, '2026-01-01T00:00:00Z')).toBe(1);
+ expect(store.claimAgentUser('agent-1', 2, '2026-01-01T00:00:01Z')).toBe(1);
+ expect(store.claimAgentUser('agent-2', 2, '2026-01-01T00:00:02Z')).toBe(2);
+ expect(store.claimAgentUser('agent-3', 2, '2026-01-01T00:00:03Z')).toBeNull();
+ store.releaseAgentUser('agent-1');
+ expect(store.agentUser('agent-1')).toBeNull();
+ expect(store.claimAgentUser('agent-3', 2, '2026-01-01T00:00:04Z')).toBe(1);
+ store.close();
+ });
+});
diff --git a/console/packages/agent-controller/src/store.ts b/console/packages/agent-controller/src/store.ts
index c2bb2f69b..9612aaecf 100644
--- a/console/packages/agent-controller/src/store.ts
+++ b/console/packages/agent-controller/src/store.ts
@@ -63,6 +63,13 @@ const MIGRATIONS: readonly string[] = [
cursor INTEGER NOT NULL,
updated_at TEXT NOT NULL
);`,
+ // Which of the Linux users set up for agents each agent runs as, when the
+ // controller runs every agent as a user of its own.
+ `CREATE TABLE agent_users (
+ slot INTEGER PRIMARY KEY,
+ agent_id TEXT NOT NULL UNIQUE,
+ claimed_at TEXT NOT NULL
+ );`,
];
export const STORE_SCHEMA_VERSION = MIGRATIONS.length;
@@ -276,6 +283,42 @@ export class ControllerStore {
.run(agentId, cursor, at);
}
+ /** The agent user (`1`…) the agent runs as, or null when it holds none. */
+ agentUser(agentId: string): number | null {
+ const row = this.db.prepare('SELECT slot FROM agent_users WHERE agent_id = ?').get(agentId) as
+ | Row
+ | undefined;
+ return row ? Number(row.slot) : null;
+ }
+
+ /**
+ * The agent user the agent runs as, claiming the lowest free one of
+ * `1`…`count` when it holds none; null when every one is taken.
+ */
+ claimAgentUser(agentId: string, count: number, at: string): number | null {
+ return this.transaction(() => {
+ const held = this.agentUser(agentId);
+ if (held !== null) return held;
+ const taken = new Set(
+ (this.db.prepare('SELECT slot FROM agent_users').all() as Row[]).map((row) =>
+ Number(row.slot)
+ )
+ );
+ for (let slot = 1; slot <= count; slot++)
+ if (!taken.has(slot)) {
+ this.db
+ .prepare('INSERT INTO agent_users (slot, agent_id, claimed_at) VALUES (?, ?, ?)')
+ .run(slot, agentId, at);
+ return slot;
+ }
+ return null;
+ });
+ }
+
+ releaseAgentUser(agentId: string): void {
+ this.db.prepare('DELETE FROM agent_users WHERE agent_id = ?').run(agentId);
+ }
+
/** The loopback port the relay last listened on, which running watchers were given. */
relayPort(): number | null {
const value = this.meta('relay_port');
diff --git a/console/packages/agent-controller/src/systemd-runtime.e2e.test.ts b/console/packages/agent-controller/src/systemd-runtime.e2e.test.ts
new file mode 100644
index 000000000..f4df3498e
--- /dev/null
+++ b/console/packages/agent-controller/src/systemd-runtime.e2e.test.ts
@@ -0,0 +1,278 @@
+import { execFileSync } from 'node:child_process';
+import { existsSync, readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { setTimeout as delay } from 'node:timers/promises';
+import { sealProviderLogin } from '@switch-console/agent-providers';
+import { afterAll, describe, expect, it } from 'vitest';
+import { type ControllerExit, runController } from './controller';
+import { silentLogger } from './log';
+import type { AgentAssignment, StatusReport } from './schemas';
+import { CONTROLLER_CREDENTIAL, FileSecretStore } from './secrets';
+import {
+ agentRoot,
+ loadSeparateUsersConfig,
+ type SeparateUsersConfig,
+ separateUserNames,
+} from './separate-users';
+import { ControllerStore } from './store';
+import { SystemdRuntime, systemctl } from './systemd-runtime';
+import { FakeCore } from './testing/fake-core';
+
+/**
+ * Runs real agents as users of their own, under systemd, against a fake Core.
+ * It needs a machine set up with `install-service --separate-users` for a
+ * data directory holding a `controller.db`, with the controller service
+ * stopped, and runs as that controller's user with the agents' group:
+ *
+ * sudo setpriv --reuid=$USER --regid=$(id -g) --groups=$(id -g),<agents gid> --reset-env \
+ * env SWITCH_SEPARATE_USERS_E2E=<data dir> SWITCH_SEPARATE_USERS_E2E_CLAUDE=<fake claude> \
+ * PATH=$PATH HOME=$HOME pnpm exec vitest run systemd-runtime.e2e
+ *
+ * The fake `claude`, outside any home directory, answers `auth status` with
+ * `{"loggedIn": true}` only when `CLAUDE_CODE_OAUTH_TOKEN` is set: the machine
+ * has no Claude login of its own, and one given to it signs in.
+ */
+const dataDir = process.env.SWITCH_SEPARATE_USERS_E2E;
+const fakeClaude = process.env.SWITCH_SEPARATE_USERS_E2E_CLAUDE ?? '/usr/bin/claude';
+
+function agent(agentId: string, name: string): AgentAssignment {
+ return {
+ agent_id: agentId,
+ revision: 1,
+ desired_state: 'running',
+ definition: {
+ name,
+ display_name: null,
+ icon_url: null,
+ provider: 'claude',
+ model: null,
+ advanced_config: {},
+ instructions: '',
+ auto_approve: false,
+ directory: null,
+ isolation: 'shared',
+ },
+ };
+}
+
+async function waitFor(condition: () => boolean, what: string, timeoutMs = 30_000): Promise<void> {
+ const deadline = Date.now() + timeoutMs;
+ while (!condition()) {
+ if (Date.now() > deadline) throw new Error(`Timed out waiting for ${what}.`);
+ await delay(100);
+ }
+}
+
+/** The agent's process in the latest status report from `since` on that names it. */
+function latestProcess(core: FakeCore, agentId: string, since = 0): string | undefined {
+ const reports: StatusReport[] = core.statusReports;
+ for (let index = reports.length - 1; index >= since; index--) {
+ const entry = reports[index]!.agents.find((status) => status.agent_id === agentId);
+ if (entry) return entry.process;
+ }
+ return undefined;
+}
+
+async function unitUser(unit: string): Promise<string> {
+ const pid = (await systemctl(['show', unit, '--property=MainPID', '--value'])).trim();
+ return execFileSync('ps', ['-o', 'user:32=', '-p', pid], { encoding: 'utf8' }).trim();
+}
+
+describe.skipIf(!dataDir)('agents as users of their own, under systemd', () => {
+ let config: SeparateUsersConfig;
+ let store: ControllerStore;
+ let core: FakeCore;
+ let stop: AbortController;
+ let running: Promise<ControllerExit>;
+
+ afterAll(async () => {
+ stop?.abort();
+ await running?.catch(() => {});
+ await core?.stop();
+ for (const slot of [1, 2, 3])
+ await systemctl(['stop', separateUserNames(config.uid).agentUnit(slot)]).catch(() => {});
+ store?.close();
+ });
+
+ it('runs each agent as its own user, and frees the user when the agent goes', async () => {
+ config = await loadSeparateUsersConfig(process.getuid!(), dataDir!);
+ const names = separateUserNames(config.uid);
+ store = ControllerStore.open(join(dataDir!, 'controller.db'));
+ core = new FakeCore();
+ await core.start();
+ store.saveIdentity({
+ controllerId: core.controllerId,
+ server: core.url,
+ name: 'e2e-box',
+ enrolledAt: '2026-01-01T00:00:00Z',
+ });
+ core.setAssignment({
+ revision: 1,
+ agents: [agent('e2e-agent-1', 'first'), agent('e2e-agent-2', 'second')],
+ });
+ // A store that outlives the process, as a machine's own does: it keeps the sealing key.
+ const secrets = new FileSecretStore(join(dataDir!, 'secrets'));
+ await secrets.set(CONTROLLER_CREDENTIAL, core.credential);
+ stop = new AbortController();
+ running = runController(
+ {
+ store,
+ secrets,
+ runtime: () =>
+ new SystemdRuntime({
+ config,
+ store,
+ systemctl,
+ env: { ...process.env, ANTHROPIC_API_KEY: "sk-e2e-not-real-'quoted'" },
+ log: silentLogger,
+ now: Date.now,
+ }),
+ locator: {
+ locate: async (provider) =>
+ provider === 'claude' ? { path: fakeClaude, version: '2.0.0' } : null,
+ },
+ fetch,
+ openWebSocket: core.openWebSocket,
+ log: silentLogger,
+ dataDir: dataDir!,
+ workspacesFor: () => join(dataDir!, 'workspaces'),
+ version: '0.1.0',
+ now: Date.now,
+ random: () => 0,
+ timing: {
+ resyncMs: 60_000,
+ statusPollMs: 200,
+ statusMinGapMs: 100,
+ defaultReportWithinS: 60,
+ streamIdleMs: 5_000,
+ streamInitialBackoffMs: 50,
+ streamMaxBackoffMs: 500,
+ eventBufferLimit: 100,
+ },
+ },
+ stop.signal
+ );
+
+ await waitFor(
+ () =>
+ latestProcess(core, 'e2e-agent-1') === 'running' &&
+ latestProcess(core, 'e2e-agent-2') === 'running',
+ 'both agents running'
+ );
+ const first = store.agentUser('e2e-agent-1');
+ const second = store.agentUser('e2e-agent-2');
+ expect(new Set([first, second])).toEqual(new Set([1, 2]));
+ expect(await unitUser(names.agentUnit(first!))).toBe(names.agentUser(first!));
+ expect(await unitUser(names.agentUnit(second!))).toBe(names.agentUser(second!));
+ const root = agentRoot(config, first!);
+ expect(readFileSync(join(root, '.switch-agent-id'), 'utf8').trim()).toBe('e2e-agent-1');
+ const environment = readFileSync(join(dataDir!, 'units', `0${first}`, 'environment'), 'utf8');
+ expect(environment).toContain('ANTHROPIC_API_KEY="sk-e2e-not-real-\'quoted\'"');
+
+ // The machine has no Claude login of its own; its owner gives it one.
+ await waitFor(() => core.publicKey !== null, 'the machine registering its key');
+ core.sealedLogins.set('claude', {
+ revision: 1,
+ sealed: sealProviderLogin({
+ publicKey: core.publicKey!,
+ controllerId: core.controllerId,
+ provider: 'claude',
+ login: { kind: 'setup-token', credential: 'sk-ant-oat-e2e-given' },
+ }),
+ });
+ core.addOperation({
+ id: 'op-give',
+ kind: 'provider.login',
+ agent_id: null,
+ params: { provider: 'claude', method: 'sealed', revision: 1 },
+ created_at: '2026-01-01T00:00:00Z',
+ });
+ core.push('operation.pending', {
+ operation_id: 'op-give',
+ kind: 'provider.login',
+ agent_id: null,
+ });
+ await waitFor(() => core.results.has('op-give'), 'the given login taken up');
+ expect(core.results.get('op-give')).toMatchObject({ outcome: 'succeeded' });
+ const configPath = join(root, 'watcher', 'config.json');
+ await waitFor(
+ () =>
+ JSON.parse(readFileSync(configPath, 'utf8')).start.input.env.CLAUDE_CODE_OAUTH_TOKEN ===
+ 'sk-ant-oat-e2e-given',
+ 'the agent configured with the given login'
+ );
+ const relay = JSON.parse(
+ readFileSync(join(dataDir!, 'units', `0${first}`, 'relay.json'), 'utf8')
+ );
+ expect(relay.providerLogin).toMatchObject({ provider: 'claude', revision: '1' });
+ await waitFor(
+ () => latestProcess(core, 'e2e-agent-1', core.statusReports.length - 1) === 'running',
+ 'the agent running again on the given login'
+ );
+
+ core.setAssignment({ revision: 2, agents: [agent('e2e-agent-1', 'first')] });
+ core.push('assignment.changed', { revision: 2 });
+ await waitFor(() => store.agentUser('e2e-agent-2') === null, 'the second agent’s user freed');
+ const state = await systemctl(['is-active', names.agentUnit(second!)]).catch(
+ (error: { stdout?: string }) => error.stdout ?? ''
+ );
+ expect(state.trim()).toBe('inactive');
+ expect(existsSync(join(config.agentsDir, 'released', 'e2e-agent-2', 'watcher'))).toBe(true);
+
+ // A new agent takes the freed user; the removed one comes back as another, with its directory.
+ core.setAssignment({
+ revision: 3,
+ agents: [agent('e2e-agent-1', 'first'), agent('e2e-agent-3', 'third')],
+ });
+ core.push('assignment.changed', { revision: 3 });
+ core.push('agent.attached', { agent_id: 'e2e-agent-3', from_seq: 0, rooms: [] });
+ await waitFor(
+ () => latestProcess(core, 'e2e-agent-3') === 'running',
+ 'the third agent running'
+ );
+ expect(store.agentUser('e2e-agent-3')).toBe(second);
+ core.setAssignment({
+ revision: 4,
+ agents: [
+ agent('e2e-agent-1', 'first'),
+ agent('e2e-agent-3', 'third'),
+ agent('e2e-agent-2', 'second'),
+ ],
+ });
+ const reported = core.statusReports.length;
+ core.push('assignment.changed', { revision: 4 });
+ core.push('agent.attached', { agent_id: 'e2e-agent-2', from_seq: 0, rooms: [] });
+ await waitFor(
+ () => latestProcess(core, 'e2e-agent-2', reported) === 'running',
+ 'the second agent back'
+ );
+ const moved = store.agentUser('e2e-agent-2')!;
+ expect(moved).not.toBe(second);
+ expect(await unitUser(names.agentUnit(moved))).toBe(names.agentUser(moved));
+ const back = JSON.parse(
+ readFileSync(join(agentRoot(config, moved), 'watcher', 'config.json'), 'utf8')
+ );
+ expect(back.start.input.cwd).toBe(join(config.agentsDir, 'agent', 'workspace'));
+ const owners = execFileSync(
+ 'find',
+ [
+ join(agentRoot(config, moved), 'watcher'),
+ '-mindepth',
+ '1',
+ '-maxdepth',
+ '1',
+ '-printf',
+ '%u\n',
+ ],
+ { encoding: 'utf8' }
+ )
+ .split('\n')
+ .filter(Boolean);
+ expect(new Set(owners)).toEqual(new Set([config.user, names.agentUser(moved)]));
+
+ stop.abort();
+ expect(await running).toBe('stopped');
+ // Agents outlive the controller.
+ expect((await systemctl(['is-active', names.agentUnit(first!)])).trim()).toBe('active');
+ }, 120_000);
+});
diff --git a/console/packages/agent-controller/src/systemd-runtime.test.ts b/console/packages/agent-controller/src/systemd-runtime.test.ts
new file mode 100644
index 000000000..9ea179a6f
--- /dev/null
+++ b/console/packages/agent-controller/src/systemd-runtime.test.ts
@@ -0,0 +1,305 @@
+import {
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ rmSync,
+ statSync,
+ writeFileSync,
+} from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { afterEach, beforeEach, describe, expect, it } from 'vitest';
+import { silentLogger } from './log';
+import type { SeparateUsersConfig } from './separate-users';
+import { ControllerStore } from './store';
+import { environmentLine, SystemdRuntime } from './systemd-runtime';
+import { buildWatcherTemplate } from './template';
+
+const RELAY = {
+ endpoint: 'http://127.0.0.1:43210',
+ token: 'swlr_relay-token-placeholder',
+ hub: 'ws://127.0.0.1:43210/hub',
+ providerLogin: null,
+};
+const LAUNCH = {
+ isolation: 'shared' as const,
+ restart: false,
+ replaceIdentity: false,
+ clearTakenOver: false,
+};
+
+/** A `systemctl` that records what it is asked and answers from `units`. */
+class FakeSystemctl {
+ readonly calls: string[][] = [];
+ readonly units = new Map<string, { activeState: string; result: string; mainPid: number }>();
+
+ readonly run = async (args: string[]): Promise<string> => {
+ this.calls.push(args);
+ const verb = args[0] === '--no-block' ? args[1] : args[0];
+ const unit = args[0] === '--no-block' ? args[2]! : args[1]!;
+ const state = this.units.get(unit) ?? {
+ activeState: 'inactive',
+ result: 'success',
+ mainPid: 0,
+ };
+ if (verb === 'show')
+ return `ActiveState=${state.activeState}\nResult=${state.result}\nMainPID=${state.mainPid}\n`;
+ if (verb === 'start') this.units.set(unit, { ...state, activeState: 'active', mainPid: 4242 });
+ if (verb === 'stop') this.units.set(unit, { ...state, activeState: 'inactive', mainPid: 0 });
+ if (verb === 'reset-failed') this.units.set(unit, { ...state, activeState: 'inactive' });
+ return '';
+ };
+
+ verbs(): string[] {
+ return this.calls.filter((call) => call[0] !== 'show').map((call) => call.join(' '));
+ }
+}
+
+let dir: string;
+let config: SeparateUsersConfig;
+let store: ControllerStore;
+let systemctl: FakeSystemctl;
+let runtime: SystemdRuntime;
+let env: NodeJS.ProcessEnv;
+
+function makeRuntime(): SystemdRuntime {
+ return new SystemdRuntime({
+ config,
+ store,
+ systemctl: systemctl.run,
+ env,
+ log: silentLogger,
+ now: () => 1_000,
+ });
+}
+
+function template(agentId: string, cwd: string, binaryPath = '/usr/bin/claude') {
+ return buildWatcherTemplate({
+ agentId,
+ provider: 'claude',
+ definition: {
+ name: agentId,
+ model: null,
+ advanced_config: {},
+ instructions: '',
+ auto_approve: false,
+ },
+ cwd,
+ credentialsPath: runtime.credentialsPath(agentId),
+ binaryPath,
+ });
+}
+
+async function start(agentId: string): Promise<string> {
+ await runtime.writeCredentials(agentId, RELAY);
+ const cwd = await runtime.workingDirectory(agentId, agentId, null);
+ await runtime.launch(agentId, template(agentId, cwd), LAUNCH);
+ return cwd;
+}
+
+beforeEach(() => {
+ dir = mkdtempSync(join(tmpdir(), 'systemd-runtime-'));
+ config = {
+ v: 1,
+ user: 'controller',
+ uid: process.getuid!(),
+ gid: process.getgid!(),
+ dataDir: join(dir, 'data'),
+ agentsDir: join(dir, 'agents'),
+ agentUsers: 2,
+ node: '/usr/bin/node',
+ bundle: '/opt/switch/shared-host.mjs',
+ };
+ mkdirSync(config.dataDir, { mode: 0o700 });
+ mkdirSync(config.agentsDir, { mode: 0o700 });
+ store = ControllerStore.open(join(config.dataDir, 'controller.db'));
+ systemctl = new FakeSystemctl();
+ env = {
+ ANTHROPIC_API_KEY: 'sk-placeholder',
+ CLAUDE_CONFIG_DIR: '/home/controller/.claude',
+ UNRELATED: 'kept out',
+ };
+ runtime = makeRuntime();
+});
+
+afterEach(() => {
+ store.close();
+ rmSync(dir, { recursive: true, force: true });
+});
+
+describe('SystemdRuntime', () => {
+ it('runs an agent as the first free agent user, from files the unit loads', async () => {
+ const cwd = await start('agent-1');
+ const root = join(config.agentsDir, '01');
+ expect(store.agentUser('agent-1')).toBe(1);
+ // As the agent sees it, whichever user it runs as.
+ expect(cwd).toBe(join(config.agentsDir, 'agent', 'workspace'));
+ expect(systemctl.verbs()).toEqual([`start switch-agent-${config.uid}@01.service`]);
+ expect(runtime.credentialsPath('agent-1')).toBe(
+ `/run/credentials/switch-agent-${config.uid}@01.service/relay`
+ );
+ expect(await runtime.readCredentials('agent-1')).toEqual(RELAY);
+ expect(readFileSync(join(root, '.switch-agent-id'), 'utf8').trim()).toBe('agent-1');
+ const saved = JSON.parse(readFileSync(join(root, 'watcher', 'config.json'), 'utf8'));
+ expect(saved.session.agentId).toBe('agent-1');
+ expect(JSON.parse(readFileSync(join(root, 'watcher', 'watch.json'), 'utf8'))).toEqual({
+ enabled: true,
+ spawn: true,
+ });
+ expect(statSync(root).mode & 0o7777).toBe(0o1770);
+ expect(statSync(join(root, 'watcher', 'config.json')).mode & 0o777).toBe(0o640);
+ // Only provider settings, and none naming a path in the controller's home.
+ expect(readFileSync(join(config.dataDir, 'units', '01', 'environment'), 'utf8')).toBe(
+ "ANTHROPIC_API_KEY='sk-placeholder'\n"
+ );
+ });
+
+ it('gives each agent a user of its own, and says so when none is left', async () => {
+ await start('agent-1');
+ await start('agent-2');
+ expect(store.agentUser('agent-2')).toBe(2);
+ await expect(runtime.writeCredentials('agent-3', RELAY)).rejects.toMatchObject({
+ reason: 'capacity_exceeded',
+ });
+ });
+
+ it('hands a running agent host its new configuration without restarting it', async () => {
+ await start('agent-1');
+ const before = JSON.parse(
+ readFileSync(join(config.agentsDir, '01', 'watcher', 'config.json'), 'utf8')
+ );
+ systemctl.calls.length = 0;
+ const cwd = await runtime.workingDirectory('agent-1', 'agent-1', null);
+ await runtime.launch('agent-1', template('agent-1', cwd), LAUNCH);
+ expect(systemctl.verbs()).toEqual([]);
+ const after = JSON.parse(
+ readFileSync(join(config.agentsDir, '01', 'watcher', 'config.json'), 'utf8')
+ );
+ // The session identity an earlier run saved is kept.
+ expect(after.session.hostId).toBe(before.session.hostId);
+ });
+
+ it('restarts through a stop when asked to', async () => {
+ await start('agent-1');
+ systemctl.calls.length = 0;
+ const cwd = await runtime.workingDirectory('agent-1', 'agent-1', null);
+ await runtime.launch('agent-1', template('agent-1', cwd), { ...LAUNCH, restart: true });
+ const unit = `switch-agent-${config.uid}@01.service`;
+ expect(systemctl.verbs()).toEqual([`stop ${unit}`, `start ${unit}`]);
+ });
+
+ it('starts a failed unit again after clearing its failure', async () => {
+ await start('agent-1');
+ const unit = `switch-agent-${config.uid}@01.service`;
+ systemctl.units.set(unit, { activeState: 'failed', result: 'exit-code', mainPid: 0 });
+ expect((await runtime.observe('agent-1')).failure).toContain('failed (exit-code)');
+ systemctl.calls.length = 0;
+ const cwd = await runtime.workingDirectory('agent-1', 'agent-1', null);
+ await runtime.launch('agent-1', template('agent-1', cwd), LAUNCH);
+ expect(systemctl.verbs()).toEqual([`reset-failed ${unit}`, `start ${unit}`]);
+ });
+
+ it('refuses a working directory outside the agent’s own', async () => {
+ await runtime.writeCredentials('agent-1', RELAY);
+ await expect(runtime.workingDirectory('agent-1', 'scout', '/srv/repo')).rejects.toMatchObject({
+ reason: 'definition_invalid',
+ });
+ await expect(runtime.workingDirectory('agent-1', 'scout', 'relative')).rejects.toMatchObject({
+ reason: 'definition_invalid',
+ });
+ const inside = join(config.agentsDir, 'agent', 'workspace', 'repo');
+ expect(await runtime.workingDirectory('agent-1', 'scout', inside)).toBe(inside);
+ // Made for the agent to write in: the agents' group, and its bits.
+ const made = statSync(join(config.agentsDir, '01', 'workspace', 'repo'));
+ expect(made.gid).toBe(config.gid);
+ expect(made.mode & 0o777).toBe(0o770);
+ });
+
+ it('refuses a provider CLI agents cannot reach', async () => {
+ await runtime.writeCredentials('agent-1', RELAY);
+ const cwd = await runtime.workingDirectory('agent-1', 'agent-1', null);
+ await expect(
+ runtime.launch('agent-1', template('agent-1', cwd, '/home/me/.local/bin/claude'), LAUNCH)
+ ).rejects.toMatchObject({ reason: 'provider_not_installed' });
+ expect(
+ await runtime.probe('claude', '/home/me/.local/bin/claude', config.dataDir, null)
+ ).toMatchObject({ status: 'unconfigured' });
+ });
+
+ it('observes the agent host’s health, current only for the unit’s process', async () => {
+ await start('agent-1');
+ const health = join(config.agentsDir, '01', 'watcher', 'health.json');
+ const body = {
+ state: 'connected',
+ detail: null,
+ since: '2026-01-01T00:00:00Z',
+ placements: {},
+ pid: 4242,
+ updatedAt: '2026-01-01T00:00:00Z',
+ };
+ writeFileSync(health, JSON.stringify(body));
+ const live = await runtime.observe('agent-1');
+ expect(live.alive).toBe(true);
+ expect(live.health).toMatchObject({ pid: 4242, current: true });
+ writeFileSync(health, JSON.stringify({ ...body, pid: 1 }));
+ expect((await runtime.observe('agent-1')).health).toMatchObject({ current: false });
+ expect(await runtime.observe('agent-never-started')).toMatchObject({ alive: false });
+ });
+
+ it('sets a removed agent’s directory aside, frees its user, and gives it back later', async () => {
+ await start('agent-1');
+ writeFileSync(join(config.agentsDir, '01', 'workspace', 'notes.md'), 'kept');
+ systemctl.calls.length = 0;
+ await runtime.deleteCredentials('agent-1');
+ const unit = `switch-agent-${config.uid}@01.service`;
+ expect(systemctl.verbs()).toEqual([`stop ${unit}`]);
+ expect(store.agentUser('agent-1')).toBeNull();
+ expect(existsSync(join(config.agentsDir, '01'))).toBe(false);
+ expect(existsSync(join(config.dataDir, 'units', '01'))).toBe(false);
+ expect(
+ readFileSync(join(config.agentsDir, 'released', 'agent-1', 'workspace', 'notes.md'), 'utf8')
+ ).toBe('kept');
+
+ // Another agent takes the freed user, in a fresh directory.
+ await start('agent-2');
+ expect(store.agentUser('agent-2')).toBe(1);
+ expect(existsSync(join(config.agentsDir, '01', 'workspace', 'notes.md'))).toBe(false);
+ // The first comes back on the next free user, with its directory.
+ await start('agent-1');
+ expect(store.agentUser('agent-1')).toBe(2);
+ expect(readFileSync(join(config.agentsDir, '02', 'workspace', 'notes.md'), 'utf8')).toBe(
+ 'kept'
+ );
+ });
+
+ it('sets aside a directory its user held for another agent', async () => {
+ const stale = join(config.agentsDir, '01');
+ mkdirSync(join(stale, 'workspace'), { recursive: true });
+ writeFileSync(join(stale, '.switch-agent-id'), 'agent-old\n');
+ await start('agent-1');
+ expect(existsSync(join(config.agentsDir, 'released', 'agent-old', 'workspace'))).toBe(true);
+ expect(readFileSync(join(stale, '.switch-agent-id'), 'utf8').trim()).toBe('agent-1');
+ });
+
+ it('stops through the watch flags and the unit', async () => {
+ await start('agent-1');
+ systemctl.calls.length = 0;
+ await runtime.stop('agent-1', { wait: false });
+ const unit = `switch-agent-${config.uid}@01.service`;
+ expect(systemctl.verbs()).toEqual([`--no-block stop ${unit}`]);
+ expect(
+ JSON.parse(readFileSync(join(config.agentsDir, '01', 'watcher', 'watch.json'), 'utf8'))
+ ).toEqual({ enabled: false, spawn: false });
+ systemctl.calls.length = 0;
+ await runtime.stop('agent-unknown', { wait: true });
+ expect(systemctl.verbs()).toEqual([]);
+ });
+});
+
+describe('environmentLine', () => {
+ it('quotes a value so systemd reads it back as is', () => {
+ expect(environmentLine('KEY', 'a b$c"d')).toBe(`KEY='a b$c"d'`);
+ expect(environmentLine('KEY', `it's $x "y" \\z`)).toBe(`KEY="it's \\$x \\"y\\" \\\\z"`);
+ });
+});
diff --git a/console/packages/agent-controller/src/systemd-runtime.ts b/console/packages/agent-controller/src/systemd-runtime.ts
new file mode 100644
index 000000000..029d02601
--- /dev/null
+++ b/console/packages/agent-controller/src/systemd-runtime.ts
@@ -0,0 +1,505 @@
+import { execFile } from 'node:child_process';
+import { randomUUID } from 'node:crypto';
+import { chmod, lstat, mkdir, open, rename, rm } from 'node:fs/promises';
+import { isAbsolute, join, relative, resolve, sep } from 'node:path';
+import { promisify } from 'node:util';
+import {
+ AGENT_ENV_VARS,
+ clearTakenOver,
+ type ProviderReadiness,
+ type SharedHostConfig,
+ sharedConfigSchema,
+ WATCH_FLAGS_FILE,
+ WATCHER_HEALTH_FILE,
+ type WatchFlags,
+ watcherHealthFileSchema,
+ watchFlagsSchema,
+} from '@switch-console/agent-providers';
+import { ReasonedError } from './errors';
+import type { Logger } from './log';
+import {
+ type AgentObservation,
+ type AgentRuntime,
+ emptyObservation,
+ isInside,
+ type LaunchOptions,
+ loginProbeEnvironment,
+ parseRelayCredentials,
+ probeProvider,
+ readOptional,
+ readRoot,
+ type RelayCredentials,
+ relayCredentialsBody,
+ removeOptional,
+ writeAtomic,
+} from './runtime';
+import type { Provider } from './schemas';
+import type { GivenLogin } from './sealed-logins';
+import {
+ agentRoot,
+ homeRootOf,
+ type SeparateUsersConfig,
+ separateUserNames,
+ unitAgentRoot,
+ unitCredentialsPath,
+ unitFilesDir,
+} from './separate-users';
+import type { ControllerStore } from './store';
+
+/** `systemctl` with `args`, answering its standard output; a non-zero exit throws. */
+export type Systemctl = (args: string[]) => Promise<string>;
+
+const execute = promisify(execFile);
+
+/** The system manager's `systemctl`, never through a shell, never asking for a password. */
+export const systemctl: Systemctl = async (args) => {
+ const { stdout } = await execute('systemctl', ['--no-ask-password', ...args], {
+ timeout: 60_000,
+ maxBuffer: 1024 * 1024,
+ });
+ return stdout;
+};
+
+const LIVE_STATES = new Set(['active', 'activating', 'deactivating', 'reloading']);
+/**
+ * Sticky, so an agent can never rename or remove what the controller wrote in
+ * its directories (its configuration, its watch flags); group-writable, so it
+ * can write its own beside them.
+ */
+const CONTROLLED_DIR_MODE = 0o1770;
+/** The agent's home and workspace: entirely the agent's to write in. */
+const AGENT_DIR_MODE = 0o770;
+const SHARED_FILE_MODE = 0o640;
+/** In each agent's directory: which agent it holds, so a reused agent user never runs in another's. */
+const AGENT_MARKER = '.switch-agent-id';
+/** Under the agents' directory: the directories of agents this controller no longer runs. */
+const RELEASED = 'released';
+/** Provider settings naming a path in the controller user's home, which an agent could not reach. */
+const PATH_VARIABLES = new Set([
+ 'CLAUDE_CONFIG_DIR',
+ 'CODEX_HOME',
+ 'CLOUDSDK_CONFIG',
+ 'GOOGLE_APPLICATION_CREDENTIALS',
+ 'AWS_PROFILE',
+]);
+
+type UnitState = { activeState: string; result: string; mainPid: number };
+
+/**
+ * Runs each agent as a Linux user of its own (see `separate-users.ts`): the
+ * agent claims a free user from the pool set up for this controller, and runs
+ * as that user's instance of the agents' template unit, with systemd its
+ * supervisor. The controller writes what the unit starts from, the agent
+ * host's `config.json` and `watch.json` in its directory and the relay
+ * credentials and environment it loads, and starts and stops the unit through
+ * `systemctl`, which polkit allows it for these units only. The agent host
+ * hears its events on the controller's hub, as one in a process of its own
+ * does, and outlives the controller.
+ *
+ * Everything an agent writes is read as untrusted: through no link, parsed
+ * against its schema, and never run.
+ */
+export class SystemdRuntime implements AgentRuntime {
+ private readonly names: ReturnType<typeof separateUserNames>;
+
+ constructor(
+ private readonly deps: {
+ config: SeparateUsersConfig;
+ store: ControllerStore;
+ systemctl: Systemctl;
+ /** The controller's own environment: the provider settings its agents get come from it. */
+ env: NodeJS.ProcessEnv;
+ log: Logger;
+ now: () => number;
+ }
+ ) {
+ this.names = separateUserNames(deps.config.uid);
+ }
+
+ /** The agent user the agent runs as, claimed now when it holds none. */
+ private claim(agentId: string): number {
+ const { config, store } = this.deps;
+ const slot = store.claimAgentUser(
+ agentId,
+ config.agentUsers,
+ new Date(this.deps.now()).toISOString()
+ );
+ if (slot === null)
+ throw new ReasonedError(
+ 'capacity_exceeded',
+ `Every one of the ${config.agentUsers} agent users set up on this machine runs an agent. Run the setup again with a larger --agent-users.`
+ );
+ return slot;
+ }
+
+ credentialsPath(agentId: string): string {
+ return unitCredentialsPath(this.deps.config, this.claim(agentId));
+ }
+
+ async readCredentials(agentId: string): Promise<RelayCredentials | null> {
+ const slot = this.deps.store.agentUser(agentId);
+ if (slot === null) return null;
+ const text = await readOptional(join(unitFilesDir(this.deps.config, slot), 'relay.json'));
+ return text === null ? null : parseRelayCredentials(text, agentId);
+ }
+
+ async writeCredentials(agentId: string, credentials: RelayCredentials): Promise<void> {
+ const directory = unitFilesDir(this.deps.config, this.claim(agentId));
+ await mkdir(directory, { recursive: true, mode: 0o700 });
+ await writeAtomic(join(directory, 'relay.json'), relayCredentialsBody(agentId, credentials));
+ }
+
+ /**
+ * Forgets an agent: its unit is stopped, the files it loaded removed, its
+ * directory set aside under `released/` (to come back if the agent is given
+ * to this controller again), and its agent user freed.
+ */
+ async deleteCredentials(agentId: string): Promise<void> {
+ const slot = this.deps.store.agentUser(agentId);
+ if (slot === null) return;
+ const unit = this.names.agentUnit(slot);
+ await this.deps.systemctl(['stop', unit]);
+ if ((await this.show(unit)).activeState === 'failed')
+ await this.deps.systemctl(['reset-failed', unit]);
+ await rm(unitFilesDir(this.deps.config, slot), { recursive: true, force: true });
+ const root = agentRoot(this.deps.config, slot);
+ if (await isDirectory(root)) await this.setAside(root, agentId);
+ this.deps.store.releaseAgentUser(agentId);
+ }
+
+ /**
+ * The definition's directory, as the agent sees it, which must be inside
+ * its own directory, the only one its unit can see; a `workspace` there
+ * when it names none.
+ */
+ async workingDirectory(agentId: string, name: string, directory: string | null): Promise<string> {
+ const onMachine = await this.prepare(agentId);
+ const seen = unitAgentRoot(this.deps.config);
+ if (directory === null) return join(seen, 'workspace');
+ if (!isAbsolute(directory))
+ throw new ReasonedError(
+ 'definition_invalid',
+ `The working directory '${directory}' is not an absolute path.`
+ );
+ const resolved = resolve(directory);
+ if (resolved !== seen && !isInside(seen, resolved))
+ throw new ReasonedError(
+ 'definition_invalid',
+ `Agent ${name} would work in ${resolved}, but this machine runs each agent as a user of its own that sees only its own directory, at ${seen}. Leave the directory empty, or name one inside it.`
+ );
+ let path = onMachine;
+ for (const part of relative(seen, resolved).split(sep).filter(Boolean)) {
+ path = join(path, part);
+ if (await isDirectory(path)) continue;
+ await this.controlledDirectory(path, AGENT_DIR_MODE);
+ }
+ return resolved;
+ }
+
+ /**
+ * The provider's login as an agent would have it: only the provider
+ * settings it is given, or `login`, and a home of its own.
+ */
+ async probe(
+ provider: Provider,
+ binaryPath: string,
+ cwd: string,
+ login: GivenLogin | null
+ ): Promise<ProviderReadiness> {
+ const home = homeRootOf(binaryPath, []);
+ if (home)
+ return {
+ status: 'unconfigured',
+ message: `${binaryPath} is under ${home}, which agents running as users of their own cannot reach. Install the ${provider} CLI system-wide.`,
+ models: [],
+ };
+ const probeHome = join(this.deps.config.dataDir, 'probe-home');
+ await mkdir(probeHome, { recursive: true, mode: 0o700 });
+ const given = login
+ ? await loginProbeEnvironment(
+ join(this.deps.config.dataDir, 'login-check', provider),
+ login,
+ binaryPath
+ )
+ : {};
+ return probeProvider(this.deps.config.bundle, provider, binaryPath, cwd, {
+ ...this.agentEnvironment(),
+ PATH: this.deps.env.PATH ?? '/usr/local/bin:/usr/bin:/bin',
+ HOME: probeHome,
+ ...given,
+ });
+ }
+
+ agentStateRoot(_agentId: string): string {
+ return join(unitAgentRoot(this.deps.config), 'watcher');
+ }
+
+ async observe(agentId: string): Promise<AgentObservation> {
+ const slot = this.deps.store.agentUser(agentId);
+ if (slot === null) return emptyObservation();
+ const unit = this.names.agentUnit(slot);
+ const state = await this.show(unit);
+ const alive = LIVE_STATES.has(state.activeState);
+ const root = join(agentRoot(this.deps.config, slot), 'watcher');
+ if (!(await isDirectory(root))) return { ...emptyObservation(), alive };
+ const onDisk = await readRoot(root);
+ let health: AgentObservation['health'] = null;
+ const healthText = await readOptional(join(root, WATCHER_HEALTH_FILE));
+ if (healthText !== null) {
+ const parsed = watcherHealthFileSchema.safeParse(parseJson(healthText));
+ if (parsed.success)
+ health = { ...parsed.data, current: alive && parsed.data.pid === state.mainPid };
+ else
+ this.deps.log.warn('An agent’s health file is not one; ignoring it', {
+ agentId,
+ error: parsed.error.message,
+ });
+ }
+ const failure =
+ onDisk.failure ??
+ (state.activeState === 'failed'
+ ? `The agent's unit ${unit} failed (${state.result}); see journalctl -u ${unit}.`
+ : null);
+ return { ...onDisk, alive, health, failure };
+ }
+
+ async launch(agentId: string, template: SharedHostConfig, options: LaunchOptions): Promise<void> {
+ const slot = this.claim(agentId);
+ const unit = this.names.agentUnit(slot);
+ const watcher = join(await this.prepare(agentId), 'watcher');
+ const seen = unitAgentRoot(this.deps.config);
+ const cwd = resolve(template.start.input.cwd);
+ if (cwd !== seen && !isInside(seen, cwd))
+ throw new ReasonedError(
+ 'definition_invalid',
+ `Agent ${agentId} would work in ${cwd}, outside ${seen}, the only directory its user sees.`
+ );
+ if (template.execution?.credentialsPath !== unitCredentialsPath(this.deps.config, slot))
+ throw new Error(
+ `The launch configuration for agent ${agentId} does not read the credentials its unit loads.`
+ );
+ const binaryPath = template.execution.binaryPath;
+ if (!binaryPath)
+ throw new Error(`The launch configuration for agent ${agentId} names no provider CLI.`);
+ const home = homeRootOf(binaryPath, []);
+ if (home)
+ throw new ReasonedError(
+ 'provider_not_installed',
+ `${binaryPath} is under ${home}, which agents running as users of their own cannot reach. Install the ${template.start.provider} CLI system-wide.`
+ );
+
+ if (options.restart || options.replaceIdentity) await this.stop(agentId, { wait: true });
+ if (options.replaceIdentity) await removeOptional(join(watcher, 'config.json'));
+ if (options.clearTakenOver) await clearTakenOver(watcher);
+ await this.writeEnvironment(slot);
+ await this.writeConfig(watcher, template);
+ await this.writeFlags(watcher, { enabled: true, spawn: true });
+ const state = await this.show(unit);
+ // A running agent host reads its new configuration itself.
+ if (LIVE_STATES.has(state.activeState)) return;
+ await rm(join(watcher, 'supervisor', 'failure.json'), { force: true });
+ if (state.activeState === 'failed') await this.deps.systemctl(['reset-failed', unit]);
+ await this.deps.systemctl(['start', unit]);
+ }
+
+ async stop(agentId: string, options: { wait: boolean }): Promise<void> {
+ const slot = this.deps.store.agentUser(agentId);
+ if (slot === null) return;
+ const watcher = join(agentRoot(this.deps.config, slot), 'watcher');
+ if (await isDirectory(watcher))
+ await this.writeFlags(watcher, { enabled: false, spawn: false });
+ const unit = this.names.agentUnit(slot);
+ await this.deps.systemctl(options.wait ? ['stop', unit] : ['--no-block', 'stop', unit]);
+ }
+
+ /** Agent units outlive the controller. */
+ async close(): Promise<void> {}
+
+ /**
+ * The agent's directory, ready for its unit: the one it had under
+ * `released/` if it ran here before, another agent's set aside if its user
+ * held one, and the directories the agent host and the agent write in.
+ */
+ private async prepare(agentId: string): Promise<string> {
+ const slot = this.claim(agentId);
+ const root = agentRoot(this.deps.config, slot);
+ if (await isDirectory(root)) {
+ const holder = (await readOptional(join(root, AGENT_MARKER)))?.trim() ?? '';
+ if (holder !== agentId) await this.setAside(root, holder || `unknown-${this.deps.now()}`);
+ }
+ if (!(await isDirectory(root))) {
+ const released = join(this.deps.config.agentsDir, RELEASED, agentId);
+ if (await isDirectory(released)) {
+ await rename(released, root);
+ this.deps.log.info('Gave an agent back the directory it had here before', {
+ agentId,
+ directory: root,
+ });
+ }
+ }
+ await this.controlledDirectory(root, CONTROLLED_DIR_MODE);
+ await this.writeShared(join(root, AGENT_MARKER), `${agentId}\n`);
+ await this.controlledDirectory(join(root, 'watcher'), CONTROLLED_DIR_MODE);
+ await this.controlledDirectory(join(root, 'home'), AGENT_DIR_MODE);
+ await this.controlledDirectory(join(root, 'workspace'), AGENT_DIR_MODE);
+ return root;
+ }
+
+ /** Moves an agent's directory to `released/<agentId>`, moving aside one already there. */
+ private async setAside(root: string, agentId: string): Promise<void> {
+ const released = join(this.deps.config.agentsDir, RELEASED);
+ await mkdir(released, { recursive: true, mode: 0o700 });
+ const target = join(released, agentId);
+ if (await isDirectory(target)) await rename(target, `${target}.${this.deps.now()}`);
+ await rename(root, target);
+ this.deps.log.info('Set an agent’s directory aside', { agentId, directory: target });
+ }
+
+ /**
+ * A directory the controller owns and the agents' group may use: made when
+ * missing, refused when it is a link or someone else's, and given the group
+ * and mode the controller's umask would not.
+ */
+ private async controlledDirectory(path: string, mode: number): Promise<void> {
+ await mkdir(path, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => {
+ if (error.code !== 'EEXIST') throw error;
+ });
+ const info = await lstat(path);
+ if (info.isSymbolicLink() || !info.isDirectory())
+ throw new Error(`${path} is not a directory; an agent's directories are never followed.`);
+ if (info.uid !== this.deps.config.uid)
+ throw new Error(`${path} is not owned by the controller's user.`);
+ if (info.gid !== this.deps.config.gid) {
+ const handle = await open(path, 'r');
+ try {
+ await handle.chown(-1, this.deps.config.gid);
+ } finally {
+ await handle.close();
+ }
+ }
+ if ((info.mode & 0o7777) !== mode) await chmod(path, mode);
+ }
+
+ /** Writes a file the agent reads: whole, owned by the controller, readable by the agents' group. */
+ private async writeShared(path: string, body: string): Promise<void> {
+ const temporary = `${path}.${randomUUID()}`;
+ const file = await open(temporary, 'wx', SHARED_FILE_MODE);
+ try {
+ await file.chown(-1, this.deps.config.gid);
+ await file.chmod(SHARED_FILE_MODE);
+ await file.writeFile(body);
+ await file.sync();
+ } finally {
+ await file.close();
+ }
+ await rename(temporary, path);
+ }
+
+ /**
+ * The configuration the agent host starts from: the template, keeping the
+ * session identity an earlier run saved, as `ensureSharedProcess` does.
+ */
+ private async writeConfig(watcher: string, template: SharedHostConfig): Promise<void> {
+ const path = join(watcher, 'config.json');
+ const text = await readOptional(path);
+ let config: SharedHostConfig = template;
+ if (text !== null) {
+ const saved = sharedConfigSchema.parse(JSON.parse(text));
+ if (
+ saved.session.agentId !== template.session.agentId ||
+ saved.start.provider !== template.start.provider ||
+ saved.start.input.cwd !== template.start.input.cwd
+ )
+ throw new Error(
+ 'The saved agent host identity or working directory differs from the one asked for.'
+ );
+ config = { ...template, session: saved.session, resumeOperationId: saved.resumeOperationId };
+ }
+ await this.writeShared(path, JSON.stringify(config));
+ }
+
+ private writeFlags(watcher: string, flags: WatchFlags): Promise<void> {
+ return this.writeShared(
+ join(watcher, WATCH_FLAGS_FILE),
+ JSON.stringify(watchFlagsSchema.parse(flags))
+ );
+ }
+
+ /**
+ * The provider settings from the controller's environment an agent is
+ * given, but those naming a path in the controller user's home.
+ */
+ private agentEnvironment(): Record<string, string> {
+ const env: Record<string, string> = {};
+ for (const name of AGENT_ENV_VARS) {
+ const value = this.deps.env[name];
+ if (value === undefined || PATH_VARIABLES.has(name)) continue;
+ env[name] = value;
+ }
+ return env;
+ }
+
+ /** The environment file the unit loads, in the syntax systemd reads. */
+ private async writeEnvironment(slot: number): Promise<void> {
+ const lines: string[] = [];
+ for (const [name, value] of Object.entries(this.agentEnvironment())) {
+ if (/[\n\r]/.test(value))
+ throw new ReasonedError(
+ 'definition_invalid',
+ `The controller's ${name} holds a line break, which an agent's environment cannot carry.`
+ );
+ lines.push(environmentLine(name, value));
+ }
+ const directory = unitFilesDir(this.deps.config, slot);
+ await mkdir(directory, { recursive: true, mode: 0o700 });
+ await writeAtomic(join(directory, 'environment'), lines.length ? `${lines.join('\n')}\n` : '');
+ }
+
+ private async show(unit: string): Promise<UnitState> {
+ const stdout = await this.deps.systemctl([
+ 'show',
+ unit,
+ '--property=ActiveState,Result,MainPID',
+ ]);
+ const values = new Map<string, string>();
+ for (const line of stdout.split('\n')) {
+ const at = line.indexOf('=');
+ if (at > 0) values.set(line.slice(0, at), line.slice(at + 1).trim());
+ }
+ const activeState = values.get('ActiveState');
+ if (!activeState) throw new Error(`systemctl show did not report the state of ${unit}.`);
+ return {
+ activeState,
+ result: values.get('Result') ?? '',
+ mainPid: Number(values.get('MainPID') ?? 0) || 0,
+ };
+ }
+}
+
+async function isDirectory(path: string): Promise<boolean> {
+ try {
+ const info = await lstat(path);
+ return info.isDirectory() && !info.isSymbolicLink();
+ } catch (error) {
+ if ((error as NodeJS.ErrnoException).code === 'ENOENT') return false;
+ throw error;
+ }
+}
+
+function parseJson(text: string): unknown {
+ try {
+ return JSON.parse(text);
+ } catch {
+ return null;
+ }
+}
+
+/**
+ * `NAME='value'` as systemd's `EnvironmentFile=` reads it: single quotes keep
+ * everything literal; a value holding one is double-quoted instead, with the
+ * characters systemd unescapes there escaped.
+ */
+export function environmentLine(name: string, value: string): string {
+ if (!value.includes("'")) return `${name}='${value}'`;
+ return `${name}="${value.replace(/(["\\$`])/g, '\\$1')}"`;
+}
diff --git a/console/packages/agent-controller/src/testing/fake-core.ts b/console/packages/agent-controller/src/testing/fake-core.ts
index d741a84a2..2549cc0f5 100644
--- a/console/packages/agent-controller/src/testing/fake-core.ts
+++ b/console/packages/agent-controller/src/testing/fake-core.ts
@@ -104,6 +104,10 @@ export class FakeCore {
readonly beats: Record<string, number>[] = [];
/** The machine's name and description, as `PATCH .../controllers/{id}` changes them. */
info: { name: string; description: string | null } = { name: 'laptop', description: null };
+ /** The key the controller registered for sealed logins. */
+ publicKey: string | null = null;
+ /** The logins given to the machine, by provider, as `GET .../provider-credentials/{provider}` answers. */
+ readonly sealedLogins = new Map<string, { revision: number; sealed: unknown }>();
/** Answers the next request to a path with this, once. */
readonly scripted: { method: string; path: string; status: number; body: unknown }[] = [];
/** What `GET .../media` sends, in these chunks; `waitBetween` holds back all but the first. */
@@ -401,11 +405,28 @@ export class FakeCore {
});
}
if (method === 'PATCH' && url.pathname === base) {
- const change = body as { name?: string; description?: string | null };
+ const change = body as {
+ name?: string;
+ description?: string | null;
+ public_key?: { alg: string; key: string };
+ };
+ if (change.public_key !== undefined) {
+ if (this.publicKey !== null && this.publicKey !== change.public_key.key)
+ return this.refuse(res, 409, 'validation_error');
+ this.publicKey = change.public_key.key;
+ }
if (change.name !== undefined) this.info.name = change.name;
if (change.description !== undefined) this.info.description = change.description;
return this.json(res, 200, { id: this.controllerId, ...this.info, state: 'online' });
}
+ const given = url.pathname.match(
+ new RegExp(`^${base.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}/provider-credentials/([^/]+)$`)
+ );
+ if (method === 'GET' && given) {
+ const login = this.sealedLogins.get(given[1]!);
+ if (!login) return this.refuse(res, 404, 'provider_login_missing');
+ return this.json(res, 200, { provider: given[1], ...login });
+ }
if (method === 'GET' && url.pathname === `${base}/assignment`) {
const etag = `"${this.assignment.revision}"`;
if (req.headers['if-none-match'] === etag) {
diff --git a/console/packages/agent-controller/src/testing/fake-runtime.ts b/console/packages/agent-controller/src/testing/fake-runtime.ts
index 12fe0f64f..f03c665fe 100644
--- a/console/packages/agent-controller/src/testing/fake-runtime.ts
+++ b/console/packages/agent-controller/src/testing/fake-runtime.ts
@@ -12,6 +12,7 @@ import {
type RelayCredentials,
} from '../runtime';
import type { Provider } from '../schemas';
+import type { GivenLogin } from '../sealed-logins';
import type { LocatedProvider, ProviderLocator } from '../status';
export type RuntimeCall =
@@ -27,6 +28,13 @@ export class FakeRuntime implements AgentRuntime {
readonly credentials = new Map<string, RelayCredentials>();
readiness: ProviderReadiness = { status: 'authenticated', message: 'Signed in.', models: [] };
probes = 0;
+ /** What a probe with a login Switch gave the machine answers. */
+ loginReadiness: ProviderReadiness = {
+ status: 'authenticated',
+ message: 'Signed in.',
+ models: [],
+ };
+ readonly loginProbes: GivenLogin[] = [];
/** Thrown from the next launch, once. */
failNextLaunch: Error | null = null;
/** What the controller gave it to open each agent's event stream. */
@@ -74,7 +82,11 @@ export class FakeRuntime implements AgentRuntime {
this.credentials.delete(agentId);
}
- async workingDirectory(name: string, directory: string | null): Promise<string> {
+ async workingDirectory(
+ _agentId: string,
+ name: string,
+ directory: string | null
+ ): Promise<string> {
if (directory === null) return `/data/workspaces/${name}`;
if (!directory.startsWith('/'))
throw new ReasonedError('definition_invalid', `'${directory}' is not absolute.`);
@@ -116,7 +128,20 @@ export class FakeRuntime implements AgentRuntime {
observation.health = { ...observation.health, state: 'disabled', current: false };
}
- async probe(_provider: Provider, _binaryPath: string, _cwd: string): Promise<ProviderReadiness> {
+ agentStateRoot(agentId: string): string {
+ return `/data/watchers/${agentId}`;
+ }
+
+ async probe(
+ _provider: Provider,
+ _binaryPath: string,
+ _cwd: string,
+ login: GivenLogin | null
+ ): Promise<ProviderReadiness> {
+ if (login) {
+ this.loginProbes.push(login);
+ return this.loginReadiness;
+ }
this.probes++;
return this.readiness;
}
diff --git a/console/packages/agent-providers/package.json b/console/packages/agent-providers/package.json
index b637628f7..7e06b1c06 100644
--- a/console/packages/agent-providers/package.json
+++ b/console/packages/agent-providers/package.json
@@ -6,9 +6,6 @@
"author": {
"name": "Switch Console Contributors"
},
- "bin": {
- "switch-hosted-bootstrap": "./dist/hosted-bootstrap.mjs"
- },
"files": [
"dist"
],
@@ -25,8 +22,7 @@
},
"./package.json": "./package.json",
"./host-daemon": "./dist/host-daemon.mjs",
- "./shared-host-daemon": "./dist/shared-host-daemon.mjs",
- "./hosted-bootstrap": "./dist/hosted-bootstrap.mjs"
+ "./shared-host-daemon": "./dist/shared-host-daemon.mjs"
},
"scripts": {
"dev": "tsdown --watch",
diff --git a/console/packages/agent-providers/src/codex/home.test.ts b/console/packages/agent-providers/src/codex/home.test.ts
index 5d9cb5802..4323722e0 100644
--- a/console/packages/agent-providers/src/codex/home.test.ts
+++ b/console/packages/agent-providers/src/codex/home.test.ts
@@ -22,7 +22,6 @@ it('isolates config, preserves rollouts and refreshed auth on resume, and protec
sessionId: '../session',
sourceHome,
config: 'model = "test-model"',
- auth: 'copy-once' as const,
};
const home = await prepareCodexSessionHome(input);
expect(parse(await readFile(join(home, 'config.toml'), 'utf8'))).toEqual({
@@ -55,7 +54,6 @@ it('allows native environment or keychain authentication when no login file exis
sessionId: 'session',
sourceHome: join(root, 'missing'),
config: '',
- auth: 'copy-once',
})
).resolves.toEqual(expect.any(String));
});
@@ -68,7 +66,6 @@ it('removes a Switch skill file an earlier build left in the session home', asyn
sessionId: 'session',
sourceHome: join(root, 'missing'),
config: '',
- auth: 'copy-once' as const,
};
const home = await prepareCodexSessionHome(input);
await mkdir(join(home, 'skills/switch'), { recursive: true });
@@ -88,7 +85,6 @@ it('links the user’s own Codex skills into a session home that has no skills f
sessionId: 'session',
sourceHome,
config: '',
- auth: 'copy-once',
});
expect(await readFile(join(home, 'skills', '.system', 'SKILL.md'), 'utf8')).toBe('system skill');
});
diff --git a/console/packages/agent-providers/src/codex/home.ts b/console/packages/agent-providers/src/codex/home.ts
index 589cb2940..68f76ecf8 100644
--- a/console/packages/agent-providers/src/codex/home.ts
+++ b/console/packages/agent-providers/src/codex/home.ts
@@ -10,19 +10,12 @@ export async function prepareCodexSessionHome(input: {
sessionId: string;
sourceHome: string;
config: string;
- /**
- * `refresh` replaces the session's login whenever the source login changes
- * (a hosted worker, whose owner can reconnect the provider); `copy-once`
- * keeps whatever the session refreshed after its first copy.
- */
- auth: 'copy-once' | 'refresh';
}): Promise<string> {
const key = createHash('sha256').update(input.sessionId).digest('hex');
const home = join(input.root, key);
await mkdir(home, { recursive: true, mode: 0o700 });
await chmod(home, 0o700);
- if (input.auth === 'refresh') await refreshCodexAuthentication(home, input.sourceHome);
- else await copyCodexAuthenticationOnce(home, input.sourceHome);
+ await copyCodexAuthenticationOnce(home, input.sourceHome);
const sourceConfig = await optionalText(join(input.sourceHome, 'config.toml'));
const config = { ...(sourceConfig ? parse(sourceConfig) : {}), ...parse(input.config) };
// The session host registers its own `switch` server; an entry of that name
diff --git a/console/packages/agent-providers/src/host/agent-env.ts b/console/packages/agent-providers/src/host/agent-env.ts
index ff55b41bc..bd8b190b9 100644
--- a/console/packages/agent-providers/src/host/agent-env.ts
+++ b/console/packages/agent-providers/src/host/agent-env.ts
@@ -24,6 +24,7 @@ export const AGENT_ENV_VARS = [
'BAILIAN_CODING_PLAN_API_KEY',
'CLAUDE_CODE_DISABLE_BACKGROUND_TASKS',
'CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS',
+ 'CLAUDE_CODE_OAUTH_TOKEN',
'CLAUDE_CODE_SKIP_BEDROCK_AUTH',
'CLAUDE_CODE_SKIP_VERTEX_AUTH',
'CLAUDE_CODE_SUBAGENT_MODEL',
diff --git a/console/packages/agent-providers/src/host/agent-host.test.ts b/console/packages/agent-providers/src/host/agent-host.test.ts
index 80b9b1a82..8290a6397 100644
--- a/console/packages/agent-providers/src/host/agent-host.test.ts
+++ b/console/packages/agent-providers/src/host/agent-host.test.ts
@@ -18,10 +18,7 @@ import {
supersededSessions,
withDefinitionOf,
} from './agent-host';
-import { AttachmentTransfers } from './attachment-transfers';
-import { WorkerObsoleteError } from './exit-codes';
import type { Handoff } from './handoff';
-import { FAILED_HOLD_MS, HostedWorker } from './hosted-worker';
import { ensureSharedProcess, type Supervision } from './launch';
import { type SessionRequest, SessionLinks } from './session-channel';
import { sharedConfigSchema } from './shared-config';
@@ -44,11 +41,6 @@ const declarations = vi.hoisted(() => [] as boolean[]);
/** Every placements map the watcher stated to Switch, in order. */
const published = vi.hoisted(() => [] as Record<string, string>[]);
const placementsRefusal = vi.hoisted(() => ({ error: null as Error | null }));
-/** A hosted worker's up-calls, and how Switch answers them. */
-const upcalls = vi.hoisted(() => ({
- calls: [] as { path: string; body: Record<string, unknown> }[],
- answer: (_path: string): unknown => ({}),
-}));
vi.mock('@sandboxaq/switch-agent-runtime', async (importOriginal) => {
const original = await importOriginal<typeof runtime>();
return {
@@ -66,12 +58,6 @@ vi.mock('@sandboxaq/switch-agent-runtime', async (importOriginal) => {
published.push(structuredClone(placements));
if (placementsRefusal.error) throw placementsRefusal.error;
}
- async workerCall(path: string, body: Record<string, unknown>): Promise<unknown> {
- upcalls.calls.push({ path, body });
- if (path.endsWith('/connection/idle'))
- return { queued_operations: [], credential_revision: 'r1' };
- return upcalls.answer(path);
- }
},
};
});
@@ -82,8 +68,6 @@ afterEach(async () => {
declarations.length = 0;
published.length = 0;
placementsRefusal.error = null;
- upcalls.calls.length = 0;
- upcalls.answer = () => ({});
vi.useRealTimers();
vi.unstubAllGlobals();
supervisors.clear();
@@ -373,7 +357,6 @@ it('stays down while a takeover marker says another client holds the connection'
new AbortController().signal,
supervision,
new WatcherControl(),
- null,
openSwitchStream
)
).resolves.toBeUndefined();
@@ -389,7 +372,6 @@ it('stays down while a takeover marker says another client holds the connection'
new AbortController().signal,
supervision,
new WatcherControl(),
- null,
openSwitchStream
)
).rejects.toThrow('credentials.json');
@@ -483,33 +465,6 @@ it('resumes at the server head after its numbering restarts', async () => {
]);
});
-it('keeps a mailbox delivery out of the stream position, and knows it again after a restart', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-wake-position-'));
- roots.push(root);
- paths.root = root;
- const config = template(root);
- const assignments = await AgentHostAssignments.open(root);
- const woken = { sequence: 1, roomId: 'room', messageId: 'woken' };
- await assignments.park(woken, true, true);
- // Assigned, and then the watcher dies before the session takes it.
- const wakeConfig = await assignments.assign(config, woken);
- // A stream event on the number the wake was handed is a different message.
- const streamed = await assignments.assign(config, {
- sequence: 1,
- roomId: 'other',
- messageId: 'streamed',
- });
- expect(streamed.session.sessionId).not.toBe(wakeConfig.session.sessionId);
- expect(assignments.cursor).toBe(0);
-
- await assignments.restart();
- const restarted = await AgentHostAssignments.open(root);
- expect(restarted.known(woken)).toBe(true);
- expect(await restarted.assign(config, woken)).toEqual(wakeConfig);
- expect(restarted.cursor).toBe(0);
-});
-
-/** The rename both Console and the SSH inline script use to replace the file. */
async function writeFlags(root: string, flags: { enabled: boolean; spawn: boolean }) {
const temporary = join(root, 'watch.json.tmp');
await writeFile(temporary, JSON.stringify(flags));
@@ -595,7 +550,6 @@ it('starts no saved session at startup; each waits until it is needed', async ()
abort.signal,
supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -634,7 +588,6 @@ it('leaves an event queued behind earlier work unstarted once spawning is turned
abort.signal,
supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -690,7 +643,6 @@ it('gives one room one session however close together its first messages arrive'
abort.signal,
supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -732,7 +684,6 @@ it('routes a room that has a session to that session, and starts no other', asyn
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -784,7 +735,6 @@ it('routes to the session Console moved the room to, and tells Switch where ever
abort.signal,
hosts.supervision,
control,
- null,
openSwitchStream
);
try {
@@ -837,7 +787,6 @@ it('puts a move Console asked for back when Switch refuses it', async () => {
abort.signal,
hosts.supervision,
control,
- null,
openSwitchStream
);
try {
@@ -906,7 +855,6 @@ it('answers its sessions’ tool calls as the calling session, placing the room
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -985,7 +933,6 @@ it('forgets a room another connection took over, and a session that was stopped'
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1053,7 +1000,6 @@ it('still owes an event its session never acknowledged', async () => {
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1186,7 +1132,6 @@ it('hands a session it has just created the event that created it', async () =>
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1236,7 +1181,6 @@ it('tells the room it is starting a session when its stream leaves that to it, b
start: () => {},
setSpawnCapable: () => {},
replacePlacements: async (placements) => void published.push(structuredClone(placements)),
- workerCall: () => Promise.reject(new Error('not a worker')),
};
};
@@ -1247,7 +1191,6 @@ it('tells the room it is starting a session when its stream leaves that to it, b
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
controllerStream
);
try {
@@ -1291,7 +1234,6 @@ it('leaves telling the room to Switch on its own connection', async () => {
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1321,7 +1263,6 @@ it('holds a room nothing can take while starting sessions is off, and delivers o
abort.signal,
supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1361,7 +1302,6 @@ it('keeps a held event, content and all, across a controller restart', async ()
first.signal,
supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1380,7 +1320,6 @@ it('keeps a held event, content and all, across a controller restart', async ()
second.signal,
supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1422,7 +1361,6 @@ it('starts the session serving a room again when its host has gone, instead of a
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1580,7 +1518,6 @@ it('restarts a room’s session under its agent’s edited definition, resuming
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1643,7 +1580,6 @@ it('brings live sessions in step with a rewritten definition, each once its turn
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1697,7 +1633,6 @@ it('gives a room a new session once the one serving it has been stopped', async
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1736,7 +1671,6 @@ it('forgets a session Console deleted, so its room starts a new one', async () =
abort.signal,
hosts.supervision,
control,
- null,
openSwitchStream
);
try {
@@ -1779,7 +1713,6 @@ it('passes an approval answer to the session it is for, and only that one', asyn
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
const outcome = (sessionId: string) => ({
@@ -1825,7 +1758,6 @@ it('hands a relayed command to the session it names, and only if it runs here',
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
const command = (sessionId: string) => ({ sessionId, commandId: `command-${sessionId}` });
@@ -1874,7 +1806,6 @@ it('hands a room control to the session working in its room, when Switch names o
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1919,7 +1850,6 @@ it('refuses to run without being the parent of its sessions', async () => {
new AbortController().signal,
detached,
new WatcherControl(),
- null,
openSwitchStream
)
).rejects.toThrow('parent of its sessions');
@@ -1939,7 +1869,6 @@ it('hands a message to its session over the IPC pipe and releases it once taken'
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -1989,7 +1918,6 @@ it('reports its connection and placements as they change, and why it stopped', a
new AbortController().signal,
hosts.supervision,
control,
- null,
openSwitchStream
);
await eventually(() => streams.length === 1);
@@ -2029,7 +1957,6 @@ it('reports a room connection that is turned off, and a watcher that failed', as
new AbortController().signal,
hosts.supervision,
control,
- null,
openSwitchStream
);
await eventually(() => streams.length === 1);
@@ -2046,7 +1973,6 @@ it('reports a room connection that is turned off, and a watcher that failed', as
new AbortController().signal,
hosts.supervision,
control,
- null,
openSwitchStream
)
).rejects.toThrow('credentials.json');
@@ -2094,7 +2020,6 @@ it('stops starting a session whose host failed, keeps its message, and tells its
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -2169,7 +2094,6 @@ it('tells the room without addressing anyone when the owner cannot be addressed,
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
@@ -2206,481 +2130,6 @@ function placementsOf(maps: Record<string, string>[]): Record<string, string> {
);
}
-/** A hosted watcher over `hosts`, whose sessions write the config a real launch would. */
-async function hostedWatcher(
- root: string,
- config: ReturnType<typeof watchable>,
- hosts: ReturnType<typeof sessionHosts>,
- credential: { revoked: boolean }
-) {
- vi.mocked(ensureSharedProcess).mockImplementation(
- async ({ root: sessionRoot, config: launched }) => {
- await mkdir(sessionRoot, { recursive: true });
- await writeFile(join(sessionRoot, 'config.json'), JSON.stringify(launched));
- return hosts.start(sessionRoot);
- }
- );
- const control = new WatcherControl();
- const hosted = new HostedWorker(
- root,
- {
- capability: 'capability-placeholder',
- bootId: 'boot',
- instanceId: 'instance',
- stateVersion: 1,
- },
- {
- agentId: config.session.agentId,
- links: hosts.links,
- ensure: async () => ({ created: false }),
- watcher: control,
- transfers: new AttachmentTransfers(root),
- },
- {
- fetch: async () => ({
- revoked: credential.revoked,
- revision: credential.revoked ? null : 'r1',
- apply: async () => {},
- }),
- }
- );
- await hosted.open();
- const abort = new AbortController();
- const run = runAgentHost(
- root,
- config,
- abort.signal,
- hosts.supervision,
- control,
- hosted,
- openSwitchStream
- );
- await eventually(() => streams.length === 1);
- const stream = streams[0]!;
- const attach = async (overrides: Record<string, unknown>) => {
- await stream.onWorkerFrame!('worker_attached', {
- launch_revision: 1,
- limits: { sessions_per_agent: 8 },
- idle: { report_every_s: 30, fresh_for_s: 75 },
- credential_revision: 'r1',
- queued_operations: [],
- relay_fence: 0,
- cancelled: [],
- ...overrides,
- });
- };
- return { stream, run, abort, attach, hosted };
-}
-
-/** Every mailbox ack a hosted watcher sent, as `outcome:message`. */
-function acks(): string[] {
- return upcalls.calls
- .filter((call) => call.path.endsWith('/connection/mailbox/ack'))
- .flatMap((call) =>
- (call.body.entries as { message_id: string; outcome: string }[]).map(
- (entry) => `${entry.outcome}:${entry.message_id}`
- )
- );
-}
-
-function notices(): string[] {
- return upcalls.calls
- .filter((call) => call.path.endsWith('/room-notices'))
- .map((call) => `${call.body.reason}:${call.body.message_id}`);
-}
-
-const wakeEntry = (messageId: string, roomId: string, origin: 'live' | 'cutover' = 'live') => ({
- room_id: roomId,
- message_id: messageId,
- thread_id: null,
- origin,
- event: {
- type: 'message',
- payload: { addressed: true, sender: '@owner:example.test', message_id: messageId, body: 'hi' },
- missed: 0,
- },
-});
-
-it('journals a hosted delivery, acks it through admission and answers a second copy as duplicate', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-wake-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const hosts = sessionHosts();
- const watcher = await hostedWatcher(root, config, hosts, { revoked: false });
- try {
- await watcher.attach({});
- await watcher.stream.onWorkerFrame!('wake', { entries: [wakeEntry('woken', 'room')] });
- await eventually(() => acks().includes('admitted:woken'));
- expect(acks().slice(0, 2)).toEqual(['journaled:woken', 'admitted:woken']);
- await watcher.stream.onEvent!(addressed(1, 'room'));
- await eventually(() => acks().includes('admitted:message-1'));
- await watcher.stream.onWorkerFrame!('wake', { entries: [wakeEntry('message-1', 'room')] });
- await eventually(() => acks().includes('duplicate:message-1'));
- const sessionId = placementsOf(published).room!;
- expect(
- hosts.to(join(root, sessionId)).map((request) => (request as { handoff: Handoff }).handoff)
- ).toMatchObject([{ messageId: 'woken' }, { messageId: 'message-1' }]);
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
-});
-
-it('hands a message imported at the cutover to its session marked as such', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-cutover-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const hosts = sessionHosts();
- const watcher = await hostedWatcher(root, config, hosts, { revoked: false });
- try {
- await watcher.attach({});
- await watcher.stream.onWorkerFrame!('wake', {
- entries: [wakeEntry('carried', 'room', 'cutover')],
- });
- await eventually(() => acks().includes('admitted:carried'));
- const sessionId = placementsOf(published).room!;
- expect(
- hosts.to(join(root, sessionId)).map((request) => (request as { handoff: Handoff }).handoff)
- ).toMatchObject([{ messageId: 'carried', event: { cutover: true } }]);
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
-});
-
-it('admits nothing from its journal until it has settled what Switch cancelled', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-cancel-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const assignments = await AgentHostAssignments.open(root);
- for (const messageId of ['stopped', 'kept'])
- await assignments.park(
- { sequence: 1, roomId: 'room', messageId, event: wakeEntry(messageId, 'room').event },
- true,
- true
- );
- const hosts = sessionHosts();
- vi.spyOn(console, 'warn').mockImplementation(() => {});
- const watcher = await hostedWatcher(root, config, hosts, { revoked: false });
- try {
- await new Promise((resolve) => setTimeout(resolve, 100));
- expect(ensureSharedProcess).not.toHaveBeenCalled();
- await watcher.attach({
- cancelled: [
- { room_id: 'room', message_id: 'stopped', reason: 'stopped' },
- { room_id: 'room', message_id: 'never-seen', reason: 'expired' },
- ],
- });
- await eventually(() => acks().includes('admitted:kept'));
- expect(acks()).toEqual(
- expect.arrayContaining(['cancelled:stopped', 'cancelled:never-seen', 'admitted:kept'])
- );
- const sessionId = placementsOf(published).room!;
- expect(
- hosts.to(join(root, sessionId)).map((request) => (request as { handoff: Handoff }).handoff)
- ).toMatchObject([{ messageId: 'kept' }]);
- // A cancel for what a session already took says so.
- await watcher.stream.onWorkerFrame!('mailbox_cancel', {
- entries: [{ room_id: 'room', message_id: 'kept' }],
- });
- await eventually(() => acks().filter((ack) => ack === 'admitted:kept').length === 2);
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
-});
-
-/** Writes what a session host had durably recorded of `messageId` when its watcher died. */
-async function hostRecorded(
- sessionRoot: string,
- config: ReturnType<typeof watchable>,
- messageId: string,
- furthest: 'dispatched' | 'finished'
-) {
- const commandId = randomUUID();
- await writeFile(
- join(sessionRoot, 'room-inbox.jsonl'),
- `${JSON.stringify({ type: 'handoff', sequence: 1, roomId: 'room', messageId, event: wakeEntry(messageId, 'room').event })}\n`
- );
- const records = [
- {
- type: 'accepted',
- command: {
- contractVersion: 1,
- commandId,
- sessionId: config.session.sessionId,
- epoch: 'epoch',
- origin: {
- surface: 'slack',
- actorId: '@owner:example.test',
- roomId: 'room',
- threadId: null,
- messageId,
- },
- body: { type: 'message.send', text: 'hi', attachments: [], delivery: 'queue' },
- },
- },
- { type: 'dispatched', commandId },
- ...(furthest === 'finished' ? [{ type: 'finished', commandId }] : []),
- ];
- await writeFile(
- join(sessionRoot, 'inbox.jsonl'),
- records.map((record) => `${JSON.stringify(record)}\n`).join('')
- );
-}
-
-it.each(['dispatched', 'finished'] as const)(
- 'answers a Stop for a message its host had %s before the watcher crashed as admitted',
- async (furthest) => {
- const root = await mkdtemp(join(tmpdir(), `shared-watch-hosted-cancel-${furthest}-`));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const assignments = await AgentHostAssignments.open(root);
- await assignments.park(
- { sequence: 1, roomId: 'room', messageId: 'taken', event: wakeEntry('taken', 'room').event },
- true,
- true
- );
- const { sessionRoot } = await existing(root, config);
- await hostRecorded(sessionRoot, config, 'taken', furthest);
- const hosts = sessionHosts();
- vi.spyOn(console, 'warn').mockImplementation(() => {});
- const watcher = await hostedWatcher(root, config, hosts, { revoked: false });
- try {
- await watcher.attach({
- cancelled: [{ room_id: 'room', message_id: 'taken', reason: 'stopped' }],
- });
- await eventually(() => acks().includes('admitted:taken'));
- expect(acks()).not.toContain('cancelled:taken');
- expect(hosts.requests).toEqual([]);
- expect(
- (await AgentHostAssignments.open(root)).deliveryState({
- roomId: 'room',
- messageId: 'taken',
- })
- ).toEqual({ state: 'released', reason: null });
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
- }
-);
-
-it('still answers a Stop for a journaled message no host took as cancelled', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-cancel-unstarted-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const assignments = await AgentHostAssignments.open(root);
- await assignments.park(
- {
- sequence: 1,
- roomId: 'room',
- messageId: 'waiting',
- event: wakeEntry('waiting', 'room').event,
- },
- true,
- true
- );
- const { sessionRoot } = await existing(root, config);
- await hostRecorded(sessionRoot, config, 'other', 'finished');
- const hosts = sessionHosts();
- vi.spyOn(console, 'warn').mockImplementation(() => {});
- const watcher = await hostedWatcher(root, config, hosts, { revoked: false });
- try {
- await watcher.attach({
- cancelled: [{ room_id: 'room', message_id: 'waiting', reason: 'stopped' }],
- });
- await eventually(() => acks().includes('cancelled:waiting'));
- expect(acks()).not.toContain('admitted:waiting');
- expect(hosts.requests).toEqual([]);
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
-});
-
-it('leaves a Stop unanswered, loudly, when a host record it would need is unreadable', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-cancel-unreadable-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const assignments = await AgentHostAssignments.open(root);
- await assignments.park(
- {
- sequence: 1,
- roomId: 'room',
- messageId: 'unclear',
- event: wakeEntry('unclear', 'room').event,
- },
- true,
- true
- );
- const { sessionRoot } = await existing(root, config);
- await writeFile(join(sessionRoot, 'inbox.jsonl'), '{"type":"accepted","comm');
- const hosts = sessionHosts();
- vi.spyOn(console, 'warn').mockImplementation(() => {});
- const watcher = await hostedWatcher(root, config, hosts, { revoked: false });
- try {
- await expect(
- watcher.attach({
- cancelled: [{ room_id: 'room', message_id: 'unclear', reason: 'stopped' }],
- })
- ).rejects.toThrow(/Cannot tell whether session .* took message unclear/);
- expect(acks()).toEqual([]);
- expect(
- (await AgentHostAssignments.open(root)).deliveryState({
- roomId: 'room',
- messageId: 'unclear',
- })
- ).toEqual({ state: 'journaled' });
- } finally {
- watcher.abort.abort();
- await watcher.run.catch(() => {});
- }
-});
-
-it('refuses a new session past the limit, and admits exactly one of two starts at the edge', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-limit-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const hosts = sessionHosts();
- vi.spyOn(console, 'warn').mockImplementation(() => {});
- upcalls.answer = (path) => {
- const id = path.match(/operations\/([^/]+)\/claim$/)?.[1];
- return id
- ? { id, session_id: `session-${id}`, action: 'start', state: 'claimed', error: null }
- : {};
- };
- const watcher = await hostedWatcher(root, config, hosts, { revoked: false });
- try {
- await watcher.attach({ limits: { sessions_per_agent: 2 } });
- await watcher.stream.onEvent!(addressed(1, 'room'));
- await eventually(() => acks().includes('admitted:message-1'));
- await Promise.all([
- watcher.stream.onWorkerFrame!('operation', { id: 'a' }),
- watcher.stream.onWorkerFrame!('operation', { id: 'b' }),
- ]);
- const results = () =>
- upcalls.calls
- .filter((call) => call.path.endsWith('/result'))
- .map((call) => call.body as { state: string; error: string | null });
- await eventually(() => results().length === 2);
- expect(
- results()
- .map((result) => result.state)
- .sort()
- ).toEqual(['applied', 'failed']);
- expect(results().find((result) => result.state === 'failed')!.error).toBe('session limit');
- await watcher.stream.onEvent!(addressed(2, 'another'));
- await eventually(() => acks().includes('refused:message-2'));
- expect(notices()).toEqual(['capacity:message-2']);
- expect(await settled(root)).toBe(true);
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
-});
-
-it('tells the room once when its host fails, and acks its messages held when the hold ends', async () => {
- vi.useFakeTimers({ shouldAdvanceTime: true, toFake: ['setTimeout', 'Date'] });
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-held-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const hosts = sessionHosts();
- hosts.fail = SIGN_IN;
- vi.spyOn(console, 'warn').mockImplementation(() => {});
- vi.spyOn(console, 'error').mockImplementation(() => {});
- const watcher = await hostedWatcher(root, config, hosts, { revoked: false });
- try {
- await watcher.attach({});
- await watcher.stream.onEvent!(addressed(1, 'room'));
- await eventually(() => notices().includes('startup:message-1'));
- watcher.hosted.report(true);
- await eventually(() =>
- upcalls.calls.some(
- (call) =>
- call.path.endsWith('/connection/idle') &&
- JSON.stringify(call.body.reasons).includes('failed_holding')
- )
- );
- expect(acks()).not.toContain('held:message-1');
- const idleCallsBefore = upcalls.calls.filter((call) =>
- call.path.endsWith('/connection/idle')
- ).length;
- await vi.advanceTimersByTimeAsync(FAILED_HOLD_MS + 30_000);
- await eventually(() => acks().includes('held:message-1'));
- await eventually(
- () =>
- upcalls.calls.filter((call) => call.path.endsWith('/connection/idle')).length >
- idleCallsBefore
- );
- const last = upcalls.calls.filter((call) => call.path.endsWith('/connection/idle')).at(-1)!;
- expect(JSON.stringify(last.body.reasons)).not.toContain('failed_holding');
- expect(notices()).toEqual(['startup:message-1']);
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
-});
-
-it('refuses addressed messages while the provider is disconnected', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-revoked-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const hosts = sessionHosts();
- vi.spyOn(console, 'error').mockImplementation(() => {});
- const watcher = await hostedWatcher(root, config, hosts, { revoked: true });
- try {
- await watcher.attach({});
- await eventually(() => watcher.hosted.revoked);
- await watcher.stream.onEvent!(addressed(1, 'room'));
- await eventually(() => acks().includes('refused:message-1'));
- expect(notices()).toEqual(['revoked:message-1']);
- expect(ensureSharedProcess).not.toHaveBeenCalled();
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
-});
-
-it('stops as obsolete when its launch moved to a newer revision', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-obsolete-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- const watcher = await hostedWatcher(root, config, sessionHosts(), { revoked: false });
- watcher.stream.onEvicted({ code: 'launch_superseded', reason: 'revision 2', roomId: null });
- await expect(watcher.run).rejects.toBeInstanceOf(WorkerObsoleteError);
-});
-
-it('answers an addressed message in the room when auto-start is off', async () => {
- const root = await mkdtemp(join(tmpdir(), 'shared-watch-hosted-spawn-off-'));
- roots.push(root);
- paths.root = root;
- const config = await spawning(root);
- await writeFlags(root, { enabled: true, spawn: false });
- const watcher = await hostedWatcher(root, config, sessionHosts(), { revoked: false });
- try {
- await watcher.attach({});
- await watcher.stream.onEvent!(addressed(1, 'room'));
- await eventually(() => acks().includes('refused:message-1'));
- await eventually(() => notices().length === 1);
- expect(notices()).toEqual(['auto_start_off:message-1']);
- expect(ensureSharedProcess).not.toHaveBeenCalled();
- } finally {
- watcher.abort.abort();
- await watcher.run;
- }
-});
-
it('tells the room once a session keeps not taking a message, instead of starting it forever', async () => {
// A host that is up but never takes the message — it hung up its pipe and
// stayed alive — used to be started again every few seconds, for ever,
@@ -2703,7 +2152,6 @@ it('tells the room once a session keeps not taking a message, instead of startin
abort.signal,
hosts.supervision,
new WatcherControl(),
- null,
openSwitchStream
);
try {
diff --git a/console/packages/agent-providers/src/host/agent-host.ts b/console/packages/agent-providers/src/host/agent-host.ts
index 1d9ccb545..ba12d96f5 100644
--- a/console/packages/agent-providers/src/host/agent-host.ts
+++ b/console/packages/agent-providers/src/host/agent-host.ts
@@ -1,31 +1,16 @@
import { createHash, randomUUID } from 'node:crypto';
-import * as nodeFs from 'node:fs';
-import { mkdir, readdir, readFile, rm, stat } from 'node:fs/promises';
-import * as nodePath from 'node:path';
+import { mkdir, readdir, readFile, rm } from 'node:fs/promises';
import { join } from 'node:path';
import { isDeepStrictEqual } from 'node:util';
import {
EVICTION_HEARTBEAT_LAPSED,
- EVICTION_LAUNCH_SUPERSEDED,
EVICTION_TAKEN_OVER,
SwitchEventStream,
type SwitchEventStreamDeps,
- WORKER_CAPABILITY_OBSOLETE,
} from '@sandboxaq/switch-agent-runtime';
import type { SwitchIdentity } from '@sandboxaq/switch-agent-runtime/hosted';
import { z } from 'zod';
-import { WorkerObsoleteError } from './exit-codes';
import type { Handoff } from './handoff';
-import {
- type CancelReason,
- FAILED_HOLD_MS,
- type HostedPort,
- type HostedWorker,
- type IdleReason,
- type MailboxAck,
- OperationRefusedError,
- type SessionCounts,
-} from './hosted-worker';
import { Journal } from './journal';
import {
ensureSharedProcess,
@@ -36,16 +21,13 @@ import {
} from './launch';
import { releaseOwner, replaceOwner, withOwnershipLockOutlasting } from './ownership-lock';
import { SessionPlacements } from './placements';
-import { roomInboxHolds, roomInputId } from './room-inbox';
+import { roomInputId } from './room-inbox';
import {
SessionHostFailedError,
type SessionRequest,
SessionUnavailableError,
} from './session-channel';
-import { hostInboxRecordSchema } from './session-host';
-import { readHostSessions } from './session-list';
import { readSharedCredentials, sharedConfigSchema, type SharedHostConfig } from './shared-config';
-import { hostParked } from './shared-state';
import { readTakenOver, recordTakenOver } from './taken-over';
import { awaitWatchChange, onConfigReplaced, readWatchFlags } from './watch-flags';
import {
@@ -62,17 +44,8 @@ const assignmentSchema = z.strictObject({
roomId: z.string().min(1),
messageId: z.string().min(1),
config: sharedConfigSchema,
- // Made for a mailbox delivery: its sequence is no position in the stream's numbering.
- wake: z.literal(true).optional(),
});
-/**
- * The sequence a mailbox delivery is handed on. A wake is not numbered in the
- * stream; the journal marks it so, and everything after it knows the delivery
- * by room and message.
- */
-const WAKE_SEQUENCE = 1;
-
/** Marks where the server's sequence numbering restarted. */
const restartSchema = z.strictObject({ restarted: z.literal(true), at: z.string().min(1) });
@@ -96,9 +69,6 @@ const parkedSchema = z.strictObject({
// The event as the stream delivered it, kept because nothing else keeps it:
// the session's prompt is built from this copy when the room gets an owner.
event: z.unknown().optional(),
- // Offered from Switch's mailbox rather than the stream: its sequence is no
- // position in the stream's numbering.
- wake: z.literal(true).optional(),
});
/**
@@ -115,23 +85,12 @@ const releasedSchema = z.strictObject({
reason: z.string().min(1).optional(),
}),
});
-const mailboxAckSchema = z.strictObject({
- roomId: z.string().min(1),
- messageId: z.string().min(1),
- outcome: z.enum(['journaled', 'admitted', 'duplicate', 'refused', 'held', 'cancelled']),
- reason: z.string().min(1).optional(),
-});
-/** A hosted worker's mailbox ack, owed to Switch until it is confirmed. */
-const ackOwedSchema = z.strictObject({ mailboxAck: mailboxAckSchema });
-const ackConfirmedSchema = z.strictObject({ mailboxAcked: mailboxAckSchema });
export const assignmentRecordSchema = z.union([
assignmentSchema,
restartSchema,
handledSchema,
parkedSchema,
releasedSchema,
- ackOwedSchema,
- ackConfirmedSchema,
]);
type Assignment = z.infer<typeof assignmentSchema>;
@@ -158,9 +117,6 @@ function isAssignment(record: WatchRecord): record is Assignment {
return 'config' in record;
}
-const ackKey = (ack: MailboxAck): string =>
- JSON.stringify([ack.roomId, ack.messageId, ack.outcome]);
-
const delivery = (event: { roomId: string; messageId: string }): string =>
JSON.stringify([event.roomId, event.messageId]);
@@ -228,48 +184,6 @@ async function stopped(sessionId: string): Promise<boolean> {
}
}
-/**
- * Whether any of this agent's session hosts has a durable record of taking
- * this room message: in its room inbox, or as a command in its inbox. A record
- * that cannot be read throws, since it may be the one that says the message
- * ran.
- */
-async function hostTook(
- agentId: string,
- event: { roomId: string; messageId: string }
-): Promise<boolean> {
- for (const listed of readHostSessions(nodeFs, nodePath, agentId, sharedSessionsBase())) {
- const { sessionId } = z.object({ sessionId: z.string().min(1) }).parse(listed.session);
- const sessionRoot = sharedSessionRoot(sessionId);
- try {
- if (await roomInboxHolds(sessionRoot, event)) return true;
- let text: string;
- try {
- text = await readFile(join(sessionRoot, 'inbox.jsonl'), 'utf8');
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code === 'ENOENT') continue;
- throw error;
- }
- if (text && !text.endsWith('\n')) throw new Error('inbox.jsonl has an incomplete record.');
- for (const line of text.split('\n').filter(Boolean)) {
- const record = hostInboxRecordSchema.parse(JSON.parse(line));
- if (
- record.type === 'accepted' &&
- record.command.origin.roomId === event.roomId &&
- record.command.origin.messageId === event.messageId
- )
- return true;
- }
- } catch (error) {
- throw new Error(
- `Cannot tell whether session ${sessionId} took message ${event.messageId} in room ${event.roomId}, so its cancel stays unanswered: ${(error as Error).message}`,
- { cause: error }
- );
- }
- }
- return false;
-}
-
/**
* The same session, reachable over the connection this controller holds.
*
@@ -520,13 +434,11 @@ export class AgentHostAssignments {
const released = this.settled;
for (const record of this.current) {
if (isHandled(record)) complete = Math.max(complete, record.handled);
- else if (isParked(record)) {
- if (!record.wake) parked.set(record.parked, delivery(record));
- }
+ else if (isParked(record)) parked.set(record.parked, delivery(record));
// A held delivery's assignment is not a position. It can be made under a
// numbering the sequence it names does not belong to, and the release
// that closes it says where it got to.
- else if (isAssignment(record) && !record.wake && !released.has(delivery(record))) {
+ else if (isAssignment(record) && !released.has(delivery(record))) {
// An assignment is only made once the one before it has been routed, so
// a journal written before routing was recorded still resumes at its
// last complete event instead of from the beginning.
@@ -604,61 +516,16 @@ export class AgentHostAssignments {
}
/** Records that the event is waiting for its room's owner to be decided. */
- async park(event: Handoff, spawning: boolean, wake: boolean): Promise<void> {
+ async park(event: Handoff, spawning: boolean): Promise<void> {
await this.journal.append({
parked: event.sequence,
roomId: event.roomId,
messageId: event.messageId,
spawning,
...(event.event === undefined ? {} : { event: event.event }),
- ...(wake ? { wake: true as const } : {}),
});
}
- /** Whether this delivery was already journaled here, whatever came of it. */
- known(event: { roomId: string; messageId: string }): boolean {
- const identity = delivery(event);
- return this.journal.records.some(
- (record) =>
- (isParked(record) && delivery(record) === identity) ||
- (isReleased(record) && delivery(record.released) === identity)
- );
- }
-
- /**
- * What became of a delivery: still journaled and waiting, released (to a
- * session when `reason` is null), or unknown here.
- */
- deliveryState(event: {
- roomId: string;
- messageId: string;
- }): { state: 'journaled' } | { state: 'released'; reason: string | null } | { state: 'unknown' } {
- const identity = delivery(event);
- const released = this.journal.records.find(
- (record) => isReleased(record) && delivery(record.released) === identity
- );
- if (released && isReleased(released))
- return { state: 'released', reason: released.released.reason ?? null };
- return this.known(event) ? { state: 'journaled' } : { state: 'unknown' };
- }
-
- async recordAck(ack: MailboxAck): Promise<void> {
- await this.journal.append({ mailboxAck: ack });
- }
-
- /** Acks owed to Switch that it has not confirmed, oldest first. */
- unconfirmedAcks(): MailboxAck[] {
- const owed = new Map<string, MailboxAck>();
- for (const record of this.journal.records)
- if ('mailboxAck' in record) owed.set(ackKey(record.mailboxAck), record.mailboxAck);
- else if ('mailboxAcked' in record) owed.delete(ackKey(record.mailboxAcked));
- return [...owed.values()];
- }
-
- async confirmAcks(acks: MailboxAck[]): Promise<void> {
- for (const ack of acks) await this.journal.append({ mailboxAcked: ack });
- }
-
/**
* Notes that the server restarted its numbering. Past sequence numbers no
* longer identify an event, so they stop being read as a saved position or
@@ -680,19 +547,12 @@ export class AgentHostAssignments {
// A held delivery is recognised by the room and message it names. The
// sequence it arrived on may belong to a numbering the server has since
// restarted, where it now stands for somebody else's message.
- // A mailbox delivery has no stream position at all, so it is only ever
- // recognised that way, under whichever numbering it was first assigned.
const identity = delivery(event);
- const wake = this.journal.records.some(
- (record) => isParked(record) && record.wake === true && delivery(record) === identity
- );
const waiting = new Set(this.pending().map(delivery));
const assignments = this.current.filter(isAssignment);
- const duplicate = wake
- ? this.every.find((record) => delivery(record) === identity)
- : waiting.has(identity)
- ? assignments.find((record) => delivery(record) === identity)
- : assignments.find((record) => !record.wake && record.sequence === event.sequence);
+ const duplicate = waiting.has(identity)
+ ? assignments.find((record) => delivery(record) === identity)
+ : assignments.find((record) => record.sequence === event.sequence);
if (duplicate) {
if (duplicate.roomId !== event.roomId || duplicate.messageId !== event.messageId)
throw new Error('Watcher sequence changed message identity.');
@@ -708,7 +568,6 @@ export class AgentHostAssignments {
roomId: event.roomId,
messageId: event.messageId,
config,
- ...(wake ? { wake: true as const } : {}),
});
return config;
}
@@ -742,8 +601,6 @@ export type AgentEventStream = {
start(): void;
setSpawnCapable(capable: boolean): void;
replacePlacements(placements: Record<string, string>): Promise<void>;
- /** For a hosted worker's calls; a stream that is not a worker's refuses them. */
- workerCall(path: string, body: Record<string, unknown>): Promise<unknown>;
};
export type OpenAgentStream = (deps: SwitchEventStreamDeps) => AgentEventStream;
@@ -757,7 +614,6 @@ export async function runAgentHost(
signal: AbortSignal,
supervision: Supervision,
control: WatcherControl,
- hosted: HostedWorker | null,
openStream: OpenAgentStream
): Promise<void> {
const ownerPath = join(root, 'shared-owner.lock');
@@ -896,7 +752,6 @@ export async function runAgentHost(
`Session ${sessionId} is running again; handing it the ${entry.queue.length} room message(s) that waited for it.`
);
entry.failed = null;
- entry.failedAt = null;
entry.restarts = 0;
pump(entry.config);
}
@@ -1009,15 +864,6 @@ export async function runAgentHost(
if (announced.has(key)) return;
announced.add(key);
try {
- if (hosted) {
- await hosted.notice({
- roomId: event.roomId,
- messageId: event.messageId,
- threadId: threadOf(event.event),
- reason: 'startup',
- });
- return;
- }
// Switch answers the call from where it holds the session placed.
await publish();
await announceStartFailure({
@@ -1055,38 +901,9 @@ export async function runAgentHost(
running: boolean;
failed: string | null;
restarts: number;
- /** When a hosted worker posted the failure, which starts its bounded hold. */
- failedAt: number | null;
- /** The queued messages already acked `held` to Switch. */
- heldAcked: Set<string>;
config: SharedHostConfig;
};
const pumps = new Map<string, Pump>();
- const ack = async (
- event: { roomId: string; messageId: string },
- outcome: MailboxAck['outcome'],
- reason: string | null
- ) => {
- await hosted?.ack({
- roomId: event.roomId,
- messageId: event.messageId,
- outcome,
- ...(reason === null ? {} : { reason }),
- });
- };
- /** When a failed host's hold ends, Switch stops counting its queued messages. */
- const endHold = (entry: Pump, failedAt: number) => {
- pending = pending.then(async () => {
- if (entry.failedAt !== failedAt) return;
- for (const event of entry.queue) {
- if (entry.heldAcked.has(event.messageId)) continue;
- entry.heldAcked.add(event.messageId);
- await ack(event, 'held', null);
- }
- hosted?.changed();
- });
- void pending.catch((error: Error) => fail(error));
- };
const pump = (config: SharedHostConfig) => {
const sessionId = config.session.sessionId;
const entry = pumps.get(sessionId)!;
@@ -1104,13 +921,8 @@ export async function runAgentHost(
);
entry.queue.shift();
entry.restarts = 0;
- entry.heldAcked.delete(event.messageId);
- pending = pending.then(async () => {
- await assignments.released(event, null);
- await ack(event, 'admitted', null);
- });
+ pending = pending.then(() => assignments.released(event, null));
await pending;
- hosted?.changed();
} catch (error) {
if (error instanceof SessionHostFailedError) {
entry.failed = error.failure;
@@ -1118,14 +930,7 @@ export async function runAgentHost(
`Session ${sessionId} could not start: ${error.failure} Its ${entry.queue.length} room message(s) stay queued; it is started again when the room next addresses the agent or the session is restarted from Console.`
);
// Answer the newest message: it is the one somebody just sent.
- const announcing = announce(config, entry.queue.at(-1) ?? event, error.failure);
- if (hosted)
- void announcing.then(() => {
- const failedAt = Date.now();
- entry.failedAt = failedAt;
- hosted.changed();
- setTimeout(() => endHold(entry, failedAt), FAILED_HOLD_MS).unref();
- });
+ void announce(config, entry.queue.at(-1) ?? event, error.failure);
break;
}
if (!(error instanceof SessionUnavailableError)) throw error;
@@ -1134,14 +939,7 @@ export async function runAgentHost(
console.error(
`Session ${sessionId}: ${entry.failed} Its ${entry.queue.length} room message(s) stay queued; it is started again when the room next addresses the agent or the session is restarted from Console.`
);
- const announcing = announce(config, entry.queue.at(-1) ?? event, entry.failed);
- if (hosted)
- void announcing.then(() => {
- const failedAt = Date.now();
- entry.failedAt = failedAt;
- hosted.changed();
- setTimeout(() => endHold(entry, failedAt), FAILED_HOLD_MS).unref();
- });
+ void announce(config, entry.queue.at(-1) ?? event, entry.failed);
break;
}
entry.restarts++;
@@ -1169,14 +967,12 @@ export async function runAgentHost(
): Promise<boolean> => {
const sessionId = config.session.sessionId;
const sessionRoot = sharedSessionRoot(sessionId);
- if (!waiting) await assignments.park(event, false, false);
+ if (!waiting) await assignments.park(event, false);
const entry = pumps.get(sessionId) ?? {
queue: [],
running: false,
failed: null,
restarts: 0,
- failedAt: null,
- heldAcked: new Set<string>(),
config,
};
pumps.set(sessionId, entry);
@@ -1191,7 +987,6 @@ export async function runAgentHost(
`Room ${event.roomId} addressed the agent again; starting session ${sessionId} again after it failed (${entry.failed}).`
);
entry.failed = null;
- entry.failedAt = null;
entry.restarts = 0;
}
if (!links.ready(sessionRoot)) await launch(config);
@@ -1208,22 +1003,7 @@ export async function runAgentHost(
* force when it is finally admitted: a room that was promised a session
* when it was addressed should still get one.
*/
- /** A hosted worker's answer to a message it will not act on. */
- const refuse = async (event: Handoff, reason: 'capacity' | 'revoked' | 'auto_start_off') => {
- await assignments.released(event, reason);
- await ack(event, 'refused', reason);
- await hosted?.notice({
- roomId: event.roomId,
- messageId: event.messageId,
- threadId: threadOf(event.event),
- reason,
- });
- };
const admit = async (event: Handoff, spawning: boolean, waiting: boolean): Promise<boolean> => {
- if (hosted?.revoked) {
- await refuse(event, 'revoked');
- return true;
- }
const placed = placements.sessionIn(event.roomId);
if (placed) {
const owner = await sessionConfig(placed);
@@ -1237,26 +1017,7 @@ export async function runAgentHost(
`Session ${placed} ${owner ? 'was stopped' : 'is not one of this agent’s sessions here'}, so room ${event.roomId} has no session attending it now.`
);
}
- if (!spawning) {
- // A hosted room with no session and no permission to start one would
- // hold its worker awake for good.
- if (!hosted) return false;
- await refuse(event, 'auto_start_off');
- return true;
- }
- if (hosted) {
- const limit = hosted.limit;
- if (limit === null)
- throw new Error('A hosted watcher admitted a message before it attached.');
- const existing = sessionIdFor(agentId, event.roomId, event.messageId);
- if (!(await sessionConfig(existing)) && (await sessionCounts()).active >= limit) {
- console.warn(
- `Room ${event.roomId} addressed the agent, but it already has ${limit} active session(s); refusing message ${event.messageId}.`
- );
- await refuse(event, 'capacity');
- return true;
- }
- }
+ if (!spawning) return false;
const config = await assignments.assign(
sharedConfigSchema.parse(JSON.parse(await readFile(join(root, 'config.json'), 'utf8'))),
event
@@ -1281,41 +1042,7 @@ export async function runAgentHost(
`No session of this agent can take room ${event.roomId}'s messages yet, and starting one is off; holding them until one can.`
);
}
- if (!parked) await assignments.park(event, spawning, false);
- };
- /**
- * A hosted delivery, from the stream or the mailbox alike: journaled and
- * acknowledged before it is routed, and recognised by room and message
- * when it comes again.
- */
- const accept = async (event: Handoff, spawning: boolean, wake: boolean) => {
- if (assignments.known(event)) {
- await ack(event, 'duplicate', null);
- return;
- }
- await assignments.park(event, spawning, wake);
- await ack(event, 'journaled', null);
- if (held.size) await resolveHeld();
- if (held.has(event.roomId) || !(await admit(event, spawning, true)))
- await hold(event, spawning, true);
- hosted?.changed();
- };
- /** This agent's sessions here, as the idle report and the session limit count them. */
- const sessionCounts = async (): Promise<SessionCounts> => {
- const running = new Set(links.live());
- const counts = { total: 0, live: 0, parked: 0, failed: 0, active: 0 };
- for (const listed of readHostSessions(nodeFs, nodePath, agentId, sharedSessionsBase())) {
- const { sessionId } = z.object({ sessionId: z.string().min(1) }).parse(listed.session);
- const sessionRoot = sharedSessionRoot(sessionId);
- const alive = listed.alive || running.has(sessionRoot);
- const parked = !alive && !listed.stopped && (await hostParked(sessionRoot));
- counts.total += 1;
- if (alive && !listed.stopped) counts.live += 1;
- if (parked) counts.parked += 1;
- if (links.failure(sessionRoot) !== null) counts.failed += 1;
- if (!listed.stopped && (alive || parked)) counts.active += 1;
- }
- return counts;
+ if (!parked) await assignments.park(event, spawning);
};
/** Looks at each held room again, and answers the ones that can now be. */
const resolveHeld = async () => {
@@ -1350,170 +1077,7 @@ export async function runAgentHost(
);
}
}
- // A hosted watcher admits nothing from its journal until it has heard
- // which of those deliveries Switch cancelled while it was away.
- const admitting = () => hosted === null || hosted.reconciled;
- if (held.size && admitting()) await resolveHeld();
- const port: HostedPort = {
- serial: <T>(work: () => Promise<T>): Promise<T> => {
- const run = pending.then(work);
- pending = run.then(
- () => {},
- () => {}
- );
- return run;
- },
- reasons: (): IdleReason[] => {
- const reasons: IdleReason[] = [];
- const now = Date.now();
- for (const [sessionId, entry] of pumps) {
- if (!entry.queue.length) continue;
- if (entry.failed === null)
- reasons.push({
- kind: 'room_pending',
- session_id: sessionId,
- count: entry.queue.length,
- });
- else if (entry.failedAt === null || now - entry.failedAt < FAILED_HOLD_MS)
- reasons.push({
- kind: 'failed_holding',
- session_id: sessionId,
- count: entry.queue.length,
- });
- }
- for (const waiting of held.values())
- reasons.push({ kind: 'room_pending', session_id: null, count: waiting.events.length });
- return reasons;
- },
- sessions: sessionCounts,
- wake: async (entry) => {
- const event = z
- .object({ type: z.string(), payload: z.unknown(), missed: z.unknown().optional() })
- .parse(entry.event);
- await accept(
- {
- sequence: WAKE_SEQUENCE,
- roomId: entry.room_id,
- messageId: entry.message_id,
- event: {
- type: event.type,
- payload: event.payload,
- missed: event.missed ?? null,
- ...(entry.origin === 'cutover' ? { cutover: true } : {}),
- },
- },
- spawn,
- true
- );
- },
- cancel: async (entries: { roomId: string; messageId: string; reason: CancelReason }[]) => {
- for (const entry of entries) {
- const known = assignments.deliveryState(entry);
- if (known.state === 'released') {
- if (known.reason === null) await ack(entry, 'admitted', null);
- else await ack(entry, 'cancelled', entry.reason);
- continue;
- }
- if (known.state === 'journaled') {
- const inFlight = [...pumps.values()].some(
- (pumpEntry) => pumpEntry.running && pumpEntry.queue[0]?.messageId === entry.messageId
- );
- // Already on its way to the host: the host's answer settles it.
- if (inFlight) continue;
- const took = await hostTook(agentId, entry);
- const waiting = held.get(entry.roomId);
- if (waiting) {
- waiting.events = waiting.events.filter(
- (event) => event.messageId !== entry.messageId
- );
- if (!waiting.events.length) held.delete(entry.roomId);
- }
- for (const pumpEntry of pumps.values())
- pumpEntry.queue = pumpEntry.queue.filter(
- (event) => event.roomId !== entry.roomId || event.messageId !== entry.messageId
- );
- await assignments.released(entry, took ? null : entry.reason);
- if (took) {
- await ack(entry, 'admitted', null);
- continue;
- }
- }
- await ack(entry, 'cancelled', entry.reason);
- }
- if (held.size) await resolveHeld();
- hosted?.changed();
- },
- operate: async (operation, limit) => {
- const sessionRoot = sharedSessionRoot(operation.sessionId);
- if (operation.action === 'start') {
- try {
- await stat(sessionRoot);
- throw new OperationRefusedError('already exists');
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
- }
- if ((await sessionCounts()).active >= limit)
- throw new OperationRefusedError('session limit');
- const config = sessionFrom(
- sharedConfigSchema.parse(JSON.parse(await readFile(join(root, 'config.json'), 'utf8'))),
- operation.sessionId
- );
- await ensureSharedProcess({
- root: sessionRoot,
- config: reachableBy(config, connectionId),
- resuming: false,
- watcher: false,
- restart: false,
- supervision,
- // A start operation is a session someone asked Switch for.
- startSource: 'user',
- });
- return;
- }
- let saved: SharedHostConfig;
- try {
- saved = sharedConfigSchema.parse(
- JSON.parse(await readFile(join(sessionRoot, 'config.json'), 'utf8'))
- );
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code === 'ENOENT')
- throw new OperationRefusedError('not found');
- throw error;
- }
- if (saved.session.agentId !== agentId) throw new OperationRefusedError('not found');
- if ((await stopped(operation.sessionId)) && (await sessionCounts()).active >= limit)
- throw new OperationRefusedError('session limit');
- await ensureSharedProcess({
- root: sessionRoot,
- config: reachableBy(withDefinitionOf(saved, await currentTemplate()), connectionId),
- resuming: true,
- watcher: false,
- restart: true,
- supervision,
- startSource: null,
- });
- },
- revoke: async () => {
- for (const sessionRoot of links.live()) await supervision.stop(sessionRoot);
- },
- restart: (sessionRoot) =>
- port.serial(async () => {
- const saved = sharedConfigSchema.parse(
- JSON.parse(await readFile(join(sessionRoot, 'config.json'), 'utf8'))
- );
- await ensureSharedProcess({
- root: sessionRoot,
- config: reachableBy(withDefinitionOf(saved, await currentTemplate()), connectionId),
- resuming: true,
- watcher: false,
- restart: true,
- supervision,
- startSource: null,
- });
- }),
- acks: assignments,
- fail,
- };
+ if (held.size) await resolveHeld();
stream = openStream({
creds: {
agentId: credentials.SWITCH_AGENT_ID,
@@ -1521,16 +1085,6 @@ export async function runAgentHost(
token: credentials.SWITCH_API_TOKEN,
},
connectionId,
- worker: hosted?.identity ?? null,
- ...(hosted
- ? {
- onWorkerFrame: (name, data) =>
- hosted.frame(name, data).catch((error: unknown) => {
- fail(error instanceof Error ? error : new Error(String(error)));
- throw error;
- }),
- }
- : {}),
scope: 'all',
filter: 'addressed',
spawnCapable: spawn,
@@ -1614,7 +1168,6 @@ export async function runAgentHost(
// Asked again here as well as on the timer: the answer a held room is
// waiting for is written by a session that is doing other work, and
// an event arriving is the cheapest evidence that time has passed.
- if (hosted) return accept(assignment, spawning, false);
if (held.size) await resolveHeld();
if (held.has(assignment.roomId)) return hold(assignment, spawning, false);
if (!(await admit(assignment, spawning, false))) await hold(assignment, spawning, false);
@@ -1667,15 +1220,10 @@ export async function runAgentHost(
);
return;
}
- if (code === EVICTION_LAUNCH_SUPERSEDED || code === WORKER_CAPABILITY_OBSOLETE) {
- fail(new WorkerObsoleteError(reason));
- return;
- }
fail(new Error(`Shared SDK watcher was evicted: ${reason}`));
},
});
const started = stream;
- if (hosted) unbind.push(hosted.bind(started, port));
started.start();
unbind.push(
control.bind({
@@ -1722,7 +1270,7 @@ export async function runAgentHost(
// takes is the same one the handler takes, and two of them at once could
// start a session for a room the other has just found an owner for.
retry = setInterval(() => {
- if (!held.size || !admitting()) return;
+ if (!held.size) return;
pending = pending.then(resolveHeld);
void pending.catch((error: Error) => fail(error));
}, OWNERSHIP_RETRY_MS);
@@ -1815,7 +1363,7 @@ export async function runAgentHost(
started.setSpawnCapable(spawn);
// What waited for permission to start a session is answered now rather
// than on the next retry.
- if (spawn && admitting()) {
+ if (spawn) {
pending = pending.then(resolveHeld);
await pending;
}
diff --git a/console/packages/agent-providers/src/host/bench/daemon.ts b/console/packages/agent-providers/src/host/bench/daemon.ts
index b2349afab..df6b9825c 100644
--- a/console/packages/agent-providers/src/host/bench/daemon.ts
+++ b/console/packages/agent-providers/src/host/bench/daemon.ts
@@ -26,8 +26,6 @@ import { join, resolve } from 'node:path';
import { openSwitchStream, runAgentHost } from '../agent-host';
import { AttachmentTransfers } from '../attachment-transfers';
import { type ControlContext, ensureSessions, serveControl } from '../control';
-import { OBSOLETE_BUNDLE_EXIT_CODE, WorkerObsoleteError } from '../exit-codes';
-import { hostedWorker } from '../hosted-watcher';
import { detachedSupervision, ensureSharedProcess, inProcessSupervision } from '../launch';
import { replaceOwner } from '../ownership-lock';
import { ownProcessGroup } from '../process-fence';
@@ -75,7 +73,6 @@ async function main(): Promise<void> {
signal: stop.signal,
build: process.argv[1]!,
links: null,
- logRedactions: [],
});
} else if (mode === '--watch-worker') {
const stop = new AbortController();
@@ -100,19 +97,11 @@ async function main(): Promise<void> {
watcher: control,
transfers,
};
- // A bootstrapped watcher attaches as the hosted worker, as the shipped one does.
- const hosted = await hostedWorker(config, resolve(root), context);
try {
await Promise.all([
- runAgentHost(
- root,
- config,
- stop.signal,
- supervision,
- control,
- hosted,
- openSwitchStream
- ).finally(() => stop.abort()),
+ runAgentHost(root, config, stop.signal, supervision, control, openSwitchStream).finally(
+ () => stop.abort()
+ ),
serveControl(resolve(root), context, stop.signal),
]);
} finally {
@@ -160,17 +149,12 @@ async function main(): Promise<void> {
try {
await main();
} catch (error) {
- if (error instanceof WorkerObsoleteError) {
- console.error(error.message);
- process.exitCode = OBSOLETE_BUNDLE_EXIT_CODE;
- } else {
- if (mode !== '--supervise' && mode !== '--watch-supervise') {
- await mkdir(join(root, 'supervisor'), { recursive: true, mode: 0o700 });
- await replaceOwner(join(root, 'supervisor', 'failure.json'), {
- message: error instanceof Error ? error.message : String(error),
- });
- }
- console.error(error);
- process.exitCode = 1;
+ if (mode !== '--supervise' && mode !== '--watch-supervise') {
+ await mkdir(join(root, 'supervisor'), { recursive: true, mode: 0o700 });
+ await replaceOwner(join(root, 'supervisor', 'failure.json'), {
+ message: error instanceof Error ? error.message : String(error),
+ });
}
+ console.error(error);
+ process.exitCode = 1;
}
diff --git a/console/packages/agent-providers/src/host/cloud-relay-client.test.ts b/console/packages/agent-providers/src/host/cloud-relay-client.test.ts
deleted file mode 100644
index a534e5a8b..000000000
--- a/console/packages/agent-providers/src/host/cloud-relay-client.test.ts
+++ /dev/null
@@ -1,435 +0,0 @@
-import { createHash } from 'node:crypto';
-import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http';
-import type { AddressInfo } from 'node:net';
-import type { ServerEvent, Session, Snapshot } from '@switch-console/shared/session-v1';
-import { afterEach, beforeEach, expect, it, vi } from 'vitest';
-import { MAX_CHUNK_BYTES } from './attachment-transfers';
-import { CloudRelayClient, CloudRelayError } from './cloud-relay-client';
-import { SessionHostFailedError, SessionUnavailableError } from './session-channel';
-
-type Reply = { status: number; body: unknown };
-type Relayed = { message: Record<string, unknown>; timeout_ms: number };
-
-const worker = { launch_revision: 3, boot_id: 'boot', generation: 1 };
-const ok = (value: unknown): Reply => ({ status: 200, body: { ok: true, value, worker } });
-const refused = (status: number, code: string, message = code, extra = {}): Reply => ({
- status,
- body: { ok: false, error: { code, message }, worker, ...extra },
-});
-
-let server: Server;
-let base: string;
-let relayed: Relayed[];
-let answer: (message: Record<string, unknown>) => Reply;
-let streams: { query: URLSearchParams; response: ServerResponse }[];
-
-beforeEach(async () => {
- relayed = [];
- streams = [];
- answer = () => refused(500, 'unexpected');
- server = createServer((request: IncomingMessage, response: ServerResponse) => {
- const url = new URL(request.url ?? '/', 'http://relay');
- if (request.method === 'GET' && url.pathname === '/launch/relay/stream') {
- response.writeHead(200, { 'content-type': 'text/event-stream' });
- response.write(': keepalive\n\n');
- streams.push({ query: url.searchParams, response });
- return;
- }
- let body = '';
- request.on('data', (chunk: Buffer) => (body += chunk.toString()));
- request.on('end', () => {
- if (url.pathname !== '/launch/relay') {
- response.writeHead(404, { 'content-type': 'application/json' });
- response.end(JSON.stringify({ detail: 'Cloud launch not found.' }));
- return;
- }
- const parsed = JSON.parse(body) as Relayed;
- relayed.push(parsed);
- const reply = answer(parsed.message);
- response.writeHead(reply.status, { 'content-type': 'application/json' });
- response.end(JSON.stringify(reply.body));
- });
- });
- await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
- base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
-});
-
-afterEach(async () => {
- for (const { response } of streams) response.destroy();
- server.closeAllConnections();
- await new Promise((resolve) => server.close(resolve));
-});
-
-function client(launch = 'launch', retryMs = 2000): CloudRelayClient {
- return new CloudRelayClient(
- (path, init) =>
- fetch(`${base}/${launch}${path}`, {
- method: init.method,
- headers: { 'content-type': 'application/json' },
- body: init.body === undefined ? undefined : JSON.stringify(init.body),
- signal: init.signal,
- }),
- { retryMs, timeoutMs: 1000 }
- );
-}
-
-function frame(index: number, event: string, data: unknown): void {
- streams[index]!.response.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);
-}
-
-const session = (sessionId: string): Session => ({
- sessionId,
- agentId: 'agent',
- hostId: 'host',
- epoch: 'epoch-1',
- provider: 'claude',
- status: 'ready',
- connectivity: 'online',
- pendingRequestIds: [],
- capabilities: {
- input: 'queue',
- approvals: true,
- questions: true,
- interrupt: true,
- reset: true,
- compact: false,
- modelChange: false,
- attachmentMimeTypes: ['text/plain'],
- },
-});
-
-const snapshot = (text: string): Snapshot => ({
- contractVersion: 1,
- throughSequence: 7,
- session: session('one'),
- turns: [{ type: 'turn.upsert', turnId: 'turn', status: 'completed', commandId: 'command' }],
- items: [
- {
- itemId: 'item',
- turnId: 'turn',
- revision: 1,
- kind: 'assistant-message',
- status: 'completed',
- title: '',
- text,
- attachments: [],
- origin: null,
- },
- ],
- requests: [],
- commandStatuses: [],
- notices: [],
- nextPageToken: null,
-});
-
-const event = (sequence: number): ServerEvent => ({
- contractVersion: 1,
- eventId: `event-${sequence}`,
- sessionId: 'one',
- sequence,
- occurredAt: '2026-09-25T12:00:00.000Z',
- body: { type: 'turn.upsert', turnId: 'turn', status: 'running', commandId: 'command' },
-});
-
-/** Answers paged the way the worker pages them, `size` bytes a page. */
-function pager(value: unknown, size: number) {
- const data = Buffer.from(JSON.stringify(value));
- const pages: Buffer[] = [];
- for (let at = 0; at < data.byteLength; at += size) pages.push(data.subarray(at, at + size));
- let pins = 0;
- const first = () => {
- pins++;
- return ok({
- snapshotId: `snap-${pins}`,
- epoch: 'epoch-1',
- throughSequence: 7,
- bytes: data.byteLength,
- sha256: createHash('sha256').update(data).digest('hex'),
- pageCount: pages.length,
- page: { index: 0, data: pages[0]!.toString('base64') },
- });
- };
- const page = (message: Record<string, unknown>) => {
- const { snapshotId, index } = message.page as { snapshotId: string; index: number };
- return ok({ snapshotId, page: { index, data: pages[index]!.toString('base64') } });
- };
- return { first, page, count: pages.length, pins: () => pins };
-}
-
-it('reassembles a paged snapshot and checks it against its digest', async () => {
- const value = snapshot('x'.repeat(5000));
- const pages = pager(value, 1500);
- answer = (message) => ('page' in message ? pages.page(message) : pages.first());
- const result = await client().request('one', { type: 'snapshot' });
- expect(result).toEqual(value);
- expect(pages.count).toBeGreaterThan(3);
- expect(relayed.map((each) => each.message)).toEqual([
- { sessionId: 'one', request: { type: 'snapshot' } },
- ...Array.from({ length: pages.count - 1 }, (_, index) => ({
- page: { snapshotId: 'snap-1', index: index + 1 },
- })),
- ]);
- expect(relayed.every((each) => each.timeout_ms === 1000)).toBe(true);
-});
-
-it('asks for a paged answer again when its pin expires, and gives up after three', async () => {
- const pages = pager([session('one'), session('two')], 200);
- let expire = 1;
- answer = (message) => {
- if (!('page' in message)) return pages.first();
- if (expire-- > 0) return refused(409, 'snapshot_expired', 'The snapshot expired.');
- return pages.page(message);
- };
- expect((await client().list()).map((each) => each.sessionId)).toEqual(['one', 'two']);
- expect(pages.pins()).toBe(2);
-
- answer = (message) => ('page' in message ? refused(409, 'snapshot_superseded') : pages.first());
- await expect(client().list()).rejects.toMatchObject({ relayCode: 'snapshot_superseded' });
- expect(pages.pins()).toBe(5);
-});
-
-it('refuses a paged answer that does not match its digest', async () => {
- const pages = pager(snapshot('text'), 100);
- answer = (message) => {
- if ('page' in message) return pages.page(message);
- const first = pages.first();
- (first.body as { value: { sha256: string } }).value.sha256 = '0'.repeat(64);
- return first;
- };
- await expect(client().journal('one')).rejects.toThrow(/does not match its digest/);
-});
-
-it('reports a sleeping launch with whether it can be woken, without asking again', async () => {
- answer = () =>
- refused(409, 'worker_sleeping', 'The cloud worker is asleep.', { wake_available: true });
- const error = await client()
- .health()
- .catch((caught: unknown) => caught);
- expect(error).toBeInstanceOf(CloudRelayError);
- expect(error).toMatchObject({ relayCode: 'worker_sleeping', status: 409, wakeAvailable: true });
- expect(relayed).toHaveLength(1);
-});
-
-it.each(['machine_stopped', 'machine_error', 'agent_stopped', 'agent_crashed'])(
- 'raises %s without asking again',
- async (code) => {
- answer = () => refused(409, code);
- const error = await client()
- .health()
- .catch((caught: unknown) => caught);
- expect(error).toBeInstanceOf(CloudRelayError);
- expect(error).toMatchObject({ relayCode: code, status: 409, wakeAvailable: false });
- expect(relayed).toHaveLength(1);
- }
-);
-
-it('asks again while the worker is waking or busy, within its window', async () => {
- const replies = [refused(409, 'worker_waking'), refused(503, 'worker_busy')];
- answer = () =>
- replies.shift() ??
- ok({ state: 'connected', detail: null, since: '2026-09-25T12:00:00Z', placements: {} });
- expect((await client().health()).state).toBe('connected');
- expect(relayed).toHaveLength(3);
-
- answer = () => refused(409, 'worker_not_attached', 'No worker is attached.');
- await expect(client('launch', 300).health()).rejects.toMatchObject({
- relayCode: 'worker_not_attached',
- });
-});
-
-it('never sends a command twice on a timeout or a changed worker', async () => {
- const command = {
- type: 'command' as const,
- command: {
- contractVersion: 1 as const,
- commandId: 'command',
- sessionId: 'one',
- origin: null,
- body: { type: 'interrupt' as const },
- },
- };
- answer = () => refused(504, 'relay_timeout', 'The worker did not answer in time.');
- await expect(client().request('one', command as never)).rejects.toMatchObject({
- relayCode: 'relay_timeout',
- status: 504,
- });
- expect(relayed).toHaveLength(1);
-
- answer = () => refused(409, 'generation_changed');
- await expect(client().place('one', '!room:example')).rejects.toMatchObject({
- relayCode: 'generation_changed',
- });
- expect(relayed).toHaveLength(2);
-
- const replies = [refused(409, 'generation_changed')];
- answer = () => replies.shift() ?? ok({ status: 'accepted' });
- await client().request('one', { type: 'commandStatus', commandId: 'command' } as never);
- expect(relayed).toHaveLength(4);
-});
-
-it('raises the errors a sidecar raises for a stopped or failed session', async () => {
- answer = () => refused(409, 'session_unavailable', 'Session one is not running.');
- await expect(client().forget('one')).rejects.toBeInstanceOf(SessionUnavailableError);
-
- answer = () => refused(409, 'session_failed', 'The session host failed: out of credits');
- const failed = await client()
- .forget('one')
- .catch((caught: unknown) => caught);
- expect(failed).toBeInstanceOf(SessionHostFailedError);
- expect((failed as SessionHostFailedError).failure).toBe('out of credits');
-
- answer = () => refused(400, 'refused_message', 'Console may not send that.');
- await expect(client().forget('one')).rejects.toMatchObject({ relayCode: 'refused_message' });
-
- await expect(client('elsewhere').forget('one')).rejects.toMatchObject({
- relayCode: 'not_found',
- status: 404,
- });
-});
-
-it('uploads an attachment in chunks, resending one whose reply was lost', async () => {
- const data = Buffer.alloc(MAX_CHUNK_BYTES * 2 + 1234);
- for (let at = 0; at < data.byteLength; at++) data[at] = at % 251;
- const received: Buffer[] = [];
- let lose = 1;
- answer = (message) => {
- const chunk = message.attachment as { index: number; count: number; chunk: string };
- if (chunk.index === 1 && lose-- > 0) return refused(504, 'relay_timeout');
- received[chunk.index] = Buffer.from(chunk.chunk, 'base64');
- return chunk.index === chunk.count - 1
- ? ok({ staged: { transferId: 'id', ref: 'ref-1' } })
- : ok({ next: chunk.index + 1 });
- };
- const attachment = await client().uploadAttachment('one', {
- name: 'notes.txt',
- mimeType: 'text/plain',
- data,
- });
- expect(attachment).toEqual({
- attachmentId: 'ref-1',
- name: 'notes.txt',
- mimeType: 'text/plain',
- bytes: data.byteLength,
- sha256: createHash('sha256').update(data).digest('hex'),
- });
- expect(Buffer.concat(received).equals(data)).toBe(true);
- const chunks = relayed.map(
- (each) => each.message.attachment as { transferId: string; index: number; count: number }
- );
- expect(chunks.map((each) => each.index)).toEqual([0, 1, 1, 2]);
- expect(new Set(chunks.map((each) => each.transferId)).size).toBe(1);
- expect(chunks.every((each) => each.count === 3)).toBe(true);
-});
-
-it('cancels a transfer the worker refused', async () => {
- answer = (message) =>
- 'attachmentCancel' in message
- ? ok(null)
- : refused(400, 'staging_full', 'Too many attachments are staged.');
- await expect(
- client().uploadAttachment('one', {
- name: 'a.txt',
- mimeType: 'text/plain',
- data: Buffer.from('hello'),
- })
- ).rejects.toMatchObject({ relayCode: 'staging_full' });
- const transferId = (relayed[0]!.message.attachment as { transferId: string }).transferId;
- expect(relayed[1]!.message).toEqual({ attachmentCancel: transferId });
-
- await expect(
- client().uploadAttachment('one', {
- name: 'big.bin',
- mimeType: 'text/plain',
- data: Buffer.alloc(10 * 1024 * 1024 + 1),
- })
- ).rejects.toThrow(/larger than 10 MiB/);
-});
-
-it('delivers a session’s events once Switch holds the view, and resets it on a resync', async () => {
- const relay = client();
- const events: number[] = [];
- const failures: (string | null)[] = [];
- const onReset = vi.fn();
- const subscribing = relay.subscribe(
- 'one',
- (each) => events.push(each.sequence),
- (failure) => failures.push(failure),
- onReset
- );
- await vi.waitFor(() => expect(streams).toHaveLength(1));
- expect(streams[0]!.query.get('subscribe')).toBe('one');
- frame(0, 'worker', worker);
- const off = await subscribing;
- frame(0, 'event', { sessionId: 'one', event: event(8) });
- frame(0, 'failure', { sessionId: 'one', failure: 'out of credits' });
- frame(0, 'event', { sessionId: 'one', event: event(9) });
- await vi.waitFor(() => expect(events).toEqual([8, 9]));
- expect(failures).toEqual(['out of credits']);
- frame(0, 'resync', { sessionId: 'one', reason: 'overflow' });
- frame(0, 'event', { sessionId: 'one', event: event(10) });
- await vi.waitFor(() => expect(onReset).toHaveBeenCalledOnce());
- expect(onReset.mock.calls[0]![0]).toMatch(/overflow/);
- expect(events).toEqual([8, 9]);
- off();
-});
-
-it('resets a view when the worker behind it changes or the stream ends', async () => {
- const relay = client();
- const onReset = vi.fn();
- const subscribing = relay.subscribe(
- 'one',
- () => {},
- () => {},
- onReset
- );
- await vi.waitFor(() => expect(streams).toHaveLength(1));
- frame(0, 'worker', worker);
- frame(0, 'worker', worker);
- await subscribing;
- frame(0, 'worker', { ...worker, generation: 2 });
- await vi.waitFor(() => expect(onReset).toHaveBeenCalledWith('The cloud worker changed.'));
-
- const again = vi.fn();
- const second = relay.subscribe(
- 'one',
- () => {},
- () => {},
- again
- );
- await vi.waitFor(() => expect(streams).toHaveLength(2));
- frame(1, 'worker', worker);
- await second;
- streams[1]!.response.end();
- await vi.waitFor(() => expect(again).toHaveBeenCalledOnce());
-});
-
-it('refuses a view of a launch Switch will not relay', async () => {
- await expect(
- client('elsewhere').subscribe(
- 'one',
- () => {},
- () => {},
- () => {}
- )
- ).rejects.toMatchObject({ relayCode: 'not_found' });
-});
-
-it('follows the watcher’s health and closes when its stream ends', async () => {
- const relay = client();
- const healths: string[] = [];
- const closed = vi.fn();
- relay.onClose(closed);
- const watching = relay.onHealth((health) => healths.push(health.state));
- await vi.waitFor(() => expect(streams).toHaveLength(1));
- expect(streams[0]!.query.get('watchHealth')).toBe('1');
- frame(0, 'worker', worker);
- await watching;
- frame(0, 'health', {
- health: { state: 'connected', detail: null, since: '2026-09-25T12:00:00Z', placements: {} },
- });
- await vi.waitFor(() => expect(healths).toEqual(['connected']));
- streams[0]!.response.end();
- await vi.waitFor(() => expect(closed).toHaveBeenCalledOnce());
- expect(relay.isClosed).toBe(true);
- await expect(relay.health()).rejects.toThrow(/closed/);
-});
diff --git a/console/packages/agent-providers/src/host/cloud-relay-client.ts b/console/packages/agent-providers/src/host/cloud-relay-client.ts
deleted file mode 100644
index 491b96701..000000000
--- a/console/packages/agent-providers/src/host/cloud-relay-client.ts
+++ /dev/null
@@ -1,562 +0,0 @@
-import { createHash, randomUUID } from 'node:crypto';
-import {
- type Attachment,
- serverEventSchema,
- type ServerEvent,
- type Session,
- sessionSchema,
- type Snapshot,
- snapshotSchema,
-} from '@switch-console/shared/session-v1';
-import { z } from 'zod';
-import { MAX_CHUNK_BYTES } from './attachment-transfers';
-import { MAX_ATTACHMENT_BYTES } from './attachments';
-import type { ControlMessage } from './control';
-import {
- SessionHostFailedError,
- SessionUnavailableError,
- type SessionRequest,
-} from './session-channel';
-import { type PlaceOutcome, type WatcherHealth, watcherHealthSchema } from './watcher-tools';
-
-/**
- * Console's end of a cloud worker's sessions, relayed through Switch.
- *
- * A cloud worker accepts no inbound connection, so where a sidecar is reached
- * over its loopback control port this goes through the Switch gateway: one
- * `POST …/relay` per request, answered by the worker's watcher, and one
- * `GET …/relay/stream` per live view. The messages are the control
- * vocabulary's (`ControlMessage`), and the methods are `ControlClient`'s, so
- * the callers of either do not tell them apart. Large answers arrive in pages
- * and are reassembled here; attachments go up in chunks.
- */
-
-/** One call to the launch's relay routes; `path` is relative to the launch. */
-export type RelayFetch = (
- path: string,
- init: { method: 'GET' | 'POST'; body: unknown; signal: AbortSignal }
-) => Promise<Response>;
-
-/** A relay the worker or Switch refused, with the code Console acts on. */
-export class CloudRelayError extends Error {
- constructor(
- readonly relayCode: string,
- message: string,
- readonly status: number,
- readonly wakeAvailable: boolean
- ) {
- super(message);
- this.name = 'CloudRelayError';
- }
-}
-
-export class CloudRelayClosedError extends Error {
- constructor(reason: string) {
- super(`The relay to the cloud worker closed: ${reason}`);
- this.name = 'CloudRelayClosedError';
- }
-}
-
-/** The longest a relay may wait for the worker, as Switch allows it. */
-export const RELAY_TIMEOUT_MS = 30_000;
-const PAGED_ATTEMPTS = 3;
-const CHUNK_ATTEMPTS = 3;
-
-/** Nothing was sent to the worker: asking again cannot act twice. */
-const RETRYABLE = new Set(['worker_waking', 'worker_not_attached', 'worker_busy', 'snapshot_busy']);
-const RESTART_PAGED = new Set(['snapshot_expired', 'snapshot_superseded', 'generation_changed']);
-
-const replySchema = z.object({
- ok: z.boolean(),
- value: z.unknown().optional(),
- error: z.object({ code: z.string(), message: z.string() }).optional(),
- wake_available: z.boolean().optional(),
-});
-
-const firstPageSchema = z.object({
- snapshotId: z.string().min(1),
- bytes: z.number().int().nonnegative(),
- sha256: z.string().regex(/^[0-9a-f]{64}$/),
- pageCount: z.number().int().positive(),
- page: z.object({ index: z.literal(0), data: z.string() }),
-});
-const pageSchema = z.object({
- snapshotId: z.string().min(1),
- page: z.object({ index: z.number().int().positive(), data: z.string() }),
-});
-
-const workerFrameSchema = z.object({
- launch_revision: z.number().nullable(),
- boot_id: z.string().nullable(),
- generation: z.number().nullable(),
-});
-const eventFrameSchema = z.object({ sessionId: z.string(), event: z.unknown() });
-const failureFrameSchema = z.object({ sessionId: z.string(), failure: z.string().nullable() });
-const healthFrameSchema = z.object({ health: watcherHealthSchema });
-const resyncFrameSchema = z.object({ sessionId: z.string().nullable(), reason: z.string() });
-const errorFrameSchema = z.object({
- sessionId: z.string().nullable(),
- code: z.string(),
- message: z.string(),
-});
-
-const placeOutcomeSchema = z.object({
- sessionId: z.string(),
- roomId: z.string(),
- previous: z.string().nullable(),
- displaced: z.string().nullable(),
-});
-const chunkAnswerSchema = z.union([
- z.object({ next: z.number().int().nonnegative() }),
- z.object({ staged: z.object({ transferId: z.string(), ref: z.string().min(1) }) }),
-]);
-
-const MUTATING = ['place', 'forget', 'attachment', 'attachmentCancel'];
-
-function mutating(message: ControlMessage): boolean {
- if ('request' in message) return message.request.type === 'command';
- return MUTATING.some((key) => key in message);
-}
-
-/** The worker's refusal as the error `ControlClient` raises for the same case. */
-function refusal(code: string, message: string, status: number, wakeAvailable: boolean): Error {
- if (code === 'session_unavailable') return new SessionUnavailableError(message);
- if (code === 'session_failed')
- return new SessionHostFailedError(message.replace(/^The session host failed: /, ''));
- return new CloudRelayError(code, message, status, wakeAvailable);
-}
-
-async function readReply(response: Response): Promise<z.infer<typeof replySchema>> {
- const text = await response.text();
- let body: unknown;
- try {
- body = JSON.parse(text);
- } catch {
- throw new CloudRelayError(
- 'http',
- `Switch answered the relay with ${response.status}: ${text.slice(0, 500)}`,
- response.status,
- false
- );
- }
- const reply = replySchema.safeParse(body);
- if (reply.success) return reply.data;
- const detail = z.object({ detail: z.unknown() }).safeParse(body);
- if (response.status === 404)
- throw new CloudRelayError('not_found', 'Cloud launch not found.', 404, false);
- throw new CloudRelayError(
- 'http',
- `Switch answered the relay with ${response.status}: ${JSON.stringify(detail.success ? detail.data.detail : body).slice(0, 500)}`,
- response.status,
- false
- );
-}
-
-type Frame = { event: string; data: unknown };
-
-/** Server-sent events from a streaming response, one `{event, data}` per frame. */
-export async function* relayFrames(response: Response): AsyncGenerator<Frame> {
- if (!response.body) throw new Error('The relay stream has no body.');
- const decoder = new TextDecoder();
- let buffered = '';
- for await (const chunk of response.body as unknown as AsyncIterable<Uint8Array>) {
- buffered += decoder.decode(chunk, { stream: true }).replace(/\r\n/g, '\n');
- let at = buffered.indexOf('\n\n');
- while (at !== -1) {
- const block = buffered.slice(0, at);
- buffered = buffered.slice(at + 2);
- let event = 'message';
- const data: string[] = [];
- for (const line of block.split('\n')) {
- if (line.startsWith(':')) continue;
- if (line.startsWith('event:')) event = line.slice(6).trim();
- else if (line.startsWith('data:')) data.push(line.slice(5).replace(/^ /, ''));
- }
- if (data.length) yield { event, data: JSON.parse(data.join('\n')) };
- at = buffered.indexOf('\n\n');
- }
- }
-}
-
-type RelayStream = { ready: Promise<void>; close: () => void };
-
-export type CloudRelayOptions = {
- /** How long a request keeps asking while the worker is starting or its queue is full. */
- retryMs: number;
- /** How long the worker has to answer one request, at most `RELAY_TIMEOUT_MS`. */
- timeoutMs: number;
-};
-
-export class CloudRelayClient {
- private closed: Error | null = null;
- private readonly closeListeners = new Set<(error: Error) => void>();
- private readonly healthListeners = new Set<(health: WatcherHealth) => void>();
- private healthStream: RelayStream | null = null;
- private readonly views = new Set<(reason: string) => void>();
-
- constructor(
- private readonly fetchRelay: RelayFetch,
- private readonly options: CloudRelayOptions
- ) {
- if (options.timeoutMs > RELAY_TIMEOUT_MS)
- throw new Error(`A relay waits at most ${RELAY_TIMEOUT_MS} ms.`);
- }
-
- get isClosed(): boolean {
- return this.closed !== null;
- }
-
- private async once(message: ControlMessage): Promise<unknown> {
- let response: Response;
- try {
- response = await this.fetchRelay('/relay', {
- method: 'POST',
- body: { message, timeout_ms: this.options.timeoutMs },
- signal: AbortSignal.timeout(this.options.timeoutMs + 15_000),
- });
- } catch (error) {
- throw new CloudRelayError(
- 'unreachable',
- `Switch could not be reached: ${error instanceof Error ? error.message : String(error)}`,
- 0,
- false
- );
- }
- const reply = await readReply(response);
- if (reply.ok) return reply.value ?? null;
- const code = reply.error?.code ?? 'failed';
- throw refusal(
- code,
- reply.error?.message ?? 'The cloud worker refused the request.',
- response.status,
- reply.wake_available === true
- );
- }
-
- /** One relayed message, asked again while the worker cannot have received it. */
- private async call(message: ControlMessage): Promise<unknown> {
- if (this.closed) throw this.closed;
- const deadline = Date.now() + this.options.retryMs;
- let pause = 250;
- for (;;) {
- try {
- return await this.once(message);
- } catch (error) {
- const again =
- error instanceof CloudRelayError &&
- (RETRYABLE.has(error.relayCode) ||
- (error.relayCode === 'generation_changed' && !mutating(message)));
- if (!again || Date.now() + pause > deadline) throw error;
- }
- await new Promise((resolve) => setTimeout(resolve, pause));
- pause = Math.min(pause * 2, 2000);
- }
- }
-
- /** A paged answer, reassembled and checked against its size and digest. */
- private async paged(message: ControlMessage): Promise<unknown> {
- for (let attempt = 1; ; attempt++) {
- try {
- const first = firstPageSchema.parse(await this.call(message));
- const parts = [Buffer.from(first.page.data, 'base64')];
- for (let index = 1; index < first.pageCount; index++) {
- const next = pageSchema.parse(
- await this.call({ page: { snapshotId: first.snapshotId, index } })
- );
- if (next.snapshotId !== first.snapshotId || next.page.index !== index)
- throw new Error(`The cloud worker answered page ${index} with another page.`);
- parts.push(Buffer.from(next.page.data, 'base64'));
- }
- const data = Buffer.concat(parts);
- const sha256 = createHash('sha256').update(data).digest('hex');
- if (data.byteLength !== first.bytes || sha256 !== first.sha256)
- throw new Error('A paged answer from the cloud worker does not match its digest.');
- return JSON.parse(data.toString('utf8')) as unknown;
- } catch (error) {
- const restart =
- error instanceof CloudRelayError &&
- RESTART_PAGED.has(error.relayCode) &&
- attempt < PAGED_ATTEMPTS;
- if (!restart) throw error;
- }
- }
- }
-
- /** A session request; a snapshot comes back whole however many pages it took. */
- async request(sessionId: string, request: SessionRequest): Promise<unknown> {
- const message = { sessionId, request };
- return request.type === 'snapshot' ? this.paged(message) : this.call(message);
- }
-
- /** The agent's sessions on the worker, as their hosts recorded them. */
- async list(): Promise<Session[]> {
- return z.array(sessionSchema).parse(await this.paged({ list: true }));
- }
-
- /** A session read from its journal on the worker, whether or not its host runs. */
- async journal(sessionId: string): Promise<Snapshot> {
- return snapshotSchema.parse(await this.paged({ journal: sessionId }));
- }
-
- async place(sessionId: string, roomId: string): Promise<PlaceOutcome> {
- return placeOutcomeSchema.parse(await this.call({ place: { sessionId, roomId } }));
- }
-
- async forget(sessionId: string): Promise<void> {
- await this.call({ forget: sessionId });
- }
-
- async health(): Promise<WatcherHealth> {
- return watcherHealthSchema.parse(await this.call({ health: true }));
- }
-
- /**
- * Upload a file to the session's worker in chunks and stage it there. The
- * attachment it returns is named by a `message.send`, which consumes it.
- */
- async uploadAttachment(
- sessionId: string,
- file: { name: string; mimeType: string; data: Uint8Array }
- ): Promise<Attachment> {
- const size = file.data.byteLength;
- if (size === 0) throw new Error(`${file.name} is empty.`);
- if (size > MAX_ATTACHMENT_BYTES)
- throw new Error(`${file.name} is larger than ${MAX_ATTACHMENT_BYTES / (1024 * 1024)} MiB.`);
- const sha256 = createHash('sha256').update(file.data).digest('hex');
- const count = Math.ceil(size / MAX_CHUNK_BYTES);
- const transferId = randomUUID();
- try {
- for (let index = 0; index < count; index++) {
- const chunk = Buffer.from(
- file.data.subarray(index * MAX_CHUNK_BYTES, (index + 1) * MAX_CHUNK_BYTES)
- ).toString('base64');
- const message = {
- attachment: {
- transferId,
- sessionId,
- name: file.name,
- mimeType: file.mimeType,
- size,
- sha256,
- index,
- count,
- chunk,
- },
- };
- let answer: z.infer<typeof chunkAnswerSchema> | null = null;
- for (let attempt = 1; answer === null; attempt++) {
- try {
- answer = chunkAnswerSchema.parse(await this.call(message));
- } catch (error) {
- // A chunk is idempotent per index, so one whose reply was lost is sent again.
- const lost =
- error instanceof CloudRelayError &&
- error.relayCode === 'relay_timeout' &&
- attempt < CHUNK_ATTEMPTS;
- if (!lost) throw error;
- }
- }
- if ('staged' in answer) {
- if (index !== count - 1)
- throw new Error(`The cloud worker staged ${file.name} before its last chunk.`);
- return {
- attachmentId: answer.staged.ref,
- name: file.name,
- mimeType: file.mimeType,
- bytes: size,
- sha256,
- };
- }
- if (answer.next !== index + 1)
- throw new Error(
- `The cloud worker expected chunk ${answer.next} of ${file.name} after chunk ${index}.`
- );
- }
- throw new Error(`The cloud worker did not stage ${file.name} after its last chunk.`);
- } catch (error) {
- // The worker also drops a partial transfer after two idle minutes.
- await this.call({ attachmentCancel: transferId }).catch(() => undefined);
- throw error;
- }
- }
-
- private stream(
- query: string,
- onFrame: (frame: Frame) => void,
- onEnd: (error: Error) => void
- ): RelayStream {
- const abort = new AbortController();
- let settle!: { resolve: () => void; reject: (error: Error) => void };
- const ready = new Promise<void>((resolve, reject) => {
- settle = { resolve, reject };
- });
- let opened = false;
- void (async () => {
- const response = await this.fetchRelay(`/relay/stream?${query}`, {
- method: 'GET',
- body: undefined,
- signal: abort.signal,
- });
- if (!response.ok) {
- const reply = await readReply(response);
- throw refusal(
- reply.error?.code ?? 'failed',
- reply.error?.message ?? `The relay stream was refused (${response.status}).`,
- response.status,
- reply.wake_available === true
- );
- }
- for await (const frame of relayFrames(response)) {
- if (!opened && frame.event === 'worker') {
- opened = true;
- settle.resolve();
- }
- onFrame(frame);
- }
- throw new CloudRelayClosedError('Switch ended the stream.');
- })().catch((error: unknown) => {
- if (abort.signal.aborted) return;
- const failure = error instanceof Error ? error : new CloudRelayClosedError(String(error));
- if (opened) onEnd(failure);
- else settle.reject(failure);
- });
- return { ready, close: () => abort.abort() };
- }
-
- /**
- * Hear every event the session's host records, and `onFailure` whenever it
- * stops on a failure or comes up again. `onReset` is called once, instead
- * of any further event, when the live view can no longer be trusted: the
- * worker changed, Switch reported a gap or an overflow, or the stream
- * closed. The view is then rebuilt from a new snapshot.
- *
- * Resolves once Switch holds the view, so a snapshot asked for after it
- * misses nothing the view does not also carry.
- */
- async subscribe(
- sessionId: string,
- listener: (event: ServerEvent) => void,
- onFailure: (failure: string | null) => void,
- onReset: (reason: string) => void
- ): Promise<() => void> {
- if (this.closed) throw this.closed;
- let worker: string | null = null;
- let done = false;
- const reset = (reason: string) => {
- if (done) return;
- done = true;
- this.views.delete(reset);
- stream.close();
- onReset(reason);
- };
- const stream = this.stream(
- new URLSearchParams({ subscribe: sessionId }).toString(),
- (frame) => {
- if (done) return;
- switch (frame.event) {
- case 'worker': {
- const current = JSON.stringify(workerFrameSchema.parse(frame.data));
- if (worker === null) worker = current;
- else if (worker !== current) reset('The cloud worker changed.');
- return;
- }
- case 'event': {
- const parsed = serverEventSchema.safeParse(eventFrameSchema.parse(frame.data).event);
- if (parsed.success) listener(parsed.data);
- else reset(`The cloud worker sent an unreadable event: ${parsed.error.message}`);
- return;
- }
- case 'failure':
- onFailure(failureFrameSchema.parse(frame.data).failure);
- return;
- case 'resync':
- reset(`Switch asked for a resync (${resyncFrameSchema.parse(frame.data).reason}).`);
- return;
- case 'error': {
- const error = errorFrameSchema.parse(frame.data);
- reset(`${error.message} (${error.code})`);
- return;
- }
- }
- },
- (error) => reset(error.message)
- );
- this.views.add(reset);
- try {
- await stream.ready;
- } catch (error) {
- done = true;
- this.views.delete(reset);
- throw error;
- }
- return () => {
- done = true;
- this.views.delete(reset);
- stream.close();
- };
- }
-
- /**
- * Called with the watcher's state whenever it changes. The stream behind it
- * ending, or the worker changing under it, closes this client.
- */
- async onHealth(listener: (health: WatcherHealth) => void): Promise<() => void> {
- if (this.closed) throw this.closed;
- this.healthListeners.add(listener);
- if (!this.healthStream) {
- let worker: string | null = null;
- const stream = this.stream(
- 'watchHealth=1',
- (frame) => {
- if (frame.event === 'worker') {
- const current = JSON.stringify(workerFrameSchema.parse(frame.data));
- if (worker === null) worker = current;
- else if (worker !== current)
- this.fail(new CloudRelayClosedError('the cloud worker changed.'));
- } else if (frame.event === 'health') {
- const { health } = healthFrameSchema.parse(frame.data);
- for (const each of this.healthListeners) each(health);
- } else if (frame.event === 'resync' || frame.event === 'error')
- this.fail(new CloudRelayClosedError(JSON.stringify(frame.data)));
- },
- (error) => this.fail(error)
- );
- this.healthStream = stream;
- try {
- await stream.ready;
- } catch (error) {
- this.healthStream = null;
- this.healthListeners.delete(listener);
- throw error;
- }
- }
- return () => {
- if (!this.healthListeners.delete(listener) || this.healthListeners.size) return;
- this.healthStream?.close();
- this.healthStream = null;
- };
- }
-
- onClose(listener: (error: Error) => void): () => void {
- if (this.closed) {
- listener(this.closed);
- return () => {};
- }
- this.closeListeners.add(listener);
- return () => this.closeListeners.delete(listener);
- }
-
- private fail(error: Error): void {
- if (this.closed) return;
- this.closed = error;
- this.healthStream?.close();
- this.healthStream = null;
- for (const reset of [...this.views]) reset(error.message);
- for (const listener of this.closeListeners) listener(error);
- this.closeListeners.clear();
- }
-
- close(): void {
- this.fail(new CloudRelayClosedError('closed by Console.'));
- }
-}
diff --git a/console/packages/agent-providers/src/host/cutover-manifest.ts b/console/packages/agent-providers/src/host/cutover-manifest.ts
deleted file mode 100644
index 6d444c634..000000000
--- a/console/packages/agent-providers/src/host/cutover-manifest.ts
+++ /dev/null
@@ -1,119 +0,0 @@
-import { randomUUID } from 'node:crypto';
-import { mkdir, open, readFile, rename, rm } from 'node:fs/promises';
-import { join } from 'node:path';
-import { z } from 'zod';
-
-/**
- * The layout version of a hosted worker's volume. The bootstrap's preflight
- * writes it once the volume is in this layout, the watcher refuses to attach
- * without it, and Switch refuses a worker that states an older one.
- */
-export const HOSTED_STATE_VERSION = 1;
-export const STATE_VERSION_FILE = 'state-version.json';
-export const PREFLIGHT_BLOCKED_FILE = 'preflight-blocked.json';
-const MANIFEST_FILE = join('cutover', 'manifest.json');
-const UPLOADED_FILE = join('cutover', 'uploaded.json');
-
-const hostState = z.enum(['accepted', 'dispatched', 'finished']);
-export const cutoverItemSchema = z.discriminatedUnion('kind', [
- z.strictObject({
- kind: z.literal('room_message'),
- session_id: z.string().min(1).nullable(),
- room_id: z.string().min(1),
- message_id: z.string().min(1),
- thread_id: z.string().min(1).nullable(),
- room_pending: z.boolean(),
- failure_notified: z.boolean(),
- host: hostState.nullable(),
- }),
- z.strictObject({
- kind: z.literal('console_command'),
- session_id: z.string().min(1),
- command_id: z.string().min(1),
- host: hostState,
- }),
- z.strictObject({
- kind: z.literal('request_open'),
- session_id: z.string().min(1),
- request_id: z.string().min(1),
- room_id: z.string().min(1).nullable(),
- thread_id: z.string().min(1).nullable(),
- }),
- z.strictObject({ kind: z.literal('reset_pending'), session_id: z.string().min(1) }),
-]);
-export type CutoverItem = z.infer<typeof cutoverItemSchema>;
-const manifestSchema = z.strictObject({
- manifest_sha256: z.string().regex(/^[0-9a-f]{64}$/),
- items: z.array(cutoverItemSchema),
-});
-export type CutoverManifest = z.infer<typeof manifestSchema>;
-const stateVersionSchema = z.strictObject({ version: z.number().int().positive() });
-
-export async function writeJsonAtomically(path: string, value: unknown): Promise<void> {
- const temporary = `${path}.${randomUUID()}.tmp`;
- const file = await open(temporary, 'wx', 0o600);
- try {
- await file.writeFile(JSON.stringify(value));
- await file.sync();
- } catch (error) {
- await file.close();
- await rm(temporary, { force: true });
- throw error;
- }
- await file.close();
- await rename(temporary, path);
-}
-
-async function optionalJson(path: string): Promise<unknown> {
- try {
- return JSON.parse(await readFile(path, 'utf8'));
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined;
- throw new Error(`${path} cannot be read: ${String(error)}`);
- }
-}
-
-/** The volume's layout version, or null for a volume no preflight has finished. */
-export async function readStateVersion(root: string): Promise<number | null> {
- const value = await optionalJson(join(root, STATE_VERSION_FILE));
- return value === undefined ? null : stateVersionSchema.parse(value).version;
-}
-
-export async function writeStateVersion(root: string): Promise<void> {
- await writeJsonAtomically(join(root, STATE_VERSION_FILE), { version: HOSTED_STATE_VERSION });
-}
-
-export async function readCutoverManifest(root: string): Promise<CutoverManifest | null> {
- const value = await optionalJson(join(root, MANIFEST_FILE));
- return value === undefined ? null : manifestSchema.parse(value);
-}
-
-/** Written once: a rerun of the preflight keeps the manifest, and its digest, it first wrote. */
-export async function writeCutoverManifest(
- root: string,
- manifest: CutoverManifest
-): Promise<CutoverManifest> {
- const saved = await readCutoverManifest(root);
- if (saved) return saved;
- await mkdir(join(root, 'cutover'), { recursive: true, mode: 0o700 });
- await writeJsonAtomically(join(root, MANIFEST_FILE), manifest);
- return manifest;
-}
-
-/** The manifest Switch has not yet confirmed, or null once it has (or there is none). */
-export async function unconfirmedCutoverManifest(root: string): Promise<CutoverManifest | null> {
- const manifest = await readCutoverManifest(root);
- if (!manifest) return null;
- const uploaded = await optionalJson(join(root, UPLOADED_FILE));
- if (
- uploaded !== undefined &&
- z.strictObject({ manifest_sha256: z.string() }).parse(uploaded).manifest_sha256 ===
- manifest.manifest_sha256
- )
- return null;
- return manifest;
-}
-
-export async function confirmCutoverManifest(root: string, sha256: string): Promise<void> {
- await writeJsonAtomically(join(root, UPLOADED_FILE), { manifest_sha256: sha256 });
-}
diff --git a/console/packages/agent-providers/src/host/exit-codes.ts b/console/packages/agent-providers/src/host/exit-codes.ts
deleted file mode 100644
index 45ddf4660..000000000
--- a/console/packages/agent-providers/src/host/exit-codes.ts
+++ /dev/null
@@ -1,15 +0,0 @@
-/**
- * EX_TEMPFAIL, reserved for a hosted worker whose capability Switch no longer
- * accepts: its launch moved to a newer revision. The worker service records
- * the bundle it booted with as obsolete and waits for a current one, so
- * nothing relaunches the daemon on the same bundle.
- */
-export const OBSOLETE_BUNDLE_EXIT_CODE = 75;
-
-/** A hosted worker was refused as obsolete; the process exits `OBSOLETE_BUNDLE_EXIT_CODE`. */
-export class WorkerObsoleteError extends Error {
- constructor(reason: string) {
- super(`This worker's bundle is obsolete: ${reason}`);
- this.name = 'WorkerObsoleteError';
- }
-}
diff --git a/console/packages/agent-providers/src/host/host-permissions.test.ts b/console/packages/agent-providers/src/host/host-permissions.test.ts
new file mode 100644
index 000000000..d785c664f
--- /dev/null
+++ b/console/packages/agent-providers/src/host/host-permissions.test.ts
@@ -0,0 +1,19 @@
+import { afterEach, describe, expect, it } from 'vitest';
+import { dirMode, fileMode, SHARED_GROUP_ENV } from './host-permissions';
+
+afterEach(() => {
+ delete process.env[SHARED_GROUP_ENV];
+});
+
+describe('host permissions', () => {
+ it('keeps what an agent host writes private by default', () => {
+ expect(fileMode(0o600)).toBe(0o600);
+ expect(dirMode(0o700)).toBe(0o700);
+ });
+
+ it('opens it to the group under the shared group, with no set-id bit', () => {
+ process.env[SHARED_GROUP_ENV] = '1';
+ expect(fileMode(0o600)).toBe(0o640);
+ expect(dirMode(0o700)).toBe(0o770);
+ });
+});
diff --git a/console/packages/agent-providers/src/host/host-permissions.ts b/console/packages/agent-providers/src/host/host-permissions.ts
new file mode 100644
index 000000000..cfce85f8e
--- /dev/null
+++ b/console/packages/agent-providers/src/host/host-permissions.ts
@@ -0,0 +1,31 @@
+/**
+ * Modes for the files and directories an agent host writes into its state.
+ *
+ * Private to the agent's own user by default. An agents controller that runs
+ * each agent as a Linux user of its own sets `SWITCH_HOST_SHARED_GROUP=1`: the
+ * agent's primary group is then one the controller belongs to and no other
+ * user does, so what the agent host writes is group-readable for the
+ * controller to observe (health, owner and failure records) and still closed
+ * to everyone else. No set-id bit is ever asked for: such an agent runs with
+ * `RestrictSUIDSGID=`.
+ */
+export const SHARED_GROUP_ENV = 'SWITCH_HOST_SHARED_GROUP';
+
+export function sharedGroupEnabled(): boolean {
+ return process.env[SHARED_GROUP_ENV] === '1';
+}
+
+/** `privateMode`, or with the owner's read given to the group under the shared group (0o600 → 0o640). */
+export function fileMode(privateMode: number): number {
+ if (!sharedGroupEnabled()) return privateMode;
+ return privateMode | ((privateMode & 0o400) >> 3);
+}
+
+/**
+ * `privateMode`, or with the owner's bits given to the group under the shared
+ * group (0o700 → 0o770), so the controller can also clear what it reads.
+ */
+export function dirMode(privateMode: number): number {
+ if (!sharedGroupEnabled()) return privateMode;
+ return privateMode | ((privateMode & 0o700) >> 3);
+}
diff --git a/console/packages/agent-providers/src/host/hosted-bootstrap-cli.ts b/console/packages/agent-providers/src/host/hosted-bootstrap-cli.ts
deleted file mode 100644
index 8d35235b8..000000000
--- a/console/packages/agent-providers/src/host/hosted-bootstrap-cli.ts
+++ /dev/null
@@ -1,63 +0,0 @@
-#!/usr/bin/env node
-import { fileURLToPath } from 'node:url';
-import { OBSOLETE_BUNDLE_EXIT_CODE, WorkerObsoleteError } from './exit-codes';
-import { runHostedBootstrap } from './hosted-bootstrap';
-import { runGitHubCli, runGitHubCredentialHelper } from './hosted-github';
-import { checkHostedPreflight } from './hosted-preflight';
-import { superviseSharedHost } from './supervisor';
-import { runCredentialVerification } from './verify-credential';
-
-async function main(): Promise<void> {
- if (process.argv[2] === '--verify-credential') {
- await runCredentialVerification();
- return;
- }
- if (process.argv[2] === '--preflight-check') {
- const [stateDirectory, scratchDirectory, ...rest] = process.argv.slice(3);
- if (!stateDirectory || !scratchDirectory || rest.length)
- throw new Error(
- 'Usage: switch-hosted-bootstrap --preflight-check <absolute-state-directory> <absolute-scratch-directory>'
- );
- const result = await checkHostedPreflight(stateDirectory, scratchDirectory);
- process.stdout.write(`${JSON.stringify(result)}\n`);
- if ('blocked' in result) process.exitCode = 2;
- return;
- }
- if (process.argv[2] === '--github-cli') {
- await runGitHubCli(process.argv.slice(3));
- return;
- }
- if (process.argv[2] === '--git-credential') {
- if (process.argv.length !== 4) throw new Error('Invalid Git credential helper arguments.');
- await runGitHubCredentialHelper(process.argv[3]);
- return;
- }
- const [stateDirectory, specPath, ...extra] = process.argv.slice(2);
- if (!stateDirectory || !specPath || extra.length)
- throw new Error(
- 'Usage: switch-hosted-bootstrap <absolute-state-directory> <deployment-spec.json>'
- );
- const stop = new AbortController();
- const abort = () => stop.abort();
- process.on('SIGINT', abort);
- process.on('SIGTERM', abort);
- try {
- await runHostedBootstrap(
- {
- stateDirectory,
- specPath,
- sharedDaemonEntrypoint: fileURLToPath(new URL('./shared-host-daemon.mjs', import.meta.url)),
- signal: stop.signal,
- },
- { supervise: superviseSharedHost }
- );
- } finally {
- process.removeListener('SIGINT', abort);
- process.removeListener('SIGTERM', abort);
- }
-}
-
-main().catch((error: unknown) => {
- console.error(error instanceof Error ? error.message : 'Hosted SDK bootstrap failed.');
- process.exitCode = error instanceof WorkerObsoleteError ? OBSOLETE_BUNDLE_EXIT_CODE : 1;
-});
diff --git a/console/packages/agent-providers/src/host/hosted-bootstrap-owner-records.test.ts b/console/packages/agent-providers/src/host/hosted-bootstrap-owner-records.test.ts
deleted file mode 100644
index b0486bea5..000000000
--- a/console/packages/agent-providers/src/host/hosted-bootstrap-owner-records.test.ts
+++ /dev/null
@@ -1,144 +0,0 @@
-import { mkdtemp, readdir, readFile, rm } from 'node:fs/promises';
-import { tmpdir } from 'node:os';
-import { basename, dirname, join } from 'node:path';
-import { fileURLToPath } from 'node:url';
-import { afterEach, expect, it, vi } from 'vitest';
-import { openSwitchStream, runAgentHost } from './agent-host';
-import type { Supervision } from './launch';
-import type * as OwnershipLock from './ownership-lock';
-import { withOwnershipLock } from './ownership-lock';
-import type { SharedHostConfig } from './shared-config';
-import type { SharedHostOptions } from './shared-host';
-import { SharedState } from './shared-state';
-import { superviseSharedHost } from './supervisor';
-import { WatcherControl } from './watcher-tools';
-
-const written = vi.hoisted(() => [] as Array<{ path: string; value: unknown }>);
-
-vi.mock('./ownership-lock', async (original) => {
- const actual = await original<typeof OwnershipLock>();
- return {
- ...actual,
- replaceOwner: async (path: string, value: unknown) => {
- written.push({ path, value: structuredClone(value) });
- await actual.replaceOwner(path, value);
- },
- };
-});
-
-/**
- * The supervisor quarantines these records after a reboot and refuses any
- * other shape, so its fixture has to match what this code writes.
- */
-const OWNER_RECORDS = join(
- dirname(fileURLToPath(import.meta.url)),
- '../../../../../deploy/hosted/worker/testdata/owner-records.json'
-);
-
-const roots: string[] = [];
-afterEach(async () => {
- written.splice(0);
- for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true });
-});
-
-async function temporary(): Promise<string> {
- const root = await mkdtemp(join(tmpdir(), 'hosted-owner-records-'));
- roots.push(root);
- return root;
-}
-
-function shape(value: unknown, nullable: ReadonlySet<string>): Record<string, string> {
- expect(value).toBeTypeOf('object');
- return Object.fromEntries(
- Object.entries(value as Record<string, unknown>)
- .map(([key, field]): [string, string] => [
- key,
- field === null && nullable.has(key) ? 'number' : typeof field,
- ])
- .sort(([a], [b]) => a.localeCompare(b))
- );
-}
-
-async function fixtureRecords(): Promise<Array<[string, unknown]>> {
- return Object.entries(JSON.parse(await readFile(OWNER_RECORDS, 'utf8')));
-}
-
-it('writes owner records in the shapes the supervisor quarantines after a reboot', async () => {
- const root = await temporary();
-
- const tickets = await withOwnershipLock(root, async () => {
- const directory = join(root, 'ownership');
- return Promise.all(
- (await readdir(directory)).map(
- async (name) => JSON.parse(await readFile(join(directory, name), 'utf8')) as unknown
- )
- );
- });
- expect(tickets).toHaveLength(1);
-
- await expect(
- runAgentHost(
- join(root, 'watcher'),
- {} as SharedHostConfig,
- new AbortController().signal,
- {} as Supervision,
- new WatcherControl(),
- null,
- openSwitchStream
- )
- ).rejects.toThrow('execution credentials');
-
- await superviseSharedHost({
- root: join(root, 'session'),
- executable: process.execPath,
- args: ['-e', 'process.exit(0)'],
- env: process.env,
- signal: new AbortController().signal,
- build: '/opt/switch/agent-providers/shared-host-daemon.mjs',
- links: null,
- logRedactions: [],
- });
-
- const state = await SharedState.open({
- root: join(root, 'host'),
- agentApiUrl: 'https://switch.example.test/agent-api',
- input: { cwd: root },
- session: {
- sessionId: 'session-id',
- agentId: 'agent-id',
- hostId: 'host-id',
- epoch: 'epoch',
- provider: 'claude',
- },
- } as unknown as SharedHostOptions);
- await state.unlock();
-
- const produced = (path: string) => written.filter((entry) => entry.path === path);
- const [watcher] = produced(join(root, 'watcher', 'shared-owner.lock'));
- const [supervisor] = produced(join(root, 'session', 'supervisor', 'owner.json'));
- const [host] = produced(join(root, 'host', 'shared-owner.lock'));
- expect(watcher && supervisor && host).toBeTruthy();
-
- const nullable = new Set(['group']);
- const actual = {
- watcher: shape(watcher!.value, nullable),
- host: shape(host!.value, nullable),
- supervisor: shape(supervisor!.value, nullable),
- ticket: shape(tickets[0], nullable),
- };
- const records = await fixtureRecords();
- expect(records.length).toBeGreaterThan(0);
- for (const [relative, record] of records) {
- const expected = shape(record, nullable);
- if (basename(relative) === 'shared-owner.lock')
- expect([actual.watcher, actual.host]).toContainEqual(expected);
- else if (basename(relative) === 'owner.json') expect(expected).toEqual(actual.supervisor);
- else if (basename(dirname(relative)) === 'ownership') expect(expected).toEqual(actual.ticket);
- else throw new Error(`Unexpected owner record in the fixture: ${relative}`);
- }
- const lockShapes = records
- .filter(([relative]) => basename(relative) === 'shared-owner.lock')
- .map(([, record]) => shape(record, nullable));
- expect(lockShapes).toContainEqual(actual.watcher);
- expect(lockShapes).toContainEqual(actual.host);
-});
diff --git a/console/packages/agent-providers/src/host/hosted-bootstrap.test.ts b/console/packages/agent-providers/src/host/hosted-bootstrap.test.ts
deleted file mode 100644
index f3a180e78..000000000
--- a/console/packages/agent-providers/src/host/hosted-bootstrap.test.ts
+++ /dev/null
@@ -1,901 +0,0 @@
-import { execFile } from 'node:child_process';
-import {
- chmod,
- mkdir,
- mkdtemp,
- readdir,
- readFile,
- realpath,
- rm,
- stat,
- writeFile,
-} from 'node:fs/promises';
-import { tmpdir } from 'node:os';
-import { dirname, join } from 'node:path';
-import { fileURLToPath } from 'node:url';
-import { promisify } from 'node:util';
-import { afterEach, beforeEach, expect, it, vi } from 'vitest';
-import { WorkerObsoleteError } from './exit-codes';
-import {
- type HostedBootstrapDependencies,
- type HostedDeploymentSpec,
- hostedDeploymentSpecSchema,
- prepareHostedDeployment,
- runHostedBootstrap,
-} from './hosted-bootstrap';
-import { githubLaunchEnvironment } from './hosted-github';
-import { hostedSkillsDirectory } from './hosted-skills';
-import type { superviseSharedHost } from './supervisor';
-
-const WORKER_DEPLOYMENT = join(
- dirname(fileURLToPath(import.meta.url)),
- '../../../../../deploy/hosted/worker/testdata/deployment.json'
-);
-
-const roots: string[] = [];
-
-beforeEach(() => {
- vi.stubEnv('SWITCH_HOST_INSTANCE_ID', 'instance-fixture');
- vi.stubEnv('SWITCH_HOST_BOOT_ID', 'boot-fixture');
-});
-
-afterEach(async () => {
- vi.unstubAllEnvs();
- vi.unstubAllGlobals();
- for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true });
-});
-
-async function fixture(): Promise<{
- root: string;
- state: string;
- workspace: string;
- providerCredential: string;
- switchCredentials: string;
- workerCapability: string;
- specPath: string;
- spec: HostedDeploymentSpec;
-}> {
- const root = await mkdtemp(join(tmpdir(), 'hosted-bootstrap-'));
- roots.push(root);
- const state = join(root, 'state');
- const workspace = join(root, 'workspace');
- const secrets = join(root, 'mounted-secrets');
- const binary = join(root, 'claude');
- const providerCredential = join(secrets, 'provider');
- const switchCredentials = join(secrets, 'switch.json');
- const workerCapability = join(secrets, 'worker-capability');
- const specPath = join(root, 'deployment.json');
- await mkdir(workspace);
- await mkdir(secrets);
- await writeFile(binary, '#!/bin/sh\necho \'{"loggedIn":true}\'\n', { mode: 0o700 });
- await chmod(binary, 0o700);
- await writeFile(providerCredential, 'provider-secret-value\n', { mode: 0o600 });
- await writeFile(workerCapability, 'worker-capability-secret\n', { mode: 0o600 });
- await writeFile(
- switchCredentials,
- JSON.stringify({
- env: {
- SWITCH_API_ENDPOINT: 'https://switch.invalid/api/agent',
- SWITCH_API_TOKEN: 'switch-secret-value',
- SWITCH_AGENT_ID: 'agent-id',
- },
- }),
- { mode: 0o600 }
- );
- const spec: HostedDeploymentSpec = {
- version: 2,
- revision: 3,
- session: { sessionId: 'session-id', agentId: 'agent-id' },
- provider: {
- kind: 'claude',
- credential: { kind: 'api-key', path: providerCredential },
- binaryPath: binary,
- context: 'Follow the mounted Switch room workflow.',
- },
- workspacePath: workspace,
- watch: true,
- runtimeMode: 'approval-required',
- switchCredentialsPath: switchCredentials,
- workerCapabilityPath: workerCapability,
- };
- await writeFile(specPath, JSON.stringify(spec));
- return {
- root,
- state,
- workspace,
- providerCredential,
- switchCredentials,
- workerCapability,
- specPath,
- spec,
- };
-}
-
-const mirrorPath = (input: { root: string }) => join(input.root, 'repos', 'example', 'project.git');
-
-async function configureGitHub(
- input: Awaited<ReturnType<typeof fixture>>,
- token = 'github-secret-value'
-): Promise<string> {
- const credentialPath = join(input.root, 'mounted-secrets', 'github');
- await writeFile(credentialPath, `${token}\n`, { mode: 0o600 });
- input.spec.github = { credentialPath, mirrorPath: mirrorPath(input) };
- return credentialPath;
-}
-
-function mockGitHubValidation(status = 200, body = '{}') {
- const request = vi.fn(async () => new Response(body, { status }));
- vi.stubGlobal('fetch', request);
- return request;
-}
-
-/** Stub Switch's `/hosted` routes; any other request is GitHub validation. */
-function mockHosted() {
- const request = vi.fn(async (url: string | URL | Request) => {
- const path = String(url);
- if (path.endsWith('/hosted/provider-credential'))
- return new Response(
- JSON.stringify({
- status: 'connected',
- revision: 'test-revision',
- provider: 'claude',
- kind: 'api-key',
- credential: 'provider-secret-value',
- })
- );
- return new Response('{}');
- });
- vi.stubGlobal('fetch', request);
- return request;
-}
-
-function run(
- input: Awaited<ReturnType<typeof fixture>>,
- dependencies: HostedBootstrapDependencies,
- signal = new AbortController().signal
-): Promise<void> {
- return runHostedBootstrap(
- {
- stateDirectory: input.state,
- specPath: input.specPath,
- sharedDaemonEntrypoint: '/opt/switch/shared-host-daemon.mjs',
- signal,
- },
- dependencies
- );
-}
-
-it('persists one stable identity without persisting either credential value', async () => {
- const input = await fixture();
- const first = await prepareHostedDeployment(input.state, input.spec);
- const second = await prepareHostedDeployment(input.state, input.spec);
- expect(second.config.session.hostId).toBe(first.config.session.hostId);
- expect(second.config.session.epoch).toBe(first.config.session.epoch);
- expect(second.config.roomConnection?.connectionId).toBe(
- first.config.roomConnection?.connectionId
- );
- expect(first.providerEnvironment.ANTHROPIC_API_KEY).toBe('provider-secret-value');
- expect(first.providerEnvironment.SWITCH_API_TOKEN).toBeUndefined();
- expect(first.providerEnvironment.GH_TOKEN).toBeUndefined();
- expect(first.config.execution?.inheritEnv).not.toContain('GH_TOKEN');
- const persisted = [
- await readFile(join(input.state, 'hosted-deployment.json'), 'utf8'),
- await readFile(join(input.state, 'config.json'), 'utf8'),
- ].join('\n');
- expect(persisted).not.toContain('provider-secret-value');
- expect(persisted).not.toContain('switch-secret-value');
- expect(persisted).not.toContain('worker-capability-secret');
- expect(persisted).not.toContain('instance-fixture');
- expect(first.config.start.input.env.HOME).toBe(join(first.root, 'home'));
- expect(first.config.start.input.env.CLAUDE_CONFIG_DIR).toBe(
- join(first.root, 'provider-home', 'claude')
- );
-});
-
-it('converges concurrent first starts on the same saved identity', async () => {
- const input = await fixture();
- const [first, second] = await Promise.all([
- prepareHostedDeployment(input.state, input.spec),
- prepareHostedDeployment(input.state, input.spec),
- ]);
- expect(second.config).toEqual(first.config);
-});
-
-it('maps a mounted setup token only into the worker environment', async () => {
- const input = await fixture();
- input.spec.provider.credential.kind = 'setup-token';
- const prepared = await prepareHostedDeployment(input.state, input.spec);
- expect(prepared.providerEnvironment.CLAUDE_CODE_OAUTH_TOKEN).toBe('provider-secret-value');
- expect(prepared.providerEnvironment.ANTHROPIC_API_KEY).toBeUndefined();
- expect(JSON.stringify(prepared.config)).not.toContain('provider-secret-value');
-});
-
-it('does not fall back to ambient provider, cloud, Node, or home credentials', async () => {
- const input = await fixture();
- vi.stubEnv('ANTHROPIC_API_KEY', 'ambient-provider-secret');
- vi.stubEnv('CLAUDE_CODE_OAUTH_TOKEN', 'ambient-setup-secret');
- vi.stubEnv('AWS_SECRET_ACCESS_KEY', 'ambient-cloud-secret');
- vi.stubEnv('NODE_OPTIONS', '--require=/tmp/ambient-hook.cjs');
- vi.stubEnv('CLAUDE_CONFIG_DIR', '/tmp/ambient-claude-home');
- const prepared = await prepareHostedDeployment(input.state, input.spec);
- expect(prepared.providerEnvironment.ANTHROPIC_API_KEY).toBe('provider-secret-value');
- expect(prepared.providerEnvironment.CLAUDE_CODE_OAUTH_TOKEN).toBeUndefined();
- expect(prepared.providerEnvironment.AWS_SECRET_ACCESS_KEY).toBeUndefined();
- expect(prepared.providerEnvironment.NODE_OPTIONS).toBeUndefined();
- expect(prepared.providerEnvironment.CLAUDE_CONFIG_DIR).toBe(
- join(prepared.root, 'provider-home', 'claude')
- );
-});
-
-it('reloads a replaced provider secret at bootstrap without changing session identity', async () => {
- const input = await fixture();
- const first = await prepareHostedDeployment(input.state, input.spec);
- await writeFile(input.providerCredential, 'rotated-provider-secret\n', { mode: 0o600 });
- const second = await prepareHostedDeployment(input.state, input.spec);
- expect(second.config).toEqual(first.config);
- expect(second.providerEnvironment.ANTHROPIC_API_KEY).toBe('rotated-provider-secret');
- const persisted = [
- await readFile(join(second.root, 'hosted-deployment.json'), 'utf8'),
- await readFile(join(second.root, 'config.json'), 'utf8'),
- ].join('\n');
- expect(persisted).not.toContain('provider-secret-value');
- expect(persisted).not.toContain('rotated-provider-secret');
-});
-
-it('validates and launches with a GitHub token without persisting it', async () => {
- const input = await fixture();
- await configureGitHub(input);
- const request = mockGitHubValidation();
- const prepared = await prepareHostedDeployment(input.state, input.spec);
-
- expect(request).toHaveBeenCalledTimes(1);
- expect(prepared.providerEnvironment.GH_TOKEN).toBe('github-secret-value');
- expect(prepared.config.execution?.inheritEnv).toContain('GH_TOKEN');
- for (const [key, value] of Object.entries(githubLaunchEnvironment())) {
- expect(prepared.config.start.input.env[key]).toBe(value);
- expect(prepared.providerEnvironment[key]).toBe(value);
- }
- const persisted = [
- await readFile(join(input.state, 'hosted-deployment.json'), 'utf8'),
- await readFile(join(input.state, 'config.json'), 'utf8'),
- ].join('\n');
- expect(persisted).not.toContain('github-secret-value');
-});
-
-it('reloads and validates a rotated GitHub token without changing the saved plan', async () => {
- const input = await fixture();
- const credentialPath = await configureGitHub(input);
- const request = mockGitHubValidation();
- const first = await prepareHostedDeployment(input.state, input.spec);
- await writeFile(credentialPath, 'rotated-github-secret\n', { mode: 0o600 });
- const second = await prepareHostedDeployment(input.state, input.spec);
-
- expect(request).toHaveBeenCalledTimes(2);
- expect(second.config).toEqual(first.config);
- expect(second.providerEnvironment.GH_TOKEN).toBe('rotated-github-secret');
- const persisted = [
- await readFile(join(input.state, 'hosted-deployment.json'), 'utf8'),
- await readFile(join(input.state, 'config.json'), 'utf8'),
- ].join('\n');
- expect(persisted).not.toContain('github-secret-value');
- expect(persisted).not.toContain('rotated-github-secret');
-});
-
-it('rejects a GitHub credential inside hosted state or workspace before validation', async () => {
- const input = await fixture();
- await mkdir(input.state, { mode: 0o700 });
- const stateCredential = join(input.state, 'github');
- await writeFile(stateCredential, 'github-secret-value\n', { mode: 0o600 });
- input.spec.github = { credentialPath: stateCredential, mirrorPath: mirrorPath(input) };
- const request = mockGitHubValidation();
-
- await expect(prepareHostedDeployment(input.state, input.spec)).rejects.toThrow(
- 'GitHub credential file must be mounted outside'
- );
- const workspaceCredential = join(input.workspace, 'github');
- await writeFile(workspaceCredential, 'github-secret-value\n', { mode: 0o600 });
- input.spec.github = { credentialPath: workspaceCredential, mirrorPath: mirrorPath(input) };
- await expect(prepareHostedDeployment(input.state, input.spec)).rejects.toThrow(
- 'GitHub credential file must be mounted outside'
- );
- expect(request).not.toHaveBeenCalled();
- await expect(readFile(join(input.state, 'hosted-deployment.json'))).rejects.toMatchObject({
- code: 'ENOENT',
- });
-});
-
-it('renews a refreshed GitHub credential from Switch without a mounted GitHub file', async () => {
- const input = await fixture();
- const agents = join(input.root, 'agents', 'agent-id');
- const workspace = join(input.root, 'worktrees', 'agent-id', 'example', 'project');
- await mkdir(workspace, { recursive: true });
- input.spec.workspacePath = workspace;
- input.spec.github = {
- credentialPath: join(input.root, 'run', 'agents', 'agent-id', 'github'),
- repository: 'example/project',
- refresh: true,
- mirrorPath: mirrorPath(input),
- };
- const request = vi.fn(async (url: string | URL | Request) =>
- String(url).endsWith('/hosted/github-credential')
- ? new Response(
- JSON.stringify({
- token: 'renewed-github-secret',
- repository: 'example/project',
- expires_at: new Date(Date.now() + 3_600_000).toISOString(),
- })
- )
- : new Response('{}')
- );
- vi.stubGlobal('fetch', request);
-
- const prepared = await prepareHostedDeployment(agents, input.spec);
- expect(request.mock.calls.map(([url]) => String(url))).toEqual([
- 'https://switch.invalid/api/agent/hosted/github-credential',
- 'https://api.github.com/repos/example/project',
- ]);
- expect(prepared.providerEnvironment.GH_TOKEN).toBeUndefined();
- expect(prepared.providerEnvironment.SWITCH_HOSTED_GITHUB_REPOSITORY).toBe('example/project');
- expect(prepared.logRedactions).toContain('renewed-github-secret');
- const persisted = await readFile(join(prepared.root, 'hosted-deployment.json'), 'utf8');
- expect(persisted).not.toContain('renewed-github-secret');
-});
-
-it('still requires the mounted GitHub file when the credential is not refreshed', async () => {
- const input = await fixture();
- input.spec.github = {
- credentialPath: join(input.root, 'mounted-secrets', 'github'),
- repository: 'example/project',
- mirrorPath: mirrorPath(input),
- };
- const request = mockGitHubValidation();
- await expect(prepareHostedDeployment(input.state, input.spec)).rejects.toThrow(
- 'GitHub credential file is missing or invalid.'
- );
- expect(request).not.toHaveBeenCalled();
-});
-
-it('accepts only version 2 and requires an absolute mirror path with a repository', async () => {
- const { spec, root } = await fixture();
- const github = {
- credentialPath: join(root, 'mounted-secrets', 'github'),
- repository: 'example/project',
- refresh: true as const,
- };
- expect(hostedDeploymentSpecSchema.safeParse(spec).success).toBe(true);
- expect(hostedDeploymentSpecSchema.safeParse({ ...spec, version: 1 }).success).toBe(false);
- expect(hostedDeploymentSpecSchema.safeParse({ ...spec, github }).success).toBe(false);
- expect(
- hostedDeploymentSpecSchema.safeParse({
- ...spec,
- github: { ...github, mirrorPath: 'repos/example/project.git' },
- }).success
- ).toBe(false);
- expect(
- hostedDeploymentSpecSchema.safeParse({
- ...spec,
- github: { ...github, mirrorPath: join(root, 'repos', 'example', 'project.git') },
- }).success
- ).toBe(true);
-});
-
-it('accepts the deployment the root supervisor writes for the Core agent list', async () => {
- const written: unknown = JSON.parse(await readFile(WORKER_DEPLOYMENT, 'utf8'));
- const parsed = hostedDeploymentSpecSchema.safeParse(written);
- expect(parsed.error).toBeUndefined();
- expect(parsed.data).toEqual(written);
-});
-
-it('sanitizes GitHub validation rejection and does not launch or persist a plan', async () => {
- const input = await fixture();
- await configureGitHub(input);
- await writeFile(input.specPath, JSON.stringify(input.spec));
- mockGitHubValidation(401, 'remote-body-that-must-not-escape github-secret-value');
- let launched = false;
- const supervise: typeof superviseSharedHost = async () => {
- launched = true;
- };
- let message = '';
- try {
- await run(input, { supervise });
- } catch (error) {
- message = error instanceof Error ? error.message : String(error);
- }
-
- expect(message).toContain('GitHub rejected the credential');
- expect(message).not.toContain('github-secret-value');
- expect(message).not.toContain('remote-body-that-must-not-escape');
- expect(launched).toBe(false);
- await expect(readFile(join(input.state, 'hosted-deployment.json'))).rejects.toMatchObject({
- code: 'ENOENT',
- });
- await expect(readFile(join(input.state, 'config.json'))).rejects.toMatchObject({
- code: 'ENOENT',
- });
-});
-
-it('rejects overlapping state and workspace paths before creating runtime homes', async () => {
- const input = await fixture();
- const state = join(input.workspace, 'state');
- await expect(prepareHostedDeployment(state, input.spec)).rejects.toThrow(
- 'state and workspace directories must not overlap'
- );
- await expect(readFile(join(state, 'config.json'))).rejects.toMatchObject({ code: 'ENOENT' });
-});
-
-it('treats dot-dot-prefixed names as children for path isolation', async () => {
- const input = await fixture();
- await mkdir(input.state, { mode: 0o700 });
- const credentialDirectory = join(input.state, '..credentials');
- await mkdir(credentialDirectory);
- const nestedCredential = join(credentialDirectory, 'provider');
- await writeFile(nestedCredential, 'nested-provider-secret', { mode: 0o600 });
- await expect(
- prepareHostedDeployment(input.state, {
- ...input.spec,
- provider: {
- ...input.spec.provider,
- credential: { kind: 'api-key', path: nestedCredential },
- },
- })
- ).rejects.toThrow('must be mounted outside');
-
- const nestedWorkspace = join(input.state, '..workspace');
- await mkdir(nestedWorkspace);
- await expect(
- prepareHostedDeployment(input.state, { ...input.spec, workspacePath: nestedWorkspace })
- ).rejects.toThrow('state and workspace directories must not overlap');
-});
-
-it('rejects malformed credential files without exposing their content', async () => {
- const input = await fixture();
- await writeFile(
- input.switchCredentials,
- JSON.stringify({ env: { SWITCH_API_TOKEN: 'secret-that-must-not-escape' } })
- );
- let message = '';
- try {
- await prepareHostedDeployment(input.state, input.spec);
- } catch (error) {
- message = error instanceof Error ? error.message : String(error);
- }
- expect(message).toContain('Switch credential file is missing or invalid');
- expect(message).not.toContain('secret-that-must-not-escape');
-});
-
-it('rejects a credential-bearing Switch endpoint before it can enter the journal', async () => {
- const input = await fixture();
- await writeFile(
- input.switchCredentials,
- JSON.stringify({
- env: {
- SWITCH_API_ENDPOINT: 'https://user:[email protected]/api',
- SWITCH_API_TOKEN: 'switch-secret-value',
- SWITCH_AGENT_ID: 'agent-id',
- },
- })
- );
- await expect(prepareHostedDeployment(input.state, input.spec)).rejects.toThrow(
- 'invalid endpoint'
- );
-});
-
-it('rejects a changed deployment spec and a tampered saved launch configuration', async () => {
- const input = await fixture();
- await prepareHostedDeployment(input.state, input.spec);
- await expect(
- prepareHostedDeployment(input.state, {
- ...input.spec,
- watch: false,
- })
- ).rejects.toThrow('differs from the saved state');
- const planPath = join(input.state, 'hosted-deployment.json');
- const plan = JSON.parse(await readFile(planPath, 'utf8'));
- plan.config.start.input.env.UNEXPECTED = 'value';
- await writeFile(planPath, JSON.stringify(plan));
- await expect(prepareHostedDeployment(input.state, input.spec)).rejects.toThrow(
- 'does not match its deployment specification'
- );
-});
-
-it('adopts a newer revision with a changed spec and keeps the session identity and homes', async () => {
- const input = await fixture();
- input.spec.provider.definition = { name: 'helper', content: 'first definition' };
- const first = await prepareHostedDeployment(input.state, input.spec);
- const definition = join(input.workspace, '.claude', 'agents', 'helper.md');
- await mkdir(join(input.workspace, '.claude', 'agents'), { recursive: true });
- await writeFile(definition, 'first definition');
- const retained = join(first.root, 'provider-home', 'claude', 'retained');
- await writeFile(retained, 'kept');
- const revised: HostedDeploymentSpec = {
- ...input.spec,
- revision: 4,
- watch: false,
- provider: {
- ...input.spec.provider,
- model: { id: 'model-next' },
- context: 'Revised instructions.',
- definition: { name: 'helper', content: 'revised definition' },
- },
- };
- const second = await prepareHostedDeployment(input.state, revised);
- expect(second.config.session.hostId).toBe(first.config.session.hostId);
- expect(second.config.session.epoch).toBe(first.config.session.epoch);
- expect(second.config.roomConnection?.connectionId).toBe(
- first.config.roomConnection?.connectionId
- );
- expect(second.config).not.toEqual(first.config);
- expect(JSON.parse(await readFile(join(second.root, 'config.json'), 'utf8'))).toEqual(
- second.config
- );
- const plan = JSON.parse(await readFile(join(second.root, 'hosted-deployment.json'), 'utf8'));
- expect(plan.spec).toEqual(revised);
- expect(await readFile(definition, 'utf8')).toBe('revised definition');
- expect(await readFile(retained, 'utf8')).toBe('kept');
- expect(JSON.parse(await readFile(join(second.root, 'watch.json'), 'utf8')).spawn).toBe(false);
- // The same revision is settled again; a changed spec at it is still refused.
- await expect(prepareHostedDeployment(input.state, revised)).resolves.toBeDefined();
- await expect(prepareHostedDeployment(input.state, { ...revised, watch: true })).rejects.toThrow(
- 'differs from the saved state'
- );
-});
-
-it('refuses a deployment older than the saved revision', async () => {
- const input = await fixture();
- await prepareHostedDeployment(input.state, input.spec);
- await expect(
- prepareHostedDeployment(input.state, { ...input.spec, revision: 2 })
- ).rejects.toThrow('revision 2 is older than the saved revision 3');
- await expect(
- prepareHostedDeployment(input.state, { ...input.spec, revision: 2, watch: false })
- ).rejects.toThrow('older than the saved revision');
-});
-
-it('wires the shared daemon to the supervisor and forwards shutdown', async () => {
- const input = await fixture();
- mockHosted();
- const stop = new AbortController();
- let launched: Parameters<typeof superviseSharedHost>[0] | undefined;
- let started!: () => void;
- const ready = new Promise<void>((resolve) => {
- started = resolve;
- });
- const supervise: typeof superviseSharedHost = async (options) => {
- launched = options;
- started();
- await new Promise<void>((resolve) => {
- if (options.signal.aborted) resolve();
- else options.signal.addEventListener('abort', () => resolve(), { once: true });
- });
- };
- const running = run(input, { supervise }, stop.signal);
- await ready;
- stop.abort();
- await running;
- const canonicalState = await realpath(input.state);
- expect(launched?.args).toEqual([
- '/opt/switch/shared-host-daemon.mjs',
- canonicalState,
- join(canonicalState, 'config.json'),
- '--watch-worker',
- ]);
- expect(launched?.args.join(' ')).not.toContain('provider-secret-value');
- expect(launched?.args.join(' ')).not.toContain('switch-secret-value');
- expect(launched?.env.ANTHROPIC_API_KEY).toBe('provider-secret-value');
- expect(launched?.env.SWITCH_HOSTED_BOOTSTRAP).toBe('1');
- expect(launched?.env.SWITCH_HOST_INSTANCE_ID).toBe('instance-fixture');
- expect(launched?.env.SWITCH_HOST_BOOT_ID).toBe('boot-fixture');
- expect(JSON.stringify(launched?.env)).not.toContain('worker-capability-secret');
- expect(launched?.logRedactions).toEqual(
- expect.arrayContaining([
- 'provider-secret-value',
- 'switch-secret-value',
- 'worker-capability-secret',
- ])
- );
- expect(launched?.signal.aborted).toBe(true);
-});
-
-it('writes the worker capability as a private file in the daemon root', async () => {
- const input = await fixture();
- const prepared = await prepareHostedDeployment(input.state, input.spec);
- const path = join(prepared.root, 'worker-capability');
- expect(await readFile(path, 'utf8')).toBe('worker-capability-secret');
- expect((await stat(path)).mode & 0o777).toBe(0o600);
- expect(JSON.stringify(prepared.providerEnvironment)).not.toContain('worker-capability-secret');
- expect(prepared.logRedactions).toContain('worker-capability-secret');
-});
-
-it('records whether the watcher may start sessions', async () => {
- const input = await fixture();
- await prepareHostedDeployment(input.state, input.spec);
- expect(JSON.parse(await readFile(join(input.state, 'watch.json'), 'utf8'))).toEqual({
- enabled: true,
- spawn: true,
- });
- const other = await fixture();
- other.spec.watch = false;
- await prepareHostedDeployment(other.state, other.spec);
- expect(JSON.parse(await readFile(join(other.state, 'watch.json'), 'utf8'))).toEqual({
- enabled: true,
- spawn: false,
- });
-});
-
-it('points a Codex session at the materialized login', async () => {
- const input = await fixture();
- input.spec.provider.kind = 'codex';
- const prepared = await prepareHostedDeployment(input.state, input.spec);
- expect(prepared.config.start.input.env.CODEX_HOME).toBe(join(prepared.root, 'provider-home'));
-});
-
-it('requires the machine identity', async () => {
- const input = await fixture();
- vi.stubEnv('SWITCH_HOST_BOOT_ID', '');
- await expect(prepareHostedDeployment(input.state, input.spec)).rejects.toThrow(
- 'SWITCH_HOST_BOOT_ID'
- );
-});
-
-it('requires the watch flag and the worker capability path', async () => {
- const { spec } = await fixture();
- const { watch: _watch, ...withoutWatch } = spec;
- expect(hostedDeploymentSpecSchema.safeParse(withoutWatch).success).toBe(false);
- const { workerCapabilityPath: _path, ...withoutCapability } = spec;
- expect(hostedDeploymentSpecSchema.safeParse(withoutCapability).success).toBe(false);
-});
-
-it('passes an obsolete worker through unchanged', async () => {
- const input = await fixture();
- mockHosted();
- const obsolete = new WorkerObsoleteError('The worker bundle is obsolete.');
- await expect(
- run(input, {
- supervise: async () => {
- throw obsolete;
- },
- })
- ).rejects.toBe(obsolete);
-});
-
-it('installs granted connection skills into the provider skills directory before launch', async () => {
- const input = await fixture();
- input.spec.skills = [
- {
- slug: 'github',
- files: { 'SKILL.md': '---\nname: github\n---\n', 'reference/gh.md': 'gh pr list\n' },
- },
- ];
- await writeFile(input.specPath, JSON.stringify(input.spec));
- const stale = join(input.state, 'provider-home', 'claude', 'skills', 'github');
- await mkdir(input.state, { mode: 0o700 });
- await mkdir(stale, { recursive: true, mode: 0o700 });
- await writeFile(join(stale, 'old.md'), 'stale');
- mockHosted();
- const supervise = vi.fn<typeof superviseSharedHost>(async () => {
- expect(await readFile(join(stale, 'SKILL.md'), 'utf8')).toBe('---\nname: github\n---\n');
- });
- await run(input, { supervise });
- expect(supervise).toHaveBeenCalledOnce();
- expect((await readdir(stale)).sort()).toEqual(['SKILL.md', 'reference']);
- expect(await readFile(join(stale, 'reference', 'gh.md'), 'utf8')).toBe('gh pr list\n');
- expect(await readdir(join(input.state, 'provider-home', 'claude', 'skills'))).toEqual(['github']);
-});
-
-it('rejects unsafe, oversized or unsupported connection skills', async () => {
- const { spec } = await fixture();
- const skill = (files: Record<string, string>, slug = 'github') => ({ slug, files });
- const valid = skill({ 'SKILL.md': 'x' });
- expect(hostedDeploymentSpecSchema.safeParse({ ...spec, skills: [valid] }).success).toBe(true);
- for (const skills of [
- [],
- [skill({ 'README.md': 'x' })],
- [skill({ 'SKILL.md': 'x', '../escape.md': 'x' })],
- [skill({ 'SKILL.md': 'x', '/abs.md': 'x' })],
- [skill({ 'SKILL.md': 'x', 'a/../b.md': 'x' })],
- [skill({ 'SKILL.md': 'x\0' })],
- [skill({ 'SKILL.md': 'x'.repeat(32 * 1024 + 1) })],
- [skill({ 'SKILL.md': 'x' }, '../github')],
- [valid, valid],
- [{ ...valid, extra: true }],
- ])
- expect(hostedDeploymentSpecSchema.safeParse({ ...spec, skills }).success).toBe(false);
- for (const kind of ['cursor', 'antigravity'] as const)
- expect(
- hostedDeploymentSpecSchema.safeParse({
- ...spec,
- provider: { ...spec.provider, kind },
- skills: [valid],
- }).success
- ).toBe(false);
-});
-
-it('maps each skills-capable provider to the directory its agent reads', () => {
- const env = { HOME: '/r/home', CLAUDE_CONFIG_DIR: '/r/claude', XDG_CONFIG_HOME: '/r/xdg' };
- expect(hostedSkillsDirectory('claude', env)).toBe('/r/claude/skills');
- expect(hostedSkillsDirectory('codex', { ...env, CODEX_HOME: '/r/codex' })).toBe(
- '/r/codex/skills'
- );
- expect(hostedSkillsDirectory('codex', env)).toBe('/r/home/.codex/skills');
- expect(hostedSkillsDirectory('opencode', env)).toBe('/r/xdg/opencode/skills');
- expect(() => hostedSkillsDirectory('opencode', { HOME: '/r/home' })).toThrow(/not configured/);
-});
-
-it('adds the worktree on a later revision after the first repository setup failed', async () => {
- const input = await fixture();
- const workspace = join(input.root, 'worktrees', 'agent-id', 'example', 'project');
- await mkdir(workspace, { recursive: true });
- input.spec.workspacePath = workspace;
- input.spec.provider.definition = { name: 'helper', content: 'helper definition' };
- input.spec.github = {
- credentialPath: join(input.root, 'run', 'agents', 'agent-id', 'github'),
- repository: 'example/project',
- refresh: true,
- mirrorPath: mirrorPath(input),
- };
- await writeFile(input.specPath, JSON.stringify(input.spec));
- const bin = join(input.root, 'bin');
- await mkdir(bin);
- await writeFile(
- join(bin, 'flock'),
- [
- '#!/usr/bin/env python3',
- 'import fcntl, os, sys',
- 'fd = os.open(sys.argv[1], os.O_RDWR | os.O_CREAT, 0o600)',
- 'os.set_inheritable(fd, True)',
- 'fcntl.flock(fd, fcntl.LOCK_EX)',
- 'os.execvp(sys.argv[2], sys.argv[2:])',
- '',
- ].join('\n'),
- { mode: 0o755 }
- );
- vi.stubEnv('PATH', `${bin}:${process.env.PATH}`);
- const upstream = join(input.root, 'upstream');
- await mkdir(join(input.state, 'home'), { recursive: true, mode: 0o700 });
- await writeFile(
- join(input.state, 'home', '.gitconfig'),
- `[url "file://${upstream}/"]\n\tinsteadOf = https://github.com/\n`
- );
- vi.stubGlobal(
- 'fetch',
- vi.fn(async (url: string | URL | Request) => {
- const path = String(url);
- if (path.endsWith('/hosted/provider-credential'))
- return new Response(
- JSON.stringify({
- status: 'connected',
- revision: 'test-revision',
- provider: 'claude',
- kind: 'api-key',
- credential: 'provider-secret-value',
- })
- );
- if (path.endsWith('/hosted/github-credential'))
- return new Response(
- JSON.stringify({
- token: 'renewed-github-secret',
- repository: 'example/project',
- expires_at: new Date(Date.now() + 3_600_000).toISOString(),
- })
- );
- return new Response('{}');
- })
- );
- const supervise = vi.fn<typeof superviseSharedHost>(async () => {});
-
- await expect(run(input, { supervise })).rejects.toThrow(
- 'Could not prepare the selected GitHub repository'
- );
- expect(await readdir(workspace)).toEqual([]);
-
- const git = (...args: string[]) =>
- promisify(execFile)('git', args, {
- env: {
- ...process.env,
- GIT_CONFIG_NOSYSTEM: '1',
- GIT_AUTHOR_NAME: 'Fixture',
- GIT_AUTHOR_EMAIL: '[email protected]',
- GIT_COMMITTER_NAME: 'Fixture',
- GIT_COMMITTER_EMAIL: '[email protected]',
- },
- });
- const remote = join(upstream, 'example', 'project.git');
- const seed = join(input.root, 'seed');
- await git('init', '--bare', '-b', 'main', remote);
- await git('init', '-b', 'main', seed);
- await writeFile(join(seed, 'README.md'), 'seed\n');
- await git('-C', seed, 'add', 'README.md');
- await git('-C', seed, 'commit', '-m', 'seed');
- await git('-C', seed, 'push', '-q', remote, 'main');
- input.spec.revision = 4;
- await writeFile(input.specPath, JSON.stringify(input.spec));
-
- await run(input, { supervise });
- expect(supervise).toHaveBeenCalledOnce();
- expect(await readFile(join(workspace, 'README.md'), 'utf8')).toBe('seed\n');
- expect(await readFile(join(workspace, '.claude', 'agents', 'helper.md'), 'utf8')).toBe(
- 'helper definition'
- );
-});
-
-it('redacts raw and encoded GitHub credentials from launcher failures', async () => {
- const input = await fixture();
- const token = 'github-secret:%value';
- await configureGitHub(input, token);
- await writeFile(input.specPath, JSON.stringify(input.spec));
- mockHosted();
- const encoded = encodeURIComponent(token);
- const basic = Buffer.from(`x-access-token:${token}`).toString('base64');
- const supervise: typeof superviseSharedHost = async () => {
- throw new Error(`github rejected ${token} encoded ${encoded} basic ${basic}`);
- };
- let message = '';
- try {
- await run(input, { supervise });
- } catch (error) {
- message = error instanceof Error ? error.message : String(error);
- }
-
- expect(message).toBe('github rejected [REDACTED] encoded [REDACTED] basic [REDACTED]');
-});
-
-it('redacts the mounted provider credential from launcher failures', async () => {
- const input = await fixture();
- mockHosted();
- const supervise: typeof superviseSharedHost = async () => {
- throw new Error('provider rejected provider-secret-value with switch-secret-value');
- };
- await expect(run(input, { supervise })).rejects.toThrow(
- 'provider rejected [REDACTED] with [REDACTED]'
- );
-});
-
-it('does not invalidate credentials when provider readiness is inconclusive', async () => {
- const input = await fixture();
- input.spec.provider.credential.refresh = true;
- await rm(input.providerCredential);
- await writeFile(input.spec.provider.binaryPath, '#!/bin/sh\nexit 0\n', { mode: 0o700 });
- await writeFile(input.specPath, JSON.stringify(input.spec));
- const request = vi.fn(
- async () =>
- new Response(
- JSON.stringify({
- status: 'connected',
- revision: 'test-revision',
- provider: 'claude',
- kind: 'api-key',
- credential: 'provider-secret-value',
- })
- )
- );
- vi.stubGlobal('fetch', request);
- const supervise = vi.fn();
- await expect(run(input, { supervise })).rejects.toThrow('Could not verify authentication');
- expect(request).toHaveBeenCalledOnce();
- expect(request).toHaveBeenCalledWith(
- expect.stringContaining('/hosted/provider-credential'),
- expect.anything()
- );
- expect(supervise).not.toHaveBeenCalled();
-});
-
-it('migrates a saved deployment from before the cutover and keeps its identity', async () => {
- const input = await fixture();
- const first = await prepareHostedDeployment(input.state, input.spec);
- expect(JSON.parse(await readFile(join(input.state, 'state-version.json'), 'utf8'))).toEqual({
- version: 1,
- });
- const planPath = join(input.state, 'hosted-deployment.json');
- const plan = JSON.parse(await readFile(planPath, 'utf8'));
- delete plan.spec.revision;
- delete plan.spec.workerCapabilityPath;
- plan.config.roomConnection.rooms = [];
- await writeFile(planPath, JSON.stringify(plan));
- await rm(join(input.state, 'state-version.json'));
- const second = await prepareHostedDeployment(input.state, input.spec);
- expect(second.config).toEqual(first.config);
- expect(JSON.parse(await readFile(planPath, 'utf8')).spec).toEqual(input.spec);
- expect(
- JSON.parse(await readFile(join(input.state, 'cutover', 'manifest.json'), 'utf8')).items
- ).toEqual([]);
-});
diff --git a/console/packages/agent-providers/src/host/hosted-bootstrap.ts b/console/packages/agent-providers/src/host/hosted-bootstrap.ts
deleted file mode 100644
index ef219d0ec..000000000
--- a/console/packages/agent-providers/src/host/hosted-bootstrap.ts
+++ /dev/null
@@ -1,686 +0,0 @@
-import { randomUUID } from 'node:crypto';
-import { constants } from 'node:fs';
-import {
- access,
- link,
- mkdir,
- open,
- readdir,
- readFile,
- realpath,
- rename,
- rm,
- stat,
- unlink,
-} from 'node:fs/promises';
-import { isAbsolute, join, relative, resolve, sep } from 'node:path';
-import { isDeepStrictEqual } from 'node:util';
-import { z } from 'zod';
-import { buildSharedHostConfig } from './build-shared-config';
-import { WorkerObsoleteError } from './exit-codes';
-import {
- ensureHostedRepository,
- githubLaunchEnvironment,
- githubRedactions,
- prepareGitHubCli,
- readGitHubCredential,
- renewGitHubCredential,
- validateGitHubCredential,
-} from './hosted-github';
-import { redactHostedText } from './hosted-log';
-import { runHostedPreflight } from './hosted-preflight';
-import { fetchHostedProvider, hostedRequest, materializeHostedProvider } from './hosted-provider';
-import {
- hostedSkillsDirectory,
- hostedSkillsSchema,
- installHostedSkills,
- supportsHostedSkills,
-} from './hosted-skills';
-import { checkProviderReadiness } from './provider-readiness';
-import { sharedConfigSchema, type SharedHostConfig } from './shared-config';
-import type { superviseSharedHost } from './supervisor';
-import { WATCH_FLAGS_FILE } from './watch-flags';
-import { writeWorkerCapability } from './worker-capability';
-
-const absolutePath = z
- .string()
- .min(1)
- .refine((value) => isAbsolute(value), 'must be an absolute path');
-const identifier = z.string().min(1).max(200);
-
-export const hostedDeploymentSpecSchema = z
- .strictObject({
- version: z.literal(2),
- /** The launch revision this deployment was issued for. */
- revision: z.number().int().positive(),
- session: z.strictObject({ sessionId: identifier, agentId: identifier }),
- provider: z.strictObject({
- kind: z.enum(['claude', 'codex', 'opencode', 'cursor', 'antigravity']),
- credential: z.strictObject({
- kind: z.enum(['api-key', 'setup-token', 'auth-json']),
- refresh: z.literal(true).optional(),
- path: absolutePath,
- }),
- binaryPath: absolutePath,
- model: z
- .strictObject({
- id: z.string().min(1),
- options: z.record(z.string(), z.string()).optional(),
- })
- .optional(),
- context: z.string(),
- definition: z
- .strictObject({
- name: z.string().regex(/^[a-z0-9][a-z0-9._-]{0,127}$/),
- content: z.string().min(1).max(65536),
- })
- .optional(),
- }),
- github: z
- .strictObject({
- credentialPath: absolutePath,
- repository: z
- .string()
- .regex(/^[A-Za-z0-9][A-Za-z0-9-]{0,38}\/(?!\.{1,2}$)[A-Za-z0-9_.-]{1,100}$/)
- .optional(),
- refresh: z.literal(true).optional(),
- /** The bare mirror the workspace is a worktree of, shared by every agent on the repository. */
- mirrorPath: absolutePath,
- })
- .optional(),
- workspacePath: absolutePath,
- /** Whether the watcher may start a session for an addressed room message. */
- watch: z.boolean(),
- runtimeMode: z.enum(['approval-required', 'auto-accept-edits', 'full-access']),
- switchCredentialsPath: absolutePath,
- workerCapabilityPath: absolutePath,
- skills: hostedSkillsSchema.optional(),
- })
- .refine((spec) => spec.skills === undefined || supportsHostedSkills(spec.provider.kind), {
- message: 'This provider has no skills directory to install connection skills into.',
- })
- .refine((spec) => !spec.github?.refresh || spec.github.repository !== undefined, {
- message: 'GitHub credential renewal requires a selected repository.',
- });
-export type HostedDeploymentSpec = z.infer<typeof hostedDeploymentSpecSchema>;
-
-const hostedDeploymentPlanSchema = z.strictObject({
- version: z.literal(1),
- spec: hostedDeploymentSpecSchema,
- config: sharedConfigSchema,
-});
-type HostedDeploymentPlan = z.infer<typeof hostedDeploymentPlanSchema>;
-
-const INHERITED_ENV = ['PATH', 'USER', 'SHELL', 'LANG', 'LC_ALL', 'TERM'] as const;
-const MAX_CREDENTIAL_BYTES = 16 * 1024;
-const PLAN_FILE = 'hosted-deployment.json';
-const CONFIG_FILE = 'config.json';
-
-function credentialVariable(provider: HostedDeploymentSpec['provider']): string {
- if (provider.kind === 'claude')
- return provider.credential.kind === 'api-key' ? 'ANTHROPIC_API_KEY' : 'CLAUDE_CODE_OAUTH_TOKEN';
- if (provider.kind === 'codex' && provider.credential.kind === 'api-key') return 'OPENAI_API_KEY';
- if (provider.kind === 'cursor') return 'CURSOR_API_KEY';
- return 'SWITCH_HOSTED_AUTH_JSON';
-}
-
-function isWithin(root: string, path: string): boolean {
- const child = relative(root, path);
- return child === '' || (child !== '..' && !child.startsWith(`..${sep}`) && !isAbsolute(child));
-}
-
-async function readJson(path: string, failure: string): Promise<unknown> {
- try {
- return JSON.parse(await readFile(path, 'utf8'));
- } catch {
- throw new Error(failure);
- }
-}
-
-async function resolveCredentialFile(
- path: string,
- stateRoot: string,
- workspaceRoot: string,
- label: string
-): Promise<string> {
- if (isWithin(stateRoot, resolve(path)) || isWithin(workspaceRoot, resolve(path)))
- throw new Error(
- `${label} credential file must be mounted outside the state and workspace directories.`
- );
- let resolved: string;
- try {
- resolved = await realpath(path);
- const details = await stat(resolved);
- if (!details.isFile() || details.size > MAX_CREDENTIAL_BYTES) throw new Error();
- } catch {
- throw new Error(`${label} credential file is missing or invalid.`);
- }
- if (isWithin(stateRoot, resolved) || isWithin(workspaceRoot, resolved))
- throw new Error(
- `${label} credential file must be mounted outside the state and workspace directories.`
- );
- return resolved;
-}
-
-async function readWorkerCapability(path: string): Promise<string> {
- try {
- const value = (await readFile(path, 'utf8')).trim();
- if (!/^[\x21-\x7e]{16,4096}$/.test(value)) throw new Error();
- return value;
- } catch {
- throw new Error('Worker capability file is missing or invalid.');
- }
-}
-
-function machineIdentity(): { instanceId: string; bootId: string } {
- const instanceId = process.env.SWITCH_HOST_INSTANCE_ID;
- const bootId = process.env.SWITCH_HOST_BOOT_ID;
- if (!instanceId || !bootId)
- throw new Error(
- 'A hosted worker requires SWITCH_HOST_INSTANCE_ID and SWITCH_HOST_BOOT_ID from its launcher.'
- );
- return { instanceId, bootId };
-}
-
-async function replaceJson(path: string, value: unknown): Promise<void> {
- const temporary = `${path}.${randomUUID()}.tmp`;
- const file = await open(temporary, 'wx', 0o600);
- try {
- await file.writeFile(JSON.stringify(value));
- await file.sync();
- } catch (error) {
- await file.close();
- await rm(temporary, { force: true });
- throw error;
- }
- await file.close();
- await rename(temporary, path);
-}
-
-async function readProviderCredential(path: string): Promise<string> {
- try {
- const value = (await readFile(path, 'utf8')).trim();
- if (!value || Buffer.byteLength(value) > MAX_CREDENTIAL_BYTES || /[\0\r\n]/.test(value))
- throw new Error();
- return value;
- } catch {
- throw new Error('Provider credential file is missing or invalid.');
- }
-}
-
-async function validateSwitchCredentials(
- path: string,
- agentId: string
-): Promise<{ token: string }> {
- let value: unknown;
- try {
- value = JSON.parse(await readFile(path, 'utf8'));
- } catch {
- throw new Error('Switch credential file is missing or invalid.');
- }
- if (!value || typeof value !== 'object' || !('env' in value)) {
- throw new Error('Switch credential file is missing or invalid.');
- }
- const env = value.env;
- if (
- !env ||
- typeof env !== 'object' ||
- !('SWITCH_API_ENDPOINT' in env) ||
- typeof env.SWITCH_API_ENDPOINT !== 'string' ||
- !env.SWITCH_API_ENDPOINT ||
- !('SWITCH_API_TOKEN' in env) ||
- typeof env.SWITCH_API_TOKEN !== 'string' ||
- !env.SWITCH_API_TOKEN ||
- !('SWITCH_AGENT_ID' in env) ||
- typeof env.SWITCH_AGENT_ID !== 'string' ||
- !env.SWITCH_AGENT_ID
- )
- throw new Error('Switch credential file is missing or invalid.');
- try {
- const endpoint = new URL(env.SWITCH_API_ENDPOINT);
- if (
- !['http:', 'https:'].includes(endpoint.protocol) ||
- endpoint.username ||
- endpoint.password ||
- endpoint.search ||
- endpoint.hash
- )
- throw new Error();
- } catch {
- throw new Error('Switch credential file contains an invalid endpoint.');
- }
- if (env.SWITCH_AGENT_ID !== agentId)
- throw new Error('Switch credential file belongs to a different agent.');
- return { token: env.SWITCH_API_TOKEN };
-}
-
-async function writeNewJson(path: string, value: unknown): Promise<boolean> {
- const temporary = `${path}.${randomUUID()}.tmp`;
- const file = await open(temporary, 'wx', 0o600);
- try {
- await file.writeFile(JSON.stringify(value));
- await file.sync();
- } finally {
- await file.close();
- }
- try {
- await link(temporary, path);
- return true;
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error;
- return false;
- } finally {
- await unlink(temporary);
- }
-}
-
-async function syncDirectory(path: string): Promise<void> {
- const directory = await open(path, 'r');
- try {
- await directory.sync();
- } finally {
- await directory.close();
- }
-}
-
-function sameValue(left: unknown, right: unknown): boolean {
- return isDeepStrictEqual(left, right);
-}
-
-/**
- * The worker's homes. A provider's native login lives where
- * `materializeHostedProvider` writes it, so the sessions read the login the
- * worker fetched.
- */
-function controlledEnvironment(
- root: string,
- provider: HostedDeploymentSpec['provider']['kind']
-): Record<string, string> {
- return {
- HOME: join(root, 'home'),
- CLAUDE_CONFIG_DIR: join(root, 'provider-home', 'claude'),
- XDG_CACHE_HOME: join(root, 'xdg', 'cache'),
- XDG_CONFIG_HOME: join(root, 'xdg', 'config'),
- XDG_DATA_HOME: join(root, 'xdg', 'data'),
- XDG_STATE_HOME: join(root, 'xdg', 'state'),
- TMPDIR: join(root, 'tmp'),
- ...(provider === 'codex' ? { CODEX_HOME: join(root, 'provider-home') } : {}),
- ...(provider === 'opencode' ? { XDG_DATA_HOME: join(root, 'provider-data') } : {}),
- ...(provider === 'antigravity' ? { GEMINI_HOME: join(root, 'provider-home') } : {}),
- };
-}
-
-async function createControlledDirectories(environment: Record<string, string>): Promise<void> {
- await Promise.all(
- Object.values(environment).map((path) => mkdir(path, { recursive: true, mode: 0o700 }))
- );
-}
-
-async function loadPlan(path: string): Promise<HostedDeploymentPlan> {
- try {
- return hostedDeploymentPlanSchema.parse(JSON.parse(await readFile(path, 'utf8')));
- } catch {
- throw new Error('Saved hosted deployment state is missing or invalid.');
- }
-}
-
-async function loadConfig(path: string): Promise<SharedHostConfig> {
- try {
- return sharedConfigSchema.parse(JSON.parse(await readFile(path, 'utf8')));
- } catch {
- throw new Error('Saved hosted launch configuration is invalid.');
- }
-}
-
-export async function readHostedDeploymentSpec(path: string): Promise<HostedDeploymentSpec> {
- const value = await readJson(path, 'Hosted deployment specification is missing or invalid.');
- const result = hostedDeploymentSpecSchema.safeParse(value);
- if (!result.success) {
- const issue = result.error.issues[0];
- const field = issue?.path.join('.') || 'root';
- throw new Error(
- `Hosted deployment specification is invalid at ${field}: ${issue?.message ?? 'invalid value'}.`
- );
- }
- return result.data;
-}
-
-export interface PreparedHostedDeployment {
- root: string;
- configPath: string;
- config: SharedHostConfig;
- providerEnvironment: NodeJS.ProcessEnv;
- logRedactions: string[];
-}
-
-/**
- * Write the deployment's agent definition into the workspace. A definition
- * already there must match unless the deployment moved to a newer revision,
- * which replaces it.
- */
-async function writeDefinition(spec: HostedDeploymentSpec, replace: boolean): Promise<void> {
- if (!spec.provider.definition) return;
- const directory = join(spec.workspacePath, '.claude', 'agents');
- await mkdir(directory, { recursive: true, mode: 0o700 });
- if (!isWithin(await realpath(spec.workspacePath), await realpath(directory)))
- throw new Error('Cloud agent definition directory must stay inside the workspace.');
- const path = join(directory, `${spec.provider.definition.name}.md`);
- if (replace) {
- const temporary = `${path}.${randomUUID()}.tmp`;
- const file = await open(temporary, 'wx', 0o600);
- try {
- await file.writeFile(spec.provider.definition.content);
- await file.sync();
- } finally {
- await file.close();
- }
- await rename(temporary, path);
- return;
- }
- try {
- const file = await open(path, 'wx', 0o600);
- try {
- await file.writeFile(spec.provider.definition.content);
- await file.sync();
- } finally {
- await file.close();
- }
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error;
- if (
- !isWithin(await realpath(spec.workspacePath), await realpath(path)) ||
- (await readFile(path, 'utf8')) !== spec.provider.definition.content
- )
- throw new Error('The saved cloud agent definition differs from the deployment.');
- }
-}
-
-export async function prepareHostedDeployment(
- stateDirectory: string,
- spec: HostedDeploymentSpec
-): Promise<PreparedHostedDeployment> {
- if (process.platform === 'win32')
- throw new Error('Hosted SDK bootstrap requires a POSIX execution environment.');
- if (!isAbsolute(stateDirectory))
- throw new Error('Hosted SDK state directory must be an absolute path.');
- if (resolve(stateDirectory) === resolve('/'))
- throw new Error('Hosted SDK state directory cannot be the filesystem root.');
- const machine = machineIdentity();
- try {
- const existing = await stat(stateDirectory);
- if (!existing.isDirectory()) throw new Error('not-directory');
- if (process.getuid && existing.uid !== process.getuid()) throw new Error('wrong-owner');
- if ((existing.mode & 0o077) !== 0) throw new Error('not-private');
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code !== 'ENOENT')
- throw new Error('Hosted SDK state directory must be a private directory owned by this user.');
- await mkdir(stateDirectory, { recursive: true, mode: 0o700 });
- }
- const root = await realpath(stateDirectory);
- let binaryPath: string;
- try {
- binaryPath = await realpath(spec.provider.binaryPath);
- if (!(await stat(binaryPath)).isFile()) throw new Error();
- await access(binaryPath, constants.X_OK);
- } catch {
- throw new Error('Claude executable is missing or is not executable.');
- }
- let workspaceRoot: string;
- try {
- workspaceRoot = await realpath(spec.workspacePath);
- if (!(await stat(workspaceRoot)).isDirectory()) throw new Error();
- } catch {
- throw new Error('Hosted workspace is missing or is not a directory.');
- }
- if (isWithin(root, workspaceRoot) || isWithin(workspaceRoot, root))
- throw new Error('Hosted state and workspace directories must not overlap.');
- const switchCredentialsPath = await resolveCredentialFile(
- spec.switchCredentialsPath,
- root,
- workspaceRoot,
- 'Switch'
- );
- const workerCapability = await readWorkerCapability(
- await resolveCredentialFile(spec.workerCapabilityPath, root, workspaceRoot, 'Worker capability')
- );
- const githubCredentialPath =
- spec.github && !spec.github.refresh
- ? await resolveCredentialFile(spec.github.credentialPath, root, workspaceRoot, 'GitHub')
- : undefined;
- const providerCredential = spec.provider.credential.refresh
- ? null
- : await readProviderCredential(
- await resolveCredentialFile(spec.provider.credential.path, root, workspaceRoot, 'Provider')
- );
- const switchCredentials = await validateSwitchCredentials(
- switchCredentialsPath,
- spec.session.agentId
- );
- const githubCredential = spec.github?.refresh
- ? await renewGitHubCredential(switchCredentialsPath, spec.github.repository)
- : githubCredentialPath
- ? await readGitHubCredential(githubCredentialPath)
- : undefined;
- if (githubCredential) await validateGitHubCredential(githubCredential, spec.github?.repository);
- const controlled = controlledEnvironment(root, spec.provider.kind);
- await createControlledDirectories(controlled);
- const environment = {
- ...controlled,
- ...(githubCredential ? githubLaunchEnvironment() : {}),
- ...(spec.github?.refresh
- ? {
- SWITCH_HOSTED_GITHUB_REFRESH_CREDENTIALS: switchCredentialsPath,
- SWITCH_HOSTED_GITHUB_REPOSITORY: spec.github.repository!,
- PATH: `${await prepareGitHubCli(root)}:${process.env.PATH ?? '/usr/bin:/bin'}`,
- }
- : {}),
- };
- const variable = credentialVariable(spec.provider);
- const candidate = hostedDeploymentPlanSchema.parse(
- JSON.parse(
- JSON.stringify({
- version: 1,
- spec,
- config: buildSharedHostConfig({
- session: {
- sessionId: spec.session.sessionId,
- agentId: spec.session.agentId,
- provider: spec.provider.kind,
- },
- launch: {
- cwd: workspaceRoot,
- runtimeMode: spec.runtimeMode,
- env: environment,
- model: spec.provider.model,
- },
- capabilities: { approvals: true, userInput: true },
- execution: {
- credentialsPath: switchCredentialsPath,
- inheritEnv: [
- ...INHERITED_ENV,
- variable,
- ...(githubCredential && !spec.github?.refresh ? ['GH_TOKEN'] : []),
- ],
- binaryPath,
- codexConfig: '',
- skill: '',
- context: spec.provider.context,
- agentDefinition: spec.provider.definition
- ? {
- name: spec.provider.definition.name,
- path: `.claude/agents/${spec.provider.definition.name}.md`,
- }
- : undefined,
- },
- ids: {
- hostId: randomUUID(),
- epoch: randomUUID(),
- connectionId: randomUUID(),
- },
- }),
- })
- )
- );
- if (await runHostedPreflight(root, candidate)) await writeDefinition(spec, true);
- const planPath = join(root, PLAN_FILE);
- let plan: HostedDeploymentPlan;
- try {
- plan = await loadPlan(planPath);
- } catch (error) {
- try {
- await stat(planPath);
- throw error;
- } catch (statError) {
- if ((statError as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
- }
- if (await writeNewJson(planPath, candidate)) {
- await syncDirectory(root);
- plan = candidate;
- } else {
- plan = await loadPlan(planPath);
- }
- }
- if (!plan.config.roomConnection)
- throw new Error('Saved hosted deployment state is missing its room connection.');
- const expectedConfig = structuredClone(candidate.config);
- expectedConfig.session.hostId = plan.config.session.hostId;
- expectedConfig.session.epoch = plan.config.session.epoch;
- expectedConfig.roomConnection!.connectionId = plan.config.roomConnection.connectionId;
- const configPath = join(root, CONFIG_FILE);
- if (spec.revision < plan.spec.revision)
- throw new Error(
- `Hosted deployment revision ${spec.revision} is older than the saved revision ${plan.spec.revision}.`
- );
- if (spec.revision > plan.spec.revision) {
- // The plan is the commit point: a crash before it is replaced repeats this revision.
- const revised = { version: 1 as const, spec, config: expectedConfig };
- // An empty workspace may still become a worktree, which needs it empty; the launch writes it.
- if ((await readdir(spec.workspacePath)).length > 0) await writeDefinition(spec, true);
- await replaceJson(configPath, revised.config);
- await replaceJson(planPath, revised);
- await syncDirectory(root);
- plan = revised;
- }
- if (!sameValue(plan.spec, spec))
- throw new Error(
- 'Hosted deployment specification differs from the saved state; review the saved assignment before starting a different deployment.'
- );
- if (!sameValue(plan.config, expectedConfig))
- throw new Error(
- 'Saved hosted deployment configuration does not match its deployment specification.'
- );
- try {
- const saved = await loadConfig(configPath);
- if (!sameValue(saved, plan.config))
- throw new Error('Saved hosted launch configuration differs from the deployment plan.');
- } catch (error) {
- try {
- await stat(configPath);
- throw error;
- } catch (statError) {
- if ((statError as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
- }
- if (await writeNewJson(configPath, plan.config)) await syncDirectory(root);
- else if (!sameValue(await loadConfig(configPath), plan.config))
- throw new Error('Saved hosted launch configuration differs from the deployment plan.');
- }
- const providerEnvironment: NodeJS.ProcessEnv = { ...environment };
- await replaceJson(join(root, WATCH_FLAGS_FILE), { enabled: true, spawn: spec.watch });
- await writeWorkerCapability(root, workerCapability);
- await syncDirectory(root);
- for (const key of INHERITED_ENV) {
- const value = process.env[key];
- if (value !== undefined && providerEnvironment[key] === undefined)
- providerEnvironment[key] = value;
- }
- if (providerCredential !== null) providerEnvironment[variable] = providerCredential;
- if (githubCredential && !spec.github?.refresh) providerEnvironment.GH_TOKEN = githubCredential;
- providerEnvironment.SWITCH_HOSTED_BOOTSTRAP = '1';
- providerEnvironment.SWITCH_HOST_INSTANCE_ID = machine.instanceId;
- providerEnvironment.SWITCH_HOST_BOOT_ID = machine.bootId;
- return {
- root,
- configPath,
- config: plan.config,
- providerEnvironment,
- logRedactions: [
- ...(providerCredential === null ? [] : [providerCredential]),
- switchCredentials.token,
- workerCapability,
- ...(githubCredential ? githubRedactions(githubCredential) : []),
- ],
- };
-}
-
-export interface HostedBootstrapDependencies {
- supervise: typeof superviseSharedHost;
-}
-
-export async function runHostedBootstrap(
- input: {
- stateDirectory: string;
- specPath: string;
- sharedDaemonEntrypoint: string;
- signal: AbortSignal;
- },
- dependencies: HostedBootstrapDependencies
-): Promise<void> {
- const spec = await readHostedDeploymentSpec(input.specPath);
- const prepared = await prepareHostedDeployment(input.stateDirectory, spec);
- try {
- const credential = await fetchHostedProvider(prepared.config);
- if (credential.status === 'connected') prepared.logRedactions.push(credential.credential);
- const env = Object.fromEntries(
- Object.entries(prepared.providerEnvironment).filter(
- (entry): entry is [string, string] => entry[1] !== undefined
- )
- );
- await materializeHostedProvider(prepared.root, env, credential, spec.provider.binaryPath);
- const readiness = await checkProviderReadiness({
- provider: spec.provider.kind,
- binaryPath: spec.provider.binaryPath,
- cwd: spec.workspacePath,
- env,
- });
- if (credential.status !== 'connected')
- throw new Error('Reconnect the provider before starting the worker.');
- if (readiness.status === 'unknown') throw new Error(readiness.message);
- await hostedRequest(prepared.config, '/provider-status', {
- authenticated: readiness.status === 'authenticated',
- revision: credential.revision,
- });
- if (readiness.status !== 'authenticated')
- throw new Error(
- 'The provider rejected the saved sign-in. Sign in again and reconnect the provider in Switch Console.'
- );
- Object.assign(prepared.providerEnvironment, env);
- if (spec.github?.refresh && spec.github.repository)
- await ensureHostedRepository({
- workspace: spec.workspacePath,
- mirror: spec.github.mirrorPath,
- repository: spec.github.repository,
- agentId: spec.session.agentId,
- env: prepared.providerEnvironment,
- });
- await writeDefinition(spec, false);
- if (spec.skills && supportsHostedSkills(spec.provider.kind))
- await installHostedSkills(
- hostedSkillsDirectory(spec.provider.kind, prepared.providerEnvironment),
- spec.skills
- );
- await dependencies.supervise({
- root: prepared.root,
- executable: process.execPath,
- args: [input.sharedDaemonEntrypoint, prepared.root, prepared.configPath, '--watch-worker'],
- env: prepared.providerEnvironment,
- signal: input.signal,
- build: input.sharedDaemonEntrypoint,
- links: null,
- logRedactions: prepared.logRedactions,
- });
- } catch (error) {
- if (error instanceof WorkerObsoleteError) throw error;
- const message = error instanceof Error ? error.message : String(error);
- throw new Error(redactHostedText(message, prepared.logRedactions));
- }
-}
diff --git a/console/packages/agent-providers/src/host/hosted-github.test.ts b/console/packages/agent-providers/src/host/hosted-github.test.ts
deleted file mode 100644
index 48297c9b5..000000000
--- a/console/packages/agent-providers/src/host/hosted-github.test.ts
+++ /dev/null
@@ -1,432 +0,0 @@
-import { execFile, spawn } from 'node:child_process';
-import { chmod, mkdir, mkdtemp, readFile, realpath, rm, writeFile } from 'node:fs/promises';
-import { tmpdir } from 'node:os';
-import { join, resolve } from 'node:path';
-import { promisify } from 'node:util';
-import { afterEach, describe, expect, it, vi } from 'vitest';
-import {
- ensureHostedRepository,
- githubLaunchEnvironment,
- githubRedactions,
- gitHubCredentialResponse,
- readGitHubCredential,
- renewGitHubCredential,
- validateGitHubCredential,
-} from './hosted-github';
-import { redactHostedText } from './hosted-log';
-
-const roots: string[] = [];
-const token = 'synthetic-github-credential';
-afterEach(async () => {
- vi.unstubAllGlobals();
- for (const path of roots.splice(0)) await rm(path, { recursive: true, force: true });
-});
-
-it('provides credentials only for the exact GitHub HTTPS host', () => {
- expect(gitHubCredentialResponse('get', 'protocol=https\nhost=github.com\n\n', token)).toBe(
- `username=x-access-token\npassword=${token}\n\n`
- );
- for (const input of [
- 'protocol=http\nhost=github.com\n',
- 'protocol=https\nhost=github.com.evil.invalid\n',
- 'protocol=https\nhost=github.com:443\n',
- 'protocol=https\nhost=other.invalid\nhost=github.com\n',
- 'protocol=https\nhost=github.com\r\n',
- 'url=https://github.com\n',
- ])
- expect(gitHubCredentialResponse('get', input, token)).toBe('');
- expect(gitHubCredentialResponse('get', 'protocol=https\nhost=github.com\n', undefined)).toBe('');
- expect(gitHubCredentialResponse('store', 'protocol=https\nhost=github.com\n', token)).toBe('');
- expect(gitHubCredentialResponse('erase', 'protocol=https\nhost=github.com\n', token)).toBe('');
-});
-
-it('renews only the assigned repository credential and hides failed response bodies', async () => {
- const root = await mkdtemp(join(tmpdir(), 'hosted-github-refresh-'));
- roots.push(root);
- const credentials = join(root, 'switch.json');
- await writeFile(
- credentials,
- JSON.stringify({
- env: {
- SWITCH_API_ENDPOINT: 'https://switch.example.com/api/agent',
- SWITCH_API_TOKEN: 'synthetic-switch-credential',
- },
- })
- );
- const request = vi.fn(
- async () =>
- new Response(
- JSON.stringify({
- token,
- repository: 'example/project',
- expires_at: new Date(Date.now() + 3_600_000).toISOString(),
- })
- )
- );
- vi.stubGlobal('fetch', request);
- expect(await renewGitHubCredential(credentials, 'example/project')).toBe(token);
- expect(request.mock.calls[0]).toEqual([
- 'https://switch.example.com/api/agent/hosted/github-credential',
- expect.objectContaining({ method: 'POST', redirect: 'error' }),
- ]);
- await expect(renewGitHubCredential(credentials, 'example/other')).rejects.toThrow(
- 'Could not renew'
- );
- request.mockImplementation(async () => new Response('remote-secret-body', { status: 403 }));
- await expect(renewGitHubCredential(credentials, 'example/project')).rejects.toThrow(
- 'Could not renew'
- );
-});
-
-it('keeps raw and common transport encodings out of redacted output', () => {
- const secrets = githubRedactions(token);
- for (const value of secrets) expect(redactHostedText(value, secrets)).toBe('[REDACTED]');
- expect(JSON.stringify(githubLaunchEnvironment())).not.toContain(token);
-});
-
-it.each([401, 403, 429, 500, 302])(
- 'rejects GitHub status %s without exposing response content',
- async (status) => {
- vi.stubGlobal('fetch', vi.fn().mockResolvedValue(new Response(token, { status })));
- await expect(validateGitHubCredential(token)).rejects.toThrow(/GitHub/);
- try {
- await validateGitHubCredential(token);
- } catch (error) {
- expect(String(error)).not.toContain(token);
- }
- }
-);
-
-it('uses only GitHub identity validation and refuses redirects', async () => {
- const request = vi.fn().mockResolvedValue(new Response('{}', { status: 200 }));
- vi.stubGlobal('fetch', request);
- await validateGitHubCredential(token);
- expect(request).toHaveBeenCalledWith(
- 'https://api.github.com/user',
- expect.objectContaining({
- redirect: 'error',
- headers: expect.objectContaining({ Authorization: `Bearer ${token}` }),
- signal: expect.any(AbortSignal),
- })
- );
-});
-
-it('sanitizes transport failures', async () => {
- vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new Error(token)));
- await expect(validateGitHubCredential(token)).rejects.toThrow('could not reach GitHub');
-});
-
-it('rejects invalid credentials before a request', async () => {
- const request = vi.fn();
- vi.stubGlobal('fetch', request);
- for (const invalid of ['', 'two words', 'one\ntwo', 'é', 'x'.repeat(16385)])
- await expect(validateGitHubCredential(invalid)).rejects.toThrow('credential is invalid');
- expect(request).not.toHaveBeenCalled();
-});
-
-it('reads a mounted token and reports a missing file without its path', async () => {
- const root = await mkdtemp(join(tmpdir(), 'github-token-'));
- roots.push(root);
- const path = join(root, 'credential');
- await writeFile(path, token + '\n', { mode: 0o600 });
- expect(await readGitHubCredential(path)).toBe(token);
- await expect(readGitHubCredential(join(root, token))).rejects.toThrow(
- 'GitHub credential file is missing or invalid.'
- );
-});
-
-it('authenticates real Git credential requests without writing credentials or consulting stored helpers', async () => {
- const root = await mkdtemp(join(tmpdir(), "hosted github's "));
- roots.push(root);
- const script = join(root, "credential helper's.mjs");
- const moduleUrl = new URL('./hosted-github.ts', import.meta.url).href;
- await writeFile(
- script,
- `import { runGitHubCredentialHelper } from ${JSON.stringify(moduleUrl)}; await runGitHubCredentialHelper(process.argv.at(-1));`
- );
- const marker = join(root, 'unexpected-helper');
- const config = join(root, '.gitconfig');
- await writeFile(config, `[credential]\n\thelper = "!touch '${marker}'"\n`);
- const env = {
- PATH: process.env.PATH,
- HOME: root,
- GIT_CONFIG_NOSYSTEM: '1',
- ...githubLaunchEnvironment(script),
- GH_TOKEN: token,
- };
- const git = (operation: string, input: string) =>
- new Promise<{ code: number; stdout: string; stderr: string }>((resolve) => {
- const child = execFile('git', ['credential', operation], { env }, (error, stdout, stderr) =>
- resolve({ code: error ? 1 : 0, stdout, stderr })
- );
- child.stdin!.end(input);
- });
- const success = await git('fill', 'protocol=https\nhost=github.com\n\n');
- expect(success.code, success.stderr).toBe(0);
- expect(success.stdout).toContain(`password=${token}`);
- const rejected = await git('fill', 'protocol=https\nhost=other.invalid\n\n');
- expect(rejected.code).toBe(1);
- expect(rejected.stdout + rejected.stderr).not.toContain(token);
- await git(
- 'approve',
- `protocol=https\nhost=github.com\nusername=x-access-token\npassword=${token}\n\n`
- );
- await git('reject', `protocol=https\nhost=github.com\n\n`);
- await expect(readFile(marker)).rejects.toMatchObject({ code: 'ENOENT' });
- expect(await readFile(config, 'utf8')).not.toContain(token);
- await expect(readFile(join(root, '.git-credentials'))).rejects.toMatchObject({ code: 'ENOENT' });
-});
-
-it('validates installation tokens against the selected repository instead of a user identity', async () => {
- const request = vi.fn().mockResolvedValue(new Response('{}', { status: 200 }));
- vi.stubGlobal('fetch', request);
- await validateGitHubCredential(token, 'example/project');
- expect(request).toHaveBeenCalledWith(
- 'https://api.github.com/repos/example/project',
- expect.objectContaining({ redirect: 'error' })
- );
-});
-
-it.each([
- '../project',
- 'example/..',
- 'example/project?token=secret',
- 'example/project/extra',
- 'https://example.com/project',
-])('rejects unsafe repository selection %s', async (repository) => {
- const request = vi.fn();
- vi.stubGlobal('fetch', request);
- await expect(validateGitHubCredential(token, repository)).rejects.toThrow('repository');
- expect(request).not.toHaveBeenCalled();
-});
-
-it.each([409, 503, 422])('retries a token renewal only for retryable status %s', async (status) => {
- const root = await mkdtemp(join(tmpdir(), 'hosted-github-retry-'));
- roots.push(root);
- const path = join(root, 'switch.json');
- await writeFile(
- path,
- JSON.stringify({
- env: {
- SWITCH_API_ENDPOINT: 'https://switch.example.com/api/agent',
- SWITCH_API_TOKEN: 'synthetic-switch-credential',
- },
- })
- );
- const request = vi
- .fn()
- .mockResolvedValueOnce(new Response('unavailable', { status }))
- .mockResolvedValueOnce(
- new Response(
- JSON.stringify({
- token,
- repository: 'example/project',
- expires_at: new Date(Date.now() + 3_600_000).toISOString(),
- })
- )
- );
- vi.stubGlobal('fetch', request);
- if (status === 422) {
- await expect(renewGitHubCredential(path, 'example/project')).rejects.toThrow('Could not renew');
- expect(request).toHaveBeenCalledTimes(1);
- } else {
- expect(await renewGitHubCredential(path, 'example/project')).toBe(token);
- expect(request).toHaveBeenCalledTimes(2);
- }
-});
-
-describe('ensureHostedRepository', () => {
- const exec = promisify(execFile);
- const AGENT = '00000000-0000-4000-8000-000000000001';
- const OTHER = '00000000-0000-4000-8000-000000000002';
-
- /** util-linux `flock <file> <command...>`, which macOS lacks. */
- const FLOCK_SHIM = [
- '#!/usr/bin/env python3',
- 'import fcntl, os, sys',
- 'fd = os.open(sys.argv[1], os.O_RDWR | os.O_CREAT, 0o600)',
- 'os.set_inheritable(fd, True)',
- 'fcntl.flock(fd, fcntl.LOCK_EX)',
- 'os.execvp(sys.argv[2], sys.argv[2:])',
- '',
- ].join('\n');
-
- async function repositories() {
- const root = await realpath(await mkdtemp(join(tmpdir(), 'hosted-repository-')));
- roots.push(root);
- const bin = join(root, 'bin');
- await mkdir(bin);
- await writeFile(join(bin, 'flock'), FLOCK_SHIM);
- await chmod(join(bin, 'flock'), 0o755);
- const upstream = join(root, 'upstream');
- const env: NodeJS.ProcessEnv = {
- PATH: `${bin}:${process.env.PATH}`,
- HOME: root,
- GIT_CONFIG_NOSYSTEM: '1',
- GIT_TERMINAL_PROMPT: '0',
- GIT_AUTHOR_NAME: 'Fixture',
- GIT_AUTHOR_EMAIL: '[email protected]',
- GIT_COMMITTER_NAME: 'Fixture',
- GIT_COMMITTER_EMAIL: '[email protected]',
- GIT_CONFIG_COUNT: '1',
- GIT_CONFIG_KEY_0: `url.file://${upstream}/.insteadOf`,
- GIT_CONFIG_VALUE_0: 'https://github.com/',
- };
- const git = async (...args: string[]) => (await exec('git', args, { env })).stdout.trim();
- const seed = join(root, 'seed');
- await git('init', '--bare', '-b', 'main', join(upstream, 'example', 'project.git'));
- await git('init', '--bare', '-b', 'main', join(upstream, 'example', 'other.git'));
- await git('init', '-b', 'main', seed);
- const commit = async (message: string) => {
- await writeFile(join(seed, 'README.md'), `${message}\n`);
- await git('-C', seed, 'add', 'README.md');
- await git('-C', seed, 'commit', '-m', message);
- await git('-C', seed, 'push', '-q', 'https://github.com/example/project.git', 'main');
- return git('-C', seed, 'rev-parse', 'HEAD');
- };
- const mirror = join(root, 'repos', 'example', 'project.git');
- const workspace = (agentId: string) => join(root, 'worktrees', agentId, 'example', 'project');
- const ensure = (agentId: string, repository = 'example/project') =>
- ensureHostedRepository({
- workspace: workspace(agentId),
- mirror,
- repository,
- agentId,
- env,
- });
- return { root, env, git, commit, mirror, workspace, ensure };
- }
-
- it('creates the mirror and a worktree on the agent branch at the default branch', async () => {
- const { git, commit, mirror, workspace, ensure } = await repositories();
- const head = await commit('first');
- await mkdir(workspace(AGENT), { recursive: true });
- await ensure(AGENT);
- expect(await git('--git-dir', mirror, 'rev-parse', '--is-bare-repository')).toBe('true');
- expect(await git('--git-dir', mirror, 'config', 'remote.origin.url')).toBe(
- 'https://github.com/example/project.git'
- );
- expect(await git('--git-dir', mirror, 'config', 'remote.origin.fetch')).toBe(
- '+refs/heads/*:refs/remotes/origin/*'
- );
- expect(await git('-C', workspace(AGENT), 'rev-parse', '--abbrev-ref', 'HEAD')).toBe(
- `switch/${AGENT}`
- );
- expect(await git('-C', workspace(AGENT), 'rev-parse', 'HEAD')).toBe(head);
- expect(await readFile(join(workspace(AGENT), 'README.md'), 'utf8')).toBe('first\n');
- });
-
- it('gives a second agent its own worktree over the same mirror', async () => {
- const { git, commit, mirror, workspace, ensure } = await repositories();
- const head = await commit('first');
- await ensure(AGENT);
- await ensure(OTHER);
- for (const agentId of [AGENT, OTHER]) {
- const common = await git('-C', workspace(agentId), 'rev-parse', '--git-common-dir');
- expect(await realpath(resolve(workspace(agentId), common))).toBe(mirror);
- expect(await git('-C', workspace(agentId), 'rev-parse', 'HEAD')).toBe(head);
- }
- expect(await git('-C', workspace(OTHER), 'rev-parse', '--abbrev-ref', 'HEAD')).toBe(
- `switch/${OTHER}`
- );
- const listed = await git('--git-dir', mirror, 'worktree', 'list', '--porcelain');
- expect(listed).toContain(`worktree ${workspace(AGENT)}`);
- expect(listed).toContain(`worktree ${workspace(OTHER)}`);
- });
-
- it('accepts its existing worktree on a rerun and still fetches', async () => {
- const { git, commit, mirror, workspace, ensure } = await repositories();
- const first = await commit('first');
- await ensure(AGENT);
- await writeFile(join(workspace(AGENT), 'work.txt'), 'in progress\n');
- const second = await commit('second');
- await ensure(AGENT);
- expect(await git('--git-dir', mirror, 'rev-parse', 'refs/remotes/origin/main')).toBe(second);
- expect(await git('-C', workspace(AGENT), 'rev-parse', 'HEAD')).toBe(first);
- expect(await readFile(join(workspace(AGENT), 'work.txt'), 'utf8')).toBe('in progress\n');
- });
-
- it('refuses a non-empty workspace that is not a worktree of the mirror', async () => {
- const { commit, mirror, workspace, ensure, git } = await repositories();
- await commit('first');
- await mkdir(workspace(AGENT), { recursive: true });
- await writeFile(join(workspace(AGENT), 'stray.txt'), 'kept\n');
- await expect(ensure(AGENT)).rejects.toThrow(
- 'Could not prepare the selected GitHub repository (the workspace holds files that are not its worktree).'
- );
- expect(await readFile(join(workspace(AGENT), 'stray.txt'), 'utf8')).toBe('kept\n');
- expect(await git('--git-dir', mirror, 'worktree', 'list', '--porcelain')).not.toContain(
- workspace(AGENT)
- );
- });
-
- it('shares the mirror with an agent that names the repository in another case', async () => {
- const { git, commit, mirror, workspace, ensure } = await repositories();
- const head = await commit('first');
- await ensure(AGENT);
- await ensure(OTHER, 'Example/Project');
- const common = await git('-C', workspace(OTHER), 'rev-parse', '--git-common-dir');
- expect(await realpath(resolve(workspace(OTHER), common))).toBe(mirror);
- expect(await git('-C', workspace(OTHER), 'rev-parse', 'HEAD')).toBe(head);
- });
-
- it('refuses a mirror whose origin is a different repository', async () => {
- const { commit, mirror, workspace, ensure, git } = await repositories();
- await commit('first');
- await git('init', '--bare', mirror);
- await git(
- '--git-dir',
- mirror,
- 'remote',
- 'add',
- 'origin',
- 'https://github.com/example/other.git'
- );
- await expect(ensure(AGENT)).rejects.toThrow(
- 'Could not prepare the selected GitHub repository (the mirror belongs to a different repository).'
- );
- await expect(readFile(join(workspace(AGENT), 'README.md'))).rejects.toMatchObject({
- code: 'ENOENT',
- });
- });
-
- it('waits for the mirror lock before touching the mirror', async () => {
- const { root, commit, mirror, ensure, git } = await repositories();
- await commit('first');
- await ensure(AGENT);
- const second = await commit('second');
- const holder = spawn(
- 'python3',
- [
- '-c',
- 'import fcntl, os, sys\n' +
- 'fd = os.open(sys.argv[1], os.O_RDWR | os.O_CREAT, 0o600)\n' +
- 'fcntl.flock(fd, fcntl.LOCK_EX)\n' +
- 'print("locked", flush=True)\n' +
- 'sys.stdin.read()\n',
- `${mirror}.lock`,
- ],
- { cwd: root, stdio: ['pipe', 'pipe', 'inherit'] }
- );
- try {
- await new Promise<void>((resolve, reject) => {
- holder.once('error', reject);
- holder.stdout!.once('data', () => resolve());
- });
- let settled = false;
- const pending = ensure(AGENT).finally(() => {
- settled = true;
- });
- await new Promise((resolve) => setTimeout(resolve, 750));
- expect(settled).toBe(false);
- expect(await git('--git-dir', mirror, 'rev-parse', 'refs/remotes/origin/main')).not.toBe(
- second
- );
- holder.stdin!.end();
- await pending;
- expect(await git('--git-dir', mirror, 'rev-parse', 'refs/remotes/origin/main')).toBe(second);
- } finally {
- holder.kill();
- }
- }, 15_000);
-});
diff --git a/console/packages/agent-providers/src/host/hosted-github.ts b/console/packages/agent-providers/src/host/hosted-github.ts
deleted file mode 100644
index 41aa19b50..000000000
--- a/console/packages/agent-providers/src/host/hosted-github.ts
+++ /dev/null
@@ -1,322 +0,0 @@
-import { execFile, spawn } from 'node:child_process';
-import { mkdir, open, readFile, readdir, realpath } from 'node:fs/promises';
-import { dirname, join, resolve } from 'node:path';
-import { setTimeout as delay } from 'node:timers/promises';
-import { fileURLToPath } from 'node:url';
-import { promisify } from 'node:util';
-import { z } from 'zod';
-
-const MAX_TOKEN_BYTES = 16 * 1024;
-
-function validToken(token: string): boolean {
- return token.length > 0 && token.length <= MAX_TOKEN_BYTES && /^[\x21-\x7e]+$/.test(token);
-}
-
-export async function readGitHubCredential(path: string): Promise<string> {
- try {
- const token = (await readFile(path, 'utf8')).trim();
- if (!validToken(token)) throw new Error();
- return token;
- } catch {
- throw new Error('GitHub credential file is missing or invalid.');
- }
-}
-
-export async function validateGitHubCredential(token: string, repository?: string): Promise<void> {
- if (!validToken(token)) throw new Error('GitHub credential is invalid.');
- if (
- repository !== undefined &&
- !/^[A-Za-z0-9][A-Za-z0-9-]{0,38}\/(?!\.{1,2}$)[A-Za-z0-9_.-]{1,100}$/.test(repository)
- )
- throw new Error('GitHub repository must be an owner/repository name.');
- let response: Response;
- try {
- response = await fetch(
- repository ? `https://api.github.com/repos/${repository}` : 'https://api.github.com/user',
- {
- headers: {
- Authorization: `Bearer ${token}`,
- Accept: 'application/vnd.github+json',
- 'X-GitHub-Api-Version': '2022-11-28',
- },
- redirect: 'error',
- signal: AbortSignal.timeout(10_000),
- }
- );
- } catch {
- throw new Error('GitHub credential validation could not reach GitHub; retry when connected.');
- }
- // Never include the remote body or a transport exception in diagnostics.
- try {
- await response.body?.cancel();
- } catch {
- throw new Error('GitHub credential validation failed while closing the response.');
- }
- if (response.status === 401)
- throw new Error('GitHub rejected the credential; replace the expired or revoked token.');
- if (response.status === 403)
- throw new Error(
- 'GitHub denied the credential check; check token permissions, organization policy or rate limits.'
- );
- if (response.status !== 200)
- throw new Error(
- 'GitHub credential validation failed; retry or review the supplied personal token.'
- );
-}
-
-function shellQuote(value: string): string {
- return `'${value.replaceAll("'", "'\\''")}'`;
-}
-
-/** Non-secret command configuration; only GH_TOKEN is supplied separately at launch. */
-export function githubLaunchEnvironment(
- entrypoint = fileURLToPath(new URL('./hosted-bootstrap.mjs', import.meta.url))
-): Record<string, string> {
- return {
- GH_HOST: 'github.com',
- GH_PROMPT_DISABLED: '1',
- GIT_TERMINAL_PROMPT: '0',
- GIT_CONFIG_COUNT: '3',
- // Clear inherited credential stores before selecting this non-persistent helper.
- GIT_CONFIG_KEY_0: 'credential.helper',
- GIT_CONFIG_VALUE_0: '',
- GIT_CONFIG_KEY_1: 'credential.https://github.com.helper',
- GIT_CONFIG_VALUE_1: `!${shellQuote(process.execPath)} ${shellQuote(entrypoint)} --git-credential`,
- GIT_CONFIG_KEY_2: 'core.askPass',
- GIT_CONFIG_VALUE_2: '',
- };
-}
-
-export function githubRedactions(token: string): string[] {
- return [
- token,
- encodeURIComponent(token),
- Buffer.from(`x-access-token:${token}`).toString('base64'),
- ];
-}
-
-/** Git credential protocol: no cache/store, no browser, only HTTPS github.com. */
-export function gitHubCredentialResponse(operation: string, input: string, token?: string): string {
- if (operation !== 'get') return '';
- if (!token || !validToken(token) || Buffer.byteLength(input) > 64 * 1024) return '';
- const fields = new Map<string, string>();
- for (const line of input.split('\n')) {
- if (line === '') break;
- const at = line.indexOf('=');
- if (at <= 0 || line.includes('\r') || line.includes('\0')) return '';
- const key = line.slice(0, at);
- if (fields.has(key)) return '';
- fields.set(key, line.slice(at + 1));
- }
- if (fields.get('protocol') !== 'https' || fields.get('host') !== 'github.com') return '';
- return `username=x-access-token\npassword=${token}\n\n`;
-}
-
-export async function runGitHubCredentialHelper(operation: string | undefined): Promise<void> {
- if (!operation || !['get', 'store', 'erase'].includes(operation))
- throw new Error('Unsupported Git credential operation.');
- let input = '';
- for await (const chunk of process.stdin) {
- input += String(chunk);
- if (Buffer.byteLength(input) > 64 * 1024)
- throw new Error('Git credential request is too large.');
- }
- const eligible = gitHubCredentialResponse(operation, input, 'validation-only');
- if (!eligible) return;
- process.stdout.write(gitHubCredentialResponse(operation, input, await currentGitHubToken()));
-}
-
-export async function currentGitHubToken(): Promise<string | undefined> {
- const credentialsPath = process.env.SWITCH_HOSTED_GITHUB_REFRESH_CREDENTIALS;
- if (!credentialsPath) return process.env.GH_TOKEN;
- return renewGitHubCredential(credentialsPath, process.env.SWITCH_HOSTED_GITHUB_REPOSITORY);
-}
-
-export async function renewGitHubCredential(
- credentialsPath: string,
- repository: string | undefined
-): Promise<string> {
- try {
- const { env } = JSON.parse(await readFile(credentialsPath, 'utf8'));
- const endpoint = new URL(env.SWITCH_API_ENDPOINT);
- if (
- endpoint.protocol !== 'https:' ||
- endpoint.username ||
- endpoint.password ||
- endpoint.search ||
- endpoint.hash ||
- !validToken(env.SWITCH_API_TOKEN)
- )
- throw new Error();
- const request = () =>
- fetch(endpoint.href.replace(/\/$/, '') + '/hosted/github-credential', {
- method: 'POST',
- headers: { Authorization: `Bearer ${env.SWITCH_API_TOKEN}` },
- redirect: 'error',
- signal: AbortSignal.timeout(180_000),
- });
- let response = await request();
- if (response.status === 409 || response.status === 503) {
- await response.body?.cancel();
- await delay(1_000);
- response = await request();
- }
- if (!response.ok) {
- await response.body?.cancel();
- throw new Error();
- }
- const credential = z
- .object({ token: z.string(), repository: z.string(), expires_at: z.string() })
- .parse(await response.json());
- if (
- !repository ||
- !validToken(credential.token) ||
- credential.repository !== repository ||
- Date.parse(credential.expires_at) < Date.now() + 60_000 ||
- !Number.isFinite(Date.parse(credential.expires_at))
- )
- throw new Error();
- return credential.token;
- } catch {
- throw new Error(
- 'Could not renew cloud repository access. Check the owner’s GitHub connection.'
- );
- }
-}
-
-export async function prepareGitHubCli(root: string): Promise<string> {
- const directory = join(root, 'bin');
- await mkdir(directory, { recursive: true, mode: 0o700 });
- const path = join(directory, 'gh');
- const entrypoint = fileURLToPath(new URL('./hosted-bootstrap.mjs', import.meta.url));
- const source = `#!/bin/sh\nexec ${shellQuote(process.execPath)} ${shellQuote(entrypoint)} --github-cli "$@"\n`;
- try {
- const file = await open(path, 'wx', 0o700);
- try {
- await file.writeFile(source);
- await file.sync();
- } finally {
- await file.close();
- }
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error;
- if ((await readFile(path, 'utf8')) !== source)
- throw new Error('Hosted GitHub CLI wrapper differs from the deployment.');
- }
- return directory;
-}
-
-export async function runGitHubCli(args: string[]): Promise<void> {
- const token = await currentGitHubToken();
- if (!token) throw new Error('Cloud repository credential is missing.');
- const child = spawn('/usr/local/bin/gh', args, {
- stdio: 'inherit',
- env: { ...process.env, GH_TOKEN: token },
- });
- await new Promise<void>((resolve, reject) => {
- child.once('error', () => reject(new Error('GitHub CLI could not start.')));
- child.once('exit', (code) => {
- process.exitCode = code ?? 1;
- resolve();
- });
- });
-}
-
-class RepositoryStepError extends Error {}
-
-/**
- * Makes `workspace` a worktree on `switch/<agentId>` over the bare mirror every
- * agent on the machine shares for `repository`. Every git command that changes
- * the mirror holds `<mirror>.lock`, so concurrent agents take turns.
- */
-export async function ensureHostedRepository(input: {
- workspace: string;
- mirror: string;
- repository: string;
- agentId: string;
- env: NodeJS.ProcessEnv;
-}): Promise<void> {
- const { workspace, mirror, env } = input;
- const url = `https://github.com/${input.repository}.git`;
- const lock = `${mirror}.lock`;
- const exec = promisify(execFile);
- const locked = async (step: string, args: string[]) => {
- try {
- await exec('flock', [lock, 'git', ...args], {
- env,
- timeout: 120_000,
- maxBuffer: 1024 * 1024,
- });
- } catch {
- throw new RepositoryStepError(step);
- }
- };
- const probe = async (args: string[]) => {
- try {
- return (await exec('git', args, { env, timeout: 10_000 })).stdout.trim();
- } catch {
- return null;
- }
- };
- const origin = () => probe(['--git-dir', mirror, 'config', '--get', 'remote.origin.url']);
- try {
- try {
- await mkdir(dirname(mirror), { recursive: true, mode: 0o700 });
- } catch {
- throw new RepositoryStepError('create the mirror directory');
- }
- const bare = await probe(['--git-dir', mirror, 'rev-parse', '--is-bare-repository']);
- if (bare === null) await locked('initialize the mirror', ['init', '--bare', '--', mirror]);
- else if (bare !== 'true') throw new RepositoryStepError('the mirror is not a bare repository');
- if ((await origin()) === null) {
- try {
- await locked('add the mirror origin', ['-C', mirror, 'remote', 'add', 'origin', url]);
- } catch (error) {
- if ((await origin()) === null) throw error;
- }
- }
- if ((await origin())?.toLowerCase() !== url.toLowerCase())
- throw new RepositoryStepError('the mirror belongs to a different repository');
- await locked('fetch the repository', ['-C', mirror, 'fetch', '--prune', 'origin']);
- await locked('resolve the default branch', [
- '-C',
- mirror,
- 'remote',
- 'set-head',
- 'origin',
- '--auto',
- ]);
- const common = await probe(['-C', workspace, 'rev-parse', '--git-common-dir']);
- if (
- common !== null &&
- (await realpath(resolve(workspace, common)).catch(() => null)) === (await realpath(mirror))
- )
- return;
- let entries: string[];
- try {
- entries = await readdir(workspace);
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code !== 'ENOENT')
- throw new RepositoryStepError('read the workspace');
- entries = [];
- }
- if (entries.length > 0)
- throw new RepositoryStepError('the workspace holds files that are not its worktree');
- await locked('add the worktree', [
- '-C',
- mirror,
- 'worktree',
- 'add',
- '-B',
- `switch/${input.agentId}`,
- '--',
- workspace,
- 'origin/HEAD',
- ]);
- } catch (error) {
- const step = error instanceof RepositoryStepError ? error.message : 'inspect the repository';
- throw new Error(
- `Could not prepare the selected GitHub repository (${step}). Check repository access and the saved workspace.`
- );
- }
-}
diff --git a/console/packages/agent-providers/src/host/hosted-log.test.ts b/console/packages/agent-providers/src/host/hosted-log.test.ts
deleted file mode 100644
index e9d0cffce..000000000
--- a/console/packages/agent-providers/src/host/hosted-log.test.ts
+++ /dev/null
@@ -1,78 +0,0 @@
-import type { FileHandle } from 'node:fs/promises';
-import { Readable } from 'node:stream';
-import { describe, expect, it } from 'vitest';
-import { HostedLogRedactor, pipeRedactedHostedLogs, redactHostedText } from './hosted-log';
-
-function redactChunks(chunks: Buffer[], values: string[]): Buffer {
- const redactor = new HostedLogRedactor(values.map((value) => Buffer.from(value)));
- return Buffer.concat([...chunks.map((chunk) => redactor.push(chunk)), redactor.finish()]);
-}
-
-describe('hosted log redaction', () => {
- it('redacts a repeated secret when the final occurrence crosses the output boundary', () => {
- const secret = 'provider-secret';
- const output = redactChunks(
- [Buffer.from(`${secret}--provider-se`), Buffer.from('cret')],
- [secret]
- );
-
- expect(output.toString()).toBe('[REDACTED]--[REDACTED]');
- expect(output.includes(Buffer.from(secret))).toBe(false);
- });
-
- it('prefers the longest match when one secret is a prefix of another', () => {
- const expected = 'before [REDACTED] after';
-
- expect(redactHostedText('before abcdef after', ['abc', 'abcdef'])).toBe(expected);
- expect(
- redactChunks(
- [Buffer.from('before abc'), Buffer.from('def after')],
- ['abc', 'abcdef']
- ).toString()
- ).toBe(expected);
- });
-
- it('redacts a UTF-8 secret split at every byte boundary', () => {
- const secret = 'tøk🔐en';
- const input = Buffer.from(`before ${secret} after`);
- const chunks = Array.from(input, (byte) => Buffer.from([byte]));
-
- const output = redactChunks(chunks, [secret]);
-
- expect(output.toString()).toBe('before [REDACTED] after');
- expect(output.includes(Buffer.from(secret))).toBe(false);
- });
-
- it('keeps pending raw data bounded by the longest secret', () => {
- const secret = 's'.repeat(64);
- const input = Buffer.alloc(1024 * 1024, 'a');
- const redactor = new HostedLogRedactor([Buffer.from(secret)]);
-
- const emitted = redactor.push(input);
- const tail = redactor.finish();
-
- expect(emitted.length).toBe(input.length - Buffer.byteLength(secret) + 1);
- expect(tail.length).toBe(Buffer.byteLength(secret) - 1);
- expect(Buffer.concat([emitted, tail]).equals(input)).toBe(true);
- });
-
- it('writes only redacted bytes to the diagnostic sink', async () => {
- const writes: Buffer[] = [];
- const file = {
- write: async (value: Uint8Array) => {
- writes.push(Buffer.from(value));
- },
- } as unknown as FileHandle;
- const secret = 'switch-secret-value';
- const stream = Readable.from([
- Buffer.from('failure: switch-se'),
- Buffer.from('cret-value; retry switch-secret-value'),
- ]);
-
- await pipeRedactedHostedLogs([stream], file, [secret]);
-
- const persisted = Buffer.concat(writes);
- expect(persisted.toString()).toBe('failure: [REDACTED]; retry [REDACTED]');
- expect(persisted.includes(Buffer.from(secret))).toBe(false);
- });
-});
diff --git a/console/packages/agent-providers/src/host/hosted-log.ts b/console/packages/agent-providers/src/host/hosted-log.ts
deleted file mode 100644
index fd73a8452..000000000
--- a/console/packages/agent-providers/src/host/hosted-log.ts
+++ /dev/null
@@ -1,99 +0,0 @@
-import type { FileHandle } from 'node:fs/promises';
-import type { Readable } from 'node:stream';
-
-const REDACTED = Buffer.from('[REDACTED]');
-
-function secretBuffers(values: readonly string[]): Buffer[] {
- return [...new Set(values.filter(Boolean))]
- .map((value) => Buffer.from(value))
- .sort((left, right) => right.length - left.length);
-}
-
-function matchAt(input: Buffer, offset: number, secrets: readonly Buffer[]): Buffer | undefined {
- return secrets.find(
- (secret) =>
- offset + secret.length <= input.length &&
- input.subarray(offset, offset + secret.length).equals(secret)
- );
-}
-
-function redactPrefix(
- input: Buffer,
- scanLimit: number,
- secrets: readonly Buffer[]
-): { output: Buffer; consumed: number } {
- const parts: Buffer[] = [];
- let literalStart = 0;
- let cursor = 0;
- while (cursor < scanLimit) {
- const secret = matchAt(input, cursor, secrets);
- if (!secret) {
- cursor += 1;
- continue;
- }
- if (literalStart < cursor) parts.push(input.subarray(literalStart, cursor));
- parts.push(REDACTED);
- cursor += secret.length;
- literalStart = cursor;
- }
- if (!parts.length) return { output: input.subarray(0, cursor), consumed: cursor };
- if (literalStart < cursor) parts.push(input.subarray(literalStart, cursor));
- return { output: Buffer.concat(parts), consumed: cursor };
-}
-
-/** Exact-value redaction that prefers the longest value when secrets overlap. */
-export function redactHostedText(text: string, values: string[]): string {
- const secrets = secretBuffers(values);
- if (!secrets.length) return text;
- return redactPrefix(Buffer.from(text), Buffer.byteLength(text), secrets).output.toString();
-}
-
-/**
- * Streaming exact-value redaction. It retains at most maxSecretLength - 1 raw bytes,
- * which is enough to decide whether a secret continues in the next chunk.
- */
-export class HostedLogRedactor {
- private pending = Buffer.alloc(0);
- private readonly secrets: Buffer[];
- private readonly retainedBytes: number;
-
- constructor(secrets: Buffer[]) {
- if (!secrets.length) throw new Error('Hosted log redaction requires at least one secret.');
- this.secrets = [...secrets].sort((left, right) => right.length - left.length);
- this.retainedBytes = Math.max(0, ...this.secrets.map((secret) => secret.length - 1));
- }
-
- push(chunk: Buffer): Buffer {
- const combined = Buffer.concat([this.pending, chunk]);
- const scanLimit = Math.max(0, combined.length - this.retainedBytes);
- const { output, consumed } = redactPrefix(combined, scanLimit, this.secrets);
- this.pending = Buffer.from(combined.subarray(consumed));
- return output;
- }
-
- finish(): Buffer {
- const output = redactPrefix(this.pending, this.pending.length, this.secrets).output;
- this.pending = Buffer.alloc(0);
- return output;
- }
-}
-
-async function consume(stream: Readable, file: FileHandle, secrets: Buffer[]): Promise<void> {
- const redactor = new HostedLogRedactor(secrets);
- for await (const chunk of stream) {
- const output = redactor.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
- if (output.length) await file.write(output);
- }
- const output = redactor.finish();
- if (output.length) await file.write(output);
-}
-
-export async function pipeRedactedHostedLogs(
- streams: Readable[],
- file: FileHandle,
- values: string[]
-): Promise<void> {
- const secrets = secretBuffers(values);
- if (!secrets.length) throw new Error('Hosted log redaction requires at least one secret.');
- await Promise.all(streams.map((stream) => consume(stream, file, secrets)));
-}
diff --git a/console/packages/agent-providers/src/host/hosted-preflight.test.ts b/console/packages/agent-providers/src/host/hosted-preflight.test.ts
deleted file mode 100644
index 6d20532b9..000000000
--- a/console/packages/agent-providers/src/host/hosted-preflight.test.ts
+++ /dev/null
@@ -1,397 +0,0 @@
-import { createHash } from 'node:crypto';
-import { mkdir, mkdtemp, readdir, readFile, rm, stat, unlink, writeFile } from 'node:fs/promises';
-import { tmpdir } from 'node:os';
-import { dirname, join, relative } from 'node:path';
-import { afterEach, beforeEach, expect, it } from 'vitest';
-import { buildSharedHostConfig } from './build-shared-config';
-import { readCutoverManifest, readStateVersion } from './cutover-manifest';
-import {
- checkHostedPreflight,
- legacySessionsBase,
- PreflightBlockedError,
- runHostedPreflight,
-} from './hosted-preflight';
-import type { SharedHostConfig } from './shared-config';
-
-const AGENT = 'agent-placeholder';
-const WATCHER = 'watcher-placeholder';
-const SESSION = 'session-placeholder';
-const ROOM = '!room:example.test';
-const JOIN = `room_join:${'0'.repeat(64)}`;
-
-let root: string;
-let scratch: string;
-beforeEach(async () => {
- root = await mkdtemp(join(tmpdir(), 'hosted-preflight-'));
- scratch = await mkdtemp(join(tmpdir(), 'hosted-preflight-scratch-'));
-});
-afterEach(async () => {
- await rm(root, { recursive: true, force: true });
- await rm(scratch, { recursive: true, force: true });
-});
-
-function config(sessionId: string, env: Record<string, string>, ids: string): SharedHostConfig {
- return buildSharedHostConfig({
- session: { sessionId, agentId: AGENT, provider: 'opencode' },
- launch: { cwd: '/workspace', runtimeMode: 'full-access', env, model: undefined },
- capabilities: { approvals: true, userInput: true },
- execution: {
- credentialsPath: '/state/switch-credentials.json',
- inheritEnv: ['PATH'],
- binaryPath: '/usr/local/bin/opencode',
- codexConfig: '',
- skill: '',
- context: '',
- agentDefinition: undefined,
- },
- ids: { hostId: `host-${ids}`, epoch: `epoch-${ids}`, connectionId: `connection-${ids}` },
- });
-}
-
-/** A config as the session-table worker wrote it: its own rooms and cursor, and the runtime it baked in. */
-function legacyConfig(sessionId: string, rooms: string[]): Record<string, unknown> {
- const value = config(sessionId, { XDG_DATA_HOME: join(root, 'xdg', 'data') }, 'legacy');
- return {
- ...value,
- roomConnection: { connectionId: 'connection-legacy', rooms, startCursor: 3 },
- execution: {
- ...value.execution,
- mcpRuntime: '@sandboxaq/[email protected]',
- mcpRuntimePath: '/opt/runtime/index.js',
- },
- };
-}
-
-function candidate() {
- return {
- version: 1 as const,
- spec: { revision: 1, watch: true, session: { sessionId: WATCHER, agentId: AGENT } },
- config: config(WATCHER, { XDG_DATA_HOME: join(root, 'provider-data') }, 'new'),
- };
-}
-
-const sessionRoot = (sessionId: string) =>
- join(legacySessionsBase(root), createHash('sha256').update(sessionId).digest('hex'));
-
-async function put(path: string, value: unknown): Promise<void> {
- await mkdir(dirname(path), { recursive: true });
- await writeFile(path, typeof value === 'string' ? value : JSON.stringify(value));
-}
-
-const journal = (records: unknown[]) =>
- records.map((record) => `${JSON.stringify(record)}\n`).join('');
-
-function command(commandId: string, origin: Record<string, unknown>) {
- return {
- type: 'accepted',
- command: {
- contractVersion: 1,
- commandId,
- sessionId: SESSION,
- epoch: 'epoch-legacy',
- origin: { actorId: '@owner:example.test', ...origin },
- body: { type: 'message.send', text: 'hi', attachments: [], delivery: 'queue' },
- },
- };
-}
-
-function event(sequence: number, body: unknown) {
- return {
- contractVersion: 1,
- eventId: `event-${sequence}`,
- sessionId: SESSION,
- occurredAt: '2026-01-01T00:00:00Z',
- sequence,
- body,
- };
-}
-
-/** A volume as #538's worker left it: one room connection per session, owners stamped with a machine. */
-async function legacyVolume(): Promise<void> {
- await put(join(root, 'hosted-deployment.json'), {
- version: 1,
- spec: {
- watch: true,
- mcpRuntime: '@sandboxaq/[email protected]',
- session: { sessionId: WATCHER, agentId: AGENT },
- },
- config: legacyConfig(WATCHER, []),
- });
- await put(join(root, 'config.json'), legacyConfig(WATCHER, []));
- await put(join(root, 'watch.json'), { enabled: true });
- await put(join(root, 'shared-owner.lock'), { machine: 'machine-placeholder', pid: 1 });
- await put(
- join(root, 'assignments.jsonl'),
- journal([{ sequence: 1, roomId: ROOM, messageId: '$m0', config: legacyConfig(SESSION, []) }])
- );
- const session = sessionRoot(SESSION);
- await put(join(session, 'config.json'), legacyConfig(SESSION, [ROOM]));
- await put(join(session, 'supervisor', 'owner.json'), { machine: 'machine-placeholder' });
- await put(
- join(session, 'inbox.jsonl'),
- journal([
- command('command-room', {
- surface: 'slack',
- roomId: ROOM,
- threadId: '$thread',
- messageId: '$m1',
- }),
- { type: 'dispatched', commandId: 'command-room' },
- command('command-console', {
- surface: 'console',
- roomId: null,
- threadId: null,
- messageId: null,
- }),
- { type: 'reset-started' },
- ])
- );
- await put(
- join(session, 'events.jsonl'),
- journal([
- event(1, {
- type: 'turn.upsert',
- turnId: 'turn-1',
- status: 'running',
- commandId: 'command-room',
- }),
- event(2, {
- type: 'request.opened',
- request: {
- requestId: 'request-1',
- turnId: 'turn-1',
- revision: 0,
- state: 'open',
- content: {
- kind: 'approval',
- title: 'Run?',
- detail: null,
- options: [{ optionId: 'yes', label: 'Yes', decision: 'accept' }],
- },
- expiresAt: null,
- },
- }),
- ])
- );
- await put(
- join(session, 'room-inbox.jsonl'),
- journal([
- { type: 'rooms', rooms: [ROOM] },
- { type: 'received', sequence: 1, roomId: ROOM, messageId: '$m1' },
- { type: 'received', sequence: 2, roomId: ROOM, messageId: '$m2', missed: 0, gap: null },
- { type: 'failure-notified', identity: JSON.stringify([ROOM, '$m2']), reason: 'startup' },
- { type: 'received', sequence: 3, roomId: ROOM, messageId: JOIN },
- { type: 'ack', sequence: 1 },
- ])
- );
- await put(join(session, 'provider-data', 'opencode', 'storage', 'session.json'), 'history');
- await put(join(session, 'provider-data', 'opencode', 'auth.json'), 'credential-placeholder');
-}
-
-async function snapshot(): Promise<Record<string, string>> {
- const files: Record<string, string> = {};
- const walk = async (directory: string): Promise<void> => {
- for (const entry of await readdir(directory, { withFileTypes: true })) {
- const path = join(directory, entry.name);
- if (entry.isDirectory()) await walk(path);
- else files[relative(root, path)] = await readFile(path, 'utf8');
- }
- };
- await walk(root);
- return files;
-}
-
-async function exists(path: string): Promise<boolean> {
- return stat(path).then(
- () => true,
- () => false
- );
-}
-
-const readJson = async (path: string) => JSON.parse(await readFile(path, 'utf8'));
-
-it('moves a #538 volume to the watcher layout and lists what it held', async () => {
- await legacyVolume();
- expect(await runHostedPreflight(root, candidate())).toBe(true);
-
- const plan = await readJson(join(root, 'hosted-deployment.json'));
- expect(plan.spec).toEqual(candidate().spec);
- expect(plan.config.session).toMatchObject({ hostId: 'host-legacy', epoch: 'epoch-legacy' });
- expect(plan.config.roomConnection).toEqual({ connectionId: 'connection-legacy' });
- expect(plan.config.start.input.env.XDG_DATA_HOME).toBe(join(root, 'provider-data'));
- expect(await readJson(join(root, 'config.json'))).toEqual(plan.config);
- expect(await exists(join(root, 'hosted-deployment.json.pre-cutover'))).toBe(true);
-
- const session = sessionRoot(SESSION);
- const migrated = await readJson(join(session, 'config.json'));
- expect(migrated.roomConnection).toEqual({
- connectionId: 'connection-legacy',
- restoreRoomId: ROOM,
- });
- expect(migrated.execution.mcpRuntimePath).toBeUndefined();
- expect(migrated.start.input.env.XDG_DATA_HOME).toBe(join(root, 'provider-data'));
- const [assignment] = (await readFile(join(root, 'assignments.jsonl'), 'utf8'))
- .split('\n')
- .filter(Boolean)
- .map((line) => JSON.parse(line));
- expect(assignment.config.roomConnection).toEqual({ connectionId: 'connection-legacy' });
- expect(await readJson(join(root, 'placements.json'))).toEqual({
- placements: { [SESSION]: ROOM },
- });
-
- expect(await exists(join(session, 'room-inbox.jsonl'))).toBe(false);
- expect(await exists(join(session, 'room-inbox.jsonl.pre-cutover'))).toBe(true);
- expect(await exists(join(root, 'shared-owner.lock'))).toBe(false);
- expect(await exists(join(session, 'supervisor', 'owner.json'))).toBe(false);
- expect(
- await readFile(join(root, 'provider-data', 'opencode', 'storage', 'session.json'), 'utf8')
- ).toBe('history');
- expect(await exists(join(root, 'provider-data', 'opencode', 'auth.json'))).toBe(false);
- expect(await exists(join(session, 'provider-data'))).toBe(false);
-
- const manifest = await readCutoverManifest(root);
- expect(manifest?.manifest_sha256).toBe(
- createHash('sha256').update(JSON.stringify(manifest?.items)).digest('hex')
- );
- expect(manifest?.items).toHaveLength(5);
- expect(manifest?.items).toEqual(
- expect.arrayContaining([
- {
- kind: 'room_message',
- session_id: SESSION,
- room_id: ROOM,
- message_id: '$m1',
- thread_id: '$thread',
- room_pending: false,
- failure_notified: false,
- host: 'dispatched',
- },
- {
- kind: 'room_message',
- session_id: SESSION,
- room_id: ROOM,
- message_id: '$m2',
- thread_id: null,
- room_pending: true,
- failure_notified: true,
- host: null,
- },
- {
- kind: 'console_command',
- session_id: SESSION,
- command_id: 'command-console',
- host: 'accepted',
- },
- {
- kind: 'request_open',
- session_id: SESSION,
- request_id: 'request-1',
- room_id: ROOM,
- thread_id: '$thread',
- },
- { kind: 'reset_pending', session_id: SESSION },
- ])
- );
- expect(await readStateVersion(root)).toBe(1);
-});
-
-it('changes nothing on a second run', async () => {
- await legacyVolume();
- await runHostedPreflight(root, candidate());
- const before = await snapshot();
- expect(await runHostedPreflight(root, candidate())).toBe(false);
- expect(await snapshot()).toEqual(before);
-});
-
-it('finishes a preflight that stopped part way with the manifest it first listed', async () => {
- await legacyVolume();
- await runHostedPreflight(root, candidate());
- const before = await snapshot();
- await unlink(join(root, 'state-version.json'));
- expect(await runHostedPreflight(root, candidate())).toBe(true);
- expect(await snapshot()).toEqual(before);
-});
-
-it('stops at a journal it cannot read, names it, and finishes once it is repaired', async () => {
- await legacyVolume();
- const events = join(sessionRoot(SESSION), 'events.jsonl');
- const good = await readFile(events, 'utf8');
- await writeFile(events, `${good}{"contractVersion":1}\n`);
- const blocked = runHostedPreflight(root, candidate());
- await expect(blocked).rejects.toBeInstanceOf(PreflightBlockedError);
- await expect(blocked).rejects.toThrow(`${events}:3`);
- expect(await readJson(join(root, 'preflight-blocked.json'))).toMatchObject({
- version: 1,
- step: 'sessions',
- file: events,
- line: 3,
- });
- expect(await readStateVersion(root)).toBeNull();
-
- await writeFile(events, good);
- expect(await runHostedPreflight(root, candidate())).toBe(true);
- expect(await exists(join(root, 'preflight-blocked.json'))).toBe(false);
- expect((await readCutoverManifest(root))?.items).toHaveLength(5);
-});
-
-it('refuses to choose between two copies of a provider file', async () => {
- await legacyVolume();
- await put(join(root, 'provider-data', 'opencode', 'storage', 'session.json'), 'other history');
- await expect(runHostedPreflight(root, candidate())).rejects.toThrow('different content');
- expect(await readJson(join(root, 'preflight-blocked.json'))).toMatchObject({
- step: 'provider-homes',
- });
-});
-
-it('refuses a single room session, which has no watcher layout', async () => {
- await legacyVolume();
- const plan = await readJson(join(root, 'hosted-deployment.json'));
- delete plan.spec.watch;
- plan.spec.room = { roomId: ROOM, startCursor: 0 };
- await put(join(root, 'hosted-deployment.json'), plan);
- await expect(runHostedPreflight(root, candidate())).rejects.toThrow('single room session');
- expect(await readJson(join(root, 'preflight-blocked.json'))).toMatchObject({ step: 'plan' });
-});
-
-it('only stamps the layout version on a volume with nothing to migrate', async () => {
- expect(await runHostedPreflight(root, candidate())).toBe(false);
- expect(Object.keys(await snapshot())).toEqual(['state-version.json']);
- expect(await readCutoverManifest(root)).toBeNull();
-});
-
-it('refuses a volume from a newer worker', async () => {
- await put(join(root, 'state-version.json'), { version: 2 });
- await expect(runHostedPreflight(root, candidate())).rejects.toThrow('newer');
-});
-
-it('checks a copy of a volume and answers the manifest its first boot will upload', async () => {
- await legacyVolume();
- const before = await snapshot();
- const checked = await checkHostedPreflight(root, join(scratch, 'check'));
- expect(await snapshot()).toEqual(before);
- expect(await exists(join(scratch, 'check'))).toBe(false);
-
- await runHostedPreflight(root, candidate());
- expect(checked).toEqual({ manifest: await readCutoverManifest(root) });
-});
-
-it('checks a volume that would block and names the file on the volume', async () => {
- await legacyVolume();
- const events = join(sessionRoot(SESSION), 'events.jsonl');
- await writeFile(events, `${await readFile(events, 'utf8')}{"contractVersion":1}\n`);
- const before = await snapshot();
- expect(await checkHostedPreflight(root, join(scratch, 'check'))).toMatchObject({
- blocked: { step: 'sessions', file: events, line: 3 },
- });
- expect(await snapshot()).toEqual(before);
-});
-
-it('checks a volume with nothing to migrate as an empty manifest', async () => {
- const checked = await checkHostedPreflight(root, join(scratch, 'check'));
- expect(checked).toEqual({
- manifest: {
- manifest_sha256: createHash('sha256').update('[]').digest('hex'),
- items: [],
- },
- });
-});
diff --git a/console/packages/agent-providers/src/host/hosted-preflight.ts b/console/packages/agent-providers/src/host/hosted-preflight.ts
deleted file mode 100644
index ccdcc5e48..000000000
--- a/console/packages/agent-providers/src/host/hosted-preflight.ts
+++ /dev/null
@@ -1,816 +0,0 @@
-import { createHash, randomUUID } from 'node:crypto';
-import {
- cp,
- lstat,
- mkdir,
- open,
- readdir,
- readFile,
- readlink,
- rename,
- rm,
- unlink,
-} from 'node:fs/promises';
-import { isAbsolute, join, relative, sep } from 'node:path';
-import { isDeepStrictEqual } from 'node:util';
-import { type Command, serverEventSchema } from '@switch-console/shared/session-v1';
-import { z } from 'zod';
-import { assignmentRecordSchema } from './agent-host';
-import {
- type CutoverItem,
- type CutoverManifest,
- HOSTED_STATE_VERSION,
- PREFLIGHT_BLOCKED_FILE,
- readCutoverManifest,
- readStateVersion,
- writeCutoverManifest,
- writeJsonAtomically,
- writeStateVersion,
-} from './cutover-manifest';
-import { PLACEMENTS_FILE } from './placements';
-import { hostInboxRecordSchema } from './session-host';
-import { sharedConfigSchema, type SharedHostConfig } from './shared-config';
-import { deliveryRecordSchema } from './shared-delivery';
-import { sharedStateRecordSchema } from './shared-state';
-
-/**
- * Moves a retained hosted volume from the layout the session-table worker
- * (one room connection per session, Switch holding each session's commands)
- * left to the watcher's, before the watcher first runs on it, and lists what
- * its journals still hold for Switch to decide once. Every step is safe to
- * repeat: a volume whose preflight stopped part way is finished by the next
- * boot, and a finished one is left alone.
- */
-
-const PLAN_FILE = 'hosted-deployment.json';
-const CONFIG_FILE = 'config.json';
-const ASSIGNMENTS_FILE = 'assignments.jsonl';
-const LEGACY_ROOM_INBOX = 'room-inbox.jsonl';
-const PRE_CUTOVER = '.pre-cutover';
-const CREDENTIAL_FILES = new Set(['opencode/auth.json', 'antigravity-acp/acp_token.json']);
-const OWNER_FILES = ['shared-owner.lock', join('supervisor', 'owner.json')];
-const TICKET_DIRECTORIES = ['ownership', join('supervisor', 'ownership')];
-
-type Step =
- | 'plan'
- | 'sessions'
- | 'assignments'
- | 'placements'
- | 'owners'
- | 'provider-homes'
- | 'manifest';
-
-class Blocked extends Error {
- constructor(
- readonly step: Step,
- readonly file: string,
- readonly line: number | null,
- detail: string
- ) {
- super(detail);
- }
-}
-
-export class PreflightBlockedError extends Error {
- constructor(message: string) {
- super(message);
- this.name = 'PreflightBlockedError';
- }
-}
-
-const legacyPlanSchema = z.object({
- version: z.literal(1),
- spec: z
- .object({
- session: z.object({ sessionId: z.string().min(1), agentId: z.string().min(1) }),
- watch: z.boolean().optional(),
- })
- .refine((spec) => !('revision' in spec)),
- config: z.object({
- session: z.object({ hostId: z.string().min(1), epoch: z.string().min(1) }),
- roomConnection: z.object({ connectionId: z.string().min(1) }),
- }),
-});
-
-const legacyGapSchema = z
- .strictObject({ fromSequence: z.number().int().nonnegative(), reason: z.string().min(1) })
- .nullable();
-const legacyRoomInboxSchema = z.discriminatedUnion('type', [
- z.strictObject({
- type: z.literal('failure-notified'),
- identity: z.string().min(1),
- reason: z.enum(['startup', 'conversation', 'delivery']),
- }),
- z.strictObject({
- type: z.literal('received'),
- sequence: z.number().int().positive(),
- roomId: z.string().min(1),
- messageId: z.string().min(1),
- missed: z.number().int().nonnegative().default(0),
- gap: legacyGapSchema.default(null),
- }),
- z.strictObject({
- type: z.literal('ack'),
- sequence: z.number().int().positive(),
- identity: z.string().min(1).optional(),
- }),
- z.strictObject({
- type: z.literal('cursor'),
- sequence: z.number().int().nonnegative(),
- reset: z.boolean(),
- gap: legacyGapSchema,
- }),
- z.strictObject({ type: z.literal('rooms'), rooms: z.array(z.string()) }),
-]);
-
-const HOST_RANK = { accepted: 1, dispatched: 2, finished: 3 } as const;
-
-/** A delivery that is not a room message: a join or task event, named by the digest of its payload. */
-const NOT_A_MESSAGE = /^(room_join|task_[a-z_]+):[0-9a-f]{64}$/;
-
-export function legacySessionsBase(root: string): string {
- return join(root, 'home', '.local', 'state', 'switch', 'sdk-sessions');
-}
-
-async function exists(path: string): Promise<boolean> {
- try {
- await lstat(path);
- return true;
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code === 'ENOENT') return false;
- throw error;
- }
-}
-
-async function optionalText(path: string): Promise<string | null> {
- try {
- return await readFile(path, 'utf8');
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code === 'ENOENT') return null;
- throw error;
- }
-}
-
-async function readJsonFile(step: Step, path: string): Promise<unknown> {
- const text = await optionalText(path);
- if (text === null) throw new Blocked(step, path, null, 'the file is missing');
- try {
- return JSON.parse(text);
- } catch (error) {
- throw new Blocked(step, path, null, `not JSON: ${String(error)}`);
- }
-}
-
-/** Each record of a journal, parsed as the watcher will parse it; a torn tail blocks as it would. */
-async function readJournal<T>(
- step: Step,
- path: string,
- parse: (value: unknown) => T
-): Promise<T[]> {
- const text = await optionalText(path);
- if (text === null) return [];
- if (text && !text.endsWith('\n'))
- throw new Blocked(step, path, null, 'the journal ends in an incomplete write');
- const records: T[] = [];
- const lines = text.split('\n');
- for (let index = 0; index < lines.length; index++) {
- if (!lines[index]) continue;
- try {
- records.push(parse(JSON.parse(lines[index]!)));
- } catch (error) {
- throw new Blocked(
- step,
- path,
- index + 1,
- error instanceof z.ZodError ? z.prettifyError(error) : String(error)
- );
- }
- }
- return records;
-}
-
-async function writeTextAtomically(path: string, text: string): Promise<void> {
- const temporary = `${path}.${randomUUID()}.tmp`;
- const file = await open(temporary, 'wx', 0o600);
- try {
- await file.writeFile(text);
- await file.sync();
- } catch (error) {
- await file.close();
- await rm(temporary, { force: true });
- throw error;
- }
- await file.close();
- await rename(temporary, path);
-}
-
-/** Keeps the file as the session-table worker left it, once, beside the migrated one. */
-async function keepOriginal(path: string): Promise<void> {
- const original = `${path}${PRE_CUTOVER}`;
- if (await exists(original)) return;
- const text = await optionalText(path);
- if (text === null) return;
- await writeTextAtomically(original, text);
-}
-
-/**
- * A config the session-table worker wrote, in the watcher's shape: its own
- * room connection's rooms and cursor are no session's to hold now, the runtime
- * path it baked in is not how the Switch tools reach a session, and the
- * provider homes are the watcher's.
- */
-function currentConfig(
- step: Step,
- file: string,
- line: number | null,
- raw: unknown,
- environment: Record<string, string>
-): SharedHostConfig {
- if (!raw || typeof raw !== 'object' || Array.isArray(raw))
- throw new Blocked(step, file, line, 'the configuration is not an object');
- const value = structuredClone(raw) as Record<string, unknown>;
- const execution = value.execution as Record<string, unknown> | undefined;
- if (execution && typeof execution === 'object') delete execution.mcpRuntimePath;
- const connection = value.roomConnection as Record<string, unknown> | undefined;
- if (connection && typeof connection === 'object') {
- const rooms = connection.rooms;
- delete connection.rooms;
- delete connection.startCursor;
- if (Array.isArray(rooms) && rooms.length === 1 && connection.restoreRoomId === undefined)
- connection.restoreRoomId = rooms[0];
- }
- const start = value.start as { input?: { env?: Record<string, string> } } | undefined;
- if (start?.input?.env && typeof start.input.env === 'object')
- Object.assign(start.input.env, environment);
- const parsed = sharedConfigSchema.safeParse(value);
- if (!parsed.success) throw new Blocked(step, file, line, z.prettifyError(parsed.error));
- return parsed.data;
-}
-
-async function migrateConfig(
- path: string,
- environment: Record<string, string>
-): Promise<SharedHostConfig> {
- const raw = await readJsonFile('sessions', path);
- const config = currentConfig('sessions', path, null, raw, environment);
- if (!isDeepStrictEqual(raw, config)) {
- await keepOriginal(path);
- await writeJsonAtomically(path, config);
- }
- return config;
-}
-
-type HostState = keyof typeof HOST_RANK;
-type RoomState = {
- roomId: string;
- messageId: string;
- threadId: string | null;
- pending: boolean;
- notified: boolean;
- host: HostState | null;
-};
-
-/** What one session root still holds for Switch, read from its journals. */
-async function sessionItems(
- sessionRoot: string,
- sessionId: string
-): Promise<{
- items: CutoverItem[];
- rooms: string[] | null;
-}> {
- const rooms = new Map<string, RoomState>();
- const room = (roomId: string, messageId: string): RoomState => {
- const key = JSON.stringify([roomId, messageId]);
- let state = rooms.get(key);
- if (!state) {
- state = { roomId, messageId, threadId: null, pending: false, notified: false, host: null };
- rooms.set(key, state);
- }
- return state;
- };
-
- const inboxPath = join(sessionRoot, LEGACY_ROOM_INBOX);
- const keptPath = `${inboxPath}${PRE_CUTOVER}`;
- const legacyPath = (await exists(inboxPath)) ? inboxPath : keptPath;
- const outstanding = new Map<string, { roomId: string; messageId: string }>();
- const notified = new Set<string>();
- const sequences = new Map<number, string>();
- let cursor: number | null = null;
- let served: string[] | null = null;
- const legacy = await readJournal('sessions', legacyPath, (value) =>
- legacyRoomInboxSchema.parse(value)
- );
- for (const [index, record] of legacy.entries()) {
- if (record.type === 'failure-notified') notified.add(record.identity);
- else if (record.type === 'received') {
- if (record.gap && cursor !== null && record.sequence < cursor) sequences.clear();
- const key = JSON.stringify([record.roomId, record.messageId]);
- outstanding.set(key, { roomId: record.roomId, messageId: record.messageId });
- sequences.set(record.sequence, key);
- cursor = record.sequence;
- } else if (record.type === 'ack') {
- const key = record.identity ?? sequences.get(record.sequence);
- if (!key)
- throw new Blocked(
- 'sessions',
- legacyPath,
- index + 1,
- 'the room inbox acknowledges a delivery it never received'
- );
- outstanding.delete(key);
- } else if (record.type === 'cursor') {
- if (record.reset) sequences.clear();
- cursor = record.sequence;
- } else served = record.rooms;
- }
- for (const [key, delivery] of outstanding) {
- if (NOT_A_MESSAGE.test(delivery.messageId)) continue;
- const state = room(delivery.roomId, delivery.messageId);
- state.pending = true;
- state.notified = notified.has(key);
- }
-
- const inboxRecords = await readJournal('sessions', join(sessionRoot, 'inbox.jsonl'), (value) =>
- hostInboxRecordSchema.parse(value)
- );
- const commands = new Map<string, { origin: Command['origin']; host: HostState }>();
- let resetPending = false;
- for (const record of inboxRecords) {
- if (record.type === 'accepted')
- commands.set(record.command.commandId, { origin: record.command.origin, host: 'accepted' });
- else if (record.type === 'dispatched' || record.type === 'finished') {
- const command = commands.get(record.commandId);
- if (command && HOST_RANK[record.type] > HOST_RANK[command.host]) command.host = record.type;
- } else if (record.type === 'reset-started') resetPending = true;
- else if (record.type === 'reset-completed') resetPending = false;
- }
- const items: CutoverItem[] = [];
- for (const [commandId, command] of commands) {
- const { origin } = command;
- if (origin.roomId !== null && origin.messageId !== null) {
- const state = room(origin.roomId, origin.messageId);
- if (state.host === null || HOST_RANK[command.host] > HOST_RANK[state.host])
- state.host = command.host;
- state.threadId ??= origin.threadId;
- } else if (origin.roomId === null && origin.surface === 'console')
- items.push({
- kind: 'console_command',
- session_id: sessionId,
- command_id: commandId,
- host: command.host,
- });
- }
-
- const events = await readJournal('sessions', join(sessionRoot, 'events.jsonl'), (value) =>
- serverEventSchema.parse(value)
- );
- const turns = new Map<string, string | null>();
- const open = new Map<string, string>();
- for (const event of events) {
- const body = event.body;
- if (body.type === 'turn.upsert') turns.set(body.turnId, body.commandId);
- else if (body.type === 'request.opened') open.set(body.request.requestId, body.request.turnId);
- else if (body.type === 'request.settled') open.delete(body.requestId);
- }
- for (const [requestId, turnId] of open) {
- const commandId = turns.get(turnId) ?? null;
- const origin = commandId ? commands.get(commandId)?.origin : undefined;
- items.push({
- kind: 'request_open',
- session_id: sessionId,
- request_id: requestId,
- room_id: origin?.roomId ?? null,
- thread_id: origin?.roomId ? origin.threadId : null,
- });
- }
- if (resetPending) items.push({ kind: 'reset_pending', session_id: sessionId });
- for (const state of rooms.values())
- items.push({
- kind: 'room_message',
- session_id: sessionId,
- room_id: state.roomId,
- message_id: state.messageId,
- thread_id: state.threadId,
- room_pending: state.pending,
- failure_notified: state.notified,
- host: state.host,
- });
- return { items, rooms: served };
-}
-
-async function validateSessionJournals(sessionRoot: string): Promise<void> {
- await readJournal('sessions', join(sessionRoot, 'shared-state.jsonl'), (value) =>
- sharedStateRecordSchema.parse(value)
- );
- for (const name of await readdir(sessionRoot))
- if (/^delivery-[0-9a-f]{64}\.jsonl$/.test(name))
- await readJournal('sessions', join(sessionRoot, name), (value) =>
- deliveryRecordSchema.parse(value)
- );
-}
-
-/**
- * Owner records the session-table worker stamped with its machine: every
- * process that wrote one is gone with the machine it ran on, and the watcher
- * reads these strictly.
- */
-async function clearMachineOwners(root: string): Promise<void> {
- const candidates = [...OWNER_FILES.map((file) => join(root, file))];
- for (const directory of TICKET_DIRECTORIES) {
- let names: string[];
- try {
- names = await readdir(join(root, directory));
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code === 'ENOENT') continue;
- throw error;
- }
- for (const name of names)
- if (name.endsWith('.json')) candidates.push(join(root, directory, name));
- }
- for (const path of candidates) {
- const text = await optionalText(path);
- if (text === null) continue;
- let value: unknown;
- try {
- value = JSON.parse(text);
- } catch {
- throw new Blocked('owners', path, null, 'the owner record is not JSON');
- }
- if (value && typeof value === 'object' && 'machine' in value) await unlink(path);
- }
-}
-
-/**
- * Moves what a session's provider kept in its own home into the watcher's.
- * A file already there with the same bytes stays; one that differs blocks,
- * since neither copy can be chosen for the provider. The credential is
- * written fresh by the bootstrap and never moved. What is left is kept under
- * the source's name with `.pre-cutover`.
- */
-async function mergeInto(source: string, destination: string): Promise<void> {
- const walk = async (relativePath: string): Promise<void> => {
- const from = join(source, relativePath);
- const to = join(destination, relativePath);
- const entry = await lstat(from);
- if (entry.isDirectory()) {
- await mkdir(to, { recursive: true, mode: 0o700 });
- for (const name of (await readdir(from)).sort()) await walk(join(relativePath, name));
- return;
- }
- if (CREDENTIAL_FILES.has(relativePath)) return;
- if (!(await exists(to))) {
- await rename(from, to);
- return;
- }
- const existing = await lstat(to);
- const same = entry.isSymbolicLink()
- ? existing.isSymbolicLink() && (await readlink(from)) === (await readlink(to))
- : existing.isFile() && (await readFile(from)).equals(await readFile(to));
- if (!same)
- throw new Blocked(
- 'provider-homes',
- from,
- null,
- `${to} already exists with different content; move one of them aside and boot again`
- );
- };
- if (!(await exists(source))) return;
- await mkdir(destination, { recursive: true, mode: 0o700 });
- for (const name of (await readdir(source)).sort()) await walk(name);
- await rename(source, `${source}${PRE_CUTOVER}`);
-}
-
-async function migrateProviderHome(
- root: string,
- sessionRoot: string,
- provider: SharedHostConfig['start']['provider']
-): Promise<void> {
- if (provider === 'opencode')
- await mergeInto(join(sessionRoot, 'provider-data'), join(root, 'provider-data'));
- else if (provider === 'antigravity')
- await mergeInto(join(sessionRoot, 'provider-home'), join(root, 'provider-home'));
-}
-
-async function migrateAssignments(
- root: string,
- environment: Record<string, string>
-): Promise<Map<string, string>> {
- const path = join(root, ASSIGNMENTS_FILE);
- const text = await optionalText(path);
- const placements = new Map<string, string>();
- if (text === null) return placements;
- if (text && !text.endsWith('\n'))
- throw new Blocked('assignments', path, null, 'the journal ends in an incomplete write');
- const lines = text.split('\n').filter(Boolean);
- const migrated: unknown[] = [];
- for (const [index, line] of lines.entries()) {
- let raw: Record<string, unknown>;
- try {
- raw = JSON.parse(line);
- } catch (error) {
- throw new Blocked('assignments', path, index + 1, `not JSON: ${String(error)}`);
- }
- const record =
- 'config' in raw
- ? { ...raw, config: currentConfig('assignments', path, index + 1, raw.config, environment) }
- : raw;
- const parsed = assignmentRecordSchema.safeParse(record);
- if (!parsed.success)
- throw new Blocked('assignments', path, index + 1, z.prettifyError(parsed.error));
- if ('config' in parsed.data) {
- const sessionId = parsed.data.config.session.sessionId;
- for (const [placed, roomId] of placements)
- if (roomId === parsed.data.roomId) placements.delete(placed);
- placements.set(sessionId, parsed.data.roomId);
- }
- migrated.push(record);
- }
- const next = migrated.map((record) => `${JSON.stringify(record)}\n`).join('');
- if (next !== text) {
- await keepOriginal(path);
- await writeTextAtomically(path, next);
- }
- return placements;
-}
-
-export interface PreflightPlan {
- version: 1;
- spec: { session: { sessionId: string; agentId: string } };
- config: SharedHostConfig;
-}
-
-async function migratePlan(root: string, candidate: PreflightPlan): Promise<boolean> {
- const planPath = join(root, PLAN_FILE);
- const kept = `${planPath}${PRE_CUTOVER}`;
- const text = await optionalText(planPath);
- if (text === null) return false;
- let raw: unknown;
- try {
- raw = JSON.parse(text);
- } catch (error) {
- throw new Blocked('plan', planPath, null, `not JSON: ${String(error)}`);
- }
- const legacy = legacyPlanSchema.safeParse(raw);
- if (!legacy.success) return exists(kept);
- if (legacy.data.spec.session.agentId !== candidate.spec.session.agentId)
- throw new Blocked(
- 'plan',
- planPath,
- null,
- `the saved deployment belongs to agent ${legacy.data.spec.session.agentId}, not ${candidate.spec.session.agentId}`
- );
- if (legacy.data.spec.watch === undefined)
- throw new Blocked(
- 'plan',
- planPath,
- null,
- 'the saved deployment is a single room session, which has no watcher layout to move to; delete the launch and start it again'
- );
- const config = structuredClone(candidate.config);
- config.session.hostId = legacy.data.config.session.hostId;
- config.session.epoch = legacy.data.config.session.epoch;
- config.roomConnection = { connectionId: legacy.data.config.roomConnection.connectionId };
- await keepOriginal(planPath);
- await keepOriginal(join(root, CONFIG_FILE));
- await writeJsonAtomically(join(root, CONFIG_FILE), config);
- await writeJsonAtomically(planPath, { ...candidate, config });
- return true;
-}
-
-async function sessionRoots(root: string): Promise<string[]> {
- const base = legacySessionsBase(root);
- let names: string[];
- try {
- names = await readdir(base);
- } catch (error) {
- if ((error as NodeJS.ErrnoException).code === 'ENOENT') return [];
- throw error;
- }
- const roots: string[] = [];
- for (const name of names.sort())
- if (/^[0-9a-f]{64}$/.test(name) && (await lstat(join(base, name))).isDirectory())
- roots.push(join(base, name));
- return roots;
-}
-
-function placementsFrom(
- served: Map<string, string[]>,
- assigned: Map<string, string>
-): Record<string, string> {
- const placements = new Map<string, string>();
- const holder = new Map<string, string>();
- for (const [sessionId, rooms] of served) {
- if (rooms.length > 1)
- throw new Blocked(
- 'placements',
- sessionId,
- null,
- `session ${sessionId} served ${rooms.length} rooms; a session attends one`
- );
- const roomId = rooms[0];
- if (roomId === undefined) continue;
- const other = holder.get(roomId);
- if (other)
- throw new Blocked(
- 'placements',
- sessionId,
- null,
- `room ${roomId} was served by two sessions (${other} and ${sessionId})`
- );
- placements.set(sessionId, roomId);
- holder.set(roomId, sessionId);
- }
- for (const [sessionId, roomId] of assigned)
- if (!placements.has(sessionId) && !holder.has(roomId)) {
- placements.set(sessionId, roomId);
- holder.set(roomId, sessionId);
- }
- return Object.fromEntries([...placements].sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)));
-}
-
-function itemKey(item: CutoverItem): string {
- return JSON.stringify(item);
-}
-
-async function migrate(root: string, candidate: PreflightPlan): Promise<boolean> {
- if (!(await migratePlan(root, candidate))) return false;
- const environment = candidate.config.start.input.env;
- const items: CutoverItem[] = [];
- const served = new Map<string, string[]>();
- await clearMachineOwners(root);
- for (const sessionRoot of await sessionRoots(root)) {
- const configPath = join(sessionRoot, CONFIG_FILE);
- if (!(await exists(configPath)) && !(await exists(`${configPath}${PRE_CUTOVER}`))) continue;
- const config = await migrateConfig(configPath, environment);
- if (config.session.agentId !== candidate.spec.session.agentId)
- throw new Blocked(
- 'sessions',
- configPath,
- null,
- `the session belongs to agent ${config.session.agentId}`
- );
- const sessionId = config.session.sessionId;
- if (createHash('sha256').update(sessionId).digest('hex') !== sessionRoot.split('/').at(-1))
- throw new Blocked(
- 'sessions',
- configPath,
- null,
- 'the session id does not name this directory'
- );
- await validateSessionJournals(sessionRoot);
- const held = await sessionItems(sessionRoot, sessionId);
- items.push(...held.items);
- if (held.rooms) served.set(sessionId, held.rooms);
- const legacyInbox = join(sessionRoot, LEGACY_ROOM_INBOX);
- if (await exists(legacyInbox)) await rename(legacyInbox, `${legacyInbox}${PRE_CUTOVER}`);
- await clearMachineOwners(sessionRoot);
- await migrateProviderHome(root, sessionRoot, config.start.provider);
- }
- const assigned = await migrateAssignments(root, environment);
- const placementsPath = join(root, PLACEMENTS_FILE);
- if (!(await exists(placementsPath)))
- await writeJsonAtomically(placementsPath, { placements: placementsFrom(served, assigned) });
- items.sort((a, b) => (itemKey(a) < itemKey(b) ? -1 : itemKey(a) > itemKey(b) ? 1 : 0));
- await writeCutoverManifest(root, {
- manifest_sha256: createHash('sha256').update(JSON.stringify(items)).digest('hex'),
- items,
- });
- return true;
-}
-
-/**
- * Brings the volume at `root` to this worker's layout before anything reads
- * it, or refuses to: a volume it cannot migrate stops at the step that failed, with
- * `preflight-blocked.json` naming the step, file and line, and the worker does
- * not start. Answers whether a pre-cutover plan was replaced.
- */
-export async function runHostedPreflight(root: string, candidate: PreflightPlan): Promise<boolean> {
- const blockedPath = join(root, PREFLIGHT_BLOCKED_FILE);
- const version = await readStateVersion(root);
- if (version !== null && version > HOSTED_STATE_VERSION)
- throw new PreflightBlockedError(
- `This volume is at layout version ${version}, newer than this worker's ${HOSTED_STATE_VERSION}; run a current worker image.`
- );
- if (version === HOSTED_STATE_VERSION) return false;
- let migrated: boolean;
- try {
- migrated = await migrate(root, candidate);
- } catch (error) {
- const blocked =
- error instanceof Blocked
- ? { step: error.step, file: error.file, line: error.line, error: error.message }
- : {
- step: 'manifest' as const,
- file: root,
- line: null,
- error: error instanceof Error ? error.message : String(error),
- };
- await writeJsonAtomically(blockedPath, { version: HOSTED_STATE_VERSION, ...blocked });
- throw new PreflightBlockedError(
- `The hosted preflight could not migrate this volume at step ${blocked.step} (${blocked.file}${blocked.line === null ? '' : `:${blocked.line}`}): ${blocked.error}. Details are in ${blockedPath}.`
- );
- }
- await writeStateVersion(root);
- if (await exists(blockedPath)) await unlink(blockedPath);
- return migrated;
-}
-
-export type PreflightCheck =
- | { manifest: CutoverManifest }
- | { blocked: { step: string; file: string; line: number | null; error: string } };
-
-const blockedFileSchema = z.object({
- step: z.string(),
- file: z.string(),
- line: z.number().int().nullable(),
- error: z.string(),
-});
-
-function emptyManifest(): CutoverManifest {
- return { manifest_sha256: createHash('sha256').update('[]').digest('hex'), items: [] };
-}
-
-function isWithin(parent: string, path: string): boolean {
- const child = relative(parent, path);
- return child === '' || (child !== '..' && !child.startsWith(`..${sep}`) && !isAbsolute(child));
-}
-
-/**
- * The plan a check migrates a copy with. The manifest does not depend on it:
- * it only has to name the agent the saved deployment belongs to and parse as
- * the watcher's configuration, which the saved one's own does.
- */
-async function checkCandidate(root: string): Promise<PreflightPlan | null> {
- const planPath = join(root, PLAN_FILE);
- if ((await optionalText(planPath)) === null) return null;
- const raw = await readJsonFile('plan', planPath);
- const parsed = z
- .object({
- spec: z.object({
- session: z.object({ sessionId: z.string().min(1), agentId: z.string().min(1) }),
- }),
- config: z.unknown(),
- })
- .safeParse(raw);
- if (!parsed.success) throw new Blocked('plan', planPath, null, z.prettifyError(parsed.error));
- return {
- version: 1,
- spec: { session: parsed.data.spec.session },
- config: currentConfig('plan', planPath, null, parsed.data.config, {}),
- };
-}
-
-/**
- * Runs the preflight on a copy of the volume at `root` under `scratch`, and
- * answers the manifest the first boot of this image will upload, or where it
- * would stop. The volume itself is only read, so this can run against a
- * stopped worker's retained disk before Switch drops the tables its
- * dispositions need.
- */
-export async function checkHostedPreflight(root: string, scratch: string): Promise<PreflightCheck> {
- if (isWithin(root, scratch) || isWithin(scratch, root))
- throw new Error('The scratch directory must be outside the volume it checks.');
- await mkdir(scratch, { mode: 0o700 });
- const copy = join(scratch, 'state');
- const original = (file: string) =>
- isWithin(copy, file) ? join(root, relative(copy, file)) : file;
- try {
- await cp(root, copy, {
- recursive: true,
- errorOnExist: true,
- force: false,
- preserveTimestamps: true,
- verbatimSymlinks: true,
- });
- await rm(join(copy, PREFLIGHT_BLOCKED_FILE), { force: true });
- let candidate: PreflightPlan | null;
- try {
- candidate = await checkCandidate(copy);
- } catch (error) {
- if (!(error instanceof Blocked)) throw error;
- return {
- blocked: {
- step: error.step,
- file: original(error.file),
- line: error.line,
- error: error.message,
- },
- };
- }
- try {
- if (candidate) await runHostedPreflight(copy, candidate);
- } catch (error) {
- if (!(error instanceof PreflightBlockedError)) throw error;
- const text = await optionalText(join(copy, PREFLIGHT_BLOCKED_FILE));
- const saved = text === null ? null : blockedFileSchema.parse(JSON.parse(text));
- return {
- blocked: saved
- ? { ...saved, file: original(saved.file) }
- : { step: 'plan', file: root, line: null, error: error.message },
- };
- }
- return { manifest: (await readCutoverManifest(copy)) ?? emptyManifest() };
- } finally {
- await rm(scratch, { recursive: true, force: true });
- }
-}
diff --git a/console/packages/agent-providers/src/host/hosted-provider.test.ts b/console/packages/agent-providers/src/host/hosted-provider.test.ts
index b8a6f8d92..b5f045185 100644
--- a/console/packages/agent-providers/src/host/hosted-provider.test.ts
+++ b/console/packages/agent-providers/src/host/hosted-provider.test.ts
@@ -3,13 +3,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
import { prepareCodexSessionHome } from '../codex/home';
-import {
- fetchHostedProvider,
- hostedRequest,
- materializeHostedProvider,
- type HostedCredential,
-} from './hosted-provider';
-import type { SharedHostConfig } from './shared-config';
+import { materializeHostedProvider, type HostedCredential } from './hosted-provider';
let root: string;
beforeEach(async () => {
@@ -20,72 +14,6 @@ afterEach(async () => {
await rm(root, { recursive: true, force: true });
});
-async function hostedConfig(
- endpoint = 'https://switch.invalid/api/agent'
-): Promise<SharedHostConfig> {
- const credentialsPath = join(root, 'switch.json');
- await writeFile(
- credentialsPath,
- JSON.stringify({
- env: {
- SWITCH_API_ENDPOINT: endpoint,
- SWITCH_API_TOKEN: 'switch-token-fixture',
- SWITCH_AGENT_ID: 'agent-id',
- },
- })
- );
- return {
- session: { agentId: 'agent-id' },
- start: { provider: 'claude' },
- execution: { credentialsPath },
- } as unknown as SharedHostConfig;
-}
-
-it('reads a connected credential and refuses one for another provider', async () => {
- const body = {
- status: 'connected',
- revision: 'revision-one',
- provider: 'claude',
- kind: 'api-key',
- credential: 'fixture-key',
- };
- const request = vi.fn(async () => new Response(JSON.stringify(body)));
- vi.stubGlobal('fetch', request);
- const config = await hostedConfig();
- expect(await fetchHostedProvider(config)).toEqual(body);
- expect(request).toHaveBeenCalledWith(
- 'https://switch.invalid/api/agent/hosted/provider-credential',
- expect.objectContaining({
- method: 'POST',
- headers: { Authorization: 'Bearer switch-token-fixture' },
- })
- );
- vi.stubGlobal(
- 'fetch',
- vi.fn(async () => new Response(JSON.stringify({ ...body, provider: 'codex' })))
- );
- await expect(fetchHostedProvider(config)).rejects.toThrow('do not match this session');
-});
-
-it('reports a forbidden credential read as revoked and other failures loudly', async () => {
- const config = await hostedConfig();
- vi.stubGlobal(
- 'fetch',
- vi.fn(async () => new Response('', { status: 403 }))
- );
- expect(await fetchHostedProvider(config)).toEqual({ status: 'revoked' });
- vi.stubGlobal(
- 'fetch',
- vi.fn(async () => new Response('', { status: 500 }))
- );
- await expect(fetchHostedProvider(config)).rejects.toThrow('HTTP 500');
- await expect(hostedRequest(config, '/provider-status', {})).rejects.toThrow('HTTP 500');
-});
-
-it('refuses a non-HTTPS Switch origin', async () => {
- const config = await hostedConfig('http://switch.invalid/api/agent');
- await expect(fetchHostedProvider(config)).rejects.toThrow('HTTPS');
-});
const credential = (value: string): Extract<HostedCredential, { status: 'connected' }> => ({
status: 'connected',
revision: 'revision-one',
@@ -163,7 +91,6 @@ it('keeps the prepared Codex home and refreshes its auth before provider startup
sessionId: 'session',
sourceHome,
config: 'model = "fixture-model"',
- auth: 'refresh',
});
env.CODEX_HOME = home;
await writeFile(join(home, 'auth.json'), '{"fixture":"native-refresh"}');
diff --git a/console/packages/agent-providers/src/host/hosted-provider.ts b/console/packages/agent-providers/src/host/hosted-provider.ts
index ec0b388e7..1315e52d7 100644
--- a/console/packages/agent-providers/src/host/hosted-provider.ts
+++ b/console/packages/agent-providers/src/host/hosted-provider.ts
@@ -2,78 +2,11 @@ import { createHash } from 'node:crypto';
import { constants } from 'node:fs';
import { lstat, mkdir, open, readFile, rename } from 'node:fs/promises';
import { join } from 'node:path';
-import { z } from 'zod';
import { refreshCodexAuthentication } from '../codex/home';
import { importOpenCodeConsole } from './opencode-console';
-import { readSharedCredentials, type SharedHostConfig } from './shared-config';
+import type { HostedCredential } from './provider-login';
-const credentialSchema = z.discriminatedUnion('status', [
- z.object({ status: z.literal('revoked') }),
- z.object({
- status: z.literal('connected'),
- revision: z.string(),
- provider: z.enum(['claude', 'codex', 'cursor', 'opencode', 'antigravity']),
- kind: z.enum(['api-key', 'setup-token', 'auth-json']),
- credential: z.string().min(1).max(16384),
- }),
-]);
-export type HostedCredential = z.infer<typeof credentialSchema>;
-
-async function hostedOrigin(config: SharedHostConfig): Promise<{ origin: string; token: string }> {
- const credentials = await readSharedCredentials(config);
- const url = new URL(credentials.SWITCH_API_ENDPOINT);
- if (url.protocol !== 'https:' || url.username || url.password || url.search || url.hash)
- throw new Error('Cloud control requires an HTTPS server origin.');
- return { origin: url.href.replace(/\/$/, ''), token: credentials.SWITCH_API_TOKEN };
-}
-
-/** A POST to the agent's `/hosted` routes, authenticated as the agent. */
-export async function hostedRequest(
- config: SharedHostConfig,
- path: string,
- body: unknown
-): Promise<unknown> {
- const { origin, token } = await hostedOrigin(config);
- const response = await fetch(`${origin}/hosted${path}`, {
- method: 'POST',
- headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
- body: JSON.stringify(body),
- redirect: 'error',
- signal: AbortSignal.timeout(30000),
- });
- if (!response.ok) {
- await response.body?.cancel();
- throw new Error(`Cloud control request failed (HTTP ${response.status}).`);
- }
- return response.json();
-}
-
-export async function fetchHostedProvider(config: SharedHostConfig): Promise<HostedCredential> {
- const { origin, token } = await hostedOrigin(config);
- let response: Response;
- try {
- response = await fetch(`${origin}/hosted/provider-credential`, {
- method: 'POST',
- headers: { Authorization: `Bearer ${token}` },
- redirect: 'error',
- signal: AbortSignal.timeout(30000),
- });
- } catch {
- throw new Error('Cloud provider access could not be checked.');
- }
- if (response.status === 403) {
- await response.body?.cancel();
- return { status: 'revoked' };
- }
- if (!response.ok) {
- await response.body?.cancel();
- throw new Error(`Cloud provider access check failed (HTTP ${response.status}).`);
- }
- const result = credentialSchema.parse(await response.json());
- if (result.status === 'connected' && result.provider !== config.start.provider)
- throw new Error('Cloud provider credentials do not match this session.');
- return result;
-}
+export type { HostedCredential } from './provider-login';
export function applyHostedProvider(
env: Record<string, string>,
@@ -84,7 +17,6 @@ export function applyHostedProvider(
'CLAUDE_CODE_OAUTH_TOKEN',
'OPENAI_API_KEY',
'CURSOR_API_KEY',
- 'SWITCH_HOSTED_AUTH_JSON',
])
delete env[key];
if (credential.status === 'revoked')
diff --git a/console/packages/agent-providers/src/host/hosted-skills.ts b/console/packages/agent-providers/src/host/hosted-skills.ts
deleted file mode 100644
index 56bda9067..000000000
--- a/console/packages/agent-providers/src/host/hosted-skills.ts
+++ /dev/null
@@ -1,77 +0,0 @@
-import { randomUUID } from 'node:crypto';
-import { mkdir, rename, rm, writeFile } from 'node:fs/promises';
-import { dirname, join } from 'node:path';
-import { z } from 'zod';
-
-export const MAX_HOSTED_SKILL_BYTES = 32 * 1024;
-const SKILL_PATH = /^[A-Za-z0-9_-][A-Za-z0-9._-]{0,99}(\/[A-Za-z0-9_-][A-Za-z0-9._-]{0,99}){0,7}$/;
-
-/** Providers whose agent reads skills from a directory bootstrap can install into. */
-export const HOSTED_SKILL_PROVIDERS = ['claude', 'codex', 'opencode'] as const;
-type HostedSkillProvider = (typeof HOSTED_SKILL_PROVIDERS)[number];
-
-export const hostedSkillsSchema = z
- .array(
- z
- .strictObject({
- slug: z.string().regex(/^[a-z0-9][a-z0-9-]{0,62}$/),
- files: z.record(
- z.string().regex(SKILL_PATH),
- z.string().refine((value) => !value.includes('\0'), 'must not contain NUL')
- ),
- })
- .refine((skill) => 'SKILL.md' in skill.files, 'must include SKILL.md')
- )
- .min(1)
- .max(16)
- .refine(
- (skills) => new Set(skills.map((skill) => skill.slug)).size === skills.length,
- 'skill names must be unique'
- )
- .refine(
- (skills) =>
- skills
- .flatMap((skill) => Object.values(skill.files))
- .reduce((total, content) => total + Buffer.byteLength(content), 0) <=
- MAX_HOSTED_SKILL_BYTES,
- `skills must fit within ${MAX_HOSTED_SKILL_BYTES} bytes`
- );
-export type HostedSkills = z.infer<typeof hostedSkillsSchema>;
-
-export function supportsHostedSkills(provider: string): provider is HostedSkillProvider {
- return (HOSTED_SKILL_PROVIDERS as readonly string[]).includes(provider);
-}
-
-/** The global skills directory each provider reads under the hosted agent's environment. */
-export function hostedSkillsDirectory(
- provider: HostedSkillProvider,
- env: NodeJS.ProcessEnv
-): string {
- const base = {
- claude: env.CLAUDE_CONFIG_DIR,
- codex: env.CODEX_HOME || (env.HOME && join(env.HOME, '.codex')),
- opencode: env.XDG_CONFIG_HOME && join(env.XDG_CONFIG_HOME, 'opencode'),
- }[provider];
- if (!base) throw new Error(`The ${provider} home for connection skills is not configured.`);
- return join(base, 'skills');
-}
-
-/** Replace each granted skill so a restart always carries the deployment's copy. */
-export async function installHostedSkills(directory: string, skills: HostedSkills): Promise<void> {
- await mkdir(directory, { recursive: true, mode: 0o700 });
- for (const skill of skills) {
- const staging = join(directory, `.${skill.slug}.${randomUUID()}`);
- try {
- for (const [path, content] of Object.entries(skill.files)) {
- const target = join(staging, path);
- await mkdir(dirname(target), { recursive: true, mode: 0o700 });
- await writeFile(target, content, { mode: 0o600, flag: 'wx' });
- }
- const destination = join(directory, skill.slug);
- await rm(destination, { recursive: true, force: true });
- await rename(staging, destination);
- } finally {
- await rm(staging, { recursive: true, force: true });
- }
- }
-}
diff --git a/console/packages/agent-providers/src/host/hosted-watcher.ts b/console/packages/agent-providers/src/host/hosted-watcher.ts
deleted file mode 100644
index 854411940..000000000
--- a/console/packages/agent-providers/src/host/hosted-watcher.ts
+++ /dev/null
@@ -1,67 +0,0 @@
-import type { ControlContext } from './control';
-import { HOSTED_STATE_VERSION, readStateVersion } from './cutover-manifest';
-import { fetchHostedProvider, materializeHostedProvider } from './hosted-provider';
-import { type HostedCredentials, HostedWorker } from './hosted-worker';
-import type { SharedHostConfig } from './shared-config';
-import { readWorkerCapability } from './worker-capability';
-
-/**
- * The provider credential Switch holds for a hosted worker's owner, applied to
- * this process's environment, which every session host it starts inherits.
- */
-function hostedCredentials(config: SharedHostConfig, stateRoot: string): HostedCredentials {
- return {
- fetch: async () => {
- const credential = await fetchHostedProvider(config);
- if (credential.status === 'revoked')
- return { revoked: true, revision: null, apply: async () => {} };
- return {
- revoked: false,
- revision: credential.revision,
- apply: async () => {
- const env = Object.fromEntries(
- Object.entries(process.env).filter(
- (entry): entry is [string, string] => entry[1] !== undefined
- )
- );
- await materializeHostedProvider(
- stateRoot,
- env,
- credential,
- config.execution?.binaryPath ?? config.start.provider
- );
- for (const key of Object.keys(process.env)) if (!(key in env)) delete process.env[key];
- Object.assign(process.env, env);
- },
- };
- },
- };
-}
-
-/** The hosted worker a bootstrapped watcher attaches as; null for any other watcher. */
-export async function hostedWorker(
- config: SharedHostConfig,
- stateRoot: string,
- context: ControlContext
-): Promise<HostedWorker | null> {
- if (process.env.SWITCH_HOSTED_BOOTSTRAP !== '1') return null;
- const bootId = process.env.SWITCH_HOST_BOOT_ID;
- const instanceId = process.env.SWITCH_HOST_INSTANCE_ID;
- if (!bootId || !instanceId)
- throw new Error(
- 'A hosted watcher requires SWITCH_HOST_BOOT_ID and SWITCH_HOST_INSTANCE_ID from its bootstrap.'
- );
- const stateVersion = await readStateVersion(stateRoot);
- if (stateVersion !== HOSTED_STATE_VERSION)
- throw new Error(
- `This volume is at layout version ${stateVersion ?? 'none'}, not ${HOSTED_STATE_VERSION}; the hosted bootstrap's preflight must finish before the watcher starts.`
- );
- const worker = new HostedWorker(
- stateRoot,
- { capability: await readWorkerCapability(stateRoot), bootId, instanceId, stateVersion },
- context,
- hostedCredentials(config, stateRoot)
- );
- await worker.open();
- return worker;
-}
diff --git a/console/packages/agent-providers/src/host/hosted-worker.test.ts b/console/packages/agent-providers/src/host/hosted-worker.test.ts
deleted file mode 100644
index 9c261ac52..000000000
--- a/console/packages/agent-providers/src/host/hosted-worker.test.ts
+++ /dev/null
@@ -1,454 +0,0 @@
-import type { ChildProcess } from 'node:child_process';
-import { createHash } from 'node:crypto';
-import { EventEmitter } from 'node:events';
-import { mkdtemp, readFile, rm } from 'node:fs/promises';
-import { tmpdir } from 'node:os';
-import { join } from 'node:path';
-import { WorkerCallError } from '@sandboxaq/switch-agent-runtime';
-import { afterEach, beforeEach, expect, it, vi } from 'vitest';
-import { AttachmentTransfers } from './attachment-transfers';
-import type { ControlContext } from './control';
-import { unconfirmedCutoverManifest, writeCutoverManifest } from './cutover-manifest';
-import {
- CONSOLE_RECENT_MS,
- type HostedPort,
- HostedWorker,
- PAGE_BYTES,
- PAGE_IDLE_MS,
- RELAY_STALE_MS,
- RelayJournal,
- RelayPages,
-} from './hosted-worker';
-import { SessionLinks } from './session-channel';
-import { WatcherControl } from './watcher-tools';
-
-const paths = vi.hoisted(() => ({ base: '' }));
-const supervisors = vi.hoisted(() => new Set<string>());
-vi.mock('./launch', () => ({
- sharedSessionRoot: (id: string) => join(paths.base, id),
- sharedSessionsBase: () => paths.base,
- liveSupervisor: (root: string) => Promise.resolve(supervisors.has(root) ? { build: 'b' } : null),
-}));
-
-let root: string;
-beforeEach(async () => {
- paths.base = await mkdtemp(join(tmpdir(), 'hosted-worker-'));
- root = join(paths.base, 'watcher');
-});
-afterEach(async () => {
- supervisors.clear();
- vi.restoreAllMocks();
- await rm(paths.base, { recursive: true, force: true });
-});
-
-type Child = EventEmitter & {
- connected: boolean;
- sent: { kind: string; id: number; request?: unknown }[];
- send: (message: unknown, callback: (error: Error | null) => void) => boolean;
-};
-
-/** A session host over IPC that records what it is sent and answers only when told to. */
-function child(links: SessionLinks, sessionRoot: string): Child {
- const host = new EventEmitter() as Child;
- host.connected = true;
- host.sent = [];
- host.send = (message, callback) => {
- host.sent.push(message as Child['sent'][number]);
- callback(null);
- return true;
- };
- links.attach(sessionRoot, host as unknown as ChildProcess);
- return host;
-}
-
-async function eventually(reached: () => boolean | Promise<boolean>): Promise<void> {
- for (let attempt = 0; attempt < 400; attempt++) {
- if (await reached()) return;
- await new Promise((resolve) => setTimeout(resolve, 5));
- }
- throw new Error('Never reached the state this test was waiting for.');
-}
-
-const attached = (overrides: Record<string, unknown> = {}) => ({
- launch_revision: 1,
- limits: { sessions_per_agent: 8 },
- idle: { report_every_s: 30, fresh_for_s: 75 },
- credential_revision: 'r1',
- queued_operations: [],
- relay_fence: 0,
- cancelled: [],
- ...overrides,
-});
-
-/** A bound worker with a fake stream and port, recording every up-call. */
-async function worker(
- answer: (path: string, body: Record<string, unknown>) => unknown = () => ({})
-) {
- const links = new SessionLinks();
- const control = new WatcherControl();
- const placed: [string, string][] = [];
- control.bind({
- forget: async () => {},
- place: async (sessionId, roomId) => {
- placed.push([sessionId, roomId]);
- return { sessionId, roomId, displaced: null, previous: null };
- },
- });
- const context: ControlContext = {
- agentId: 'agent',
- links,
- ensure: async () => ({ created: false }),
- watcher: control,
- transfers: new AttachmentTransfers(root),
- };
- const calls: { path: string; body: Record<string, unknown> }[] = [];
- const operated: string[] = [];
- const port: HostedPort = {
- serial: (work) => work(),
- reasons: () => [],
- sessions: async () => ({ total: 0, live: 0, parked: 0, failed: 0, active: 0 }),
- wake: async () => {},
- cancel: async () => {},
- operate: async (operation) => {
- operated.push(`${operation.action}:${operation.sessionId}`);
- },
- revoke: async () => {},
- restart: async () => {},
- acks: {
- recordAck: async () => {},
- unconfirmedAcks: () => [],
- confirmAcks: async () => {},
- },
- fail: (error) => {
- throw error;
- },
- };
- const hosted = new HostedWorker(
- root,
- {
- capability: 'capability-placeholder',
- bootId: 'boot',
- instanceId: 'instance',
- stateVersion: 1,
- },
- context,
- { fetch: async () => ({ revoked: false, revision: 'r1', apply: async () => {} }) }
- );
- await hosted.open();
- const unbind = hosted.bind(
- {
- workerCall: async (path, body) => {
- calls.push({ path, body });
- if (path.endsWith('/connection/idle'))
- return { queued_operations: [], credential_revision: 'r1' };
- return answer(path, body);
- },
- },
- port
- );
- return { hosted, links, calls, placed, operated, port, unbind };
-}
-
-const idles = (calls: { path: string; body: Record<string, unknown> }[]) =>
- calls.filter((call) => call.path.endsWith('/connection/idle')).map((call) => call.body);
-
-/** Attaches, and waits for the report that closes the attach's catch-up. */
-async function attach(
- hosted: HostedWorker,
- calls: { path: string; body: Record<string, unknown> }[],
- overrides: Record<string, unknown> = {}
-): Promise<void> {
- const before = idles(calls).length;
- await hosted.frame('worker_attached', attached(overrides));
- await eventually(() => idles(calls).length > before);
-}
-
-const replies = (calls: { path: string; body: Record<string, unknown> }[], id: string) =>
- calls.filter((call) => call.path === `/agents/agent/connection/relay/${id}`).map((c) => c.body);
-
-it('advances the relay watermark only over contiguous resolved numbers, fenced gaps included', async () => {
- const relays = await RelayJournal.open(root);
- await relays.receive(2, 'b');
- await relays.resolve(2, 'taken');
- expect(relays.through).toBe(0);
- await relays.receive(1, 'a');
- expect(relays.through).toBe(0);
- await relays.resolve(1, 'refused');
- expect(relays.through).toBe(2);
- await relays.receive(5, 'e');
- // 3 and 4 never arrived; the fence says they never will.
- await relays.fence(4);
- expect(relays.through).toBe(4);
- // Received and unresolved: the watermark stops before it, fence or not.
- await relays.fence(6);
- expect(relays.through).toBe(4);
-});
-
-it('resolves what the last process received and never resolved as interrupted', async () => {
- const first = await RelayJournal.open(root);
- await first.receive(1, 'a');
- const reopened = await RelayJournal.open(root);
- expect(reopened.isResolved(1)).toBe(true);
- expect(reopened.through).toBe(1);
- expect(await readFile(join(root, 'relays.jsonl'), 'utf8')).toContain('"interrupted"');
-});
-
-it('pages a large answer from one pinned copy and expires it when idle', async () => {
- const pages = new RelayPages(root, () => null);
- const value = { rows: 'x'.repeat(PAGE_BYTES * 2) };
- const first = (await pages.pin(value, null)) as {
- snapshotId: string;
- pageCount: number;
- sha256: string;
- page: { data: string };
- };
- expect(first.pageCount).toBe(3);
- const parts = [Buffer.from(first.page.data, 'base64')];
- for (const index of [1, 2]) {
- const next = (await pages.page(first.snapshotId, index)) as { page: { data: string } };
- parts.push(Buffer.from(next.page.data, 'base64'));
- }
- const whole = Buffer.concat(parts);
- expect(createHash('sha256').update(whole).digest('hex')).toBe(first.sha256);
- expect(JSON.parse(whole.toString())).toEqual(value);
- await expect(pages.page(first.snapshotId, 3)).rejects.toMatchObject({ code: 'invalid_page' });
- await pages.sweep(Date.now() + PAGE_IDLE_MS);
- await expect(pages.page(first.snapshotId, 1)).rejects.toMatchObject({
- code: 'snapshot_expired',
- });
-});
-
-it('refuses a fifth pinned answer, and a page of a session reset since', async () => {
- let epoch = 'one';
- const pages = new RelayPages(root, () => epoch);
- const large = 'x'.repeat(PAGE_BYTES + 1);
- const snapshot = (await pages.pin(
- { throughSequence: 3, session: { epoch: 'one' }, events: large },
- 'session'
- )) as { snapshotId: string; epoch: string; throughSequence: number };
- expect(snapshot).toMatchObject({ epoch: 'one', throughSequence: 3 });
- for (let pinned = 1; pinned < 4; pinned++) await pages.pin({ large }, null);
- await expect(pages.pin({ large }, null)).rejects.toMatchObject({ code: 'snapshot_busy' });
- // A one-page answer pins nothing, so it is never refused.
- await expect(pages.pin({ small: true }, null)).resolves.toMatchObject({ pageCount: 1 });
- epoch = 'two';
- await expect(pages.page(snapshot.snapshotId, 1)).rejects.toMatchObject({
- code: 'snapshot_superseded',
- });
-});
-
-it('answers a relayed mutating message, journals it and reports it in the watermark', async () => {
- const { hosted, calls, placed } = await worker();
- await attach(hosted, calls);
- await hosted.frame('relay', {
- id: 'r1',
- deadline_ms: 30000,
- relay_seq: 1,
- message: { place: { sessionId: 's', roomId: 'room' } },
- });
- await eventually(() => replies(calls, 'r1').length === 1);
- expect(placed).toEqual([['s', 'room']]);
- expect(replies(calls, 'r1')[0]).toMatchObject({ ok: true, value: { sessionId: 's' } });
- await eventually(() => idles(calls).at(-1)!.relays_through === 1);
- expect(idles(calls).at(-1)).toMatchObject({
- busy: true,
- reasons: [{ kind: 'console_recent', session_id: null, count: 1 }],
- });
- const reported = idles(calls).length;
- vi.spyOn(Date, 'now').mockReturnValue(Date.now() + CONSOLE_RECENT_MS);
- hosted.report(true);
- await eventually(() => idles(calls).length > reported);
- expect(idles(calls).at(-1)).toMatchObject({ busy: false, reasons: [] });
- const seqs = idles(calls).map((report) => report.report_seq as number);
- expect(seqs).toEqual([...seqs].sort((a, b) => a - b));
-});
-
-it('refuses what only the watcher sends, and an unreadable message', async () => {
- const { hosted, calls } = await worker();
- await attach(hosted, calls);
- await hosted.frame('relay', {
- id: 'ensure',
- deadline_ms: 30000,
- relay_seq: null,
- message: { ensure: { config: {}, resuming: false, restart: false } },
- });
- await hosted.frame('relay', {
- id: 'room',
- deadline_ms: 30000,
- relay_seq: null,
- message: { sessionId: 's', request: { type: 'approvals' } },
- });
- await hosted.frame('relay', { id: 'junk', deadline_ms: 30000, relay_seq: null, message: 7 });
- await eventually(() => ['ensure', 'room', 'junk'].every((id) => replies(calls, id).length));
- for (const id of ['ensure', 'room', 'junk'])
- expect(replies(calls, id)[0]).toMatchObject({ ok: false, error: { code: 'refused_message' } });
-});
-
-it('abandons a relay still waiting for its host, which then never receives it', async () => {
- const { hosted, links, calls } = await worker();
- await attach(hosted, calls);
- const sessionRoot = join(paths.base, 's');
- supervisors.add(sessionRoot);
- await hosted.frame('relay', {
- id: 'r1',
- deadline_ms: 30000,
- relay_seq: 1,
- message: { sessionId: 's', request: { type: 'command', command: {}, requesterName: null } },
- });
- await new Promise((resolve) => setTimeout(resolve, 20));
- await hosted.sweep(Date.now() + RELAY_STALE_MS);
- const host = child(links, sessionRoot);
- host.emit('message', { kind: 'ready' });
- await eventually(() => replies(calls, 'r1').length === 1);
- expect(replies(calls, 'r1')[0]).toMatchObject({ ok: false, error: { code: 'relay_abandoned' } });
- expect(host.sent).toEqual([]);
- expect(await readFile(join(root, 'relays.jsonl'), 'utf8')).toContain('"abandoned"');
-});
-
-it('closes a dispatched relay its host never answered with a busy barrier', async () => {
- const { hosted, links, calls } = await worker();
- await attach(hosted, calls);
- const sessionRoot = join(paths.base, 's');
- const host = child(links, sessionRoot);
- host.emit('message', { kind: 'ready' });
- supervisors.add(sessionRoot);
- await hosted.frame('relay', {
- id: 'r1',
- deadline_ms: 30000,
- relay_seq: 1,
- message: { sessionId: 's', request: { type: 'command', command: {}, requesterName: null } },
- });
- await eventually(() => host.sent.length === 1);
- const sweeping = hosted.sweep(Date.now() + RELAY_STALE_MS);
- await eventually(() => host.sent.some((message) => message.kind === 'busyBarrier'));
- const barrier = host.sent.find((message) => message.kind === 'busyBarrier')!;
- host.emit('message', { kind: 'busy', busy: false, reasons: [], barrier: barrier.id });
- await sweeping;
- expect(await readFile(join(root, 'relays.jsonl'), 'utf8')).toContain('"barrier"');
-});
-
-it('counts a running host that has not said whether it is busy as busy', async () => {
- const { hosted, links, calls } = await worker();
- await attach(hosted, calls);
- const host = child(links, join(paths.base, 's'));
- host.emit('message', {
- kind: 'identity',
- identity: { agentId: 'agent', sessionId: 's', hostId: 'h', epoch: 'e' },
- });
- hosted.report(true);
- await eventually(
- () =>
- JSON.stringify(idles(calls).at(-1)!.reasons) ===
- JSON.stringify([{ kind: 'host_unknown', session_id: 's', count: 1 }])
- );
- host.emit('message', {
- kind: 'busy',
- busy: true,
- reasons: [{ kind: 'turn_running', count: 1 }],
- });
- await eventually(
- () =>
- JSON.stringify(idles(calls).at(-1)!.reasons) ===
- JSON.stringify([{ kind: 'turn_running', session_id: 's', count: 1 }])
- );
-});
-
-it('claims a rung operation once, runs it and posts its result', async () => {
- const { hosted, calls, operated } = await worker((path) =>
- path.endsWith('/claim')
- ? { id: 'op', session_id: 's', action: 'start', state: 'claimed', error: null }
- : {}
- );
- await attach(hosted, calls);
- await hosted.claim('op');
- await hosted.claim('op');
- expect(operated).toEqual(['start:s']);
- expect(calls.filter((call) => call.path === '/hosted/operations/op/claim')).toHaveLength(1);
- expect(calls.find((call) => call.path === '/hosted/operations/op/result')!.body).toMatchObject({
- state: 'applied',
- error: null,
- });
-});
-
-it('posts an operation interrupted by a restart as unknown, and stops re-posting on a 409', async () => {
- const first = await worker((path) =>
- path.endsWith('/claim')
- ? { id: 'op', session_id: 's', action: 'restart', state: 'claimed', error: null }
- : {}
- );
- first.port.operate = () => new Promise(() => {});
- await attach(first.hosted, first.calls);
- void first.hosted.claim('op');
- await eventually(async () =>
- (await readFile(join(root, 'operations.jsonl'), 'utf8')).includes('claimed')
- );
- first.unbind();
- const second = await worker((path) => {
- if (path.endsWith('/result')) throw new WorkerCallError(409, 'operation_not_claimable', path);
- return {};
- });
- await second.hosted.frame('worker_attached', attached());
- await eventually(async () =>
- (await readFile(join(root, 'operations.jsonl'), 'utf8')).includes('"posted"')
- );
- expect(
- second.calls.find((call) => call.path === '/hosted/operations/op/result')!.body
- ).toMatchObject({ state: 'unknown' });
- const third = await worker();
- await attach(third.hosted, third.calls);
- await new Promise((resolve) => setTimeout(resolve, 20));
- expect(third.calls.some((call) => call.path.endsWith('/result'))).toBe(false);
-});
-
-it('posts each notice once, keeps an unsent one for the next attach and drops a 404', async () => {
- let refuse: 'down' | 'missing' | null = 'down';
- const { hosted, calls } = await worker((path) => {
- if (path.endsWith('/room-notices') && refuse === 'down')
- throw new WorkerCallError(503, null, path);
- if (path.endsWith('/room-notices') && refuse === 'missing')
- throw new WorkerCallError(404, null, path);
- return {};
- });
- vi.spyOn(console, 'warn').mockImplementation(() => {});
- vi.spyOn(console, 'error').mockImplementation(() => {});
- await attach(hosted, calls);
- const notice = { roomId: 'room', messageId: 'm', threadId: null, reason: 'startup' as const };
- await hosted.notice(notice);
- await hosted.notice(notice);
- const posted = () => calls.filter((call) => call.path === '/agents/agent/room-notices');
- expect(posted()).toHaveLength(1);
- refuse = null;
- await attach(hosted, calls);
- await eventually(() => posted().length === 2);
- expect(posted()[1]!.body).toMatchObject({ room_id: 'room', message_id: 'm', reason: 'startup' });
- refuse = 'missing';
- await hosted.notice({ ...notice, reason: 'capacity' });
- await attach(hosted, calls);
- await new Promise((resolve) => setTimeout(resolve, 20));
- expect(posted()).toHaveLength(3);
-});
-
-it('uploads the cutover manifest on attach until Switch confirms it, then never again', async () => {
- let refuse = true;
- const { hosted, calls } = await worker((path) => {
- if (path.endsWith('/cutover-manifest') && refuse) throw new WorkerCallError(503, null, path);
- return {};
- });
- const manifest = {
- manifest_sha256: 'a'.repeat(64),
- items: [{ kind: 'reset_pending' as const, session_id: 'session-placeholder' }],
- };
- await writeCutoverManifest(root, manifest);
- vi.spyOn(console, 'warn').mockImplementation(() => {});
- const uploads = () => calls.filter((call) => call.path.endsWith('/connection/cutover-manifest'));
- await attach(hosted, calls);
- expect(uploads()).toHaveLength(1);
- expect(await unconfirmedCutoverManifest(root)).toEqual(manifest);
- refuse = false;
- await attach(hosted, calls);
- expect(uploads()).toHaveLength(2);
- expect(uploads()[1]!.body).toEqual(manifest);
- expect(await unconfirmedCutoverManifest(root)).toBeNull();
- await attach(hosted, calls);
- expect(uploads()).toHaveLength(2);
-});
diff --git a/console/packages/agent-providers/src/host/hosted-worker.ts b/console/packages/agent-providers/src/host/hosted-worker.ts
deleted file mode 100644
index 7e607a7a7..000000000
--- a/console/packages/agent-providers/src/host/hosted-worker.ts
+++ /dev/null
@@ -1,1191 +0,0 @@
-import { createHash, randomUUID } from 'node:crypto';
-import { mkdir, open, rm, writeFile } from 'node:fs/promises';
-import { join } from 'node:path';
-import {
- WORKER_CAPABILITY_OBSOLETE,
- WorkerCallError,
- type WorkerFrameName,
- type WorkerIdentity,
-} from '@sandboxaq/switch-agent-runtime';
-import { z } from 'zod';
-import { ControlError } from './attachment-transfers';
-import {
- type ControlContext,
- type ControlMessage,
- controlMessageSchema,
- ControlPeer,
- type ControlPush,
- handleControlMessage,
-} from './control';
-import { confirmCutoverManifest, unconfirmedCutoverManifest } from './cutover-manifest';
-import { WorkerObsoleteError } from './exit-codes';
-import { Journal } from './journal';
-import { sharedSessionRoot } from './launch';
-import {
- SessionHostFailedError,
- SessionUnavailableError,
- type SessionLinks,
-} from './session-channel';
-
-/**
- * The hosted half of a watcher: what an agent's cloud worker does on top of
- * routing room messages. It answers Console requests Switch relays to it,
- * reports whether it is idle so the launch can sleep, claims the operations
- * Switch rings it for, applies provider credentials, posts the room notices it
- * owes and acknowledges mailbox deliveries. Everything it must not lose across
- * a restart is journaled beside `assignments.jsonl`.
- */
-
-/** How long a mutating relay may stay unresolved before the watcher tries to close it. */
-export const RELAY_STALE_MS = 5 * 60_000;
-/** How often a hung host is asked again for a busy barrier. */
-const BARRIER_RETRY_MS = 60_000;
-const BARRIER_TIMEOUT_MS = 10_000;
-/** A mutating Console relay keeps the worker busy for this long. */
-export const CONSOLE_RECENT_MS = 10 * 60_000;
-/** How long a session whose host failed holds the worker awake after its notice. */
-export const FAILED_HOLD_MS = 15 * 60_000;
-const PUSH_BATCH_MS = 250;
-const PUSH_BATCH_BYTES = 64 * 1024;
-const ACKS_PER_CALL = 200;
-/** A page's slice; its base64 fits a 1 MiB relay reply. */
-export const PAGE_BYTES = 768 * 1024;
-export const PAGE_IDLE_MS = 120_000;
-const MAX_PINS = 4;
-const MAX_PINNED_BYTES = 64 * 1024 * 1024;
-const REPLY_ERROR_CHARS = 2048;
-
-const attachedSchema = z.object({
- launch_revision: z.number().int(),
- limits: z.object({ sessions_per_agent: z.number().int().positive() }),
- idle: z.object({ report_every_s: z.number().positive(), fresh_for_s: z.number().positive() }),
- credential_revision: z.string().nullable(),
- queued_operations: z.array(z.string().min(1)),
- relay_fence: z.number().int().nonnegative(),
- cancelled: z.array(
- z.object({
- room_id: z.string().min(1),
- message_id: z.string().min(1),
- reason: z.enum(['stopped', 'expired']),
- })
- ),
-});
-const relaySchema = z.object({
- id: z.string().min(1),
- deadline_ms: z.number(),
- relay_seq: z.number().int().positive().nullable(),
- message: z.unknown(),
-});
-const wakeEntrySchema = z.object({
- room_id: z.string().min(1),
- message_id: z.string().min(1),
- thread_id: z.string().nullable(),
- event: z.unknown(),
- origin: z.enum(['live', 'cutover']),
-});
-const deliverySchema = z.object({ room_id: z.string().min(1), message_id: z.string().min(1) });
-const operationSummarySchema = z.object({
- id: z.string().min(1),
- session_id: z.string().min(1),
- action: z.enum(['start', 'restart']),
-});
-const idleAnswerSchema = z.object({
- queued_operations: z.array(z.string().min(1)),
- credential_revision: z.string().nullable(),
-});
-const pushAnswerSchema = z.object({ unsubscribe: z.array(z.string()) }).partial();
-
-export type WakeEntry = z.infer<typeof wakeEntrySchema>;
-export type HostedOperation = { id: string; sessionId: string; action: 'start' | 'restart' };
-export type CancelReason = 'stopped' | 'expired';
-export type NoticeReason =
- | 'startup'
- | 'delivery'
- | 'conversation'
- | 'capacity'
- | 'auto_start_off'
- | 'stopped'
- | 'expired'
- | 'cancelled'
- | 'revoked'
- | 'upgrade';
-export type Notice = {
- roomId: string;
- messageId: string;
- threadId: string | null;
- reason: NoticeReason;
-};
-export type MailboxOutcome =
- | 'journaled'
- | 'admitted'
- | 'duplicate'
- | 'refused'
- | 'held'
- | 'cancelled';
-export type MailboxAck = {
- roomId: string;
- messageId: string;
- outcome: MailboxOutcome;
- reason?: string;
-};
-export type IdleReason = { kind: string; session_id: string | null; count: number };
-export type SessionCounts = {
- total: number;
- live: number;
- parked: number;
- failed: number;
- /** Live, starting or parked, and not stopped: what the session limit counts. */
- active: number;
-};
-
-/** An operation the watcher will not run, with the reason Switch records as its failure. */
-export class OperationRefusedError extends Error {
- constructor(message: string) {
- super(message);
- this.name = 'OperationRefusedError';
- }
-}
-
-/** Where the watcher keeps its unconfirmed mailbox acks: `assignments.jsonl`. */
-export type AckStore = {
- recordAck(ack: MailboxAck): Promise<void>;
- unconfirmedAcks(): MailboxAck[];
- confirmAcks(acks: MailboxAck[]): Promise<void>;
-};
-
-/** What the watcher lends the hosted worker. */
-export type HostedPort = {
- /** Runs `work` in the watcher's serial section, behind every routing decision before it. */
- serial: <T>(work: () => Promise<T>) => Promise<T>;
- /** Why the watcher itself is busy: messages it still has to hand a session. */
- reasons: () => IdleReason[];
- sessions: () => Promise<SessionCounts>;
- /** A mailbox row, journaled and routed like a live event. Called in `serial`. */
- wake: (entry: WakeEntry) => Promise<void>;
- /** Settle deliveries Switch cancelled. Called in `serial`. */
- cancel: (entries: { roomId: string; messageId: string; reason: CancelReason }[]) => Promise<void>;
- /** Start or restart a session; an `OperationRefusedError` is a failed operation. Called in `serial`. */
- operate: (operation: HostedOperation, limit: number) => Promise<void>;
- /** Stop every host: the provider was disconnected. */
- revoke: () => Promise<void>;
- /** Restart the idle host at this root so it starts under the current credentials. */
- restart: (root: string) => Promise<void>;
- acks: AckStore;
- fail: (error: Error) => void;
-};
-
-/** What the hosted worker needs of the stream it rides on. */
-export type WorkerStream = {
- workerCall: (path: string, body: Record<string, unknown>) => Promise<unknown>;
-};
-
-/** The provider credential as Switch holds it now, and how to put it in place. */
-export type HostedCredentials = {
- fetch: () => Promise<{ revoked: boolean; revision: string | null; apply: () => Promise<void> }>;
-};
-
-const RESOLUTIONS = [
- 'taken',
- 'refused',
- 'interrupted',
- 'not_delivered',
- 'abandoned',
- 'barrier',
-] as const;
-export type RelayResolution = (typeof RESOLUTIONS)[number];
-
-const relayRecordSchema = z.union([
- z.strictObject({
- received: z.strictObject({ seq: z.number().int().positive(), id: z.string().min(1) }),
- }),
- z.strictObject({
- resolved: z.strictObject({ seq: z.number().int().positive(), how: z.enum(RESOLUTIONS) }),
- }),
- /** Every number at or below it that was never received is `not_delivered`. */
- z.strictObject({ fence: z.number().int().nonnegative() }),
-]);
-type RelayRecord = z.infer<typeof relayRecordSchema>;
-
-/**
- * `relays.jsonl`: each mutating relay's number as it is received and as it is
- * resolved, and the reattach fences. `through` is the contiguous resolved
- * watermark the idle report carries.
- */
-export class RelayJournal {
- private readonly receivedIds = new Map<number, string>();
- private readonly resolvedSeqs = new Set<number>();
- private fenced = 0;
- private mark = 0;
-
- private constructor(private readonly journal: Journal<RelayRecord>) {
- for (const record of journal.records) this.apply(record);
- }
-
- /** Opens the journal; what the last process received and never resolved is `interrupted`. */
- static async open(root: string): Promise<RelayJournal> {
- const relays = new RelayJournal(
- await Journal.load(join(root, 'relays.jsonl'), (value) => relayRecordSchema.parse(value))
- );
- for (const seq of relays.unresolved()) await relays.resolve(seq, 'interrupted');
- return relays;
- }
-
- private apply(record: RelayRecord): void {
- if ('received' in record) this.receivedIds.set(record.received.seq, record.received.id);
- else if ('resolved' in record) this.resolvedSeqs.add(record.resolved.seq);
- else this.fenced = Math.max(this.fenced, record.fence);
- }
-
- private async append(record: RelayRecord): Promise<void> {
- await this.journal.append(record);
- this.apply(record);
- }
-
- unresolved(): number[] {
- return [...this.receivedIds.keys()].filter((seq) => !this.resolvedSeqs.has(seq));
- }
-
- isResolved(seq: number): boolean {
- return this.resolvedSeqs.has(seq);
- }
-
- async receive(seq: number, id: string): Promise<void> {
- if (this.receivedIds.has(seq)) return;
- await this.append({ received: { seq, id } });
- }
-
- async resolve(seq: number, how: RelayResolution): Promise<void> {
- if (this.resolvedSeqs.has(seq)) return;
- await this.append({ resolved: { seq, how } });
- }
-
- async fence(relaySeq: number): Promise<void> {
- if (relaySeq <= this.fenced) return;
- await this.append({ fence: relaySeq });
- }
-
- /** The largest N with every number up to N resolved, a fenced one never received counting as resolved. */
- get through(): number {
- let at = this.mark;
- for (;;) {
- const next = at + 1;
- if (this.resolvedSeqs.has(next) || (!this.receivedIds.has(next) && next <= this.fenced))
- at = next;
- else break;
- }
- this.mark = at;
- return at;
- }
-}
-
-type Pin = {
- bytes: number;
- pageCount: number;
- touchedAt: number;
- sessionId: string | null;
- epoch: string | null;
-};
-
-const pinnedSnapshotSchema = z.object({
- throughSequence: z.number().int(),
- session: z.object({ epoch: z.string() }),
-});
-
-/**
- * Relayed answers too large for one reply, serialized once and pinned under
- * `relay-pages/` so every page is a slice of the same bytes.
- */
-export class RelayPages {
- private readonly pins = new Map<string, Pin>();
-
- constructor(
- private readonly root: string,
- private readonly epochOf: (sessionId: string) => string | null
- ) {}
-
- private get directory(): string {
- return join(this.root, 'relay-pages');
- }
-
- async clear(): Promise<void> {
- this.pins.clear();
- await rm(this.directory, { recursive: true, force: true });
- }
-
- private async drop(snapshotId: string): Promise<void> {
- this.pins.delete(snapshotId);
- await rm(join(this.directory, snapshotId), { force: true });
- }
-
- async sweep(now: number): Promise<void> {
- for (const [snapshotId, pin] of this.pins)
- if (now - pin.touchedAt >= PAGE_IDLE_MS) await this.drop(snapshotId);
- }
-
- /** The first page of `value`; the rest are pinned for `page`. */
- async pin(value: unknown, sessionId: string | null): Promise<unknown> {
- await this.sweep(Date.now());
- const data = Buffer.from(JSON.stringify(value ?? null));
- const snapshot = pinnedSnapshotSchema.safeParse(value);
- const epoch = snapshot.success ? snapshot.data.session.epoch : null;
- const pageCount = Math.max(1, Math.ceil(data.byteLength / PAGE_BYTES));
- const snapshotId = randomUUID();
- if (pageCount > 1) {
- const pinned = [...this.pins.values()].reduce((total, pin) => total + pin.bytes, 0);
- if (this.pins.size >= MAX_PINS || pinned + data.byteLength > MAX_PINNED_BYTES)
- throw new ControlError(
- 'snapshot_busy',
- 'Too many paged answers are pinned; retry shortly.'
- );
- await mkdir(this.directory, { recursive: true, mode: 0o700 });
- await writeFile(join(this.directory, snapshotId), data, { mode: 0o600 });
- this.pins.set(snapshotId, {
- bytes: data.byteLength,
- pageCount,
- touchedAt: Date.now(),
- sessionId,
- epoch,
- });
- }
- return {
- snapshotId,
- epoch,
- throughSequence: snapshot.success ? snapshot.data.throughSequence : null,
- bytes: data.byteLength,
- sha256: createHash('sha256').update(data).digest('hex'),
- pageCount,
- page: { index: 0, data: data.subarray(0, PAGE_BYTES).toString('base64') },
- };
- }
-
- async page(snapshotId: string, index: number): Promise<unknown> {
- await this.sweep(Date.now());
- const pin = this.pins.get(snapshotId);
- if (!pin)
- throw new ControlError('snapshot_expired', 'That paged answer has expired; start over.');
- if (pin.sessionId !== null && pin.epoch !== null) {
- const epoch = this.epochOf(pin.sessionId);
- if (epoch !== null && epoch !== pin.epoch) {
- await this.drop(snapshotId);
- throw new ControlError(
- 'snapshot_superseded',
- 'The session was reset since this answer was taken; start over.'
- );
- }
- }
- if (index >= pin.pageCount)
- throw new ControlError('invalid_page', `That answer has ${pin.pageCount} page(s).`);
- const length = Math.min(PAGE_BYTES, pin.bytes - index * PAGE_BYTES);
- const buffer = Buffer.alloc(length);
- const file = await open(join(this.directory, snapshotId), 'r');
- try {
- await file.read(buffer, 0, length, index * PAGE_BYTES);
- } finally {
- await file.close();
- }
- pin.touchedAt = Date.now();
- return { snapshotId, page: { index, data: buffer.toString('base64') } };
- }
-}
-
-const operationRecordSchema = z.union([
- z.strictObject({
- claimed: z.strictObject({
- id: z.string().min(1),
- sessionId: z.string().min(1),
- action: z.enum(['start', 'restart']),
- }),
- }),
- z.strictObject({
- outcome: z.strictObject({
- id: z.string().min(1),
- state: z.enum(['applied', 'failed', 'unknown']),
- error: z.string().nullable(),
- }),
- }),
- z.strictObject({ posted: z.string().min(1) }),
-]);
-type OperationRecord = z.infer<typeof operationRecordSchema>;
-type OperationOutcome = {
- id: string;
- state: 'applied' | 'failed' | 'unknown';
- error: string | null;
-};
-
-const noticeRecordSchema = z.union([
- z.strictObject({
- notice: z.strictObject({
- roomId: z.string().min(1),
- messageId: z.string().min(1),
- threadId: z.string().nullable(),
- reason: z.enum([
- 'startup',
- 'delivery',
- 'conversation',
- 'capacity',
- 'auto_start_off',
- 'stopped',
- 'expired',
- 'cancelled',
- 'revoked',
- 'upgrade',
- ]),
- }),
- }),
- z.strictObject({ sent: z.string().min(1) }),
-]);
-type NoticeRecord = z.infer<typeof noticeRecordSchema>;
-
-const noticeKey = (notice: Notice): string =>
- JSON.stringify([notice.roomId, notice.messageId, notice.reason]);
-
-type Inflight = {
- id: string;
- seq: number;
- sessionId: string | null;
- root: string | null;
- receivedAt: number;
- dispatched: boolean;
- cancelled: boolean;
- warned: boolean;
- nextBarrierAt: number;
-};
-
-/** A handler cancelled before it reached the host. */
-class RelayAbandonedError extends Error {
- constructor() {
- super('The relay was cancelled before it reached the session host.');
- this.name = 'RelayAbandonedError';
- }
-}
-
-function sessionOf(message: ControlMessage): string | null {
- if ('sessionId' in message) return message.sessionId;
- if ('place' in message) return message.place.sessionId;
- if ('forget' in message) return message.forget;
- if ('attachment' in message) return message.attachment.sessionId;
- if ('journal' in message) return message.journal;
- return null;
-}
-
-function isPaged(message: ControlMessage): boolean {
- return (
- ('request' in message && message.request.type === 'snapshot') ||
- 'journal' in message ||
- 'list' in message
- );
-}
-
-function describe(error: unknown): string {
- return error instanceof Error ? error.message : String(error);
-}
-
-/**
- * One per watcher process. `open` reads its journals; `bind` attaches it to
- * the watcher's stream and serial section for as long as the watcher runs.
- */
-export class HostedWorker {
- private relays!: RelayJournal;
- private operations!: Journal<OperationRecord>;
- private notices!: Journal<NoticeRecord>;
- private readonly pages: RelayPages;
- private readonly peer: ControlPeer;
- private readonly inflight = new Map<string, Inflight>();
- private readonly hungHosts = new Set<string>();
- private readonly claiming = new Set<string>();
- private readonly staleHosts = new Set<string>();
- private stream: WorkerStream | null = null;
- private port: HostedPort | null = null;
- private attachedState: z.infer<typeof attachedSchema> | null = null;
- private lastMutatingAt = Number.NEGATIVE_INFINITY;
- private reportSeq = 0;
- private lastReport: { key: string; at: number } | null = null;
- private reporting: Promise<void> = Promise.resolve();
- private reportSoon: NodeJS.Timeout | null = null;
- private reportEvery: NodeJS.Timeout | null = null;
- private sweeper: NodeJS.Timeout | null = null;
- private pushSeqs = new Map<string, number>();
- private pushQueue: Record<string, unknown>[] = [];
- private pushBytes = 0;
- private pushTimer: NodeJS.Timeout | null = null;
- private pushing: Promise<void> = Promise.resolve();
- private acking: Promise<void> = Promise.resolve();
- private credentialState: { revision: string | null; revoked: boolean } | null = null;
- private fetchingCredential: Promise<void> | null = null;
- /** Set once the first `worker_attached` has been reconciled: the watcher may admit from its journal. */
- reconciled = false;
-
- constructor(
- private readonly root: string,
- readonly identity: WorkerIdentity,
- private readonly context: ControlContext,
- private readonly credentials: HostedCredentials
- ) {
- this.pages = new RelayPages(root, (sessionId) => this.epochOf(sessionId));
- this.peer = new ControlPeer((push) => this.enqueuePush(push));
- }
-
- private get links(): SessionLinks {
- return this.context.links;
- }
-
- private get agentId(): string {
- return this.context.agentId;
- }
-
- async open(): Promise<void> {
- this.relays = await RelayJournal.open(this.root);
- this.operations = await Journal.load(join(this.root, 'operations.jsonl'), (value) =>
- operationRecordSchema.parse(value)
- );
- this.notices = await Journal.load(join(this.root, 'notices.jsonl'), (value) =>
- noticeRecordSchema.parse(value)
- );
- for (const claimed of this.claimedWithoutOutcome())
- await this.operations.append({
- outcome: {
- id: claimed,
- state: 'unknown',
- error:
- 'The worker restarted before it could confirm the operation. Inspect the session before retrying.',
- },
- });
- await this.pages.clear();
- }
-
- /** The session limit Switch set, or null before the worker has attached. */
- get limit(): number | null {
- return this.attachedState?.limits.sessions_per_agent ?? null;
- }
-
- /** Whether the owner's provider connection is gone: no session may start. */
- get revoked(): boolean {
- return this.credentialState?.revoked === true;
- }
-
- bind(stream: WorkerStream, port: HostedPort): () => void {
- this.stream = stream;
- this.port = port;
- const offBusy = this.links.onBusy(() => {
- this.restartStale();
- this.changed();
- });
- const offExit = this.links.onExit((root) => {
- this.hungHosts.delete(root);
- this.staleHosts.delete(root);
- for (const relay of this.inflight.values())
- if (relay.root === root && relay.dispatched) void this.settle(relay, 'interrupted');
- });
- this.sweeper = setInterval(() => {
- void this.sweep(Date.now()).catch((error: unknown) => this.background(error, 'sweep'));
- }, 60_000);
- this.sweeper.unref();
- return () => {
- offBusy();
- offExit();
- this.peer.close();
- for (const timer of [this.sweeper, this.reportEvery, this.reportSoon, this.pushTimer])
- if (timer) clearTimeout(timer);
- this.sweeper = this.reportEvery = this.reportSoon = this.pushTimer = null;
- this.stream = null;
- this.port = null;
- };
- }
-
- private call(path: string, body: Record<string, unknown>): Promise<unknown> {
- if (!this.stream) throw new Error('The hosted worker is not bound to a stream.');
- return this.stream.workerCall(path, body);
- }
-
- /** Logs a background failure, and stops the worker for good on an obsolete capability. */
- private background(error: unknown, what: string): void {
- if (error instanceof WorkerCallError && error.code === WORKER_CAPABILITY_OBSOLETE) {
- this.port?.fail(new WorkerObsoleteError(error.message));
- return;
- }
- console.warn(`Hosted worker ${what} failed: ${describe(error)}`);
- }
-
- /** One protocol-7 frame, in the order the stream delivered it. */
- async frame(name: WorkerFrameName, data: Record<string, unknown>): Promise<void> {
- const port = this.port;
- if (!port) throw new Error('The hosted worker received a frame before it was bound.');
- switch (name) {
- case 'worker_attached':
- return this.attach(attachedSchema.parse(data));
- case 'relay':
- return this.relay(relaySchema.parse(data));
- case 'relay_cancel': {
- const { id } = z.object({ id: z.string().min(1) }).parse(data);
- const relay = this.inflight.get(id);
- if (relay && !relay.dispatched) {
- relay.cancelled = true;
- await this.settle(relay, 'abandoned');
- }
- return;
- }
- case 'wake': {
- const { entries } = z.object({ entries: z.array(wakeEntrySchema).max(50) }).parse(data);
- for (const entry of entries) await port.serial(() => port.wake(entry));
- this.changed();
- return;
- }
- case 'mailbox_cancel': {
- const { entries } = z.object({ entries: z.array(deliverySchema) }).parse(data);
- await port.serial(() =>
- port.cancel(
- entries.map((entry) => ({
- roomId: entry.room_id,
- messageId: entry.message_id,
- reason: 'stopped',
- }))
- )
- );
- return;
- }
- case 'operation': {
- const { id } = z.object({ id: z.string().min(1) }).parse(data);
- void this.claim(id);
- return;
- }
- case 'credential': {
- const { revision } = z.object({ revision: z.string().nullable() }).parse(data);
- if (revision !== this.credentialState?.revision || revision === null)
- void this.refreshCredential();
- return;
- }
- }
- }
-
- private async attach(attached: z.infer<typeof attachedSchema>): Promise<void> {
- const port = this.port!;
- this.attachedState = attached;
- await this.pages.clear();
- await this.context.transfers.clear();
- this.pushSeqs = new Map();
- this.pushQueue = [];
- this.pushBytes = 0;
- await this.relays.fence(attached.relay_fence);
- await port.serial(() =>
- port.cancel(
- attached.cancelled.map((entry) => ({
- roomId: entry.room_id,
- messageId: entry.message_id,
- reason: entry.reason,
- }))
- )
- );
- this.reconciled = true;
- if (this.reportEvery) clearInterval(this.reportEvery);
- this.reportEvery = setInterval(() => this.report(true), attached.idle.report_every_s * 1000);
- this.reportEvery.unref();
- this.lastReport = null;
- void this.catchUp(attached.queued_operations);
- }
-
- /** What a reattach owes Switch: claims, the credential, and every unconfirmed notice, result and ack. */
- private async catchUp(queued: string[]): Promise<void> {
- await this.refreshCredential();
- await this.uploadCutoverManifest();
- for (const id of queued) await this.claim(id);
- await this.postResults();
- await this.sendNotices();
- await this.flushAcks();
- this.report(true);
- }
-
- /** Hands Switch what this volume held before the cutover, until Switch confirms it once. */
- private async uploadCutoverManifest(): Promise<void> {
- const manifest = await unconfirmedCutoverManifest(this.root);
- if (!manifest) return;
- try {
- await this.call(`/agents/${this.agentId}/connection/cutover-manifest`, manifest);
- } catch (error) {
- this.background(error, 'cutover manifest upload; it is retried on reattach');
- return;
- }
- await confirmCutoverManifest(this.root, manifest.manifest_sha256);
- }
-
- private async relay(frame: z.infer<typeof relaySchema>): Promise<void> {
- const parsed = controlMessageSchema.safeParse(frame.message);
- const message = parsed.success ? (parsed.data as ControlMessage) : null;
- const sessionId = message ? sessionOf(message) : null;
- let relay: Inflight | null = null;
- if (frame.relay_seq !== null) {
- await this.relays.receive(frame.relay_seq, frame.id);
- this.lastMutatingAt = Date.now();
- relay = {
- id: frame.id,
- seq: frame.relay_seq,
- sessionId,
- root: null,
- receivedAt: Date.now(),
- dispatched: false,
- cancelled: false,
- warned: false,
- nextBarrierAt: 0,
- };
- this.inflight.set(frame.id, relay);
- this.changed();
- }
- void this.answer(frame.id, message, relay);
- }
-
- private async answer(
- id: string,
- message: ControlMessage | null,
- relay: Inflight | null
- ): Promise<void> {
- let reply: Record<string, unknown>;
- try {
- const value = await this.handle(message, relay);
- reply = { ok: true, value: value ?? null };
- if (relay) await this.settle(relay, 'taken');
- } catch (error) {
- reply = {
- ok: false,
- error: {
- code: this.codeOf(error),
- message: describe(error).slice(0, REPLY_ERROR_CHARS),
- },
- };
- if (relay) await this.settleFailure(relay, error);
- }
- try {
- await this.call(`/agents/${this.agentId}/connection/relay/${id}`, reply);
- } catch (error) {
- if (error instanceof WorkerCallError && (error.status === 404 || error.status === 409))
- return;
- this.background(error, `reply to relay ${id}`);
- }
- }
-
- private async handle(message: ControlMessage | null, relay: Inflight | null): Promise<unknown> {
- if (!message) throw new ControlError('refused_message', 'The relayed message is unreadable.');
- if (
- 'ensure' in message ||
- ('request' in message && ['room', 'approvals'].includes(message.request.type))
- )
- throw new ControlError('refused_message', 'Only the watcher sends that message.');
- if ('page' in message) return this.pages.page(message.page.snapshotId, message.page.index);
- if (relay && 'request' in message) relay.root = sharedSessionRoot(message.sessionId);
- else if (relay) relay.dispatched = true;
- const value = await handleControlMessage(this.context, this.peer, message, () => {
- if (!relay) return;
- if (relay.cancelled) throw new RelayAbandonedError();
- relay.dispatched = true;
- });
- return isPaged(message) ? this.pages.pin(value, sessionOf(message)) : value;
- }
-
- private codeOf(error: unknown): string {
- if (error instanceof ControlError) return error.code;
- if (error instanceof SessionHostFailedError) return 'session_failed';
- if (error instanceof SessionUnavailableError) return 'session_unavailable';
- if (error instanceof RelayAbandonedError) return 'relay_abandoned';
- return 'failed';
- }
-
- private async settleFailure(relay: Inflight, error: unknown): Promise<void> {
- if (error instanceof RelayAbandonedError) return this.settle(relay, 'abandoned');
- if (!relay.dispatched) return this.settle(relay, 'refused');
- if (error instanceof SessionHostFailedError) return this.settle(relay, 'interrupted');
- if (error instanceof SessionUnavailableError) {
- // Sent and not answered: the host may still act on it, unless it is gone.
- if (relay.root && this.links.ready(relay.root)) return;
- return this.settle(relay, 'interrupted');
- }
- return this.settle(relay, 'refused');
- }
-
- private async settle(relay: Inflight, how: RelayResolution): Promise<void> {
- this.inflight.delete(relay.id);
- await this.relays.resolve(relay.seq, how);
- this.changed();
- }
-
- /** Closes relays unresolved past `RELAY_STALE_MS` where that can be done safely. */
- async sweep(now: number): Promise<void> {
- await this.pages.sweep(now);
- for (const relay of [...this.inflight.values()]) {
- if (now - relay.receivedAt < RELAY_STALE_MS) continue;
- if (!relay.warned) {
- relay.warned = true;
- console.warn(
- `Console relay ${relay.id} (#${relay.seq}) has been unresolved for five minutes; closing it.`
- );
- }
- if (!relay.dispatched) {
- relay.cancelled = true;
- await this.settle(relay, 'abandoned');
- continue;
- }
- const root = relay.root;
- if (!root || !this.links.ready(root)) {
- await this.settle(relay, 'interrupted');
- continue;
- }
- if (now < relay.nextBarrierAt) continue;
- relay.nextBarrierAt = now + BARRIER_RETRY_MS;
- try {
- await this.links.barrier(root, BARRIER_TIMEOUT_MS);
- this.hungHosts.delete(root);
- await this.settle(relay, 'barrier');
- } catch (error) {
- if (!this.links.ready(root)) {
- this.hungHosts.delete(root);
- await this.settle(relay, 'interrupted');
- continue;
- }
- if (!this.hungHosts.has(root))
- console.warn(
- `The session host at ${root} did not answer a busy barrier (${describe(error)}); it counts as busy until it does or it stops.`
- );
- this.hungHosts.add(root);
- this.changed();
- }
- }
- }
-
- private enqueuePush(push: ControlPush): void {
- const subscription = 'health' in push ? 'health' : push.sessionId;
- const seq = (this.pushSeqs.get(subscription) ?? 0) + 1;
- this.pushSeqs.set(subscription, seq);
- const entry: Record<string, unknown> =
- 'health' in push
- ? { subscription, seq, health: push.health }
- : 'event' in push
- ? { subscription, seq, event: push.event }
- : { subscription, seq, failure: push.failure };
- this.pushQueue.push(entry);
- this.pushBytes += JSON.stringify(entry).length;
- if (this.pushBytes >= PUSH_BATCH_BYTES) this.flushPushes();
- else this.pushTimer ??= setTimeout(() => this.flushPushes(), PUSH_BATCH_MS);
- }
-
- private flushPushes(): void {
- if (this.pushTimer) clearTimeout(this.pushTimer);
- this.pushTimer = null;
- const pushes = this.pushQueue;
- this.pushQueue = [];
- this.pushBytes = 0;
- if (!pushes.length) return;
- this.pushing = this.pushing.then(async () => {
- try {
- const answer = pushAnswerSchema.parse(
- await this.call(`/agents/${this.agentId}/connection/relay/push`, { pushes })
- );
- for (const subscription of answer.unsubscribe ?? []) {
- if (subscription === 'health') {
- this.peer.unwatchHealth?.();
- this.peer.unwatchHealth = null;
- } else {
- this.peer.subscriptions.get(subscription)?.();
- this.peer.subscriptions.delete(subscription);
- }
- }
- } catch (error) {
- // Switch sees the gap in `seq` and has its views resynchronize.
- this.background(error, `push of ${pushes.length} live update(s)`);
- }
- });
- }
-
- /** Something that goes into the idle report may have changed. */
- changed(): void {
- if (!this.attachedState || this.reportSoon) return;
- this.reportSoon = setTimeout(() => {
- this.reportSoon = null;
- this.report(false);
- }, 100);
- this.reportSoon.unref();
- }
-
- private reasons(port: HostedPort, now: number): IdleReason[] {
- const reasons = new Map<string, IdleReason>();
- const add = (kind: string, sessionId: string | null, count: number) => {
- const key = JSON.stringify([kind, sessionId]);
- const existing = reasons.get(key);
- if (existing) existing.count += count;
- else reasons.set(key, { kind, session_id: sessionId, count });
- };
- for (const root of this.links.live()) {
- const sessionId = this.links.identity(root)?.sessionId ?? null;
- const busy = this.links.busy(root);
- if (busy === null || this.hungHosts.has(root)) add('host_unknown', sessionId, 1);
- else for (const reason of busy.reasons) add(reason.kind, sessionId, reason.count);
- }
- for (const reason of port.reasons()) add(reason.kind, reason.session_id, reason.count);
- for (const relay of this.inflight.values()) add('relay_inflight', relay.sessionId, 1);
- for (const claimed of this.claimedWithoutOutcome()) {
- const record = this.operations.records.find(
- (entry) => 'claimed' in entry && entry.claimed.id === claimed
- );
- add('operation_claimed', record && 'claimed' in record ? record.claimed.sessionId : null, 1);
- }
- if (now - this.lastMutatingAt < CONSOLE_RECENT_MS) add('console_recent', null, 1);
- return [...reasons.values()];
- }
-
- /** Sends the idle report if it changed, or regardless when `force`d; one at a time. */
- report(force: boolean): void {
- this.reporting = this.reporting.then(async () => {
- const port = this.port;
- if (!port || !this.attachedState) return;
- try {
- const now = Date.now();
- const reasons = this.reasons(port, now);
- const { active: _active, ...sessions } = await port.sessions();
- const body = {
- relays_through: this.relays.through,
- busy: reasons.length > 0,
- reasons,
- sessions,
- };
- const key = JSON.stringify(body);
- if (!force && this.lastReport?.key === key) return;
- this.reportSeq += 1;
- const answer = idleAnswerSchema.parse(
- await this.call(`/agents/${this.agentId}/connection/idle`, {
- report_seq: this.reportSeq,
- ...body,
- })
- );
- this.lastReport = { key, at: now };
- for (const id of answer.queued_operations) void this.claim(id);
- if (answer.credential_revision !== (this.credentialState?.revision ?? null))
- void this.refreshCredential();
- void this.flushAcks();
- } catch (error) {
- this.background(error, 'idle report');
- }
- });
- }
-
- private claimedWithoutOutcome(): string[] {
- const finished = new Set(
- this.operations.records.flatMap((record) => ('outcome' in record ? [record.outcome.id] : []))
- );
- return this.operations.records.flatMap((record) =>
- 'claimed' in record && !finished.has(record.claimed.id) ? [record.claimed.id] : []
- );
- }
-
- /** Claims the operation Switch rang for and runs it, unless it is already this worker's. */
- async claim(id: string): Promise<void> {
- const port = this.port;
- if (!port) return;
- if (this.claiming.has(id)) return;
- if (this.operations.records.some((record) => 'claimed' in record && record.claimed.id === id))
- return;
- this.claiming.add(id);
- try {
- let summary: z.infer<typeof operationSummarySchema>;
- try {
- summary = operationSummarySchema.parse(
- await this.call(`/hosted/operations/${id}/claim`, {})
- );
- } catch (error) {
- if (error instanceof WorkerCallError && error.status === 409) return;
- throw error;
- }
- const operation: HostedOperation = {
- id,
- sessionId: summary.session_id,
- action: summary.action,
- };
- await this.operations.append({ claimed: operation });
- this.changed();
- let outcome: OperationOutcome;
- try {
- await port.serial(() => port.operate(operation, this.limit ?? Number.POSITIVE_INFINITY));
- outcome = { id, state: 'applied', error: null };
- } catch (error) {
- if (error instanceof OperationRefusedError)
- outcome = { id, state: 'failed', error: error.message.slice(0, 512) };
- else {
- console.error(`Operation ${id} (${operation.action}) failed: ${describe(error)}`);
- outcome = {
- id,
- state: 'unknown',
- error:
- 'The worker could not confirm session startup. Inspect its transcript before retrying.',
- };
- }
- }
- await this.operations.append({ outcome });
- this.changed();
- await this.postResult(outcome);
- } catch (error) {
- this.background(error, `operation ${id}`);
- } finally {
- this.claiming.delete(id);
- }
- }
-
- private async postResult(outcome: OperationOutcome): Promise<void> {
- try {
- await this.call(`/hosted/operations/${outcome.id}/result`, {
- state: outcome.state,
- error: outcome.error,
- });
- } catch (error) {
- if (!(error instanceof WorkerCallError && error.status === 409)) throw error;
- console.warn(
- `Switch no longer takes the result of operation ${outcome.id} (${error.message}); it is not posted again.`
- );
- }
- await this.operations.append({ posted: outcome.id });
- }
-
- private async postResults(): Promise<void> {
- const posted = new Set(
- this.operations.records.flatMap((record) => ('posted' in record ? [record.posted] : []))
- );
- for (const record of this.operations.records)
- if ('outcome' in record && !posted.has(record.outcome.id)) {
- try {
- await this.postResult(record.outcome);
- } catch (error) {
- this.background(error, `result of operation ${record.outcome.id}`);
- }
- }
- }
-
- /** Tells the room why its message was not processed: once per message and reason. */
- async notice(notice: Notice): Promise<void> {
- const key = noticeKey(notice);
- if (
- this.notices.records.some((record) => 'notice' in record && noticeKey(record.notice) === key)
- )
- return;
- await this.notices.append({ notice });
- await this.sendNotice(notice);
- }
-
- private async sendNotice(notice: Notice): Promise<void> {
- try {
- await this.call(`/agents/${this.agentId}/room-notices`, {
- room_id: notice.roomId,
- message_id: notice.messageId,
- thread_id: notice.threadId,
- reason: notice.reason,
- });
- } catch (error) {
- if (!(error instanceof WorkerCallError && error.status === 404)) {
- this.background(error, `notice to room ${notice.roomId}; it is retried on reattach`);
- return;
- }
- console.error(
- `Switch cannot post the ${notice.reason} notice for message ${notice.messageId} in room ${notice.roomId} (${error.message}); it is not retried.`
- );
- }
- await this.notices.append({ sent: noticeKey(notice) });
- }
-
- private async sendNotices(): Promise<void> {
- const sent = new Set(
- this.notices.records.flatMap((record) => ('sent' in record ? [record.sent] : []))
- );
- for (const record of this.notices.records)
- if ('notice' in record && !sent.has(noticeKey(record.notice)))
- await this.sendNotice(record.notice);
- }
-
- /** Records a mailbox ack and sends it with any others still unconfirmed. */
- async ack(ack: MailboxAck): Promise<void> {
- const port = this.port;
- if (!port) throw new Error('The hosted worker is not bound.');
- await port.acks.recordAck(ack);
- void this.flushAcks();
- }
-
- private flushAcks(): Promise<void> {
- this.acking = this.acking.then(async () => {
- const port = this.port;
- if (!port || !this.attachedState) return;
- for (;;) {
- const batch = port.acks.unconfirmedAcks().slice(0, ACKS_PER_CALL);
- if (!batch.length) return;
- try {
- await this.call(`/agents/${this.agentId}/connection/mailbox/ack`, {
- entries: batch.map((ack) => ({
- room_id: ack.roomId,
- message_id: ack.messageId,
- outcome: ack.outcome,
- ...(ack.reason === undefined ? {} : { reason: ack.reason }),
- })),
- });
- } catch (error) {
- this.background(
- error,
- `mailbox ack of ${batch.length} entr(ies); retried with the next report`
- );
- return;
- }
- await port.acks.confirmAcks(batch);
- }
- });
- return this.acking;
- }
-
- /** Fetches the provider credential and applies it if it changed; one fetch at a time. */
- refreshCredential(): Promise<void> {
- this.fetchingCredential ??= (async () => {
- const port = this.port;
- if (!port) return;
- try {
- const fetched = await this.credentials.fetch();
- const previous = this.credentialState;
- if (fetched.revoked) {
- if (previous?.revoked) return;
- this.credentialState = { revision: null, revoked: true };
- console.error(
- "The owner's provider connection was removed; stopping every session until it is reconnected."
- );
- await port.revoke();
- this.changed();
- return;
- }
- if (previous && !previous.revoked && previous.revision === fetched.revision) return;
- await fetched.apply();
- this.credentialState = { revision: fetched.revision, revoked: false };
- // The first fetch confirms what the bootstrap put in place before any session started.
- if (!previous) return;
- console.warn(
- 'The provider credential changed; each running session restarts under it once it is idle.'
- );
- for (const root of this.links.live()) this.staleHosts.add(root);
- this.restartStale();
- } catch (error) {
- this.background(error, 'provider credential refresh');
- }
- })().finally(() => {
- this.fetchingCredential = null;
- });
- return this.fetchingCredential;
- }
-
- private restartStale(): void {
- const port = this.port;
- if (!port) return;
- for (const root of [...this.staleHosts]) {
- if (!this.links.ready(root)) {
- this.staleHosts.delete(root);
- continue;
- }
- if (this.links.busy(root)?.busy !== false) continue;
- this.staleHosts.delete(root);
- void port
- .restart(root)
- .catch((error: unknown) =>
- this.background(error, `restart of the session host at ${root}`)
- );
- }
- }
-
- private epochOf(sessionId: string): string | null {
- return this.links.identity(sharedSessionRoot(sessionId))?.epoch ?? null;
- }
-}
diff --git a/console/packages/agent-providers/src/host/hub-stream.ts b/console/packages/agent-providers/src/host/hub-stream.ts
index d66d6dce5..aef82fa32 100644
--- a/console/packages/agent-providers/src/host/hub-stream.ts
+++ b/console/packages/agent-providers/src/host/hub-stream.ts
@@ -116,12 +116,6 @@ class HubEventStream implements AgentEventStream {
if (error !== undefined) throw new Error(error);
}
- workerCall(): Promise<unknown> {
- return Promise.reject(
- new Error('An agent run by an agents controller is not a hosted worker.')
- );
- }
-
private async run(): Promise<void> {
const { signal, log } = this.deps;
let backoff = INITIAL_BACKOFF_MS;
diff --git a/console/packages/agent-providers/src/host/launch.ts b/console/packages/agent-providers/src/host/launch.ts
index 4655751ce..d39bed82a 100644
--- a/console/packages/agent-providers/src/host/launch.ts
+++ b/console/packages/agent-providers/src/host/launch.ts
@@ -128,7 +128,6 @@ export function inProcessSupervision(
signal: stop.signal,
build: entrypoint,
links,
- logRedactions: [],
})
.catch((error: unknown) => {
console.error(`Session host at ${root} stopped: ${String(error)}`);
diff --git a/console/packages/agent-providers/src/host/ownership-lock.ts b/console/packages/agent-providers/src/host/ownership-lock.ts
index 0ea70dce2..3e05e4d94 100644
--- a/console/packages/agent-providers/src/host/ownership-lock.ts
+++ b/console/packages/agent-providers/src/host/ownership-lock.ts
@@ -20,9 +20,17 @@ function alive(pid: number): boolean {
}
}
+/**
+ * `fileMode(0o600)` from `host-permissions`, written out: this module takes
+ * no relative import, since its tests load it on its own in a child process.
+ */
+function recordMode(): number {
+ return process.env.SWITCH_HOST_SHARED_GROUP === '1' ? 0o640 : 0o600;
+}
+
async function save(path: string, value: unknown, replace: boolean): Promise<void> {
const temporary = `${path}.${randomUUID()}.tmp`;
- const file = await open(temporary, 'wx', 0o600);
+ const file = await open(temporary, 'wx', recordMode());
try {
await file.writeFile(JSON.stringify(value));
await file.sync();
diff --git a/console/packages/agent-providers/src/host/provider-login.test.ts b/console/packages/agent-providers/src/host/provider-login.test.ts
new file mode 100644
index 000000000..fdb31366b
--- /dev/null
+++ b/console/packages/agent-providers/src/host/provider-login.test.ts
@@ -0,0 +1,58 @@
+import { mkdtemp, rm, writeFile } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { afterEach, beforeEach, describe, expect, it } from 'vitest';
+import { providerLoginEnvironment } from './provider-login';
+import { readProviderLogin, type SharedHostConfig } from './shared-config';
+
+const login = {
+ status: 'connected' as const,
+ provider: 'claude' as const,
+ revision: '1',
+ kind: 'setup-token' as const,
+ credential: 'sk-ant-oat-given',
+};
+
+let dir: string;
+beforeEach(async () => {
+ dir = await mkdtemp(join(tmpdir(), 'provider-login-'));
+});
+afterEach(async () => {
+ await rm(dir, { recursive: true, force: true });
+});
+
+function config(provider: string): SharedHostConfig {
+ return {
+ start: { provider },
+ session: { agentId: 'agent-1' },
+ execution: { credentialsPath: join(dir, 'credentials.json') },
+ } as unknown as SharedHostConfig;
+}
+
+describe('provider logins handed to an agent host', () => {
+ it('reads the login its controller hands over with its credentials', async () => {
+ await writeFile(
+ join(dir, 'credentials.json'),
+ JSON.stringify({ env: {}, providerLogin: login })
+ );
+ expect(await readProviderLogin(config('claude'))).toEqual(login);
+ await expect(readProviderLogin(config('codex'))).rejects.toThrow(/another provider/);
+ await writeFile(join(dir, 'credentials.json'), JSON.stringify({ env: {} }));
+ expect(await readProviderLogin(config('claude'))).toBeNull();
+ });
+
+ it('gives sessions the token, or the directory the login file is written in', () => {
+ expect(providerLoginEnvironment('/root', login)).toEqual({
+ CLAUDE_CODE_OAUTH_TOKEN: 'sk-ant-oat-given',
+ });
+ expect(providerLoginEnvironment('/root', { ...login, kind: 'api-key' })).toEqual({
+ ANTHROPIC_API_KEY: 'sk-ant-oat-given',
+ });
+ expect(
+ providerLoginEnvironment('/root', { ...login, provider: 'codex', kind: 'auth-json' })
+ ).toEqual({ CODEX_HOME: '/root/provider-home' });
+ expect(
+ providerLoginEnvironment('/root', { ...login, provider: 'opencode', kind: 'auth-json' })
+ ).toEqual({ XDG_DATA_HOME: '/root/provider-data' });
+ });
+});
diff --git a/console/packages/agent-providers/src/host/provider-login.ts b/console/packages/agent-providers/src/host/provider-login.ts
new file mode 100644
index 000000000..42df97ca2
--- /dev/null
+++ b/console/packages/agent-providers/src/host/provider-login.ts
@@ -0,0 +1,42 @@
+import { join } from 'node:path';
+import { z } from 'zod';
+
+/** A provider login Switch holds for an agent, as an agent host applies it (`materializeHostedProvider`). */
+export const hostedCredentialSchema = z.discriminatedUnion('status', [
+ z.object({ status: z.literal('revoked') }),
+ z.object({
+ status: z.literal('connected'),
+ revision: z.string(),
+ provider: z.enum(['claude', 'codex', 'cursor', 'opencode', 'antigravity']),
+ kind: z.enum(['api-key', 'setup-token', 'auth-json']),
+ credential: z.string().min(1).max(16384),
+ }),
+]);
+export type HostedCredential = z.infer<typeof hostedCredentialSchema>;
+
+/**
+ * The environment the sessions of an agent host rooted at `root` run with to
+ * use `credential`: the provider's token variable, or the directory
+ * `materializeHostedProvider` writes its login file into under `root`. What
+ * an agents controller puts in the agent's configuration, so the sessions
+ * get it whatever environment they would otherwise inherit.
+ */
+export function providerLoginEnvironment(
+ root: string,
+ credential: Extract<HostedCredential, { status: 'connected' }>
+): Record<string, string> {
+ switch (credential.provider) {
+ case 'claude':
+ return credential.kind === 'api-key'
+ ? { ANTHROPIC_API_KEY: credential.credential }
+ : { CLAUDE_CODE_OAUTH_TOKEN: credential.credential };
+ case 'cursor':
+ return { CURSOR_API_KEY: credential.credential };
+ case 'codex':
+ return { CODEX_HOME: join(root, 'provider-home') };
+ case 'opencode':
+ return { XDG_DATA_HOME: join(root, 'provider-data') };
+ case 'antigravity':
+ return { GEMINI_HOME: join(root, 'provider-home') };
+ }
+}
diff --git a/console/packages/agent-providers/src/host/room-prompt.test.ts b/console/packages/agent-providers/src/host/room-prompt.test.ts
index 5d00df00d..5466e14b8 100644
--- a/console/packages/agent-providers/src/host/room-prompt.test.ts
+++ b/console/packages/agent-providers/src/host/room-prompt.test.ts
@@ -1,5 +1,5 @@
import { describe, expect, it } from 'vitest';
-import { cutoverCommandId, roomCommand, roomCommandId, roomMessageSchema } from './room-prompt';
+import { roomCommand, roomCommandId, roomMessageSchema } from './room-prompt';
describe('a room command', () => {
const event = {
@@ -21,10 +21,4 @@ describe('a room command', () => {
it('is named by the room message it answers', () => {
expect(command(event).commandId).toBe(roomCommandId('agent', '!room:example.test', '$m'));
});
-
- it('is named apart from the live delivery when imported at the cutover', () => {
- const imported = command({ ...event, cutover: true }).commandId;
- expect(imported).toBe(cutoverCommandId('agent', '!room:example.test', '$m'));
- expect(imported).not.toBe(roomCommandId('agent', '!room:example.test', '$m'));
- });
});
diff --git a/console/packages/agent-providers/src/host/room-prompt.ts b/console/packages/agent-providers/src/host/room-prompt.ts
index 114c53476..5aaa4bff8 100644
--- a/console/packages/agent-providers/src/host/room-prompt.ts
+++ b/console/packages/agent-providers/src/host/room-prompt.ts
@@ -35,8 +35,6 @@ export const roomMessageSchema = z.object({
missed: z
.object({ count: z.number().int().nonnegative().nullable(), reason: z.string().nullable() })
.nullish(),
- /** Carried over from the session-table worker at the cutover, and named apart from a live delivery. */
- cutover: z.literal(true).optional(),
});
export type RoomMessage = z.infer<typeof roomMessageSchema>;
@@ -53,15 +51,6 @@ export function roomCommandId(agentId: string, roomId: string, messageId: string
return uuidFrom(`switch-room:${agentId}:${roomId}:${messageId}`);
}
-/**
- * The command id of a room message imported at the cutover. A session the
- * old worker left may already hold the live id for it, as a command it
- * accepted and will report unknown rather than run again.
- */
-export function cutoverCommandId(agentId: string, roomId: string, messageId: string): string {
- return uuidFrom(`switch-room-cutover:${agentId}:${roomId}:${messageId}`);
-}
-
/** The fresh start a room message triggers on a conversation that cannot continue: once per message. */
export function roomFreshStartCommandId(
agentId: string,
@@ -174,11 +163,7 @@ export function roomCommand(input: {
unreadNotice(input.message.missed);
return {
contractVersion: 1,
- commandId: (input.message.cutover ? cutoverCommandId : roomCommandId)(
- input.agentId,
- input.roomId,
- payload.message_id
- ),
+ commandId: roomCommandId(input.agentId, input.roomId, payload.message_id),
sessionId: input.sessionId,
epoch: input.epoch,
origin: {
diff --git a/console/packages/agent-providers/src/host/shared-config.ts b/console/packages/agent-providers/src/host/shared-config.ts
index b34d5cc4a..9de330304 100644
--- a/console/packages/agent-providers/src/host/shared-config.ts
+++ b/console/packages/agent-providers/src/host/shared-config.ts
@@ -8,6 +8,7 @@ import { sessionSchema } from '@switch-console/shared/session-v1';
import { z } from 'zod';
import type { HttpMcpServerSpec } from '../adapter';
import { prepareCodexSessionHome } from '../codex/home';
+import { type HostedCredential, hostedCredentialSchema } from './provider-login';
import { roomConnectionSchema } from './room-inbox';
import { startSchema } from './server';
@@ -99,7 +100,6 @@ export async function prepareSharedConfig(
sessionId: config.session.sessionId,
sourceHome: input.env.CODEX_HOME || join(homedir(), '.codex'),
config: execution.codexConfig,
- auth: process.env.SWITCH_HOSTED_BOOTSTRAP === '1' ? 'refresh' : 'copy-once',
});
input.systemContext = execution.context;
}
@@ -109,6 +109,24 @@ export async function prepareSharedConfig(
return { agentApiUrl, token, input };
}
+/**
+ * The provider login an agents controller hands this agent host with its
+ * credentials, when the machine has none of its own for the provider and
+ * Switch gave it one; null when it hands none.
+ */
+export async function readProviderLogin(
+ config: SharedHostConfig
+): Promise<HostedCredential | null> {
+ if (!config.execution) return null;
+ const { providerLogin } = z
+ .object({ providerLogin: hostedCredentialSchema.nullable().optional() })
+ .parse(JSON.parse(await readFile(config.execution.credentialsPath, 'utf8')));
+ if (!providerLogin) return null;
+ if (providerLogin.status === 'connected' && providerLogin.provider !== config.start.provider)
+ throw new Error('The provider login handed over is for another provider than this agent’s.');
+ return providerLogin;
+}
+
export async function readSharedCredentials(config: SharedHostConfig) {
if (!config.execution) throw new Error('Shared execution credentials are required.');
const credentials = z
diff --git a/console/packages/agent-providers/src/host/shared-daemon.ts b/console/packages/agent-providers/src/host/shared-daemon.ts
index 190e66ab7..e82b45e41 100644
--- a/console/packages/agent-providers/src/host/shared-daemon.ts
+++ b/console/packages/agent-providers/src/host/shared-daemon.ts
@@ -5,8 +5,8 @@ import { join, resolve } from 'node:path';
import { openSwitchStream, runAgentHost } from './agent-host';
import { AttachmentTransfers } from './attachment-transfers';
import { type ControlContext, ensureSessions, serveControl } from './control';
-import { OBSOLETE_BUNDLE_EXIT_CODE, WorkerObsoleteError } from './exit-codes';
-import { hostedWorker } from './hosted-watcher';
+import { dirMode } from './host-permissions';
+import { materializeHostedProvider } from './hosted-provider';
import { openHubStream } from './hub-stream';
import { detachedSupervision, ensureSharedProcess, inProcessSupervision } from './launch';
import { replaceOwner } from './ownership-lock';
@@ -14,7 +14,7 @@ import { ownProcessGroup } from './process-fence';
import { checkProviderReadiness } from './provider-readiness';
import { adapterFor } from './server';
import { HOST_EXIT_GRACE_MS, SessionLinks } from './session-channel';
-import { readSharedCredentials, sharedConfigSchema } from './shared-config';
+import { readProviderLogin, readSharedCredentials, sharedConfigSchema } from './shared-config';
import { hostSessionProcess } from './shared-host';
import { superviseSharedHost } from './supervisor';
import { recordWatcherHealth } from './watcher-health-file';
@@ -113,7 +113,6 @@ async function main(): Promise<void> {
signal: stop.signal,
build: process.argv[1]!,
links: null,
- logRedactions: [],
});
} else if (mode === '--watch-worker') {
const stop = new AbortController();
@@ -135,7 +134,17 @@ async function main(): Promise<void> {
watcher: control,
transfers,
};
- const hosted = await hostedWorker(config, resolve(root), context);
+ // A login Switch gave the machine for this agent's provider: its files
+ // are written here, as this agent's own; its environment is already in
+ // the configuration the controller wrote.
+ const login = await readProviderLogin(config);
+ if (login?.status === 'connected')
+ await materializeHostedProvider(
+ resolve(root),
+ {},
+ login,
+ config.execution?.binaryPath ?? config.start.provider
+ );
// An agents controller running this agent host in a process of its own
// names its hub: the agent's events come from there, not from Switch.
const hub = config.execution
@@ -156,7 +165,6 @@ async function main(): Promise<void> {
stop.signal,
supervision,
control,
- hosted,
hub ? openHubStream(hub) : openSwitchStream
).finally(() => stop.abort()),
serveControl(resolve(root), context, stop.signal),
@@ -234,23 +242,17 @@ async function main(): Promise<void> {
try {
await main();
} catch (error) {
- if (error instanceof WorkerObsoleteError) {
- // Not a failure of this bundle's to record: the worker service waits for a current one.
- console.error(error.message);
- process.exitCode = OBSOLETE_BUNDLE_EXIT_CODE;
- } else {
- if (
- root !== '--probe' &&
- root !== '--models' &&
- mode !== '--supervise' &&
- mode !== '--watch-supervise'
- ) {
- await mkdir(join(root, 'supervisor'), { recursive: true, mode: 0o700 });
- await replaceOwner(join(root, 'supervisor', 'failure.json'), {
- message: error instanceof Error ? error.message : String(error),
- });
- }
- console.error(error);
- process.exitCode = 1;
+ if (
+ root !== '--probe' &&
+ root !== '--models' &&
+ mode !== '--supervise' &&
+ mode !== '--watch-supervise'
+ ) {
+ await mkdir(join(root, 'supervisor'), { recursive: true, mode: dirMode(0o700) });
+ await replaceOwner(join(root, 'supervisor', 'failure.json'), {
+ message: error instanceof Error ? error.message : String(error),
+ });
}
+ console.error(error);
+ process.exitCode = 1;
}
diff --git a/console/packages/agent-providers/src/host/shared-host.test.ts b/console/packages/agent-providers/src/host/shared-host.test.ts
index ef483cb4e..1743329a0 100644
--- a/console/packages/agent-providers/src/host/shared-host.test.ts
+++ b/console/packages/agent-providers/src/host/shared-host.test.ts
@@ -1002,14 +1002,14 @@ it('says a session is busy for each thing that keeps it from parking, and idle o
busy: false,
reasons: [],
});
- // Subagents still at work after the turn ended keep it from parking, and its worker awake.
+ // Subagents still at work after the turn ended keep it from parking.
expect(sessionBusy(snapshot({ status: 'ready' }, ['completed'], []), 'none', 0, true)).toEqual({
busy: true,
reasons: [{ kind: 'turn_running', count: 1 }],
});
});
-it('stops holding its worker awake for a reset decision after the bound, and keeps the decision', async () => {
+it('stops holding its host busy for a reset decision after the bound, and keeps the decision', async () => {
const lastBusy = (host: Harness) =>
host.parent.sent.filter((m) => m.kind === 'busy').at(-1) as unknown as
| { busy: boolean; reasons: { kind: string; count: number }[] }
diff --git a/console/packages/agent-providers/src/host/shared-host.ts b/console/packages/agent-providers/src/host/shared-host.ts
index f64bb6b02..110da854c 100644
--- a/console/packages/agent-providers/src/host/shared-host.ts
+++ b/console/packages/agent-providers/src/host/shared-host.ts
@@ -93,9 +93,9 @@ export function parkAfterMs(): number | null {
}
/**
- * How long a session waiting on a fresh-conversation decision keeps its worker
- * awake. Past it the decision and the room messages it holds stay recorded,
- * but no longer count as busy, so the worker may sleep.
+ * How long a session waiting on a fresh-conversation decision keeps its host
+ * busy. Past it the decision and the room messages it holds stay recorded,
+ * but no longer count as busy, so the host may park.
*/
export const RESET_HOLD_MS = 15 * 60_000;
@@ -103,9 +103,8 @@ export const RESET_HOLD_MS = 15 * 60_000;
export type ResetHold = 'none' | 'holding' | 'ended';
/**
- * Why a session is busy, if it is: what keeps a hosted worker awake and what
- * keeps a host from parking. A session that is stopped or failed with nothing
- * left to decide is not busy. Work the provider is still doing after its turn
+ * Why a session is busy, if it is: what keeps a host from parking. A session
+ * that is stopped or failed with nothing left to decide is not busy. Work the provider is still doing after its turn
* ended, such as background subagents, counts as a running turn.
*/
export function sessionBusy(
@@ -546,7 +545,7 @@ export async function runSharedHost(
if (key === announcedBusy) return;
if (resetHold() === 'ended')
console.warn(
- `Session ${options.session.sessionId} has waited ${RESET_HOLD_MS / 60_000} min for a fresh-conversation decision; it no longer keeps its worker awake. The decision and its ${rooms?.pending().length ?? 0} held room message(s) are kept for when it next runs.`
+ `Session ${options.session.sessionId} has waited ${RESET_HOLD_MS / 60_000} min for a fresh-conversation decision; it no longer keeps its host busy. The decision and its ${rooms?.pending().length ?? 0} held room message(s) are kept for when it next runs.`
);
announcedBusy = key;
options.parent?.busy(state, null);
diff --git a/console/packages/agent-providers/src/host/supervisor.test.ts b/console/packages/agent-providers/src/host/supervisor.test.ts
index ccc8b2c6b..097d443b6 100644
--- a/console/packages/agent-providers/src/host/supervisor.test.ts
+++ b/console/packages/agent-providers/src/host/supervisor.test.ts
@@ -2,7 +2,6 @@ import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, expect, it } from 'vitest';
-import { OBSOLETE_BUNDLE_EXIT_CODE, WorkerObsoleteError } from './exit-codes';
import { CHILD_STOP_GRACE_MS, superviseSharedHost } from './supervisor';
const roots: string[] = [];
@@ -36,7 +35,6 @@ it('restarts a crashed isolated worker and exits after a clean stop', async () =
signal: new AbortController().signal,
build: 'test-bundle.mjs',
links: null,
- logRedactions: [],
});
expect(await readFile(join(root, 'attempts'), 'utf8')).toBe('2');
await expect(readFile(join(root, 'supervisor', 'owner.json'))).rejects.toMatchObject({
@@ -55,7 +53,6 @@ it('reports a fatal worker failure instead of restarting it repeatedly', async (
signal: new AbortController().signal,
build: 'test-bundle.mjs',
links: null,
- logRedactions: [],
})
).rejects.toThrow('exit code 1');
expect(
@@ -80,7 +77,6 @@ it('reaps provider descendants even after a clean worker exit', async () => {
signal: new AbortController().signal,
build: 'test-bundle.mjs',
links: null,
- logRedactions: [],
});
const pid = Number(await readFile(join(root, 'provider.pid'), 'utf8'));
expect(() => process.kill(pid, 0)).toThrow();
@@ -102,7 +98,6 @@ it('preserves the worker failure reason for Console startup', async () => {
signal: new AbortController().signal,
build: 'test-bundle.mjs',
links: null,
- logRedactions: [],
})
).rejects.toThrow('exit code 1');
expect(JSON.parse(await readFile(join(root, 'supervisor', 'failure.json'), 'utf8')).message).toBe(
@@ -136,27 +131,6 @@ async function leaseWorker(root: string, firstExit: number): Promise<string[]> {
return [path, root];
}
-it('stops without relaunching or recording a failure when the worker is obsolete', async () => {
- const root = await fixture();
- await expect(
- superviseSharedHost({
- root,
- executable: process.execPath,
- args: await leaseWorker(root, OBSOLETE_BUNDLE_EXIT_CODE),
- env: process.env,
- signal: new AbortController().signal,
- build: 'test-bundle.mjs',
- links: null,
- logRedactions: [],
- })
- ).rejects.toBeInstanceOf(WorkerObsoleteError);
- expect(await readFile(join(root, 'attempts'), 'utf8')).toBe('1');
- await expect(readFile(join(root, 'supervisor', 'failure.json'))).rejects.toMatchObject({
- code: 'ENOENT',
- });
- await expect(readFile(join(root, 'shared-owner.lock'))).rejects.toMatchObject({ code: 'ENOENT' });
-});
-
it('does not relaunch a worker that exits with a fatal code', async () => {
const root = await fixture();
await expect(
@@ -168,7 +142,6 @@ it('does not relaunch a worker that exits with a fatal code', async () => {
signal: new AbortController().signal,
build: 'test-bundle.mjs',
links: null,
- logRedactions: [],
})
).rejects.toThrow('exit code 1');
expect(await readFile(join(root, 'attempts'), 'utf8')).toBe('1');
@@ -177,32 +150,6 @@ it('does not relaunch a worker that exits with a fatal code', async () => {
).toContain('worker.log');
});
-it('scrubs known secrets from the worker output before it reaches worker.log', async () => {
- const root = await fixture();
- const script = `
- process.stdout.write('token=synthetic-');
- setTimeout(() => {
- process.stdout.write('secret-value done\\n');
- process.stderr.write('failed with synthetic-secret-value\\n');
- }, 20);
- `;
- await superviseSharedHost({
- root,
- executable: process.execPath,
- args: ['-e', script],
- env: process.env,
- signal: new AbortController().signal,
- build: 'test-bundle.mjs',
- links: null,
- logRedactions: ['synthetic-secret-value'],
- });
- const log = await readFile(join(root, 'supervisor', 'worker.log'), 'utf8');
- expect(log).not.toContain('synthetic-secret-value');
- expect(log.match(/\[REDACTED\]/g)).toHaveLength(2);
- expect(log).toContain('token=[REDACTED]');
- expect(log).toContain('failed with [REDACTED]');
-});
-
it('kills a worker that will not stop when asked, and everything it started', async () => {
// A watcher waiting on a session host that hung up and stayed alive never
// exits on SIGTERM, and a replacement waiting for it used to give up.
@@ -228,7 +175,6 @@ it('kills a worker that will not stop when asked, and everything it started', as
signal: abort.signal,
build: 'test-bundle.mjs',
links: null,
- logRedactions: [],
});
let pid = 0;
for (let attempt = 0; attempt < 100 && !pid; attempt++) {
diff --git a/console/packages/agent-providers/src/host/supervisor.ts b/console/packages/agent-providers/src/host/supervisor.ts
index d36526fb7..ea2072aa1 100644
--- a/console/packages/agent-providers/src/host/supervisor.ts
+++ b/console/packages/agent-providers/src/host/supervisor.ts
@@ -5,8 +5,6 @@ import { mkdir, open, readFile } from 'node:fs/promises';
import { join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
import { z } from 'zod';
-import { OBSOLETE_BUNDLE_EXIT_CODE, WorkerObsoleteError } from './exit-codes';
-import { pipeRedactedHostedLogs } from './hosted-log';
import { releaseOwner, replaceOwner, withOwnershipLock } from './ownership-lock';
import { fenceDeadOwner, killProcessTree } from './process-fence';
import type { SessionLinks } from './session-channel';
@@ -55,8 +53,6 @@ export async function superviseSharedHost(input: {
* process does, and the host is then started without one.
*/
links: SessionLinks | null;
- /** Exact values scrubbed from the host's output before it reaches worker.log. */
- logRedactions: string[];
}): Promise<void> {
const directory = join(input.root, 'supervisor');
await mkdir(directory, { recursive: true, mode: 0o700 });
@@ -77,24 +73,14 @@ export async function superviseSharedHost(input: {
continue;
}
const log = await open(join(directory, 'worker.log'), 'a', 0o600);
- const redacting = input.logRedactions.length > 0;
- const output = redacting ? 'pipe' : log.fd;
const child = spawn(input.executable, input.args, {
detached: true,
env: input.env,
- stdio: input.links ? ['ignore', output, output, 'ipc'] : ['ignore', output, output],
+ stdio: input.links ? ['ignore', log.fd, log.fd, 'ipc'] : ['ignore', log.fd, log.fd],
});
input.links?.attach(input.root, child);
const exited = once(child, 'exit');
- const logged = redacting
- ? pipeRedactedHostedLogs([child.stdout!, child.stderr!], log, input.logRedactions).catch(
- (error: unknown) => {
- child.kill('SIGKILL');
- throw error;
- }
- )
- : log.close();
- logged.catch(() => {});
+ log.close().catch(() => {});
// Asked first. A host that has not gone `CHILD_STOP_GRACE_MS` later is
// waiting on something that will not finish — a session host that hung
// up and stayed alive, say — and is killed with everything it started,
@@ -133,16 +119,8 @@ export async function superviseSharedHost(input: {
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
}
}
- if (redacting)
- try {
- await logged;
- } finally {
- await log.close();
- }
if (input.signal.aborted) return;
if (code === 0) return;
- if (code === OBSOLETE_BUNDLE_EXIT_CODE)
- throw new WorkerObsoleteError('the watcher was refused as obsolete and stopped.');
if (code !== null) {
try {
await readFile(join(directory, 'failure.json'));
diff --git a/console/packages/agent-providers/src/host/taken-over.ts b/console/packages/agent-providers/src/host/taken-over.ts
index 2f54c4e46..20c38d422 100644
--- a/console/packages/agent-providers/src/host/taken-over.ts
+++ b/console/packages/agent-providers/src/host/taken-over.ts
@@ -2,6 +2,7 @@ import { randomUUID } from 'node:crypto';
import { open, readFile, rename, unlink } from 'node:fs/promises';
import { join } from 'node:path';
import { z } from 'zod';
+import { fileMode } from './host-permissions';
/**
* The record a watcher leaves behind when something else took its connection.
@@ -30,7 +31,7 @@ const FILE = 'taken-over.json';
export async function recordTakenOver(root: string, info: TakenOver): Promise<void> {
const path = join(root, FILE);
const temporary = `${path}.${randomUUID()}.tmp`;
- const file = await open(temporary, 'wx', 0o600);
+ const file = await open(temporary, 'wx', fileMode(0o600));
try {
await file.writeFile(JSON.stringify(takenOverSchema.parse(info)));
await file.sync();
diff --git a/console/packages/agent-providers/src/host/verify-credential.test.ts b/console/packages/agent-providers/src/host/verify-credential.test.ts
deleted file mode 100644
index 94b49f908..000000000
--- a/console/packages/agent-providers/src/host/verify-credential.test.ts
+++ /dev/null
@@ -1,78 +0,0 @@
-import { afterEach, expect, it, vi } from 'vitest';
-import type { ProviderAdapter, ProviderSessionStartInput } from '../adapter';
-import type { ProviderRuntimeEvent } from '../events';
-import { verifyModelTurn } from './verify-credential';
-
-const input: ProviderSessionStartInput = {
- sessionId: 'check',
- cwd: '/tmp/check',
- runtimeMode: 'approval-required',
- env: {},
- mcpServers: {},
-};
-
-function adapter(reply: string | null, outcome = 'completed') {
- let listener: (event: ProviderRuntimeEvent) => void;
- const unsubscribe = vi.fn();
- const stopAll = vi.fn().mockResolvedValue(undefined);
- const value = {
- subscribe: vi.fn((callback) => {
- listener = callback;
- return unsubscribe;
- }),
- startSession: vi.fn().mockResolvedValue({ sessionId: input.sessionId }),
- sendTurn: vi.fn(async ({ turnId }) => {
- if (reply !== null) {
- listener({
- type: 'content.delta',
- sessionId: input.sessionId,
- turnId,
- delta: reply,
- } as ProviderRuntimeEvent);
- listener({
- type: 'turn.completed',
- sessionId: input.sessionId,
- turnId,
- outcome,
- } as ProviderRuntimeEvent);
- }
- return { turnId };
- }),
- stopAll,
- } as unknown as ProviderAdapter;
- return { value, stopAll, unsubscribe };
-}
-
-afterEach(() => vi.useRealTimers());
-
-it('requires a real completed reply and always stops the adapter', async () => {
- const fake = adapter('SWITCH_CONNECTION_OK');
- await verifyModelTurn(fake.value, input);
- expect(fake.stopAll).toHaveBeenCalledOnce();
- expect(fake.unsubscribe).toHaveBeenCalledOnce();
-});
-
-it.each([
- ['', 'completed'],
- ['SWITCH_CONNECTION_OK', 'failed'],
-])('rejects an unsuccessful check (%s, %s)', async (reply, outcome) => {
- const fake = adapter(reply, outcome);
- await expect(verifyModelTurn(fake.value, input)).rejects.toThrow('did not complete');
- expect(fake.stopAll).toHaveBeenCalledOnce();
-});
-
-it('stops a hung provider after the deadline', async () => {
- vi.useFakeTimers();
- const fake = adapter(null);
- const check = expect(verifyModelTurn(fake.value, input)).rejects.toThrow('timed out');
- await vi.advanceTimersByTimeAsync(90001);
- await check;
- expect(fake.stopAll).toHaveBeenCalledOnce();
-});
-
-it('cleans up when starting the session fails', async () => {
- const fake = adapter(null);
- vi.mocked(fake.value.startSession).mockRejectedValue(new Error('Unauthorized'));
- await expect(verifyModelTurn(fake.value, input)).rejects.toThrow('Unauthorized');
- expect(fake.stopAll).toHaveBeenCalledOnce();
-});
diff --git a/console/packages/agent-providers/src/host/verify-credential.ts b/console/packages/agent-providers/src/host/verify-credential.ts
deleted file mode 100644
index bc02993f6..000000000
--- a/console/packages/agent-providers/src/host/verify-credential.ts
+++ /dev/null
@@ -1,144 +0,0 @@
-import { randomUUID } from 'node:crypto';
-import { mkdir, readFile } from 'node:fs/promises';
-import { join } from 'node:path';
-import { z } from 'zod';
-import type { ProviderAdapter, ProviderSessionStartInput } from '../adapter';
-import { materializeHostedProvider } from './hosted-provider';
-import { exportOpenCodeConsole } from './opencode-console';
-import { checkProviderReadiness } from './provider-readiness';
-import { adapterFor } from './server';
-
-const inputSchema = z.object({
- provider: z.enum(['claude', 'codex', 'cursor', 'opencode', 'antigravity']),
- kind: z.enum(['api-key', 'setup-token', 'auth-json']),
- credential: z.string().min(1).max(16384),
- binaryPath: z.string().startsWith('/opt/switch/'),
- home: z.literal('/run/switch-verification'),
-});
-
-export async function verifyModelTurn(
- adapter: ProviderAdapter,
- input: ProviderSessionStartInput
-): Promise<void> {
- const turnId = randomUUID();
- const marker = 'SWITCH_CONNECTION_OK';
- let text = '';
- let resolveResult: () => void;
- let rejectResult: (error: Error) => void;
- const result = new Promise<void>((resolve, reject) => {
- resolveResult = resolve;
- rejectResult = reject;
- });
- // The session may fail before sendTurn returns; keep the rejection observed.
- void result.catch(() => {});
- const timeout = setTimeout(() => rejectResult(new Error('Connection check timed out.')), 90000);
- const unsubscribe = adapter.subscribe((event) => {
- if (event.sessionId !== input.sessionId) return;
- if (
- event.type === 'request.opened' ||
- event.type === 'user-input.requested' ||
- event.type === 'session.exited'
- ) {
- rejectResult(new Error('Connection check could not complete.'));
- }
- if (event.turnId !== turnId) return;
- if (event.type === 'content.delta') text += event.delta;
- if (event.type === 'item.completed' && event.item.type === 'assistant_message' && !text)
- text = event.item.text ?? '';
- if (event.type === 'turn.completed') {
- if (event.outcome === 'completed' && text.includes(marker)) resolveResult();
- else rejectResult(new Error('The provider did not complete the connection check.'));
- }
- });
- try {
- await Promise.race([
- adapter.startSession(input).then(async () => {
- await adapter.sendTurn({
- sessionId: input.sessionId,
- turnId,
- text: `Reply with exactly ${marker}. Do not use tools, read files or run commands.`,
- });
- await result;
- }),
- result,
- ]);
- } finally {
- clearTimeout(timeout);
- unsubscribe();
- await adapter.stopAll();
- }
-}
-
-export async function runCredentialVerification(): Promise<void> {
- try {
- const chunks: Buffer[] = [];
- let size = 0;
- for await (const chunk of process.stdin) {
- size += chunk.length;
- if (size > 20000) throw new Error('Verification input too large.');
- chunks.push(chunk);
- }
- const input = inputSchema.parse(JSON.parse(Buffer.concat(chunks).toString('utf8')));
- const env: Record<string, string> = {
- HOME: input.home,
- PATH: process.env.PATH ?? '/usr/bin:/bin',
- };
- await materializeHostedProvider(
- input.home,
- env,
- {
- status: 'connected',
- provider: input.provider,
- kind: input.kind,
- credential: input.credential,
- revision: randomUUID(),
- },
- input.binaryPath
- );
- const cwd = join(input.home, 'workspace');
- await mkdir(cwd, { recursive: true, mode: 0o700 });
- let model: { id: string } | undefined;
- if (input.provider === 'opencode') {
- const parsed = JSON.parse(input.credential);
- const providers =
- parsed.format === 'switch-opencode-console-v1' ? ['opencode'] : Object.keys(parsed);
- const readiness = await checkProviderReadiness({
- provider: input.provider,
- binaryPath: input.binaryPath,
- cwd,
- env,
- });
- const candidate = readiness.models.find((entry) =>
- providers.some((provider) => entry.id.startsWith(provider + '/'))
- );
- if (!candidate) throw new Error('No model is available for the configured provider.');
- model = { id: candidate.id };
- }
- await verifyModelTurn(adapterFor(input.provider, input.binaryPath, ''), {
- sessionId: randomUUID(),
- cwd,
- runtimeMode: 'approval-required',
- env,
- mcpServers: {},
- model,
- });
- let credential = input.credential;
- if (
- input.provider === 'opencode' &&
- JSON.parse(input.credential).format === 'switch-opencode-console-v1'
- ) {
- credential = exportOpenCodeConsole(env.XDG_DATA_HOME);
- } else if (input.kind === 'auth-json') {
- const authPath =
- input.provider === 'codex'
- ? join(env.CODEX_HOME, 'auth.json')
- : input.provider === 'opencode'
- ? join(env.XDG_DATA_HOME, 'opencode/auth.json')
- : join(env.GEMINI_HOME, 'antigravity-acp/acp_token.json');
- credential = await readFile(authPath, 'utf8');
- }
- process.stdout.write(JSON.stringify({ succeeded: true, credential }) + '\n');
- } catch {
- process.stdout.write(JSON.stringify({ succeeded: false }) + '\n');
- }
-}
diff --git a/console/packages/agent-providers/src/host/watcher-health-file.ts b/console/packages/agent-providers/src/host/watcher-health-file.ts
index bc92e179b..55a37dce4 100644
--- a/console/packages/agent-providers/src/host/watcher-health-file.ts
+++ b/console/packages/agent-providers/src/host/watcher-health-file.ts
@@ -2,6 +2,7 @@ import { randomUUID } from 'node:crypto';
import { rename, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { z } from 'zod';
+import { fileMode } from './host-permissions';
import { type WatcherControl, watcherHealthSchema } from './watcher-tools';
/**
@@ -46,7 +47,7 @@ export function recordWatcherHealth(root: string, control: WatcherControl): () =
writing = writing
.then(async () => {
const temporary = `${path}.${randomUUID()}`;
- await writeFile(temporary, JSON.stringify(body), { mode: 0o600 });
+ await writeFile(temporary, JSON.stringify(body), { mode: fileMode(0o600) });
await rename(temporary, path);
})
.catch((error: unknown) => {
diff --git a/console/packages/agent-providers/src/host/worker-capability.ts b/console/packages/agent-providers/src/host/worker-capability.ts
deleted file mode 100644
index f77ebd694..000000000
--- a/console/packages/agent-providers/src/host/worker-capability.ts
+++ /dev/null
@@ -1,33 +0,0 @@
-import { randomUUID } from 'node:crypto';
-import { open, readFile, rename, rm } from 'node:fs/promises';
-import { join } from 'node:path';
-
-/**
- * The hosted worker's capability, written by the bootstrap on every boot and
- * read by the watcher when it attaches. It never enters an environment or a log.
- */
-export const WORKER_CAPABILITY_FILE = 'worker-capability';
-
-export async function writeWorkerCapability(root: string, capability: string): Promise<void> {
- if (!/^[\x21-\x7e]{16,4096}$/.test(capability))
- throw new Error('The worker capability is malformed.');
- const target = join(root, WORKER_CAPABILITY_FILE);
- const temporary = `${target}.${randomUUID()}.tmp`;
- const handle = await open(temporary, 'wx', 0o600);
- try {
- await handle.writeFile(capability);
- await handle.sync();
- } catch (error) {
- await handle.close();
- await rm(temporary, { force: true });
- throw error;
- }
- await handle.close();
- await rename(temporary, target);
-}
-
-export async function readWorkerCapability(root: string): Promise<string> {
- const capability = (await readFile(join(root, WORKER_CAPABILITY_FILE), 'utf8')).trim();
- if (!capability) throw new Error('The worker capability file is empty.');
- return capability;
-}
diff --git a/console/packages/agent-providers/src/index.ts b/console/packages/agent-providers/src/index.ts
index 0e9cfb689..683b5c03b 100644
--- a/console/packages/agent-providers/src/index.ts
+++ b/console/packages/agent-providers/src/index.ts
@@ -89,7 +89,26 @@ export {
} from './host/watch-flags';
export { superviseSharedHost } from './host/supervisor';
export { controllerConnectionId } from './host/connection-id';
-export { EXECUTION_INHERIT_ENV } from './host/agent-env';
+export { AGENT_ENV_VARS, EXECUTION_INHERIT_ENV } from './host/agent-env';
+export { SHARED_GROUP_ENV } from './host/host-permissions';
+export {
+ type HostedCredential,
+ hostedCredentialSchema,
+ providerLoginEnvironment,
+} from './host/provider-login';
+export { materializeHostedProvider } from './host/hosted-provider';
+export {
+ generateSealingKeyPair,
+ openProviderLogin,
+ type ProviderLogin,
+ providerLoginSchema,
+ SEALED_LOGIN_ALG,
+ type SealedLogin,
+ sealedLoginSchema,
+ sealingKeyId,
+ sealProviderLogin,
+ type SealingKeyPair,
+} from './sealed-login';
export {
SessionHostFailedError,
SessionLinks,
@@ -111,13 +130,6 @@ export type { SharedHostConfig } from './host/shared-config';
export { providerReadinessSchema } from './host/provider-readiness';
export type { ProviderReadiness } from './host/provider-readiness';
export { CONTROL_FILE, ControlClient, SidecarConnectionClosedError } from './host/control';
-export {
- CloudRelayClient,
- CloudRelayClosedError,
- CloudRelayError,
- RELAY_TIMEOUT_MS,
- type RelayFetch,
-} from './host/cloud-relay-client';
export { hostSessions, hostSessionsByAgent, LIST_SCRIPT } from './host/session-list';
export { JournalUnavailableError, replayJournal } from './host/journal-snapshot';
export { openCodeConsoleCredentialSchema } from './opencode/console-credential';
diff --git a/console/packages/agent-providers/src/sealed-login.test.ts b/console/packages/agent-providers/src/sealed-login.test.ts
new file mode 100644
index 000000000..a6d020f58
--- /dev/null
+++ b/console/packages/agent-providers/src/sealed-login.test.ts
@@ -0,0 +1,62 @@
+import { describe, expect, it } from 'vitest';
+import {
+ generateSealingKeyPair,
+ openProviderLogin,
+ sealedLoginSchema,
+ sealingKeyId,
+ sealProviderLogin,
+} from './sealed-login';
+
+const LOGIN = { kind: 'setup-token' as const, credential: 'sk-ant-oat-placeholder' };
+
+describe('sealed provider logins', () => {
+ it('opens only for the controller and provider it was sealed for', () => {
+ const keys = generateSealingKeyPair();
+ const sealed = sealProviderLogin({
+ publicKey: keys.publicKey,
+ controllerId: 'controller-1',
+ provider: 'claude',
+ login: LOGIN,
+ });
+ expect(sealedLoginSchema.parse(sealed).key_id).toBe(sealingKeyId(keys.publicKey));
+ expect(sealed.ciphertext).not.toContain('sk-ant');
+ expect(
+ openProviderLogin({ keys, controllerId: 'controller-1', provider: 'claude', sealed })
+ ).toEqual(LOGIN);
+ expect(() =>
+ openProviderLogin({ keys, controllerId: 'controller-2', provider: 'claude', sealed })
+ ).toThrow(/does not open/);
+ expect(() =>
+ openProviderLogin({ keys, controllerId: 'controller-1', provider: 'codex', sealed })
+ ).toThrow(/does not open/);
+ });
+
+ it('refuses another key and a tampered envelope', () => {
+ const keys = generateSealingKeyPair();
+ const other = generateSealingKeyPair();
+ const sealed = sealProviderLogin({
+ publicKey: keys.publicKey,
+ controllerId: 'controller-1',
+ provider: 'claude',
+ login: LOGIN,
+ });
+ expect(() =>
+ openProviderLogin({ keys: other, controllerId: 'controller-1', provider: 'claude', sealed })
+ ).toThrow(/another key/);
+ const body = Buffer.from(sealed.ciphertext, 'base64');
+ body[0] = body[0]! ^ 1;
+ expect(() =>
+ openProviderLogin({
+ keys,
+ controllerId: 'controller-1',
+ provider: 'claude',
+ sealed: { ...sealed, ciphertext: body.toString('base64') },
+ })
+ ).toThrow(/does not open/);
+ });
+
+ it('names a key by the same id Core computes', () => {
+ // sha256 of 32 zero bytes, as core/switch_core/management/sealed_logins.key_id gives it.
+ expect(sealingKeyId(Buffer.alloc(32).toString('base64'))).toBe('66687aadf862bd77');
+ });
+});
diff --git a/console/packages/agent-providers/src/sealed-login.ts b/console/packages/agent-providers/src/sealed-login.ts
new file mode 100644
index 000000000..6d41936fb
--- /dev/null
+++ b/console/packages/agent-providers/src/sealed-login.ts
@@ -0,0 +1,177 @@
+import {
+ createCipheriv,
+ createDecipheriv,
+ createHash,
+ createPrivateKey,
+ createPublicKey,
+ diffieHellman,
+ generateKeyPairSync,
+ hkdfSync,
+ type KeyObject,
+ randomBytes,
+} from 'node:crypto';
+import { z } from 'zod';
+
+/**
+ * Provider logins sealed to an agents controller's own key, so Switch stores
+ * and relays only ciphertext. The controller makes an X25519 keypair and
+ * gives Switch the public half; Console seals a login to it; only the
+ * controller can open it. Switch Core checks the envelope's shape and the key
+ * it names (`core/switch_core/management/sealed_logins.py`).
+ *
+ * - an ephemeral X25519 keypair, and its shared secret with the controller's key;
+ * - `HKDF-SHA256(shared, salt = ephemeral public || controller public,
+ * info = "switch provider login v1")`, a 32-byte key;
+ * - `AES-256-GCM` with a random 12-byte nonce, the tag appended to the
+ * ciphertext, and `aad = "switch-provider-login-v1\n<controller id>\n<provider>"`,
+ * so a login opens only for the controller and provider it was sealed for.
+ */
+export const SEALED_LOGIN_ALG = 'X25519-HKDF-SHA256-A256GCM';
+const INFO = Buffer.from('switch provider login v1');
+const TAG_BYTES = 16;
+
+const base64 = z.string().regex(/^[A-Za-z0-9+/]*={0,2}$/);
+
+export const sealedLoginSchema = z.strictObject({
+ alg: z.literal(SEALED_LOGIN_ALG),
+ key_id: z.string().min(1),
+ ephemeral_key: base64,
+ nonce: base64,
+ ciphertext: base64,
+});
+export type SealedLogin = z.infer<typeof sealedLoginSchema>;
+
+/** What is sealed: a login as Switch's provider connections hold one. */
+export const providerLoginSchema = z.strictObject({
+ kind: z.enum(['api-key', 'setup-token', 'auth-json']),
+ credential: z.string().min(1).max(16384),
+});
+export type ProviderLogin = z.infer<typeof providerLoginSchema>;
+
+/** A controller's keypair, each half the raw 32 bytes in base64. */
+export type SealingKeyPair = { publicKey: string; privateKey: string };
+
+export function generateSealingKeyPair(): SealingKeyPair {
+ const { publicKey, privateKey } = generateKeyPairSync('x25519');
+ return {
+ publicKey: rawPublic(publicKey).toString('base64'),
+ privateKey: Buffer.from(privateKey.export({ format: 'jwk' }).d!, 'base64url').toString(
+ 'base64'
+ ),
+ };
+}
+
+/** The id a sealed login names its controller's key by: the first 16 hex digits of its SHA-256. */
+export function sealingKeyId(publicKey: string): string {
+ return createHash('sha256').update(rawKey(publicKey, 'public key')).digest('hex').slice(0, 16);
+}
+
+export function sealProviderLogin(input: {
+ publicKey: string;
+ controllerId: string;
+ provider: string;
+ login: ProviderLogin;
+}): SealedLogin {
+ const controller = importPublic(input.publicKey);
+ const ephemeral = generateKeyPairSync('x25519');
+ const ephemeralPublic = rawPublic(ephemeral.publicKey);
+ const key = derive(
+ diffieHellman({ privateKey: ephemeral.privateKey, publicKey: controller }),
+ ephemeralPublic,
+ rawKey(input.publicKey, 'public key')
+ );
+ const nonce = randomBytes(12);
+ const cipher = createCipheriv('aes-256-gcm', key, nonce, { authTagLength: TAG_BYTES });
+ cipher.setAAD(aad(input.controllerId, input.provider));
+ const plaintext = Buffer.from(JSON.stringify(providerLoginSchema.parse(input.login)));
+ const ciphertext = Buffer.concat([cipher.update(plaintext), cipher.final(), cipher.getAuthTag()]);
+ return {
+ alg: SEALED_LOGIN_ALG,
+ key_id: sealingKeyId(input.publicKey),
+ ephemeral_key: ephemeralPublic.toString('base64'),
+ nonce: nonce.toString('base64'),
+ ciphertext: ciphertext.toString('base64'),
+ };
+}
+
+/** Opens a login sealed to `keys`; a wrong key, controller, provider or a tampered envelope throws. */
+export function openProviderLogin(input: {
+ keys: SealingKeyPair;
+ controllerId: string;
+ provider: string;
+ sealed: SealedLogin;
+}): ProviderLogin {
+ const sealed = sealedLoginSchema.parse(input.sealed);
+ if (sealed.key_id !== sealingKeyId(input.keys.publicKey))
+ throw new Error(
+ 'The login was sealed to another key than this controller’s; give the machine the login again.'
+ );
+ const ephemeralPublic = rawKey(sealed.ephemeral_key, 'ephemeral key');
+ const key = derive(
+ diffieHellman({
+ privateKey: importPrivate(input.keys),
+ publicKey: importPublic(sealed.ephemeral_key),
+ }),
+ ephemeralPublic,
+ rawKey(input.keys.publicKey, 'public key')
+ );
+ const body = Buffer.from(sealed.ciphertext, 'base64');
+ if (body.length <= TAG_BYTES) throw new Error('The sealed login holds no ciphertext.');
+ const decipher = createDecipheriv('aes-256-gcm', key, Buffer.from(sealed.nonce, 'base64'), {
+ authTagLength: TAG_BYTES,
+ });
+ decipher.setAAD(aad(input.controllerId, input.provider));
+ decipher.setAuthTag(body.subarray(body.length - TAG_BYTES));
+ let plaintext: Buffer;
+ try {
+ plaintext = Buffer.concat([
+ decipher.update(body.subarray(0, body.length - TAG_BYTES)),
+ decipher.final(),
+ ]);
+ } catch {
+ throw new Error(
+ 'The sealed login does not open with this controller’s key for this provider; it was tampered with or sealed for another.'
+ );
+ }
+ return providerLoginSchema.parse(JSON.parse(plaintext.toString('utf8')));
+}
+
+function aad(controllerId: string, provider: string): Buffer {
+ return Buffer.from(`switch-provider-login-v1\n${controllerId}\n${provider}`);
+}
+
+function derive(shared: Buffer, ephemeralPublic: Buffer, controllerPublic: Buffer): Buffer {
+ return Buffer.from(
+ hkdfSync('sha256', shared, Buffer.concat([ephemeralPublic, controllerPublic]), INFO, 32)
+ );
+}
+
+function rawKey(value: string, what: string): Buffer {
+ const raw = Buffer.from(value, 'base64');
+ if (raw.length !== 32 || raw.toString('base64') !== value)
+ throw new Error(`The ${what} is not 32 bytes of base64.`);
+ return raw;
+}
+
+function rawPublic(key: KeyObject): Buffer {
+ return Buffer.from(key.export({ format: 'jwk' }).x!, 'base64url');
+}
+
+function importPublic(value: string): KeyObject {
+ return createPublicKey({
+ key: { kty: 'OKP', crv: 'X25519', x: rawKey(value, 'public key').toString('base64url') },
+ format: 'jwk',
+ });
+}
+
+function importPrivate(keys: SealingKeyPair): KeyObject {
+ return createPrivateKey({
+ key: {
+ kty: 'OKP',
+ crv: 'X25519',
+ d: rawKey(keys.privateKey, 'private key').toString('base64url'),
+ x: rawKey(keys.publicKey, 'public key').toString('base64url'),
+ },
+ format: 'jwk',
+ });
+}
diff --git a/console/packages/agent-providers/tsdown.bench.config.ts b/console/packages/agent-providers/tsdown.bench.config.ts
index f249cebe8..2c7fca53c 100644
--- a/console/packages/agent-providers/tsdown.bench.config.ts
+++ b/console/packages/agent-providers/tsdown.bench.config.ts
@@ -15,10 +15,6 @@ import { defineConfig } from 'tsdown';
export default defineConfig({
entry: {
'bench-host-daemon': 'src/host/bench/daemon.ts',
- // The hosted bootstrap supervises the daemon beside it by this name, so a
- // hosted acceptance run boots the real bootstrap onto the stand-in provider.
- 'shared-host-daemon': 'src/host/bench/daemon.ts',
- 'hosted-bootstrap': 'src/host/hosted-bootstrap-cli.ts',
},
outDir: 'dist-bench',
format: ['esm'],
diff --git a/console/packages/agent-providers/tsdown.config.ts b/console/packages/agent-providers/tsdown.config.ts
index 93d7114d2..64b901079 100644
--- a/console/packages/agent-providers/tsdown.config.ts
+++ b/console/packages/agent-providers/tsdown.config.ts
@@ -6,7 +6,6 @@ export default defineConfig({
testing: 'src/testing/index.ts',
'host-daemon': 'src/host/daemon.ts',
'shared-host-daemon': 'src/host/shared-daemon.ts',
- 'hosted-bootstrap': 'src/host/hosted-bootstrap-cli.ts',
},
format: ['esm'],
dts: true,
diff --git a/console/packages/switch-agent-runtime/src/event-stream.old-server.test.ts b/console/packages/switch-agent-runtime/src/event-stream.old-server.test.ts
index d53477775..3b6d93837 100644
--- a/console/packages/switch-agent-runtime/src/event-stream.old-server.test.ts
+++ b/console/packages/switch-agent-runtime/src/event-stream.old-server.test.ts
@@ -3,11 +3,8 @@ import {
BEAT_INTERVAL_MS,
BEAT_SETTLE_LIMIT_MS,
EVICTION_CREDENTIALS_REJECTED,
- EVICTION_LAUNCH_SUPERSEDED,
EVICTION_TAKEN_OVER,
SwitchEventStream,
- WORKER_CAPABILITY_OBSOLETE,
- WorkerCallError,
type Eviction,
type SwitchEventStreamDeps,
} from './event-stream';
@@ -162,7 +159,6 @@ function makeStream(
onEvent: () => {},
onGap: () => {},
onEvicted: () => {},
- worker: null,
log,
signal: abort.signal,
...deps,
@@ -1797,164 +1793,6 @@ it('states placements on the attached incarnation, and raises on a refusal', asy
}
});
-describe('a hosted worker', () => {
- const worker = { capability: 'cap-1', bootId: 'boot-1', instanceId: 'i-1', stateVersion: 1 };
-
- function refused(status: number, code: string) {
- return {
- ok: false,
- status,
- body: null,
- text: async (): Promise<string> => JSON.stringify({ detail: { code, message: code } }),
- };
- }
-
- it('states its capability and host on the open', async () => {
- const fetchMock = vi.fn(async (_url: string, _init?: RequestInit) => ({
- ok: true,
- status: 200,
- body: openForever(),
- text: async (): Promise<string> => '',
- }));
- const { abort } = makeStream(fetchMock, { rooms: [], worker });
- await flush();
- const open = fetchMock.mock.calls.find((c) => String(c[0]).includes('/events'));
- expect(open?.[1]?.headers).toMatchObject({
- 'X-Switch-Worker-Capability': 'cap-1',
- 'X-Switch-Host-Boot-Id': 'boot-1',
- 'X-Switch-Host-Instance-Id': 'i-1',
- 'X-Switch-Worker-State-Version': '1',
- });
- abort.abort();
- });
-
- it.each([
- [403, WORKER_CAPABILITY_OBSOLETE],
- [403, 'worker_capability_required'],
- [426, 'upgrade_required'],
- [403, 'hosted_worker_only'],
- ])('stops for good on a %i %s open', async (status, code) => {
- vi.useFakeTimers();
- const fetchMock = vi.fn(async () => refused(status, code));
- const evicted: Eviction[] = [];
- const { abort } = makeStream(fetchMock, {
- rooms: [],
- worker,
- onEvicted: (e) => evicted.push(e),
- });
- await vi.advanceTimersByTimeAsync(5 * 60_000);
- expect(urlsFor(fetchMock, '/events')).toHaveLength(1);
- expect(evicted.map((e) => e.code)).toEqual([code]);
- abort.abort();
- });
-
- it('backs off and retries while another worker is attached', async () => {
- vi.useFakeTimers();
- const fetchMock = vi.fn(async () => refused(409, 'worker_already_attached'));
- const evicted: Eviction[] = [];
- const { abort } = makeStream(fetchMock, {
- rooms: [],
- worker,
- onEvicted: (e) => evicted.push(e),
- });
- await vi.advanceTimersByTimeAsync(10_000);
- expect(urlsFor(fetchMock, '/events').length).toBeGreaterThan(1);
- expect(urlsFor(fetchMock, '/events').length).toBeLessThan(6);
- expect(evicted).toEqual([]);
- abort.abort();
- });
-
- it('halts on a launch_superseded eviction', async () => {
- vi.useFakeTimers();
- const fetchMock = vi.fn(async (url: string) =>
- String(url).includes('/events')
- ? {
- ok: true,
- status: 200,
- body: frameThenClose('evicted', { code: 'launch_superseded', reason: 'bumped' }),
- text: async (): Promise<string> => '',
- }
- : { ok: true, status: 200, text: async (): Promise<string> => '' }
- );
- const evicted: Eviction[] = [];
- const { abort } = makeStream(fetchMock, {
- rooms: [],
- worker,
- onEvicted: (e) => evicted.push(e),
- });
- await vi.advanceTimersByTimeAsync(5 * 60_000);
- expect(urlsFor(fetchMock, '/events')).toHaveLength(1);
- expect(evicted.map((e) => e.code)).toEqual([EVICTION_LAUNCH_SUPERSEDED]);
- abort.abort();
- });
-
- it('hands worker frames over in order and not as room events', async () => {
- const body = new ReadableStream<Uint8Array>({
- start(controller) {
- controller.enqueue(connected(4));
- controller.enqueue(encodeFrame('worker_attached', { launch_revision: 3 }));
- controller.enqueue(encodeFrame('operation', { id: 'op-1' }));
- },
- });
- const fetchMock = vi.fn(async (url: string) =>
- String(url).includes('/events')
- ? { ok: true, status: 200, body, text: async (): Promise<string> => '' }
- : { ok: true, status: 200, text: async (): Promise<string> => '' }
- );
- const frames: [string, unknown][] = [];
- const events: unknown[] = [];
- const { abort } = makeStream(fetchMock, {
- rooms: [],
- worker,
- onEvent: (e) => void events.push(e),
- onWorkerFrame: (name, data) => void frames.push([name, data]),
- });
- await flush();
- expect(frames).toEqual([
- ['worker_attached', { launch_revision: 3 }],
- ['operation', { id: 'op-1' }],
- ]);
- expect(events).toEqual([]);
- abort.abort();
- });
-
- it('sends an up-call as its connection and incarnation, and raises a refusal with its code', async () => {
- const fetchMock = vi.fn(async (url: string, _init?: RequestInit) => {
- const u = String(url);
- if (u.includes('/events'))
- return {
- ok: true,
- status: 200,
- body: openForever(),
- text: async (): Promise<string> => '',
- };
- if (u.endsWith('/claim')) return refused(409, 'already_claimed');
- return {
- ok: true,
- status: 200,
- text: async (): Promise<string> => '{"queued_operations":[]}',
- };
- });
- const { stream, abort } = makeStream(fetchMock, { rooms: [], worker });
- await expect(
- stream.workerCall('/agents/agent-1/connection/idle', { busy: false })
- ).resolves.toEqual({
- queued_operations: [],
- });
- const idle = fetchMock.mock.calls.find((c) => String(c[0]).endsWith('/connection/idle'));
- expect(String(idle?.[0])).toBe('https://switch.test/agents/agent-1/connection/idle');
- expect(JSON.parse(String(idle?.[1]?.body))).toEqual({
- busy: false,
- connection_id: 'conn-1',
- generation: 0,
- });
- const claim = stream.workerCall('/hosted/operations/op-1/claim', {});
- await expect(claim).rejects.toBeInstanceOf(WorkerCallError);
- await expect(claim).rejects.toMatchObject({ status: 409, code: 'already_claimed' });
- abort.abort();
- });
-});
-
describe('test_gap_above_cursor_is_not_cursor_reset', () => {
it('reports a restart gap resuming above the cursor as no cursor reset', async () => {
const body = new ReadableStream<Uint8Array>({
diff --git a/console/packages/switch-agent-runtime/src/event-stream.test.ts b/console/packages/switch-agent-runtime/src/event-stream.test.ts
index 2fa9b01c1..fe93623e9 100644
--- a/console/packages/switch-agent-runtime/src/event-stream.test.ts
+++ b/console/packages/switch-agent-runtime/src/event-stream.test.ts
@@ -1,10 +1,8 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
EVICTION_CREDENTIALS_REJECTED,
- EVICTION_LAUNCH_SUPERSEDED,
EVICTION_TAKEN_OVER,
SwitchEventStream,
- WORKER_CAPABILITY_OBSOLETE,
type Eviction,
type SwitchEventStreamDeps,
} from './event-stream';
@@ -214,7 +212,6 @@ function makeStream(
onEvent: () => {},
onGap: () => {},
onEvicted: () => {},
- worker: null,
log,
signal: abort.signal,
...deps,
@@ -1296,101 +1293,3 @@ describe('placements', () => {
}
});
});
-
-describe('a hosted worker over the socket', () => {
- const worker = { capability: 'cap-1', bootId: 'boot-1', instanceId: 'i-1', stateVersion: 1 };
-
- it('states its capability and host on the socket it opens', async () => {
- const { abort } = makeStream({ rooms: [], worker });
- await flush();
- expect(server.sockets[0]?.headers).toEqual({
- Authorization: 'Bearer tok',
- 'X-Switch-Worker-Capability': 'cap-1',
- 'X-Switch-Host-Boot-Id': 'boot-1',
- 'X-Switch-Host-Instance-Id': 'i-1',
- 'X-Switch-Worker-State-Version': '1',
- });
- abort.abort();
- });
-
- it('sends no worker headers when it is not a worker', async () => {
- const { abort } = makeStream({ rooms: [] });
- await flush();
- expect(server.sockets[0]?.headers).toEqual({ Authorization: 'Bearer tok' });
- abort.abort();
- });
-
- it.each([
- [403, WORKER_CAPABILITY_OBSOLETE],
- [403, 'worker_capability_required'],
- [426, 'upgrade_required'],
- [403, 'hosted_worker_only'],
- ])('stops for good on a %i %s refusal, and does not try the stream', async (status, code) => {
- vi.useFakeTimers();
- serve((socket) => socket.refuse(status, { code, message: code }));
- const evicted: Eviction[] = [];
- const { abort, fetchMock } = makeStream({
- rooms: [],
- worker,
- onEvicted: (e) => evicted.push(e),
- });
- await vi.advanceTimersByTimeAsync(5 * 60_000);
- expect(server.sockets).toHaveLength(1);
- expect(postsTo(fetchMock, '/events')).toEqual([]);
- expect(evicted.map((e) => e.code)).toEqual([code]);
- abort.abort();
- });
-
- it('backs off and retries while another worker is attached', async () => {
- vi.useFakeTimers();
- serve((socket) => socket.refuse(409, { code: 'worker_already_attached', message: 'attached' }));
- const evicted: Eviction[] = [];
- const { abort } = makeStream({ rooms: [], worker, onEvicted: (e) => evicted.push(e) });
- await vi.advanceTimersByTimeAsync(10_000);
- expect(server.sockets.length).toBeGreaterThan(1);
- expect(server.sockets.length).toBeLessThan(6);
- expect(evicted).toEqual([]);
- abort.abort();
- });
-
- it('halts on a launch_superseded eviction', async () => {
- vi.useFakeTimers();
- serve((socket) => {
- attach(socket, 0);
- socket.frame('evicted', { code: 'launch_superseded', reason: 'bumped', room_id: null });
- socket.drop();
- });
- const evicted: Eviction[] = [];
- const { abort } = makeStream({ rooms: [], worker, onEvicted: (e) => evicted.push(e) });
- await vi.advanceTimersByTimeAsync(5 * 60_000);
- expect(server.sockets).toHaveLength(1);
- expect(evicted.map((e) => e.code)).toEqual([EVICTION_LAUNCH_SUPERSEDED]);
- abort.abort();
- });
-
- it('hands worker frames over in order and not as room events', async () => {
- serve((socket) => {
- attach(socket, 0);
- socket.frame('worker_attached', { launch_revision: 3 });
- socket.frame('operation', { id: 'op-1' });
- });
- const frames: [string, unknown][] = [];
- const onEvent = vi.fn();
- const { abort } = makeStream({
- rooms: [],
- worker,
- onEvent,
- onWorkerFrame: (name, data) => void frames.push([name, data]),
- });
- try {
- await vi.waitFor(() => expect(frames).toHaveLength(2));
- expect(frames).toEqual([
- ['worker_attached', { launch_revision: 3 }],
- ['operation', { id: 'op-1' }],
- ]);
- expect(onEvent).not.toHaveBeenCalled();
- } finally {
- abort.abort();
- }
- });
-});
diff --git a/console/packages/switch-agent-runtime/src/event-stream.ts b/console/packages/switch-agent-runtime/src/event-stream.ts
index e7d963c4a..b0dbc5c44 100644
--- a/console/packages/switch-agent-runtime/src/event-stream.ts
+++ b/console/packages/switch-agent-runtime/src/event-stream.ts
@@ -123,49 +123,6 @@ export const EVICTION_TAKEN_OVER = 'taken_over';
export const EVICTION_HEARTBEAT_LAPSED = 'heartbeat_lapsed';
export const EVICTION_CLOSED = 'closed';
export const EVICTION_CREDENTIALS_REJECTED = 'credentials_rejected';
-/** A hosted worker's launch moved to a newer revision. Terminal, like `taken_over`. */
-export const EVICTION_LAUNCH_SUPERSEDED = 'launch_superseded';
-/** Switch refused a hosted worker's open or up-call for good; `code` names why. */
-export const WORKER_CAPABILITY_OBSOLETE = 'worker_capability_obsolete';
-const TERMINAL_WORKER_REFUSALS = new Set([
- WORKER_CAPABILITY_OBSOLETE,
- 'worker_capability_required',
- 'upgrade_required',
- 'hosted_worker_only',
-]);
-
-/** The frames of agent-protocol 7, sent only to an attached hosted worker. */
-export const WORKER_FRAMES = [
- 'worker_attached',
- 'relay',
- 'relay_cancel',
- 'wake',
- 'mailbox_cancel',
- 'operation',
- 'credential',
-] as const;
-export type WorkerFrameName = (typeof WORKER_FRAMES)[number];
-
-/** Who a hosted worker is, stated on every open. */
-export interface WorkerIdentity {
- capability: string;
- bootId: string;
- instanceId: string;
- /** The layout version of the worker's volume; Switch refuses one older than it requires. */
- stateVersion: number;
-}
-
-/** A hosted up-call Switch answered with anything but success. */
-export class WorkerCallError extends Error {
- constructor(
- readonly status: number,
- readonly code: string | null,
- detail: string
- ) {
- super(`Switch refused ${detail} (HTTP ${status}${code ? `, ${code}` : ''})`);
- this.name = 'WorkerCallError';
- }
-}
/**
* Read the code off an `evicted` frame, falling back to its prose.
@@ -352,12 +309,6 @@ export interface SwitchEventStreamDeps {
* A `taken_over` eviction has already halted the connection before this runs:
* there is nothing left to reconnect, only something to report. */
onEvicted(eviction: Eviction): void;
- /**
- * Open as the agent's hosted worker. Null for every other client. A worker
- * is handed the protocol-7 frames through `onWorkerFrame`, in order.
- */
- worker: WorkerIdentity | null;
- onWorkerFrame?: (frame: WorkerFrameName, data: Record<string, unknown>) => Promise<void> | void;
log: EventStreamLogger;
/** Aborts the connection. */
signal: AbortSignal;
@@ -644,34 +595,10 @@ export class SwitchEventStream {
return true;
}
- /**
- * A refusal a hosted worker cannot reopen its way past: its capability is
- * obsolete or missing, or it speaks too old a protocol. Ends both loops and
- * reports the refusal's own code.
- */
- private refuseWorker(status: number, body: string): boolean {
- const code = refusalCode(body);
- if (code === null || !TERMINAL_WORKER_REFUSALS.has(code)) return false;
- if (this.halt.signal.aborted) return true;
- this.deps.log.error('SwitchEventStream: Switch refused this worker — stopping', {
- event: 'switch_stream_worker_refused',
- status,
- code,
- });
- this.halt.abort();
- this.deps.onEvicted({
- code,
- reason: `Switch refused this worker (HTTP ${status}, ${code})`,
- roomId: null,
- });
- return true;
- }
-
/** A rejected credential is not an outage: every reopen would carry the same
* token, so end the connection and tell the owner once. */
private rejectCredentials(status: number, body: string): void {
if (this.halt.signal.aborted) return;
- if (this.refuseWorker(status, body)) return;
const detail = body.slice(0, 500);
this.deps.log.error('SwitchEventStream: the server rejected our credentials — stopping', {
event: 'switch_stream_credentials_rejected',
@@ -806,32 +733,6 @@ export class SwitchEventStream {
return attached && !this.halt.signal.aborted;
}
- /**
- * A hosted worker's up-call: `path` under the API endpoint, sent as this
- * connection and the incarnation it holds. Waits for the stream to be
- * attached like a placements update. Answers the parsed body; any other
- * status raises a `WorkerCallError` carrying the refusal's code.
- */
- async workerCall(path: string, body: Record<string, unknown>): Promise<unknown> {
- if (!(await this.attached()))
- throw new Error(`the connection to Switch is not open, so ${path} cannot be sent`);
- const { creds } = this.deps;
- const resp = await fetch(`${creds.apiEndpoint}${path}`, {
- method: 'POST',
- headers: { Authorization: `Bearer ${creds.token}`, 'Content-Type': 'application/json' },
- body: JSON.stringify({
- ...body,
- connection_id: this.deps.connectionId,
- generation: this.generation,
- }),
- redirect: 'error',
- signal: AbortSignal.any([AbortSignal.timeout(30_000), this.deps.signal, this.halt.signal]),
- });
- const text = await resp.text();
- if (!resp.ok) throw new WorkerCallError(resp.status, refusalCode(text), path);
- return text ? JSON.parse(text) : null;
- }
-
private post(path: string, body: unknown, timeoutMs = REQUEST_TIMEOUT_MS) {
const { creds } = this.deps;
return fetch(`${creds.apiEndpoint}/agents/${creds.agentId}/${path}`, {
@@ -937,7 +838,6 @@ export class SwitchEventStream {
} catch (error) {
if (error instanceof OpenRefused) {
const { status, body } = error;
- if (this.refuseWorker(status, body)) return;
// Reopening without the refused room is a different request from the
// one that just failed, and the declared set strictly shrinks, so
// this cannot spin: retry now rather than serving the backoff a
@@ -1031,18 +931,6 @@ export class SwitchEventStream {
yield* this.readEventStream(params, abort, onOpen, null);
}
- /** Who a hosted worker is, as the headers every open of its connection carries. */
- private workerHeaders(): Record<string, string> {
- const worker = this.deps.worker;
- if (!worker) return {};
- return {
- 'X-Switch-Worker-Capability': worker.capability,
- 'X-Switch-Host-Boot-Id': worker.bootId,
- 'X-Switch-Host-Instance-Id': worker.instanceId,
- 'X-Switch-Worker-State-Version': String(worker.stateVersion),
- };
- }
-
/**
* Open the connection as a Server-Sent Events stream, the way a server from
* before the socket serves it, and yield its frames until it ends.
@@ -1066,7 +954,6 @@ export class SwitchEventStream {
Authorization: `Bearer ${creds.token}`,
Accept: 'text/event-stream',
...(this.cursor > 0 ? { 'Last-Event-ID': String(this.cursor) } : {}),
- ...this.workerHeaders(),
},
signal: AbortSignal.any([abort, signal, this.halt.signal]),
});
@@ -1106,9 +993,8 @@ export class SwitchEventStream {
`this runtime needs Node 22 or newer to connect to Switch (it has ${process.version}, with no WebSocket)`
);
}
- // A hosted worker proves who it is on the socket exactly as on the stream.
const socket = new Socket(url, {
- headers: { Authorization: `Bearer ${this.deps.creds.token}`, ...this.workerHeaders() },
+ headers: { Authorization: `Bearer ${this.deps.creds.token}` },
});
const pending: SocketFrame[] = [];
let refused: unknown = null;
@@ -1281,7 +1167,7 @@ export class SwitchEventStream {
// Halt before reporting: a takeover is the one ending that reopening
// cannot recover, because reopening is itself a takeover. The
// connection ends here, and nothing the callback does can restart them.
- if (code === EVICTION_TAKEN_OVER || code === EVICTION_LAUNCH_SUPERSEDED) this.halt.abort();
+ if (code === EVICTION_TAKEN_OVER) this.halt.abort();
onEvicted({
code,
reason: String(frame.data.reason ?? 'connection closed'),
@@ -1309,16 +1195,6 @@ export class SwitchEventStream {
return;
}
default:
- if ((WORKER_FRAMES as readonly string[]).includes(frame.event)) {
- if (this.deps.onWorkerFrame)
- await this.deps.onWorkerFrame(frame.event as WorkerFrameName, frame.data);
- else
- log.warn('SwitchEventStream: worker frame on a connection that is not a worker', {
- event: 'switch_stream_unexpected_worker_frame',
- frame: frame.event,
- });
- return;
- }
if (typeof frame.data.type !== 'string') {
// A frame this runtime does not know and cannot read as a room event.
log.warn('SwitchEventStream: unknown frame dropped', {
diff --git a/console/packages/switch-agent-runtime/src/index.ts b/console/packages/switch-agent-runtime/src/index.ts
index 5ee1940f4..d7dc5550c 100644
--- a/console/packages/switch-agent-runtime/src/index.ts
+++ b/console/packages/switch-agent-runtime/src/index.ts
@@ -30,14 +30,9 @@ export {
EVICTION_CLOSED,
EVICTION_CREDENTIALS_REJECTED,
EVICTION_HEARTBEAT_LAPSED,
- EVICTION_LAUNCH_SUPERSEDED,
EVICTION_TAKEN_OVER,
PlacementsRefusedError,
SwitchEventStream,
- WORKER_CAPABILITY_OBSOLETE,
- WorkerCallError,
- type WorkerFrameName,
- type WorkerIdentity,
type ApprovalOutcome,
type SessionCommand,
type DeliveryFilter,
diff --git a/core/pyproject.toml b/core/pyproject.toml
index b1240c371..189d0de2f 100644
--- a/core/pyproject.toml
+++ b/core/pyproject.toml
@@ -52,7 +52,6 @@ Issues = "https://github.com/sandbox-quantum/switch/issues"
[project.scripts]
switch-core = "switch_core.main:main"
switch-migrate = "switch_core.main:migrate"
-switch-hosted-cutover-upgrade = "switch_core.hosted_cutover_upgrade:main"
[build-system]
requires = ["hatchling"]
diff --git a/core/switch_core/bridges/agent/api/handlers.py b/core/switch_core/bridges/agent/api/handlers.py
index 2e170e7ce..9ae01e01c 100644
--- a/core/switch_core/bridges/agent/api/handlers.py
+++ b/core/switch_core/bridges/agent/api/handlers.py
@@ -23,10 +23,6 @@ from pydantic import ValidationError
from sqlalchemy.ext.asyncio import AsyncSession
from starlette.responses import Response, StreamingResponse
-from switch_core.bridges.agent.api.hosted_worker_routes import (
- admit_worker,
- hosted_worker_only,
-)
from switch_core.bridges.agent.api.schemas import (
AcceptTaskRequest,
AgentInfo,
@@ -88,10 +84,8 @@ from switch_core.bridges.agent.dependencies import (
get_protocol,
get_session,
)
-from switch_core.bridges.agent.hosted_mailbox import deliver_on_attach
from switch_core.bridges.agent.protocol.agent_connections import (
HEARTBEAT_INTERVAL_SECONDS,
- TAKEN_OVER,
AgentConnection,
ClientDeclaration,
Closure,
@@ -112,7 +106,6 @@ from switch_core.bridges.agent.protocol.agent_connections import (
)
from switch_core.bridges.agent.protocol.agent_core import AgentCore, AgentExistsError
from switch_core.bridges.agent.protocol.event_buffer import Reader
-from switch_core.bridges.agent.protocol.hosted_workers import hosted_launch_of
from switch_core.bridges.agent.protocol.stream import (
KEEPALIVE,
Frame,
@@ -127,8 +120,7 @@ from switch_core.bridges.agent.registration_bootstrap import (
)
from switch_core.budgets import BudgetExceeded
from switch_core.config import SwitchConfig
-from switch_core.db.models import Agent, HostedLaunch, Task, require_tenant_id
-from switch_core.db.session_scope import tenant_session
+from switch_core.db.models import Agent, Task
from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.db.stores.feature_flag_store import FeatureFlagStore
from switch_core.feature_flags import is_known_flag
@@ -473,12 +465,6 @@ async def delete_agent(
) -> dict[str, bool]:
if agent.id != agent_id:
raise HTTPException(status_code=403, detail="Not authorized for this agent")
- if hosted_launch_of(agent.metadata_) is not None:
- raise HTTPException(
- status_code=409,
- detail="This is a cloud agent. Remove it from Switch Console's cloud agents instead.",
- )
-
try:
await protocol.delete_agent(agent_id=agent_id)
except ValueError as e:
@@ -760,16 +746,6 @@ async def poll_events(
client_version: Annotated[str | None, Query()] = None,
rooms: Annotated[str | None, Query()] = None,
last_event_id: Annotated[str | None, Header(alias="last-event-id")] = None,
- worker_capability: Annotated[
- str | None, Header(alias="x-switch-worker-capability")
- ] = None,
- host_boot_id: Annotated[str | None, Header(alias="x-switch-host-boot-id")] = None,
- host_instance_id: Annotated[
- str | None, Header(alias="x-switch-host-instance-id")
- ] = None,
- worker_state_version: Annotated[
- int | None, Header(alias="x-switch-worker-state-version")
- ] = None,
) -> EventResponse | Response:
"""Deliver the agent's events, as a push stream or a long poll.
@@ -803,14 +779,8 @@ async def poll_events(
rooms=rooms,
last_event_id=last_event_id,
expected_generation=expected_generation,
- worker_capability=worker_capability,
- host_boot_id=host_boot_id,
- host_instance_id=host_instance_id,
- worker_state_version=worker_state_version,
)
- if hosted_launch_of(agent.metadata_) is not None:
- raise hosted_worker_only()
events = await protocol.poll_events(agent.id, timeout=timeout)
if not events:
return Response(status_code=204)
@@ -910,17 +880,13 @@ async def _open_connection(
declaration: ClientDeclaration,
rooms: str | None,
expected_generation: int | None,
- worker_capability: str | None = None,
- host_boot_id: str | None = None,
- host_instance_id: str | None = None,
- worker_state_version: int | None = None,
last_event_id: str | None = None,
transport: Transport = "websocket",
) -> tuple[AgentConnection, AsyncGenerator[Frame]]:
"""Open or reattach the connection and claim its declared rooms.
- Shared by every transport, so the socket and the event stream fence,
- admit a hosted worker and claim rooms in exactly the same way. Returns
+ Shared by every transport, so the socket and the event stream fence and
+ claim rooms in exactly the same way. Returns
the connection and its frames, for the transport to encode.
"""
if not connection_id:
@@ -942,47 +908,18 @@ async def _open_connection(
cursor = _resolve_start_cursor(protocol, agent.id, start_from, last_event_id)
- def register() -> AgentConnection:
- return _register_connection(
- protocol=protocol,
- agent=agent,
- connection_id=connection_id,
- scope=scope,
- event_filter=event_filter,
- spawn_capable=spawn_capable,
- cursor=cursor,
- declaration=declaration,
- expected_generation=expected_generation,
- transport=transport,
- )
-
- launch_id = hosted_launch_of(agent.metadata_)
- if launch_id is None:
- conn = register()
- else:
- # Admission, the open and the binding all happen under the launch
- # lock, so no revision bump lands between the check and the bind.
- async with tenant_session(protocol.session_factory, require_tenant_id()) as db:
- attach = await admit_worker(
- session=db,
- registry=protocol.connections,
- config=config,
- agent=agent,
- launch_id=launch_id,
- connection_id=connection_id,
- declaration=declaration,
- capability=worker_capability,
- boot_id=host_boot_id,
- instance_id=host_instance_id,
- state_version=worker_state_version,
- )
- conn = register()
- if attach.takes_over is not None and attach.takes_over.id != conn.id:
- protocol.connections.close(attach.takes_over.id, TAKEN_OVER)
- protocol.connections.bind_worker(conn, attach.binding, attach.attached)
- launch = await db.get(HostedLaunch, (require_tenant_id(), launch_id))
- assert launch is not None
- await deliver_on_attach(db, protocol, launch)
+ conn = _register_connection(
+ protocol=protocol,
+ agent=agent,
+ connection_id=connection_id,
+ scope=scope,
+ event_filter=event_filter,
+ spawn_capable=spawn_capable,
+ cursor=cursor,
+ declaration=declaration,
+ expected_generation=expected_generation,
+ transport=transport,
+ )
# Built before anything below can yield, so it holds the generation this
# open produced; a reconnect during the bookkeeping supersedes it rather
@@ -1071,10 +1008,6 @@ async def _open_event_stream(
rooms: str | None,
last_event_id: str | None,
expected_generation: int | None,
- worker_capability: str | None,
- host_boot_id: str | None,
- host_instance_id: str | None,
- worker_state_version: int | None,
) -> StreamingResponse:
"""The connection as a Server-Sent Events stream, for an old client.
@@ -1095,10 +1028,6 @@ async def _open_event_stream(
declaration=declaration,
rooms=rooms,
expected_generation=expected_generation,
- worker_capability=worker_capability,
- host_boot_id=host_boot_id,
- host_instance_id=host_instance_id,
- worker_state_version=worker_state_version,
last_event_id=last_event_id,
transport="sse",
)
@@ -1201,16 +1130,6 @@ async def connection_socket(
client: Annotated[str | None, Query()] = None,
client_version: Annotated[str | None, Query()] = None,
rooms: Annotated[str | None, Query()] = None,
- worker_capability: Annotated[
- str | None, Header(alias="x-switch-worker-capability")
- ] = None,
- host_boot_id: Annotated[str | None, Header(alias="x-switch-host-boot-id")] = None,
- host_instance_id: Annotated[
- str | None, Header(alias="x-switch-host-instance-id")
- ] = None,
- worker_state_version: Annotated[
- int | None, Header(alias="x-switch-worker-state-version")
- ] = None,
) -> None:
"""The agent's connection over one WebSocket: events down, heartbeats up.
@@ -1227,12 +1146,6 @@ async def connection_socket(
A refusal on opening is sent as a `refused` frame carrying the status and
detail an HTTP request would have been answered with, then the socket closes
with code 4000 plus that status.
-
- A hosted agent's worker opens it as it opens the stream, with its
- capability and host identity in the `X-Switch-Worker-*` and
- `X-Switch-Host-*` headers, and is admitted and refused by the same rules
- (agent-protocol 7). Its frames come down the socket; its up-calls stay
- HTTP requests.
"""
await websocket.accept()
try:
@@ -1253,10 +1166,6 @@ async def connection_socket(
),
rooms=rooms,
expected_generation=expected_generation,
- worker_capability=worker_capability,
- host_boot_id=host_boot_id,
- host_instance_id=host_instance_id,
- worker_state_version=worker_state_version,
)
except HTTPException as exc:
metrics().increment(AGENT_CONNECTIONS_REFUSED, {"reason": _refusal_reason(exc)})
@@ -1389,10 +1298,6 @@ def _current_connection(
the client on it. Asked again after any wait, because what a caller was
admitted on is not what it is still holding.
"""
- if hosted_launch_of(agent.metadata_) is not None:
- named = protocol.connections.get(req.connection_id)
- if named is None or named.agent_id != agent.id or named.worker is None:
- raise hosted_worker_only()
try:
return protocol.connections.require_current(
agent.id, req.connection_id, generation=req.generation
diff --git a/core/switch_core/bridges/agent/api/hosted_cutover_routes.py b/core/switch_core/bridges/agent/api/hosted_cutover_routes.py
deleted file mode 100644
index 7fb138c32..000000000
--- a/core/switch_core/bridges/agent/api/hosted_cutover_routes.py
+++ /dev/null
@@ -1,87 +0,0 @@
-"""The worker's cutover manifest: what its retained volume held before the upgrade."""
-
-from __future__ import annotations
-
-import logging
-from typing import Annotated, Any
-
-from fastapi import APIRouter, Depends
-from pydantic import ConfigDict
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.bridges.agent.api.hosted_worker_routes import (
- refusal,
- require_self,
- require_worker,
-)
-from switch_core.bridges.agent.auth import get_agent_from_scope
-from switch_core.bridges.agent.dependencies import get_protocol, get_session
-from switch_core.bridges.agent.hosted_cutover import (
- CutoverConflict,
- CutoverManifest,
- CutoverUnrecorded,
- confirm_manifest,
- owed_notices,
- post_cutover_notices,
- queue_imports,
-)
-from switch_core.bridges.agent.hosted_mailbox import offer_pending
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.db.models import Agent, HostedLaunch, require_tenant_id
-from switch_core.db.stores.hosted_launch_store import lock_launch
-from switch_core.db.stores.hosted_mailbox_store import MailboxFull
-
-logger = logging.getLogger(__name__)
-
-router = APIRouter()
-
-
-class CutoverUpload(CutoverManifest):
- model_config = ConfigDict(extra="forbid")
- connection_id: str
- generation: int
-
-
[email protected]("/{agent_id}/connection/cutover-manifest")
-async def cutover_manifest(
- agent_id: str,
- body: CutoverUpload,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict[str, Any]:
- """Confirm the volume's manifest is the one recorded before the upgrade, and deliver what it owes."""
- require_self(agent_id, agent)
- conn = require_worker(
- protocol.connections, agent, body.connection_id, body.generation
- )
- assert conn.worker is not None
- await lock_launch(session, conn.worker.launch_id)
- launch = await session.get(
- HostedLaunch,
- (require_tenant_id(), conn.worker.launch_id),
- populate_existing=True,
- )
- if launch is None or launch.revision != conn.worker.launch_revision:
- raise refusal(
- 409,
- "generation_changed",
- "The launch moved to a newer revision; upload the manifest again after "
- "reattaching.",
- )
- try:
- await confirm_manifest(session, launch.id, body.manifest_sha256)
- await queue_imports(session, launch.id)
- except CutoverUnrecorded as exc:
- raise refusal(409, "cutover_manifest_unrecorded", str(exc)) from exc
- except CutoverConflict as exc:
- raise refusal(409, "cutover_manifest_conflict", str(exc)) from exc
- except MailboxFull as exc:
- raise refusal(503, "mailbox_full", str(exc)) from exc
- notices = await owed_notices(session, agent.id)
- await session.commit()
- unposted = await post_cutover_notices(protocol, agent, notices)
- await offer_pending(
- session, protocol.connections, protocol.event_buffer.boot, launch
- )
- return {"manifest_sha256": body.manifest_sha256, "unposted_notices": unposted}
diff --git a/core/switch_core/bridges/agent/api/hosted_machine_routes.py b/core/switch_core/bridges/agent/api/hosted_machine_routes.py
deleted file mode 100644
index d4dcbe077..000000000
--- a/core/switch_core/bridges/agent/api/hosted_machine_routes.py
+++ /dev/null
@@ -1,427 +0,0 @@
-"""The machine supervisor's routes: its agent list and its heartbeat.
-
-Authenticated by the machine capability rather than an agent key, so the
-Bearer middleware lets these paths through and the dependency below does the
-check.
-"""
-
-import logging
-from collections.abc import AsyncIterator
-from dataclasses import dataclass
-from datetime import UTC, datetime, timedelta
-from pathlib import Path
-from typing import Annotated, Literal
-
-from fastapi import APIRouter, Depends, HTTPException, Request, Response
-from pydantic import AwareDatetime, BaseModel, Field
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-from switch_core.bridges.agent.dependencies import (
- get_config,
- get_protocol,
- get_session_factory,
-)
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.config import SwitchConfig
-from switch_core.connections.loader import CATALOG, SKILL_PROVIDERS, deployment_skills
-from switch_core.db.models import (
- Agent,
- ApiKey,
- HostedLaunch,
- HostedMachine,
- ProviderConnection,
- require_tenant_id,
-)
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
-from switch_core.db.stores.hosted_machine_store import HostedMachineStore, lock_launch
-from switch_core.providers.hosted import HostedControllerSettings
-from switch_core.tenant_context import tenant_scope
-
-logger = logging.getLogger(__name__)
-
-router = APIRouter(prefix="/hosted/machines")
-
-HEARTBEAT_EVERY_S = 15
-DISK_FULL_BELOW_BYTES = 1 << 30
-RETIRED_STATES = {"retained", "deleting", "deleted"}
-WORKER_ATTACH_TIMEOUT = timedelta(minutes=10)
-IDENTITY_REGISTRATION_GRACE = timedelta(minutes=1)
-
-
-@dataclass(frozen=True)
-class MachineRequest:
- session: AsyncSession
- machine: HostedMachine
- settings: HostedControllerSettings
-
-
-def _settings(config: SwitchConfig) -> HostedControllerSettings:
- if not config.hosted_controller_config_path:
- raise HTTPException(503, "Cloud machines are not enabled on this server.")
- return HostedControllerSettings.model_validate_json(
- Path(config.hosted_controller_config_path).read_text()
- )
-
-
-def _capability_valid(machine: HostedMachine | None, capability: str) -> bool:
- return (
- bool(capability)
- and machine is not None
- and HostedMachineStore.capability_matches(machine, capability)
- )
-
-
-async def machine_request(
- machine_id: str,
- request: Request,
- config: Annotated[SwitchConfig, Depends(get_config)],
- factory: Annotated[async_sessionmaker[AsyncSession], Depends(get_session_factory)],
-) -> AsyncIterator[MachineRequest]:
- """The supervisor's machine, locked, once its capability checks out."""
- settings = _settings(config)
- supplied = request.headers.get("authorization", "")
- capability = supplied[7:] if supplied.startswith("Bearer ") else ""
- with tenant_scope(settings.tenant_id):
- async with factory() as session:
- machine = await HostedMachineStore().get(session, machine_id)
- if not _capability_valid(machine, capability):
- raise HTTPException(401, "invalid machine capability")
- if not request.headers.get(
- "x-switch-host-boot-id"
- ) or not request.headers.get("x-switch-host-instance-id"):
- raise HTTPException(
- 400,
- "X-Switch-Host-Boot-Id and X-Switch-Host-Instance-Id are required.",
- )
- machine = await HostedMachineStore().locked(session, machine_id)
- if machine is None or not _capability_valid(machine, capability):
- raise HTTPException(401, "invalid machine capability")
- if (
- machine.state in RETIRED_STATES
- or machine.desired_state in RETIRED_STATES
- ):
- raise HTTPException(410, "machine retired")
- yield MachineRequest(session, machine, settings)
-
-
-async def _machine_launch(
- session: AsyncSession, machine: HostedMachine, launch_id: str
-) -> HostedLaunch | None:
- await lock_launch(session, launch_id)
- launch = await session.get(
- HostedLaunch, (require_tenant_id(), launch_id), populate_existing=True
- )
- if launch is None or launch.machine_id != machine.id:
- return None
- return launch
-
-
-UNAVAILABLE_ERRORS = {
- "agent_key_missing": (
- "Switch lost this cloud agent's credential, so its machine cannot run it. "
- "Remove the agent and create it again."
- ),
- "agent_identity_missing": (
- "Switch lost this cloud agent's identity, so its machine cannot run it. "
- "Retry it in Switch Console."
- ),
-}
-
-
-def _unavailable_entry(
- launch: HostedLaunch,
- code: Literal["agent_key_missing", "agent_identity_missing"],
- now: datetime,
-) -> dict:
- """List a launch the machine cannot run, and put it in error.
-
- A queued launch whose identity is missing is left alone for
- `IDENTITY_REGISTRATION_GRACE`: launch creation commits `agent_id` before
- it registers the agent. Past the grace it goes to error like any other
- state, and retry registers the reserved id again.
- """
- logger.error(
- "Cloud launch %s: agent %s is unavailable (%s); the machine cannot run it.",
- launch.id,
- launch.agent_id,
- code,
- )
- exempt = (
- code == "agent_identity_missing"
- and launch.state == "queued"
- and now - launch.updated_at < IDENTITY_REGISTRATION_GRACE
- )
- already = launch.state == "error" and launch.error_code == code
- if not exempt and not already:
- launch.state = "error"
- launch.error_code = code
- launch.error = UNAVAILABLE_ERRORS[code]
- launch.updated_at = now
- return {
- "launch_id": launch.id,
- "agent_id": launch.agent_id,
- "name": launch.name,
- "revision": launch.revision,
- "desired_state": launch.desired_state,
- "unavailable": code,
- }
-
-
-async def _agent_entry(
- session: AsyncSession,
- launch: HostedLaunch,
- key: ApiKey,
- settings: HostedControllerSettings,
- config: SwitchConfig,
-) -> dict:
- assert launch.agent_id is not None
- provider = launch.spec.get("provider", "claude")
- connection = await session.get(
- ProviderConnection, (require_tenant_id(), launch.owner_id, provider)
- )
- if connection is None:
- logger.warning(
- "Cloud launch %s: the owner's %s connection is gone; listing it without a credential kind.",
- launch.id,
- provider,
- )
- if provider not in SKILL_PROVIDERS:
- logger.warning(
- "Cloud launch %s: %s has no skills directory; granted connection skills are not installed.",
- launch.id,
- provider,
- )
- return {
- "launch_id": launch.id,
- "agent_id": launch.agent_id,
- "name": launch.name,
- "revision": launch.revision,
- "desired_state": launch.desired_state,
- "provider": provider,
- "provider_credential_kind": None if connection is None else connection.kind,
- "worker_capability": HostedLaunchStore().issue_worker_capability(
- launch, config.keyring
- ),
- "switch_credentials": {
- "env": {
- "SWITCH_API_ENDPOINT": settings.agent_api_endpoint,
- "SWITCH_API_TOKEN": config.keyring.decrypt(key.encrypted_key),
- "SWITCH_AGENT_ID": launch.agent_id,
- }
- },
- "repository": launch.repository,
- "spec": launch.spec,
- "skills": deployment_skills(CATALOG, ["github"])
- if provider in SKILL_PROVIDERS
- else [],
- }
-
-
[email protected]("/{machine_id}/agents")
-async def agents(
- response: Response,
- current: Annotated[MachineRequest, Depends(machine_request)],
- config: Annotated[SwitchConfig, Depends(get_config)],
-) -> dict:
- response.headers["Cache-Control"] = "no-store"
- session, machine = current.session, current.machine
- now = datetime.now(UTC)
- entries = []
- for candidate in await HostedMachineStore().launches(session, machine.id):
- launch = await _machine_launch(session, machine, candidate.id)
- if (
- launch is None
- or launch.state in {"deleting", "deleted"}
- or launch.desired_state == "deleted"
- or launch.agent_id is None
- ):
- continue
- agent = await session.get(Agent, launch.agent_id)
- if agent is None:
- entries.append(_unavailable_entry(launch, "agent_identity_missing", now))
- continue
- key = await session.get(ApiKey, agent.api_key_id)
- if key is None or not key.encrypted_key:
- entries.append(_unavailable_entry(launch, "agent_key_missing", now))
- continue
- entries.append(
- await _agent_entry(session, launch, key, current.settings, config)
- )
- if launch.state == "queued":
- launch.state = "provisioning"
- launch.updated_at = now
- result = {
- "machine_id": machine.id,
- "revision": machine.revision,
- "desired_state": machine.desired_state,
- "agents_version": machine.agents_version,
- "agents": entries,
- }
- await session.commit()
- return result
-
-
-class DiskReport(BaseModel):
- path: str = Field(max_length=4096)
- total_bytes: int = Field(ge=0)
- used_bytes: int = Field(ge=0)
- available_bytes: int = Field(ge=0)
-
-
-class MemoryReport(BaseModel):
- total_bytes: int = Field(ge=0)
- available_bytes: int = Field(ge=0)
-
-
-class ProcessExit(BaseModel):
- code: int | None
- signal: int | None
- result: str | None = Field(max_length=128)
-
-
-class AgentReport(BaseModel):
- launch_id: str = Field(max_length=128)
- agent_id: str = Field(max_length=128)
- revision: int = Field(ge=1)
- process_state: Literal[
- "pending",
- "starting",
- "running",
- "stopping",
- "stopped",
- "restarting",
- "crashed",
- "failed",
- ]
- restarts: int = Field(ge=0)
- oom_kills: int = Field(ge=0)
- exit: ProcessExit | None
- since: AwareDatetime
-
-
-class Heartbeat(BaseModel):
- boot_id: str = Field(max_length=128)
- instance_id: str = Field(max_length=128)
- supervisor_version: str = Field(max_length=128)
- runtime_fingerprint: str = Field(max_length=256)
- disk: DiskReport
- memory: MemoryReport
- agents: list[AgentReport] = Field(max_length=100)
-
-
-CRASHED_ERROR = "The agent crashed 5 times in 10 minutes. Retry it in Switch Console."
-FAILED_ERROR = (
- "The agent stopped with an error and was not restarted. Retry it in Switch Console."
-)
-ATTACH_TIMEOUT_ERROR = (
- "The agent started but did not connect to Switch within 10 minutes. "
- "Retry it in Switch Console, or check its provider login."
-)
-
-
-def _apply_process_state(
- launch: HostedLaunch,
- report: AgentReport,
- protocol: AgentCore,
- now: datetime,
-) -> None:
- """Move the launch on the process state its supervisor reported."""
- previous = launch.state
- if report.process_state == "running":
- worker = (
- protocol.connections.attached_worker(launch.agent_id)
- if launch.agent_id
- else None
- )
- listening = (
- worker is not None
- and worker.worker is not None
- and worker.worker.launch_id == launch.id
- and worker.worker.launch_revision == launch.revision
- and (worker.spawn_capable or not launch.spec["auto_session"])
- )
- if (
- listening
- and launch.desired_state == "running"
- and launch.state not in {"ready", "error"}
- ):
- launch.state = "ready"
- launch.error = None
- launch.error_code = None
- launch.active_at = now
- elif (
- not listening
- and launch.desired_state == "running"
- and launch.state == "provisioning"
- and now - max(report.since, launch.updated_at) > WORKER_ATTACH_TIMEOUT
- ):
- launch.state = "error"
- launch.error_code = "worker_attach_timeout"
- launch.error = ATTACH_TIMEOUT_ERROR
- elif (
- report.process_state in {"crashed", "failed"}
- and launch.desired_state == "running"
- ):
- crashed = report.process_state == "crashed"
- launch.state = "error"
- launch.error_code = "agent_crashed" if crashed else "agent_failed"
- launch.error = CRASHED_ERROR if crashed else FAILED_ERROR
- elif (
- report.process_state in {"stopped", "crashed", "failed"}
- and launch.desired_state == "stopped"
- ):
- launch.state = "stopped"
- launch.error = None
- launch.error_code = None
- if launch.state != previous:
- launch.updated_at = now
-
-
[email protected]("/{machine_id}/heartbeat")
-async def heartbeat(
- body: Heartbeat,
- current: Annotated[MachineRequest, Depends(machine_request)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict:
- session, machine = current.session, current.machine
- now = datetime.now(UTC)
- machine.heartbeat = body.model_dump(mode="json")
- machine.heartbeat_at = now
- if (
- machine.state == "provisioning"
- and machine.running_observed_at is not None
- and now > machine.running_observed_at
- ):
- machine.state = "ready"
- machine.error = None
- machine.error_code = None
- machine.updated_at = now
- if body.disk.available_bytes < DISK_FULL_BELOW_BYTES:
- if machine.error_code is None:
- machine.error_code = "disk_full"
- elif machine.error_code == "disk_full":
- machine.error_code = None
- for report in body.agents:
- launch = await _machine_launch(session, machine, report.launch_id)
- if (
- launch is None
- or launch.agent_id != report.agent_id
- or launch.revision != report.revision
- ):
- continue
- launch.process_state = report.process_state
- launch.process_restarts = report.restarts
- launch.process_oom_kills = report.oom_kills
- launch.process_exit = (
- None if report.exit is None else report.exit.model_dump(mode="json")
- )
- launch.process_reported_at = now
- if launch.desired_state != "deleted":
- _apply_process_state(launch, report, protocol, now)
- result = {
- "agents_version": machine.agents_version,
- "machine_desired_state": machine.desired_state,
- "heartbeat_every_s": HEARTBEAT_EVERY_S,
- }
- await session.commit()
- return result
diff --git a/core/switch_core/bridges/agent/api/hosted_routes.py b/core/switch_core/bridges/agent/api/hosted_routes.py
deleted file mode 100644
index 402c84434..000000000
--- a/core/switch_core/bridges/agent/api/hosted_routes.py
+++ /dev/null
@@ -1,339 +0,0 @@
-from datetime import UTC, datetime
-from pathlib import Path
-from typing import Annotated, Literal
-from uuid import UUID
-
-from fastapi import APIRouter, Depends, HTTPException, Response
-from pydantic import BaseModel, ConfigDict, Field
-from sqlalchemy import select, text
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.bridges.agent.api.hosted_worker_routes import refusal, require_worker
-from switch_core.bridges.agent.auth import get_agent_from_scope
-from switch_core.bridges.agent.dependencies import get_config, get_protocol, get_session
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.config import SwitchConfig
-from switch_core.db.models import (
- Agent,
- HostedLaunch,
- HostedOperation,
- ProviderConnection,
- TenantMember,
- require_tenant_id,
-)
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.gateway.github_connections import (
- conditions,
- discard_github_credential,
- github_credentials,
-)
-from switch_core.gateway.github_connections import lock as github_lock
-from switch_core.gateway.hosted_launches import operation_summary
-from switch_core.providers.github import (
- GitHubConnections,
- GitHubError,
- GitHubUnavailableError,
-)
-from switch_core.providers.github_installation import GitHubInstallationCredentials
-from switch_core.providers.github_revocations import remember_repository_token
-from switch_core.providers.hosted import HostedControllerSettings
-
-router = APIRouter(prefix="/hosted")
-
-
[email protected]("/provider-credential")
-async def provider_credential(
- response: Response,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- session: Annotated[AsyncSession, Depends(get_session)],
- config: Annotated[SwitchConfig, Depends(get_config)],
-) -> dict:
- response.headers["Cache-Control"] = "no-store"
- launch = await worker_launch(session, agent)
- connection = await session.scalar(
- select(ProviderConnection).where(
- ProviderConnection.tenant_id == require_tenant_id(),
- ProviderConnection.user_id == launch.owner_id,
- ProviderConnection.provider == launch.spec.get("provider", "claude"),
- )
- )
- if connection is None:
- return {"status": "revoked"}
- return {
- "status": "connected",
- "kind": connection.kind,
- "provider": connection.provider,
- "revision": str(connection.verified_at),
- "credential": config.keyring.decrypt(connection.encrypted_credential),
- }
-
-
-async def worker_launch(session: AsyncSession, agent: Agent) -> HostedLaunch:
- launch = await session.scalar(
- select(HostedLaunch).where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.agent_id == agent.id,
- HostedLaunch.owner_id == agent.owner_id,
- HostedLaunch.desired_state == "running",
- HostedLaunch.state != "error",
- )
- )
- if (
- launch is None
- or await session.get(TenantMember, (require_tenant_id(), launch.owner_id))
- is None
- ):
- raise HTTPException(
- 403, "This worker is not authorized to execute cloud operations."
- )
- return launch
-
-
-async def operation_launch(session: AsyncSession, agent: Agent) -> HostedLaunch:
- launch = await worker_launch(session, agent)
- await session.execute(
- text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
- {"key": f"hosted-launch:{require_tenant_id()}:{launch.id}"},
- )
- session.expire(launch)
- return await worker_launch(session, agent)
-
-
-class ProviderStatus(BaseModel):
- model_config = ConfigDict(extra="forbid")
- authenticated: bool
- revision: str
-
-
[email protected]("/provider-status")
-async def provider_status(
- body: ProviderStatus,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict:
- current = await worker_launch(session, agent)
- await ProviderConnectionStore().wait_user(session, current.owner_id)
- launch = await session.scalar(
- select(HostedLaunch)
- .where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.id == current.id,
- HostedLaunch.desired_state == "running",
- HostedLaunch.state != "error",
- )
- .with_for_update()
- .execution_options(populate_existing=True)
- )
- if launch is None:
- raise HTTPException(409, "The worker changed during verification.")
- connection = await session.get(
- ProviderConnection,
- (require_tenant_id(), launch.owner_id, launch.spec.get("provider", "claude")),
- )
- if connection is None:
- raise HTTPException(409, "The provider was disconnected.")
- if str(connection.verified_at) != body.revision:
- raise HTTPException(409, "The provider credential changed during verification.")
- connection.verification_status = "verified" if body.authenticated else "configured"
- if not body.authenticated:
- launch.error = (
- "The provider could not authenticate on the worker. "
- "Sign in again locally, then reconnect it and retry."
- )
- launch.state = "error"
- await session.commit()
- return {"verified": body.authenticated}
-
-
-class WorkerFence(BaseModel):
- model_config = ConfigDict(extra="forbid")
- connection_id: str = Field(min_length=1, max_length=128)
- generation: int = Field(ge=0)
-
-
-async def locked_operation(
- session: AsyncSession, launch: HostedLaunch, operation_id: UUID
-) -> HostedOperation:
- operation = await session.scalar(
- select(HostedOperation)
- .where(
- HostedOperation.tenant_id == require_tenant_id(),
- HostedOperation.id == str(operation_id),
- HostedOperation.launch_id == launch.id,
- )
- .with_for_update()
- )
- if operation is None:
- raise HTTPException(404, "Cloud operation not found.")
- return operation
-
-
[email protected]("/operations/{operation_id}/claim")
-async def claim_operation(
- operation_id: UUID,
- body: WorkerFence,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- session: Annotated[AsyncSession, Depends(get_session)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict:
- conn = require_worker(
- protocol.connections, agent, body.connection_id, body.generation
- )
- launch = await operation_launch(session, agent)
- operation = await locked_operation(session, launch, operation_id)
- assert conn.worker is not None
- reclaim = (
- operation.state == "claimed"
- and operation.claimed_boot_id == conn.worker.boot_id
- )
- if (
- (operation.state != "queued" and not reclaim)
- or operation.launch_revision != launch.revision
- or conn.worker.launch_revision != launch.revision
- ):
- raise refusal(
- 409, "operation_not_claimable", "This operation cannot be claimed."
- )
- operation.state = "claimed"
- operation.claimed_by = f"{protocol.event_buffer.boot}:{conn.id}:{body.generation}"
- operation.claimed_boot_id = conn.worker.boot_id
- operation.updated_at = datetime.now(UTC)
- await session.commit()
- return operation_summary(operation)
-
-
-class OperationResult(BaseModel):
- model_config = ConfigDict(extra="forbid")
- connection_id: str = Field(min_length=1, max_length=128)
- generation: int = Field(ge=0)
- state: Literal["applied", "failed", "unknown"]
- error: str | None = Field(max_length=512)
-
-
[email protected]("/operations/{operation_id}/result")
-async def operation_result(
- operation_id: UUID,
- body: OperationResult,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- session: Annotated[AsyncSession, Depends(get_session)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict:
- conn = require_worker(
- protocol.connections, agent, body.connection_id, body.generation
- )
- launch = await operation_launch(session, agent)
- operation = await locked_operation(session, launch, operation_id)
- assert conn.worker is not None
- if operation.launch_revision != launch.revision:
- raise HTTPException(
- 409, "The operation belongs to an earlier worker generation."
- )
- if operation.claimed_boot_id != conn.worker.boot_id:
- raise refusal(
- 409, "operation_not_claimed", "This worker did not claim the operation."
- )
- if operation.state not in ("claimed", body.state):
- raise HTTPException(409, "This operation is no longer awaiting a result.")
- operation.state = body.state
- operation.error = body.error
- operation.updated_at = datetime.now(UTC)
- await session.commit()
- return operation_summary(operation)
-
-
[email protected]("/github-credential")
-async def repository_credential(
- response: Response,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- session: Annotated[AsyncSession, Depends(get_session)],
- config: Annotated[SwitchConfig, Depends(get_config)],
-) -> dict:
- response.headers["Cache-Control"] = "no-store"
- if not config.hosted_controller_config_path or not config.hosted_github_config_path:
- raise HTTPException(503, "Cloud repository credentials are not enabled.")
- settings = HostedControllerSettings.model_validate_json(
- Path(config.hosted_controller_config_path).read_text() # nosemgrep
- )
- if settings.tenant_id != require_tenant_id():
- raise HTTPException(403, "This agent is not a managed cloud worker.")
- launch = await session.scalar(
- select(HostedLaunch).where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.agent_id == agent.id,
- HostedLaunch.owner_id == agent.owner_id,
- HostedLaunch.state != "error",
- HostedLaunch.desired_state == "running",
- )
- )
- if launch is None:
- raise HTTPException(403, "This agent is not a managed cloud worker.")
- if await session.get(TenantMember, (require_tenant_id(), launch.owner_id)) is None:
- raise HTTPException(
- 403, "The cloud agent owner is no longer a workspace member."
- )
- revision = launch.revision
- launch_id = launch.id
- owner_id = launch.owner_id
- installation_id = launch.spec["installation_id"]
- repository_id = launch.spec["repository_id"]
- github = GitHubConnections(config.hosted_github_config_path)
- await session.commit()
- saved_github = await github_credentials(owner_id, session, config, github)
- if saved_github is None:
- raise HTTPException(422, "The owner must reconnect GitHub.")
- credentials, github_revision = saved_github
- signer = GitHubInstallationCredentials(
- github.client_id, str(settings.github_private_key_path)
- )
- try:
- credential = await signer.issue(
- github,
- credentials["access_token"],
- installation_id,
- repository_id,
- )
- except GitHubUnavailableError as error:
- raise HTTPException(503, str(error)) from None
- except GitHubError as error:
- raise HTTPException(422, str(error)) from None
- try:
- await github_lock(session, owner_id)
- await session.execute(
- text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
- {"key": f"hosted-launch:{require_tenant_id()}:{launch_id}"},
- )
- launch = await session.get(
- HostedLaunch, (require_tenant_id(), launch_id), populate_existing=True
- )
- row = await session.scalar(
- select(ProviderConnection)
- .where(*conditions(owner_id))
- .execution_options(populate_existing=True)
- )
- if (
- launch is None
- or launch.revision != revision
- or launch.desired_state != "running"
- or launch.state == "error"
- or row is None
- or row.verified_at != github_revision
- or await session.get(
- TenantMember, (require_tenant_id(), owner_id), populate_existing=True
- )
- is None
- ):
- raise HTTPException(
- 409, "Cloud launch or GitHub connection changed. Please retry."
- )
- remember_repository_token(session, launch, credential, config)
- await session.commit()
- return {
- "token": credential.token,
- "expires_at": credential.expires_at.isoformat(),
- "repository": credential.repository_name,
- }
- except BaseException as error:
- await discard_github_credential(
- session, signer, credential, config, launch_id, error
- )
- raise
diff --git a/core/switch_core/bridges/agent/api/hosted_worker_routes.py b/core/switch_core/bridges/agent/api/hosted_worker_routes.py
deleted file mode 100644
index e3e19254f..000000000
--- a/core/switch_core/bridges/agent/api/hosted_worker_routes.py
+++ /dev/null
@@ -1,638 +0,0 @@
-"""A hosted agent's worker: attaching, and the up-calls only the worker may make.
-
-A hosted agent's connections are all workers: every open proves the
-capability the controller minted for the launch's current revision. Each
-up-call below names the connection and generation it comes from and is
-refused unless that is the attached worker still.
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-import time
-from collections.abc import Sequence
-from dataclasses import dataclass
-from datetime import UTC, datetime
-from typing import Annotated, Any, Literal, cast
-
-from fastapi import APIRouter, Depends, HTTPException, Request
-from pydantic import BaseModel, ConfigDict, Field, ValidationError
-from sqlalchemy import select, text
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.bridges.agent.auth import get_agent_from_scope
-from switch_core.bridges.agent.dependencies import get_protocol, get_session
-from switch_core.bridges.agent.protocol.agent_connections import (
- AgentConnection,
- AgentConnectionRegistry,
- ClientDeclaration,
- SupersededControlError,
- UnfencedControlError,
- UnknownConnectionError,
- WorkerAlreadyAttachedError,
-)
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.bridges.agent.protocol.hosted_workers import (
- HOSTED_PROTOCOL_REVISION,
- HOSTED_WORKER_ONLY_MESSAGE,
- HOSTED_WORKER_STATE_VERSION,
- IDLE_FRESH_FOR_SECONDS,
- IDLE_REPORT_EVERY_SECONDS,
- NOTICE_MESSAGES,
- RELAY_REPLY_ENVELOPE_BYTES,
- RELAY_REPLY_LIMIT_BYTES,
- IdleReport,
- WorkerBinding,
- hosted_launch_of,
-)
-from switch_core.config import SwitchConfig
-from switch_core.db.models import (
- Agent,
- Client,
- HostedLaunch,
- Message,
- TenantMember,
- require_tenant_id,
-)
-from switch_core.db.session_scope import tenant_session
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore, lock_launch
-from switch_core.db.stores.hosted_mailbox_store import (
- ACKS_PER_CALL,
- HostedMailboxStore,
- MailboxNotice,
- MailboxOutcome,
- by_room,
-)
-
-logger = logging.getLogger(__name__)
-
-router = APIRouter()
-
-
-def refusal(status: int, code: str, message: str) -> HTTPException:
- return HTTPException(status_code=status, detail={"code": code, "message": message})
-
-
-def hosted_worker_only() -> HTTPException:
- return refusal(403, "hosted_worker_only", HOSTED_WORKER_ONLY_MESSAGE)
-
-
-def require_worker(
- registry: AgentConnectionRegistry, agent: Agent, connection_id: str, generation: int
-) -> AgentConnection:
- """The attached worker making this call, or the refusal saying why not."""
- if hosted_launch_of(agent.metadata_) is None:
- raise hosted_worker_only()
- try:
- conn = registry.require_current(agent.id, connection_id, generation=generation)
- except (
- SupersededControlError,
- UnfencedControlError,
- UnknownConnectionError,
- ) as exc:
- raise refusal(409, "generation_changed", str(exc)) from exc
- if conn.worker is None:
- raise hosted_worker_only()
- return conn
-
-
-def require_self(agent_id: str, agent: Agent) -> None:
- if agent_id != agent.id:
- raise HTTPException(
- status_code=403, detail=f"authenticated as agent {agent.id}, not {agent_id}"
- )
-
-
-@dataclass(frozen=True, slots=True)
-class WorkerAttach:
- """What a worker attach proved, and who it takes over once opened."""
-
- binding: WorkerBinding
- attached: dict[str, Any]
- takes_over: AgentConnection | None
-
-
-async def admit_worker(
- *,
- session: AsyncSession,
- registry: AgentConnectionRegistry,
- config: SwitchConfig,
- agent: Agent,
- launch_id: str,
- connection_id: str,
- declaration: ClientDeclaration,
- capability: str | None,
- boot_id: str | None,
- instance_id: str | None,
- state_version: int | None,
-) -> WorkerAttach:
- """Check a hosted agent's stream open, under the launch lock the caller holds open.
-
- The lock is taken here and released with the caller's transaction, so the
- open and the binding happen before any revision bump can land.
- """
- if not capability:
- raise refusal(
- 403,
- "worker_capability_required",
- "This agent runs on a cloud worker; opening its stream needs the "
- "worker capability.",
- )
- if (declaration.speaks or 0) < HOSTED_PROTOCOL_REVISION:
- raise refusal(
- 426,
- "upgrade_required",
- f"A cloud worker must speak agent-protocol {HOSTED_PROTOCOL_REVISION} "
- "or later.",
- )
- if (state_version or 0) < HOSTED_WORKER_STATE_VERSION:
- raise refusal(
- 426,
- "upgrade_required",
- f"A cloud worker must have migrated its volume to state version "
- f"{HOSTED_WORKER_STATE_VERSION} before it attaches.",
- )
- if not boot_id or not instance_id:
- raise HTTPException(
- status_code=400,
- detail="X-Switch-Host-Boot-Id and X-Switch-Host-Instance-Id are required "
- "to attach a cloud worker",
- )
- store = HostedLaunchStore()
- launch = await store.locked(session, launch_id)
- if (
- launch is None
- or launch.agent_id != agent.id
- or launch.owner_id != agent.owner_id
- or launch.desired_state != "running"
- or launch.state == "error"
- or not store.capability_matches(launch, capability)
- or await session.get(TenantMember, (require_tenant_id(), launch.owner_id))
- is None
- ):
- raise refusal(
- 403,
- "worker_capability_obsolete",
- "This worker capability is not the one for the launch's current "
- "revision, or the launch is not running.",
- )
- try:
- takes_over = registry.admit_worker(agent.id, connection_id, boot_id)
- except WorkerAlreadyAttachedError as exc:
- raise refusal(409, exc.code, str(exc)) from exc
- return WorkerAttach(
- binding=WorkerBinding(
- launch_id=launch.id,
- launch_revision=launch.revision,
- boot_id=boot_id,
- instance_id=instance_id,
- ),
- attached={
- "launch_revision": launch.revision,
- "limits": {"sessions_per_agent": config.hosted_sessions_per_agent},
- "idle": {
- "report_every_s": IDLE_REPORT_EVERY_SECONDS,
- "fresh_for_s": IDLE_FRESH_FOR_SECONDS,
- },
- "credential_revision": await store.credential_revision(session, launch),
- "queued_operations": await store.queued_operation_ids(
- session, launch, boot_id
- ),
- "relay_fence": launch.relay_seq,
- "cancelled": await HostedMailboxStore().cancelled_entries(
- session, agent.id
- ),
- },
- takes_over=takes_over,
- )
-
-
-class RelayReplyError(BaseModel):
- model_config = ConfigDict(extra="forbid")
- code: str = Field(min_length=1, max_length=64)
- message: str = Field(max_length=2048)
-
-
-class RelayReply(BaseModel):
- model_config = ConfigDict(extra="forbid")
- connection_id: str
- generation: int
- ok: bool
- value: Any = None
- error: RelayReplyError | None = None
-
-
-class RelayPush(BaseModel):
- model_config = ConfigDict(extra="forbid")
- subscription: str = Field(min_length=1)
- seq: int = Field(ge=0)
- event: Any = None
- failure: Any = None
- health: Any = None
-
-
-class RelayPushes(BaseModel):
- model_config = ConfigDict(extra="forbid")
- connection_id: str
- generation: int
- pushes: list[RelayPush]
-
-
[email protected]("/{agent_id}/connection/relay/push")
-async def relay_push(
- agent_id: str,
- body: RelayPushes,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict[str, list[str]]:
- """Live events for Console views, forwarded in order per subscription."""
- require_self(agent_id, agent)
- conn = require_worker(
- protocol.connections, agent, body.connection_id, body.generation
- )
- pushes = [push.model_dump(exclude_unset=True) for push in body.pushes]
- unsubscribe = protocol.connections.relay_views.deliver(
- agent.id, conn.stream_generation, pushes
- )
- return {"unsubscribe": unsubscribe}
-
-
[email protected]("/{agent_id}/connection/relay/{relay_id}")
-async def relay_reply(
- agent_id: str,
- relay_id: str,
- request: Request,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict[str, bool]:
- """The worker's answer to one relayed Console request."""
- require_self(agent_id, agent)
- raw = await request.body()
- if len(raw) > RELAY_REPLY_LIMIT_BYTES + RELAY_REPLY_ENVELOPE_BYTES:
- raise refusal(
- 413, "too_large", "A relay reply is at most 1 MiB; page the answer."
- )
- try:
- body = RelayReply.model_validate_json(raw)
- except ValidationError as exc:
- raise HTTPException(status_code=422, detail=exc.errors()) from exc
- conn = require_worker(
- protocol.connections, agent, body.connection_id, body.generation
- )
- relay = protocol.connections.relays.get(relay_id)
- if (
- relay is None
- or relay.agent_id != agent.id
- or relay.tenant_id != require_tenant_id()
- ):
- raise HTTPException(status_code=404, detail="Relay not found.")
- if relay.connection_id != conn.id or relay.generation != conn.stream_generation:
- raise refusal(
- 409,
- "generation_changed",
- "This relay was dispatched to another connection or generation.",
- )
- if relay.future.done():
- raise refusal(409, "relay_resolved", "This relay has already been answered.")
- answer: dict[str, Any] = {"ok": body.ok}
- if body.ok:
- answer["value"] = body.value
- else:
- if body.error is None:
- raise HTTPException(
- status_code=422, detail="A failed relay reply carries an error."
- )
- answer["error"] = body.error.model_dump()
- protocol.connections.relays.resolve(relay, answer)
- return {"ok": True}
-
-
-class IdleReason(BaseModel):
- model_config = ConfigDict(extra="forbid")
- kind: str = Field(min_length=1, max_length=64)
- session_id: str | None
- count: int = Field(ge=0)
-
-
-class IdleSessions(BaseModel):
- model_config = ConfigDict(extra="forbid")
- total: int = Field(ge=0)
- live: int = Field(ge=0)
- parked: int = Field(ge=0)
- failed: int = Field(ge=0)
-
-
-class IdleReportBody(BaseModel):
- model_config = ConfigDict(extra="forbid")
- connection_id: str
- generation: int
- report_seq: int = Field(ge=0)
- relays_through: int = Field(ge=0)
- busy: bool
- reasons: list[IdleReason] = Field(max_length=1000)
- sessions: IdleSessions
-
-
[email protected]("/{agent_id}/connection/idle")
-async def idle_report(
- agent_id: str,
- body: IdleReportBody,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict[str, Any]:
- """Keep the worker's latest idle report; answer the doorbells it may have missed."""
- require_self(agent_id, agent)
- conn = require_worker(
- protocol.connections, agent, body.connection_id, body.generation
- )
- assert conn.worker is not None
- protocol.connections.record_idle_report(
- conn,
- IdleReport(
- report_seq=body.report_seq,
- relays_through=body.relays_through,
- busy=body.busy,
- reasons=[reason.model_dump() for reason in body.reasons],
- sessions=body.sessions.model_dump(),
- launch_revision=conn.worker.launch_revision,
- generation=conn.stream_generation,
- received_monotonic=time.monotonic(),
- received_at=datetime.now(UTC),
- ),
- )
- store = HostedLaunchStore()
- launch = await session.get(
- HostedLaunch, (require_tenant_id(), conn.worker.launch_id)
- )
- if launch is None or launch.revision != conn.worker.launch_revision:
- return {"queued_operations": [], "credential_revision": None}
- return {
- "queued_operations": await store.queued_operation_ids(
- session, launch, conn.worker.boot_id
- ),
- "credential_revision": await store.credential_revision(session, launch),
- }
-
-
-NoticeReason = Literal[
- "startup",
- "delivery",
- "conversation",
- "capacity",
- "auto_start_off",
- "stopped",
- "expired",
- "cancelled",
- "revoked",
- "upgrade",
-]
-
-
-#: Reasons only Core posts, for outcomes only the wake mailbox knows.
-CoreNoticeReason = Literal[
- "removed",
- "started_before_stop",
- "started_before_expiry",
- "expired_uncertain",
- "cutover_uncertain",
- "cutover_unrecoverable",
- "cutover_run_now",
-]
-
-
-class RoomNotice(BaseModel):
- model_config = ConfigDict(extra="forbid")
- connection_id: str
- generation: int
- room_id: str = Field(min_length=1)
- message_id: str = Field(min_length=1)
- thread_id: str | None
- reason: NoticeReason
-
-
-async def post_room_notice(
- protocol: AgentCore,
- agent: Agent,
- room_id: str,
- message_id: str,
- thread_id: str | None,
- reason: NoticeReason | CoreNoticeReason,
-) -> bool:
- """Tell a room why a message was not processed, once per message and reason.
-
- True when this call posted it. Core's own reasons go through here too.
- """
- return await post_notice_once(
- protocol,
- agent,
- room_id,
- key=json.dumps([agent.id, room_id, message_id, reason]),
- body=NOTICE_MESSAGES[reason].format(name=agent.name),
- thread_id=thread_id,
- anchor=message_id,
- )
-
-
-async def post_notice_once(
- protocol: AgentCore,
- agent: Agent,
- room_id: str,
- *,
- key: str,
- body: str,
- thread_id: str | None,
- anchor: str | None,
-) -> bool:
- """Post `body` to the room unless the agent already posted one under `key`.
-
- `anchor`, when set, is the message the notice is about, and must be in
- the room. True when this call posted it.
- """
- async with tenant_session(protocol.session_factory, require_tenant_id()) as db:
- await db.execute(
- text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
- {"key": f"room-failure:{require_tenant_id()}:{key}"},
- )
- if anchor is not None:
- target = await db.scalar(
- select(Message.id).where(
- Message.room_id == room_id, Message.transport_event_id == anchor
- )
- )
- if target is None:
- raise ValueError(f"Message {anchor} is not in room {room_id}")
- sender = (
- select(Client.transport_user_id)
- .join(Agent, Agent.client_id == Client.id)
- .where(Agent.id == agent.id)
- .scalar_subquery()
- )
- previous = await db.scalar(
- select(Message.id)
- .where(
- Message.room_id == room_id,
- Message.sender_id == sender,
- Message.content["switch_room_failure"].astext == key,
- )
- .limit(1)
- )
- if previous is not None:
- return False
- await protocol.send_message(
- agent.id,
- room_id,
- body,
- thread_id=thread_id,
- extra_content={"switch_room_failure": key},
- )
- await db.commit()
- return True
-
-
[email protected]("/{agent_id}/room-notices")
-async def room_notice(
- agent_id: str,
- body: RoomNotice,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict[str, Any]:
- """A failure notice the worker owes a room, posted at most once."""
- require_self(agent_id, agent)
- require_worker(protocol.connections, agent, body.connection_id, body.generation)
- try:
- posted = await post_room_notice(
- protocol, agent, body.room_id, body.message_id, body.thread_id, body.reason
- )
- except ValueError as exc:
- raise HTTPException(status_code=404, detail=str(exc)) from exc
- except PermissionError as exc:
- raise HTTPException(status_code=403, detail=str(exc)) from exc
- return {"room_id": body.room_id, "message_id": body.message_id, "posted": posted}
-
-
-async def post_mailbox_notices(
- protocol: AgentCore, notices: Sequence[MailboxNotice]
-) -> None:
- """Post what the mailbox owes rooms, one notice per room and reason, after the commit.
-
- Each row keeps its `notice_owed` mark until the room has the notice, so a
- send that fails is logged and left for the upkeep to retry; the per
- message and reason receipt keeps a retry from posting it twice. A notice
- for a deleted agent can never be posted: it is marked dropped, not posted.
- """
- agents: dict[str, Agent | None] = {}
- store = HostedMailboxStore()
- for notice, message_ids in by_room(notices):
- if notice.agent_id not in agents:
- async with tenant_session(
- protocol.session_factory, require_tenant_id()
- ) as db:
- agents[notice.agent_id] = await db.get(Agent, notice.agent_id)
- agent = agents[notice.agent_id]
- if agent is None:
- logger.warning(
- "Mailbox notice %s for room %s dropped: agent %s was deleted",
- notice.reason,
- notice.room_id,
- notice.agent_id,
- )
- async with tenant_session(
- protocol.session_factory, require_tenant_id()
- ) as db:
- await store.notice_dropped(
- db,
- notice.agent_id,
- notice.room_id,
- notice.reason,
- message_ids,
- "agent_deleted",
- )
- await db.commit()
- continue
- try:
- await post_room_notice(
- protocol,
- agent,
- notice.room_id,
- notice.message_id,
- notice.thread_id,
- cast(NoticeReason | CoreNoticeReason, notice.reason),
- )
- except Exception:
- logger.warning(
- "Could not post the %s notice for message %s in room %s; "
- "the mailbox upkeep retries it",
- notice.reason,
- notice.message_id,
- notice.room_id,
- exc_info=True,
- )
- continue
- async with tenant_session(protocol.session_factory, require_tenant_id()) as db:
- await store.notice_posted(
- db, notice.agent_id, notice.room_id, notice.reason, message_ids
- )
- await db.commit()
-
-
-class MailboxAckEntry(BaseModel):
- model_config = ConfigDict(extra="forbid")
- room_id: str = Field(min_length=1)
- message_id: str = Field(min_length=1)
- outcome: MailboxOutcome
- reason: str | None = Field(default=None, max_length=64)
-
-
-class MailboxAcks(BaseModel):
- model_config = ConfigDict(extra="forbid")
- connection_id: str
- generation: int
- entries: list[MailboxAckEntry] = Field(max_length=ACKS_PER_CALL)
-
-
[email protected]("/{agent_id}/connection/mailbox/ack")
-async def mailbox_ack(
- agent_id: str,
- body: MailboxAcks,
- agent: Annotated[Agent, Depends(get_agent_from_scope)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict[str, Any]:
- """The attached worker says how far each delivered row has got; forward moves only."""
- require_self(agent_id, agent)
- conn = require_worker(
- protocol.connections, agent, body.connection_id, body.generation
- )
- assert conn.worker is not None
- await lock_launch(session, conn.worker.launch_id)
- launch = await session.get(
- HostedLaunch,
- (require_tenant_id(), conn.worker.launch_id),
- populate_existing=True,
- )
- if launch is None or launch.revision != conn.worker.launch_revision:
- raise refusal(
- 409,
- "generation_changed",
- "The launch moved to a newer revision; acknowledge again after "
- "reattaching.",
- )
- states, notices = await HostedMailboxStore().ack(
- session,
- agent.id,
- [(entry.room_id, entry.message_id, entry.outcome) for entry in body.entries],
- )
- await session.commit()
- await post_mailbox_notices(protocol, notices)
- return {
- "entries": [
- {
- "room_id": entry.room_id,
- "message_id": entry.message_id,
- "state": states.get((entry.room_id, entry.message_id)),
- }
- for entry in body.entries
- ]
- }
diff --git a/core/switch_core/bridges/agent/api/operations.py b/core/switch_core/bridges/agent/api/operations.py
index 952746706..561b6f92c 100644
--- a/core/switch_core/bridges/agent/api/operations.py
+++ b/core/switch_core/bridges/agent/api/operations.py
@@ -35,7 +35,6 @@ from switch_core.bridges.agent.operations.callctx import (
)
from switch_core.bridges.agent.protocol.agent_connections import UnknownConnectionError
from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.bridges.agent.protocol.hosted_workers import CodedPermissionError
from switch_core.budgets import BudgetExceeded
from switch_core.db.models import Agent
from switch_core.observability.catalogue import (
@@ -311,8 +310,6 @@ async def post_operation(
raise HTTPException(status_code=404, detail=str(exc)) from exc
except BadArgumentsError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
- except CodedPermissionError as exc:
- raise HTTPException(status_code=403, detail=exc.detail) from exc
except BudgetExceeded as exc:
raise HTTPException(status_code=429, detail=str(exc)) from exc
except GuardrailBlockedError as exc:
diff --git a/core/switch_core/bridges/agent/app.py b/core/switch_core/bridges/agent/app.py
index 960e7a9a6..61e4b6c32 100644
--- a/core/switch_core/bridges/agent/app.py
+++ b/core/switch_core/bridges/agent/app.py
@@ -9,16 +9,6 @@ from starlette.types import ASGIApp, Receive, Scope, Send
from switch_core.bridges.agent.api.activity_routes import router as activity_router
from switch_core.bridges.agent.api.handlers import router as api_router
-from switch_core.bridges.agent.api.hosted_cutover_routes import (
- router as hosted_cutover_router,
-)
-from switch_core.bridges.agent.api.hosted_machine_routes import (
- router as hosted_machine_router,
-)
-from switch_core.bridges.agent.api.hosted_routes import router as hosted_router
-from switch_core.bridges.agent.api.hosted_worker_routes import (
- router as hosted_worker_router,
-)
from switch_core.bridges.agent.api.operations import router as operations_router
from switch_core.bridges.agent.api.version_routes import router as version_router
from switch_core.bridges.agent.api_key_cache import ApiKeyCache
@@ -157,10 +147,6 @@ def create_agent_bridge_app(
app.add_exception_handler(SessionError, session_error_response)
app.include_router(activity_router, tags=["session activity"])
app.include_router(api_router, prefix="/agents", tags=["api"])
- app.include_router(hosted_worker_router, prefix="/agents", tags=["hosted"])
- app.include_router(hosted_cutover_router, prefix="/agents", tags=["hosted"])
- app.include_router(hosted_router, tags=["hosted"])
- app.include_router(hosted_machine_router, tags=["hosted"])
app.include_router(operations_router)
app.include_router(deeplink_router, tags=["deeplink"])
app.include_router(version_router, tags=["version"])
diff --git a/core/switch_core/bridges/agent/commands.py b/core/switch_core/bridges/agent/commands.py
index aa251bb66..7d3cad181 100644
--- a/core/switch_core/bridges/agent/commands.py
+++ b/core/switch_core/bridges/agent/commands.py
@@ -13,7 +13,6 @@ from switch_core.aliases import (
check_alias_collisions,
validate_alias_format,
)
-from switch_core.bridges.agent.protocol.hosted_workers import hosted_launch_of
from switch_core.bridges.agent.protocol.statuses import compute_agent_statuses
from switch_core.bridges.agent.protocol.types import (
AgentEvent,
@@ -21,10 +20,8 @@ from switch_core.bridges.agent.protocol.types import (
CommandPayload,
)
from switch_core.clients.mentions import mention_tokens as _mention_tokens
-from switch_core.db.models import CollaborationBridge, HostedLaunch, HostedMachine, Room
-from switch_core.db.session_scope import tenant_session
+from switch_core.db.models import CollaborationBridge, Room
from switch_core.db.stores.agent_runtime_state_store import AgentRuntimeStateStore
-from switch_core.db.stores.hosted_machine_store import idle_sleeping
from switch_core.events import CommandEvent
from switch_core.gateway.known_agents import known_agent_for
from switch_core.transport import RoomRef
@@ -401,62 +398,6 @@ async def _cmd_help(
await _reply(client, room, event, "\n".join(lines))
-async def _reply_hosted_asleep(
- client: AgentConsumer,
- room: RoomRef,
- event: CommandEvent,
- agent: Agent,
- launch_id: str,
- command: str,
-) -> bool:
- """Answer an undelivered room control for a hosted agent that is asleep.
-
- A sleeping worker holds no placement, so this is decided from the launch,
- not from whether a session was placed. `!reset` wakes the worker but is
- not queued: a destructive command is never run later than it was asked.
- A machine in error is never woken; the room is told of the error instead.
- Returns False when the worker is awake, so the ordinary reply applies.
- """
- async with tenant_session(client.session_factory, client.tenant_id) as session:
- launch = await session.get(HostedLaunch, (agent.tenant_id, launch_id))
- machine = (
- None
- if launch is None or launch.machine_id is None
- else await session.get(HostedMachine, (agent.tenant_id, launch.machine_id))
- )
- if launch is None or machine is None or launch.desired_state != "running":
- return False
- if machine.state == "error":
- await _reply(
- client,
- room,
- event,
- f"@{agent.name}'s cloud machine has a problem, so the {command} was not sent. Its owner can check it in Switch Console.",
- )
- return True
- if not idle_sleeping(machine):
- return False
- if command == "reset":
- hosted = await client._note_hosted_addressed(agent, None)
- if hosted is not None and hosted.refusal is not None:
- await _reply(client, room, event, hosted.refusal)
- return True
- await _reply(
- client,
- room,
- event,
- f"The cloud worker is waking up. The reset was not queued. Wait until the agent is back (usually about a minute), then send !reset @{agent.name} again to start a fresh conversation.",
- )
- return True
- await _reply(
- client,
- room,
- event,
- f"@{agent.name} is asleep, so nothing is running. The {command} was not sent.",
- )
- return True
-
-
async def _dispatch_control_command(
client: AgentConsumer,
room: RoomRef,
@@ -493,7 +434,6 @@ async def _dispatch_control_command(
# know which session that is, so the frame names none.
controller_backed = client._connections.controllers.is_bound(agent.id)
placed = client._connections.session_in_room(agent.id, meta.room_id)
- launch_id = hosted_launch_of(agent.metadata_)
if placed is not None or controller_backed:
if not event.message_id:
await _reply(
@@ -514,16 +454,9 @@ async def _dispatch_control_command(
surface=await _room_surface(client, meta.room_id),
requester_name=event.user_name,
)
- delivered = client._connections.relay_session_command(
- agent.id, frame, worker_only=launch_id is not None
- )
- if delivered:
+ if client._connections.relay_session_command(agent.id, frame):
await _reply(client, room, event, ack)
return
- if launch_id is not None and await _reply_hosted_asleep(
- client, room, event, agent, launch_id, command
- ):
- return
if placed is not None or controller_backed:
await _reply(
client,
diff --git a/core/switch_core/bridges/agent/hosted_cutover.py b/core/switch_core/bridges/agent/hosted_cutover.py
deleted file mode 100644
index 9249459ef..000000000
--- a/core/switch_core/bridges/agent/hosted_cutover.py
+++ /dev/null
@@ -1,776 +0,0 @@
-"""Merging a retained worker volume's pre-cutover work with what Core captured.
-
-The cutover-manifest revision copied the old server-side session tables into
-`hosted_cutover_items`; a preflight check of the stopped volume lists what its
-journals hold, and `hosted-cutover-upgrade record` applies it before the old
-tables are dropped. Every record for one room message is merged by
-`(agent, room, message)` and decided once, strongest evidence first, so a
-message runs at most once: only an `import` becomes a mailbox row, under
-`origin = cutover`, queued once the mailbox exists.
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-from collections.abc import Iterable
-from dataclasses import dataclass
-from datetime import UTC, datetime
-from typing import Annotated, Any, Literal
-from uuid import uuid4
-
-from pydantic import BaseModel, ConfigDict, Field
-from sqlalchemy import ColumnElement, distinct, or_, select, update
-from sqlalchemy.ext.asyncio import AsyncSession
-from sqlalchemy.orm import load_only
-
-from switch_core.attachments import parse_attachment_group
-from switch_core.bridges.agent.api.hosted_worker_routes import post_notice_once
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.bridges.agent.protocol.hosted_workers import NOTICE_MESSAGES
-from switch_core.bridges.agent.protocol.types import (
- AgentEvent,
- AttachmentRef,
- MessagePayload,
-)
-from switch_core.clients.admin_messages import (
- PLATFORM_MARKER,
- platform_on_behalf_of,
- platform_replies_in_channel,
-)
-from switch_core.db.models import (
- Agent,
- HostedCutoverItem,
- HostedCutoverVolume,
- MediaBlob,
- Message,
- MessageAttachment,
- Room,
- require_tenant_id,
-)
-from switch_core.db.session_scope import tenant_session
-from switch_core.db.stores.hosted_mailbox_store import HostedMailboxStore, MailboxEntry
-from switch_core.transport.stored import to_inbound
-from switch_core.transport.types import InboundMedia, InboundMessage
-
-logger = logging.getLogger(__name__)
-
-HostState = Literal["accepted", "dispatched", "finished"]
-Disposition = Literal[
- "ran",
- "uncertain",
- "unrecoverable",
- "import",
- "settled_by_host",
- "owner_notice",
- "interrupted",
- "preserved",
-]
-CutoverNoticeReason = Literal[
- "cutover_uncertain",
- "cutover_unrecoverable",
- "cutover_interrupted",
- "cutover_run_now",
- "conversation",
-]
-
-_HOST_RANK: dict[str | None, int] = {
- None: 0,
- "accepted": 1,
- "dispatched": 2,
- "finished": 3,
-}
-
-MANIFEST_ITEM_LIMIT = 100_000
-
-
-class RoomMessageRecord(BaseModel):
- """A room message the volume still holds: pending in its room inbox, or a host command."""
-
- model_config = ConfigDict(extra="forbid")
- kind: Literal["room_message"]
- session_id: str | None
- room_id: str = Field(min_length=1)
- message_id: str = Field(min_length=1)
- thread_id: str | None
- room_pending: bool
- failure_notified: bool
- host: HostState | None
-
-
-class ConsoleCommandRecord(BaseModel):
- model_config = ConfigDict(extra="forbid")
- kind: Literal["console_command"]
- session_id: str = Field(min_length=1)
- command_id: str = Field(min_length=1)
- host: HostState
-
-
-class RequestOpenRecord(BaseModel):
- model_config = ConfigDict(extra="forbid")
- kind: Literal["request_open"]
- session_id: str = Field(min_length=1)
- request_id: str = Field(min_length=1)
- room_id: str | None
- thread_id: str | None
-
-
-class ResetPendingRecord(BaseModel):
- model_config = ConfigDict(extra="forbid")
- kind: Literal["reset_pending"]
- session_id: str = Field(min_length=1)
-
-
-ManifestItem = Annotated[
- RoomMessageRecord | ConsoleCommandRecord | RequestOpenRecord | ResetPendingRecord,
- Field(discriminator="kind"),
-]
-
-
-class CutoverManifest(BaseModel):
- model_config = ConfigDict(extra="forbid")
- manifest_sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
- items: list[ManifestItem] = Field(max_length=MANIFEST_ITEM_LIMIT)
-
-
-class CutoverConflict(Exception):
- """The volume's manifest was already applied with a different digest."""
-
-
-class CutoverUnrecorded(Exception):
- """The volume has no recorded manifest to confirm."""
-
-
-@dataclass(frozen=True)
-class CutoverNotice:
- """A notice owed to a room for one cutover item, posted after the commit."""
-
- item_id: str
- agent_id: str
- room_id: str
- message_id: str | None
- request_id: str | None
- thread_id: str | None
- reason: CutoverNoticeReason
-
-
-@dataclass(frozen=True)
-class WorkerEvidence:
- """Every record the volume holds for one room message, merged."""
-
- session_id: str | None
- thread_id: str | None
- room_pending: bool
- failure_notified: bool
- host: HostState | None
-
- def wire(self) -> dict[str, Any]:
- return {
- "session_id": self.session_id,
- "room_pending": self.room_pending,
- "failure_notified": self.failure_notified,
- "host": self.host,
- }
-
-
-def merge_worker(records: Iterable[RoomMessageRecord]) -> WorkerEvidence | None:
- merged: WorkerEvidence | None = None
- for record in records:
- if merged is None:
- merged = WorkerEvidence(
- session_id=record.session_id,
- thread_id=record.thread_id,
- room_pending=record.room_pending,
- failure_notified=record.failure_notified,
- host=record.host,
- )
- continue
- stronger = _HOST_RANK[record.host] > _HOST_RANK[merged.host]
- merged = WorkerEvidence(
- session_id=record.session_id if stronger else merged.session_id,
- thread_id=merged.thread_id or record.thread_id,
- room_pending=merged.room_pending or record.room_pending,
- failure_notified=merged.failure_notified or record.failure_notified,
- host=record.host if stronger else merged.host,
- )
- return merged
-
-
-def decide_room_message(
- core_status: str | None, worker: WorkerEvidence | None
-) -> Disposition:
- """The one disposition for a room message, strongest evidence across all sources first."""
- host = None if worker is None else worker.host
- if host == "finished" or core_status == "applied":
- return "ran"
- if host == "dispatched" or core_status in ("dispatched", "unknown"):
- return "uncertain"
- if core_status == "rejected":
- return "unrecoverable"
- return "import"
-
-
-def cutover_event(
- room: Room, row: Message, files: list[MessageAttachment]
-) -> dict[str, Any]:
- """The mailbox event for a stored message, as the agent client would have built it."""
- inbound = to_inbound(row, files, transport_room_id=room.transport_room_id)
- if not isinstance(inbound, InboundMessage):
- raise ValueError(f"event {row.transport_event_id} is not a room message")
- thread_id = inbound.thread_root_id
- if platform_replies_in_channel(inbound.content):
- thread_id = None
- sender_name = inbound.sender_name or inbound.sender
- sender_kind: str | None = None
- on_behalf_of: str | None = None
- if PLATFORM_MARKER in inbound.content:
- sender_kind = "platform"
- person = platform_on_behalf_of(inbound.content)
- if person is not None:
- on_behalf_of = person.name
- sender_name = person.name
- attachments = (
- [
- AttachmentRef(
- filename=inbound.filename or inbound.body,
- mimetype=inbound.mimetype or "",
- size=inbound.size or 0,
- mxc=inbound.uri,
- msgtype=inbound.msgtype,
- )
- ]
- if isinstance(inbound, InboundMedia)
- else []
- )
- event = AgentEvent(
- type="message",
- room_id=room.id,
- bridge_id=room.bridge_id,
- channel_type=room.channel_type,
- payload=MessagePayload(
- addressed=True,
- sender=inbound.sender,
- sender_name=sender_name,
- sender_kind=sender_kind,
- on_behalf_of=on_behalf_of,
- message_id=inbound.event_id,
- body=inbound.body,
- timestamp=inbound.timestamp,
- thread_id=thread_id,
- attachments=attachments,
- ),
- )
- entry = MailboxEntry.of(event)
- assert entry is not None
- return entry.event
-
-
-async def rebuild_event(
- session: AsyncSession, room_id: str, message_id: str
-) -> dict[str, Any] | str:
- """The stored message rebuilt as a mailbox event, or why it cannot be."""
- # Only the columns the event needs: `record` runs this against a database
- # still at the pilot revision, before later migrations add room columns.
- room = await session.scalar(
- select(Room)
- .options(
- load_only(
- Room.id, Room.transport_room_id, Room.bridge_id, Room.channel_type
- )
- )
- .where(Room.id == room_id)
- )
- if room is None:
- return "the room is gone"
- row = await session.scalar(
- select(Message).where(
- Message.room_id == room_id, Message.transport_event_id == message_id
- )
- )
- if row is None:
- return "the message is gone"
- group = parse_attachment_group(row.content)
- if group is not None and group[2] > 1:
- return "it is part of a multi-file message"
- files = list(
- await session.scalars(
- select(MessageAttachment)
- .where(MessageAttachment.message_id == row.id)
- .order_by(MessageAttachment.position)
- )
- )
- if len(files) > 1:
- return "it carries several files"
- for file in files:
- blob = await session.scalar(
- select(MediaBlob.id).where(MediaBlob.uri == file.uri)
- )
- if blob is None:
- return f"its attachment {file.uri} is gone"
- try:
- return cutover_event(room, row, files)
- except ValueError as exc:
- return str(exc)
-
-
-def _item(
- *,
- agent_id: str,
- launch_id: str,
- session_id: str | None,
- kind: str,
- evidence: dict[str, Any],
- disposition: Disposition,
- room_id: str | None,
- message_id: str | None,
- thread_id: str | None,
-) -> HostedCutoverItem:
- return HostedCutoverItem(
- tenant_id=require_tenant_id(),
- id=str(uuid4()),
- agent_id=agent_id,
- launch_id=launch_id,
- session_id=session_id,
- kind=kind,
- room_id=room_id,
- message_id=message_id,
- thread_id=thread_id,
- evidence=evidence,
- disposition=disposition,
- )
-
-
-async def _locked_volume(session: AsyncSession, launch_id: str) -> HostedCutoverVolume:
- tenant_id = require_tenant_id()
- volume = await session.scalar(
- select(HostedCutoverVolume)
- .where(
- HostedCutoverVolume.tenant_id == tenant_id,
- HostedCutoverVolume.launch_id == launch_id,
- )
- .with_for_update()
- .execution_options(populate_existing=True)
- )
- if volume is None:
- raise CutoverUnrecorded(
- f"launch {launch_id} has no cutover volume; the cutover-manifest "
- "revision creates one for each hosted agent's latest launch"
- )
- return volume
-
-
-async def record_blocked(session: AsyncSession, launch_id: str, reason: str) -> None:
- """Record that the volume's preflight check failed; a later `apply_manifest` clears it."""
- volume = await _locked_volume(session, launch_id)
- if volume.preflight_state == "complete":
- raise CutoverConflict(
- f"launch {launch_id} already applied manifest {volume.manifest_sha256}"
- )
- volume.preflight_state = "blocked"
- volume.blocked_reason = reason
- await session.flush()
-
-
-async def apply_manifest(
- session: AsyncSession,
- *,
- agent_id: str,
- launch_id: str,
- manifest: CutoverManifest,
-) -> None:
- """Decide every pre-cutover item of the agent once, keeping each import's event.
-
- Idempotent by digest: the same manifest again changes nothing, a different
- one for a completed volume raises `CutoverConflict`. The caller commits.
- """
- volume = await _locked_volume(session, launch_id)
- if volume.preflight_state == "complete":
- if volume.manifest_sha256 != manifest.manifest_sha256:
- raise CutoverConflict(
- f"launch {launch_id} already applied manifest {volume.manifest_sha256}, "
- f"not {manifest.manifest_sha256}"
- )
- return
-
- core = list(
- await session.scalars(
- select(HostedCutoverItem)
- .where(
- HostedCutoverItem.tenant_id == require_tenant_id(),
- HostedCutoverItem.agent_id == agent_id,
- HostedCutoverItem.disposition.is_(None),
- )
- .with_for_update()
- )
- )
-
- rooms: dict[tuple[str, str], list[RoomMessageRecord]] = {}
- consoles: dict[str, ConsoleCommandRecord] = {}
- requests: dict[str, RequestOpenRecord] = {}
- for record in manifest.items:
- if isinstance(record, RoomMessageRecord):
- rooms.setdefault((record.room_id, record.message_id), []).append(record)
- elif isinstance(record, ConsoleCommandRecord):
- consoles[record.command_id] = record
- elif isinstance(record, RequestOpenRecord):
- requests[record.request_id] = record
- else:
- session.add(
- _item(
- agent_id=agent_id,
- launch_id=launch_id,
- session_id=record.session_id,
- kind="reset_pending",
- evidence={"worker": {}},
- disposition="preserved",
- room_id=None,
- message_id=None,
- thread_id=None,
- )
- )
-
- core_rooms = {
- (item.room_id, item.message_id): item
- for item in core
- if item.kind == "room_message" and item.room_id and item.message_id
- }
- for key in sorted(set(core_rooms) | set(rooms)):
- room_id, message_id = key
- existing = core_rooms.get(key)
- worker = merge_worker(rooms.get(key, []))
- core_status = None if existing is None else existing.evidence["core"]["status"]
- disposition = decide_room_message(core_status, worker)
- event: dict[str, Any] | None = None
- if disposition == "import":
- rebuilt = await rebuild_event(session, room_id, message_id)
- if isinstance(rebuilt, str):
- logger.warning(
- "Pre-cutover message %s in room %s for agent %s cannot be run: %s",
- message_id,
- room_id,
- agent_id,
- rebuilt,
- )
- disposition = "unrecoverable"
- else:
- event = rebuilt
- if existing is None:
- assert worker is not None
- existing = _item(
- agent_id=agent_id,
- launch_id=launch_id,
- session_id=worker.session_id,
- kind="room_message",
- evidence={},
- disposition=disposition,
- room_id=room_id,
- message_id=message_id,
- thread_id=worker.thread_id,
- )
- session.add(existing)
- existing.evidence = {
- **existing.evidence,
- **({} if worker is None else {"worker": worker.wire()}),
- }
- existing.disposition = disposition
- existing.payload = event
-
- for item in core:
- if item.kind == "console_command":
- command_id = item.evidence["core"]["command_id"]
- console = consoles.pop(command_id, None)
- if console is None:
- logger.warning(
- "Console command %s for agent %s was accepted by Core before the "
- "cutover and never reached its worker; it did not run",
- command_id,
- agent_id,
- )
- item.disposition = "owner_notice"
- else:
- item.evidence = {**item.evidence, "worker": {"host": console.host}}
- item.disposition = "settled_by_host"
- elif item.kind == "request_open":
- request_id = item.evidence["core"]["request_id"]
- worker_request = requests.pop(request_id, None)
- if worker_request is not None:
- item.evidence = {**item.evidence, "worker": {}}
- item.room_id = item.room_id or worker_request.room_id
- item.thread_id = item.thread_id or worker_request.thread_id
- item.disposition = "interrupted"
- elif item.kind in ("session", "operation", "reset_pending"):
- item.disposition = "preserved"
- for command_id, console in consoles.items():
- session.add(
- _item(
- agent_id=agent_id,
- launch_id=launch_id,
- session_id=console.session_id,
- kind="console_command",
- evidence={"worker": {"command_id": command_id, "host": console.host}},
- disposition="settled_by_host",
- room_id=None,
- message_id=None,
- thread_id=None,
- )
- )
- for request_id, request in requests.items():
- session.add(
- _item(
- agent_id=agent_id,
- launch_id=launch_id,
- session_id=request.session_id,
- kind="request_open",
- evidence={"worker": {"request_id": request_id}},
- disposition="interrupted",
- room_id=request.room_id,
- message_id=None,
- thread_id=request.thread_id,
- )
- )
-
- volume.preflight_state = "complete"
- volume.manifest_sha256 = manifest.manifest_sha256
- volume.completed_at = datetime.now(UTC)
- volume.blocked_reason = None
- await session.flush()
-
-
-async def confirm_manifest(
- session: AsyncSession, launch_id: str, manifest_sha256: str
-) -> None:
- """Check the worker's manifest is the one recorded for its volume before the upgrade."""
- volume = await _locked_volume(session, launch_id)
- if volume.preflight_state != "complete":
- raise CutoverUnrecorded(
- f"launch {launch_id} has no recorded cutover manifest (preflight "
- f"{volume.preflight_state}); record the volume's preflight check first"
- )
- if volume.manifest_sha256 != manifest_sha256:
- raise CutoverConflict(
- f"launch {launch_id} recorded manifest {volume.manifest_sha256}, "
- f"not {manifest_sha256}; the volume changed after its preflight check"
- )
-
-
-async def queue_imports(session: AsyncSession, launch_id: str) -> int:
- """Write the volume's imports to the mailbox once; the caller holds the launch lock.
-
- Returns how many rows were written. Raises `CutoverUnrecorded` for a
- volume that is not complete, and `MailboxFull` at the mailbox limit.
- """
- volume = await _locked_volume(session, launch_id)
- if volume.preflight_state != "complete":
- raise CutoverUnrecorded(
- f"launch {launch_id} cannot queue its imports: preflight "
- f"{volume.preflight_state}"
- )
- if volume.imports_queued_at is not None:
- return 0
- imports = await session.scalars(
- select(HostedCutoverItem)
- .where(
- HostedCutoverItem.tenant_id == require_tenant_id(),
- HostedCutoverItem.launch_id == launch_id,
- HostedCutoverItem.disposition == "import",
- )
- .order_by(HostedCutoverItem.created_at, HostedCutoverItem.id)
- )
- written = 0
- for item in imports:
- if item.payload is None or item.room_id is None or item.message_id is None:
- raise ValueError(f"cutover import {item.id} has no event to queue")
- thread_id = item.payload["payload"].get("thread_id")
- if await HostedMailboxStore().write(
- session,
- agent_id=item.agent_id,
- launch_id=launch_id,
- entry=MailboxEntry(
- room_id=item.room_id,
- message_id=item.message_id,
- thread_id=thread_id if isinstance(thread_id, str) else None,
- event=item.payload,
- origin="cutover",
- ),
- offered_to=None,
- ):
- written += 1
- volume.imports_queued_at = datetime.now(UTC)
- await session.flush()
- return written
-
-
-_NOTICE_REASONS: dict[str, CutoverNoticeReason] = {
- "uncertain": "cutover_uncertain",
- "unrecoverable": "cutover_unrecoverable",
- "interrupted": "cutover_interrupted",
- "import": "cutover_run_now",
-}
-
-
-async def owed_notices(session: AsyncSession, agent_id: str) -> list[CutoverNotice]:
- """The agent's cutover notices not yet posted, oldest item first.
-
- An import into a session whose reset was interrupted does not run now:
- the host holds it until someone decides the reset, so the room is told
- that, not that it will run.
- """
- awaiting_reset = set(
- await session.scalars(
- select(HostedCutoverItem.session_id).where(
- HostedCutoverItem.tenant_id == require_tenant_id(),
- HostedCutoverItem.agent_id == agent_id,
- HostedCutoverItem.kind == "reset_pending",
- HostedCutoverItem.session_id.is_not(None),
- )
- )
- )
- items = await session.scalars(
- select(HostedCutoverItem)
- .where(
- HostedCutoverItem.tenant_id == require_tenant_id(),
- HostedCutoverItem.agent_id == agent_id,
- HostedCutoverItem.notice_posted_at.is_(None),
- HostedCutoverItem.notice_dropped.is_(None),
- HostedCutoverItem.room_id.is_not(None),
- or_(
- HostedCutoverItem.disposition.in_(
- ("uncertain", "unrecoverable", "interrupted")
- ),
- (HostedCutoverItem.disposition == "import")
- & or_(
- HostedCutoverItem.evidence["worker"]["failure_notified"].astext
- == "true",
- HostedCutoverItem.session_id.in_(awaiting_reset),
- ),
- ),
- )
- .order_by(HostedCutoverItem.created_at, HostedCutoverItem.id)
- )
- notices = []
- for item in items:
- assert item.room_id is not None and item.disposition is not None
- request_id = None
- if item.kind == "request_open":
- request_id = (item.evidence.get("core") or item.evidence["worker"])[
- "request_id"
- ]
- notices.append(
- CutoverNotice(
- item_id=item.id,
- agent_id=item.agent_id,
- room_id=item.room_id,
- message_id=item.message_id,
- request_id=request_id,
- thread_id=item.thread_id,
- reason="conversation"
- if item.disposition == "import" and item.session_id in awaiting_reset
- else _NOTICE_REASONS[item.disposition],
- )
- )
- return notices
-
-
-async def mark_notice_posted(session: AsyncSession, item_id: str) -> None:
- item = await session.get(HostedCutoverItem, (require_tenant_id(), item_id))
- assert item is not None
- item.notice_posted_at = datetime.now(UTC)
-
-
-async def drop_notices(
- session: AsyncSession, item_ids: Iterable[str], reason: str
-) -> None:
- """These items' notices will never be posted, for `reason`; nothing retries them."""
- await session.execute(
- update(HostedCutoverItem)
- .where(
- HostedCutoverItem.tenant_id == require_tenant_id(),
- HostedCutoverItem.id.in_(list(item_ids)),
- HostedCutoverItem.notice_posted_at.is_(None),
- )
- .values(notice_dropped=reason)
- .execution_options(synchronize_session=False)
- )
-
-
-async def post_cutover_notices(
- protocol: AgentCore, agent: Agent, notices: list[CutoverNotice]
-) -> int:
- """Post each owed notice once, and record it; one that fails stays owed for the next pass."""
- unposted = 0
- for notice in notices:
- subject = (
- notice.message_id
- if notice.request_id is None
- else f"request:{notice.request_id}"
- )
- try:
- await post_notice_once(
- protocol,
- agent,
- notice.room_id,
- key=json.dumps([agent.id, notice.room_id, subject, notice.reason]),
- body=NOTICE_MESSAGES[notice.reason].format(name=agent.name),
- thread_id=notice.thread_id,
- anchor=None,
- )
- except Exception:
- logger.error(
- "Could not post the %s cutover notice for %s in room %s",
- notice.reason,
- subject,
- notice.room_id,
- exc_info=True,
- )
- unposted += 1
- continue
- async with tenant_session(protocol.session_factory, require_tenant_id()) as db:
- await mark_notice_posted(db, notice.item_id)
- await db.commit()
- return unposted
-
-
-def cutover_notice_candidates() -> tuple[ColumnElement[bool], ...]:
- """The bound tenant's cutover items that may still owe a room notice."""
- return (
- HostedCutoverItem.tenant_id == require_tenant_id(),
- HostedCutoverItem.notice_posted_at.is_(None),
- HostedCutoverItem.notice_dropped.is_(None),
- HostedCutoverItem.room_id.is_not(None),
- HostedCutoverItem.disposition.in_(_NOTICE_REASONS),
- )
-
-
-async def post_owed_cutover_notices(protocol: AgentCore) -> int:
- """Post every cutover notice the bound tenant still owes; returns how many stay owed."""
- async with tenant_session(protocol.session_factory, require_tenant_id()) as db:
- agent_ids = list(
- await db.scalars(
- select(distinct(HostedCutoverItem.agent_id)).where(
- *cutover_notice_candidates()
- )
- )
- )
- owed: list[tuple[Agent | None, str, list[CutoverNotice]]] = []
- for agent_id in agent_ids:
- notices = await owed_notices(db, agent_id)
- if notices:
- owed.append((await db.get(Agent, agent_id), agent_id, notices))
- unposted = 0
- for agent, agent_id, notices in owed:
- if agent is None:
- logger.warning(
- "%d cutover notice(s) for agent %s dropped: the agent was deleted",
- len(notices),
- agent_id,
- )
- async with tenant_session(
- protocol.session_factory, require_tenant_id()
- ) as db:
- await drop_notices(
- db, [notice.item_id for notice in notices], "agent_deleted"
- )
- await db.commit()
- continue
- unposted += await post_cutover_notices(protocol, agent, notices)
- return unposted
diff --git a/core/switch_core/bridges/agent/hosted_mailbox.py b/core/switch_core/bridges/agent/hosted_mailbox.py
deleted file mode 100644
index 6ba4c934e..000000000
--- a/core/switch_core/bridges/agent/hosted_mailbox.py
+++ /dev/null
@@ -1,224 +0,0 @@
-"""Delivering the wake mailbox to a hosted agent's worker, and keeping it tidy.
-
-Delivery is always mark offered, then send: a row is leased to one worker
-stream in a committed transaction before its `wake` frame is queued, so a
-stream that dies or a Core that restarts leaves a lease that reclaim returns
-to `pending`, never an event nobody holds.
-"""
-
-from __future__ import annotations
-
-import logging
-from datetime import UTC, datetime
-
-from sqlalchemy import exists, or_, select
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.bridges.agent.api.hosted_worker_routes import post_mailbox_notices
-from switch_core.bridges.agent.hosted_cutover import (
- cutover_notice_candidates,
- post_owed_cutover_notices,
-)
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.bridges.agent.protocol.hosted_workers import (
- FrameSlot,
- WorkerBusyError,
- attached_worker_for,
- frame_size,
- offer_key,
-)
-from switch_core.config import hosted_configured
-from switch_core.db.models import HostedLaunch, HostedWakeMailbox, require_tenant_id
-from switch_core.db.session_scope import tenant_session
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
-from switch_core.db.stores.hosted_mailbox_store import (
- NOTICE_RETRIES_PER_PASS,
- WAKE_ENTRIES_PER_FRAME,
- HostedMailboxStore,
- MailboxEntry,
-)
-
-logger = logging.getLogger(__name__)
-
-
-async def offer_pending(
- session: AsyncSession,
- registry: AgentConnectionRegistry,
- boot: int,
- launch: HostedLaunch,
-) -> int:
- """Lease the agent's `pending` rows to its worker, then queue them as `wake` frames.
-
- Oldest first, up to 50 entries a frame. A frame that does not fit the
- worker's queue is not offered: its rows stay `pending` for the next pass.
- Commits the session. Returns how many rows were sent.
- """
- conn = attached_worker_for(registry, launch)
- if conn is None or launch.agent_id is None:
- await session.commit()
- return 0
- rows = await HostedMailboxStore().pending(session, launch.agent_id)
- frames: list[tuple[FrameSlot, list[MailboxEntry]]] = []
- for start in range(0, len(rows), WAKE_ENTRIES_PER_FRAME):
- entries = [
- MailboxEntry.of_row(row)
- for row in rows[start : start + WAKE_ENTRIES_PER_FRAME]
- ]
- try:
- slot = conn.worker_frames.reserve(
- frame_size({"entries": [entry.wire() for entry in entries]})
- )
- except WorkerBusyError:
- logger.warning(
- "Worker queue of agent %s is full; %d mailbox row(s) stay pending",
- launch.agent_id,
- len(rows) - start,
- )
- break
- frames.append((slot, entries))
- try:
- offered = await HostedMailboxStore().mark_offered(
- session,
- launch.agent_id,
- [
- (entry.room_id, entry.message_id)
- for _, group in frames
- for entry in group
- ],
- offer_key(boot, conn),
- )
- await session.commit()
- except BaseException:
- for slot, _ in frames:
- slot.release()
- raise
- sent = 0
- for slot, group in frames:
- entries = [e for e in group if (e.room_id, e.message_id) in offered]
- if not entries:
- slot.release()
- continue
- slot.put("wake", {"entries": [entry.wire() for entry in entries]})
- sent += len(entries)
- return sent
-
-
-async def deliver_on_attach(
- session: AsyncSession, protocol: AgentCore, launch: HostedLaunch
-) -> int:
- """At attach: take back every offer of the agent's, then offer all `pending` rows.
-
- The attaching stream is the only one that can hold a lease from now on,
- so every earlier offer (another boot, an older generation) is reclaimed.
- """
- assert launch.agent_id is not None
- conn = attached_worker_for(protocol.connections, launch)
- await HostedMailboxStore().reclaim(
- session,
- launch.agent_id,
- {}
- if conn is None
- else {launch.agent_id: offer_key(protocol.event_buffer.boot, conn)},
- )
- return await offer_pending(
- session, protocol.connections, protocol.event_buffer.boot, launch
- )
-
-
-async def retains_hosted_work(session: AsyncSession) -> bool:
- """Whether the bound tenant has a hosted launch, a wake mailbox row or an owed cutover notice."""
- tenant_id = require_tenant_id()
- return bool(
- await session.scalar(
- select(
- or_(
- exists().where(HostedLaunch.tenant_id == tenant_id),
- exists().where(HostedWakeMailbox.tenant_id == tenant_id),
- exists().where(*cutover_notice_candidates()),
- )
- )
- )
- )
-
-
-async def mailbox_upkeep(protocol: AgentCore, since: datetime) -> None:
- """One pass over the bound tenant: reclaim, expire, prune, post owed notices, re-offer, log the backlog and post owed cutover notices.
-
- Skipped on a server that does not run cloud agents, unless the tenant still
- holds hosted work from when it did.
- """
- if not hosted_configured(protocol.config):
- async with tenant_session(
- protocol.session_factory, require_tenant_id()
- ) as session:
- if not await retains_hosted_work(session):
- return
- store = HostedMailboxStore()
- registry = protocol.connections
- boot = protocol.event_buffer.boot
- now = datetime.now(UTC)
- async with tenant_session(protocol.session_factory, require_tenant_id()) as session:
- launches = list(
- await session.scalars(
- select(HostedLaunch).where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.agent_id.is_not(None),
- )
- )
- )
- live: dict[str, str] = {}
- for launch in launches:
- conn = attached_worker_for(registry, launch)
- if conn is not None and launch.agent_id is not None:
- live[launch.agent_id] = offer_key(boot, conn)
- reclaimed = await store.reclaim(session, None, live)
- notices, tombstoned = await store.expire(session, now)
- pruned = await store.prune(session, now)
- backlog = await store.backlog(session, since)
- waiting = await store.agents_with_pending(session)
- owed = await store.owed_notices(session, NOTICE_RETRIES_PER_PASS)
- await session.commit()
- if reclaimed or notices or tombstoned or pruned:
- logger.info(
- "Wake mailbox upkeep in tenant %s: reclaimed=%d expired=%d "
- "tombstoned=%d pruned=%d",
- require_tenant_id(),
- reclaimed,
- len(notices),
- tombstoned,
- pruned,
- )
- for agent in backlog:
- logger.info(
- "Wake mailbox backlog agent=%s waiting=%d oldest=%s refused=%d",
- agent.agent_id,
- agent.waiting,
- agent.oldest.isoformat() if agent.oldest else "-",
- agent.refused,
- )
- fresh = {(notice.agent_id, notice.room_id, notice.message_id) for notice in notices}
- retried = [
- notice
- for notice in owed
- if (notice.agent_id, notice.room_id, notice.message_id) not in fresh
- ]
- if retried:
- logger.warning(
- "Wake mailbox in tenant %s retries %d room notice(s) a failed send still owes",
- require_tenant_id(),
- len(retried),
- )
- await post_mailbox_notices(protocol, owed)
- by_agent = {launch.agent_id: launch.id for launch in launches}
- for agent_id in waiting:
- launch_id = by_agent.get(agent_id)
- if launch_id is None or agent_id not in live:
- continue
- async with tenant_session(
- protocol.session_factory, require_tenant_id()
- ) as session:
- current = await HostedLaunchStore().locked(session, launch_id)
- if current is not None:
- await offer_pending(session, registry, boot, current)
- await post_owed_cutover_notices(protocol)
diff --git a/core/switch_core/bridges/agent/mcp/server.py b/core/switch_core/bridges/agent/mcp/server.py
index 3e7c2cb05..e914cb2c1 100644
--- a/core/switch_core/bridges/agent/mcp/server.py
+++ b/core/switch_core/bridges/agent/mcp/server.py
@@ -30,7 +30,6 @@ from switch_core.bridges.agent.operations import (
)
from switch_core.bridges.agent.operations.callctx import CallContext, call_context
from switch_core.bridges.agent.operations.context import init_operations_protocol
-from switch_core.bridges.agent.protocol.hosted_workers import CodedPermissionError
if TYPE_CHECKING:
from switch_core.bridges.agent.protocol.agent_core import AgentCore
@@ -78,14 +77,7 @@ class CallContextMiddleware(Middleware):
session=None,
)
):
- try:
- return await call_next(context)
- except ToolError as exc:
- # The server prefixes a tool's error with its name; a coded
- # refusal is sent as its bare `{code, message}` object instead.
- if isinstance(exc.__cause__, CodedPermissionError):
- raise ToolError(str(exc.__cause__)) from exc.__cause__
- raise
+ return await call_next(context)
class ExistingOperationsOnly(Middleware):
diff --git a/core/switch_core/bridges/agent/operations/definitions.py b/core/switch_core/bridges/agent/operations/definitions.py
index 280208d13..708ee3ebe 100644
--- a/core/switch_core/bridges/agent/operations/definitions.py
+++ b/core/switch_core/bridges/agent/operations/definitions.py
@@ -44,11 +44,6 @@ from switch_core.bridges.agent.protocol.agent_connections import (
from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.bridges.agent.protocol.agent_detail import AgentProfileUpdate
from switch_core.bridges.agent.protocol.agent_management import ManagedAgentChanges
-from switch_core.bridges.agent.protocol.hosted_workers import (
- HOSTED_WORKER_ONLY_MESSAGE,
- CodedPermissionError,
- hosted_launch_of,
-)
from switch_core.bridges.agent.protocol.instructions import build_room_instructions
from switch_core.bridges.agent.protocol.types import IntegrationProfile
from switch_core.db.models import CollaborationBridge, User
@@ -293,12 +288,6 @@ async def connect_to_room(
if room_model.bridge_id:
bridge = await session.get(CollaborationBridge, room_model.bridge_id)
- if hosted_launch_of(agent.metadata_) is not None:
- key = session_key()
- caller_connection = protocol.connections.get(key) if key else None
- if caller_connection is None or caller_connection.worker is None:
- raise CodedPermissionError("hosted_worker_only", HOSTED_WORKER_ONLY_MESSAGE)
-
profile = IntegrationProfile(**agent.integration_profile)
instructions = build_room_instructions(
agent,
diff --git a/core/switch_core/bridges/agent/protocol/agent_connections.py b/core/switch_core/bridges/agent/protocol/agent_connections.py
index 784299116..acb996f8d 100644
--- a/core/switch_core/bridges/agent/protocol/agent_connections.py
+++ b/core/switch_core/bridges/agent/protocol/agent_connections.py
@@ -33,15 +33,6 @@ from typing import Any, Literal
from switch_core.artifacts import ARTIFACT_VERSIONS, contract_range
from switch_core.bridges.agent.protocol.controller_presence import ControllerPresence
-from switch_core.bridges.agent.protocol.hosted_workers import (
- IDLE_FRESH_FOR_SECONDS,
- IdleReport,
- PendingRelay,
- PendingRelays,
- RelayViews,
- WorkerBinding,
- WorkerFrames,
-)
from switch_core.bridges.agent.protocol.liveness import (
HEARTBEAT_INTERVAL_SECONDS as HEARTBEAT_INTERVAL_SECONDS,
)
@@ -263,34 +254,11 @@ class UnfencedBeatError(ConnectionError_):
self.speaks = speaks
-class WorkerAlreadyAttachedError(ConnectionError_):
- """Another host is attached as this hosted agent's worker and still alive."""
-
- code = "worker_already_attached"
-
- def __init__(self, agent_id: str) -> None:
- super().__init__(
- f"agent {agent_id} already has a worker attached from another host "
- "boot; retry with backoff once it has lapsed"
- )
- self.agent_id = agent_id
-
-
# How many closed connection ids are remembered for the reconnect counter.
# Far more than the live fleet, so a lapse storm is still recognised when its
# agents come back.
_RECENTLY_CLOSED_LIMIT = 10_000
-#: The hosted launch moved to a newer revision than the one this worker was
-#: attached for. Terminal: the worker's capability is obsolete, and only a
-#: restart onto the current bundle can attach again.
-LAUNCH_SUPERSEDED = Closure(
- code="launch_superseded",
- message="the hosted launch moved to a newer revision; this worker's "
- "capability is obsolete",
- room_id=None,
-)
-
@dataclass(frozen=True, slots=True)
class Released:
@@ -455,15 +423,6 @@ class AgentConnection:
# Rooms another connection took off this one, not yet written to its
# stream, with the session of this connection that was placed there.
released_rooms: dict[str, str | None] = field(default_factory=dict)
- # Set when a hosted agent's worker attached with a valid capability. Only
- # this connection is sent the hosted frames, relays and doorbells.
- worker: WorkerBinding | None = None
- # The worker's latest accepted idle report, for this generation only.
- idle_report: IdleReport | None = None
- worker_frames: WorkerFrames = field(init=False, repr=False)
-
- def __post_init__(self) -> None:
- self.worker_frames = WorkerFrames(self.wake)
def is_alive(self, now: float) -> bool:
return self.closure is None and (now - self.last_beat) < HEARTBEAT_TTL_SECONDS
@@ -512,8 +471,6 @@ class AgentConnectionRegistry:
# replaced wholesale, and a placement another connection takes is
# reported to the one that lost it.
self._placement_owners: dict[str, dict[str, str]] = {}
- self.relays = PendingRelays(on_expire=self._cancel_relay)
- self.relay_views = RelayViews()
# Agents run by an agents controller hold no connection here: their
# presence is their controller's. Every presence question below asks
# it first for those agents, and this registry for the rest.
@@ -617,7 +574,6 @@ class AgentConnectionRegistry:
# is replaced rather than kept. The connection outlives the socket;
# what is on the other end of it need not.
existing.declaration = declaration
- self._reset_worker(existing)
metrics().increment(AGENT_CONNECTIONS_OPENED, {"kind": "reattach"})
logger.info(
"[CONN] reattached agent=%s connection=%s scope=%s generation=%s "
@@ -680,8 +636,6 @@ class AgentConnectionRegistry:
"""
if self._by_id.get(conn.id) is conn and conn.stream_generation == generation:
conn.stream_attached = False
- conn.idle_report = None
- self.relays.fail_connection(conn.id, generation)
logger.info(
"[CONN] stream detached agent=%s connection=%s (connection still "
"alive until heartbeat lapses)",
@@ -710,12 +664,6 @@ class AgentConnectionRegistry:
self._by_agent.pop(conn.agent_id, None)
conn.closure = closure
conn.stream_attached = False
- conn.idle_report = None
- if conn.worker is not None:
- # A hosted worker restates its placements on every attach, and a
- # sleeping agent must not look present in the rooms it last held.
- self._drop_placements(conn)
- self.relays.fail_connection(conn.id, None)
conn.wake.set()
lapsed = closure is HEARTBEAT_LAPSED
self._remember_closed(connection_id, "after_lapse" if lapsed else "after_close")
@@ -1001,14 +949,6 @@ class AgentConnectionRegistry:
self.release_room(conn, room_id)
return released
- def _drop_placements(self, conn: AgentConnection) -> None:
- """Forget every session placement this connection made."""
- placed = self._session_rooms.get(conn.agent_id, {})
- owners = self._placement_owners.get(conn.agent_id, {})
- for session_id in [s for s, owner in owners.items() if owner == conn.id]:
- placed.pop(session_id, None)
- owners.pop(session_id, None)
-
def _unplace(
self, agent_id: str, session_id: str, room_id: str, *, taker: str
) -> Released | None:
@@ -1085,21 +1025,14 @@ class AgentConnectionRegistry:
None,
)
- def relay_session_command(
- self, agent_id: str, frame: dict[str, Any], *, worker_only: bool
- ) -> bool:
+ def relay_session_command(self, agent_id: str, frame: dict[str, Any]) -> bool:
"""Hand a session command to the agent's watcher stream. False if none is attached.
Not stored anywhere else: a command relayed to nobody is refused to
whoever sent it, rather than held for a watcher that may not return.
- `worker_only` narrows it to a hosted agent's attached worker, so a
- holder of the agent key cannot receive room controls meant for it. A
- controller-backed agent's command goes to its controller's stream.
+ A controller-backed agent's command goes to its controller's stream.
"""
- if worker_only:
- worker = self.attached_worker(agent_id)
- watchers = [worker] if worker is not None else []
- elif self.controllers.is_bound(agent_id):
+ if self.controllers.is_bound(agent_id):
return self.controllers.relay_session_command(agent_id, frame)
else:
watchers = [
@@ -1114,124 +1047,6 @@ class AgentConnectionRegistry:
conn.wake.set()
return bool(watchers)
- # ------------------------------------------------------------------
- # Hosted workers
- # ------------------------------------------------------------------
-
- def _reset_worker(self, conn: AgentConnection) -> None:
- """Forget what the previous generation of this connection was bound to.
-
- Its relays fail and its queued frames are dropped with it; a reattach
- is bound again only if it proves a capability.
- """
- self.relays.fail_connection(conn.id, None)
- conn.worker = None
- conn.idle_report = None
- conn.worker_frames = WorkerFrames(conn.wake)
-
- def worker_of(self, agent_id: str) -> AgentConnection | None:
- """The agent's live worker connection, attached or between streams."""
- return next(
- (conn for conn in self.for_agent(agent_id) if conn.worker is not None),
- None,
- )
-
- def attached_worker(self, agent_id: str) -> AgentConnection | None:
- """The agent's worker, when its stream is attached and can take frames."""
- conn = self.worker_of(agent_id)
- if conn is None or not conn.stream_attached:
- return None
- return conn
-
- def admit_worker(
- self, agent_id: str, connection_id: str, boot_id: str
- ) -> AgentConnection | None:
- """Refuse a worker attach, or name the worker it will take over.
-
- One worker per agent. A live worker from another host boot keeps its
- place; one from the same boot is a restarted daemon, and the new
- connection takes over once it has opened (the caller closes the one
- returned, with `TAKEN_OVER`). Reattaching the same connection id is
- `open`'s to fence.
- """
- current = self.worker_of(agent_id)
- if current is None or current.id == connection_id:
- return None
- assert current.worker is not None
- if current.worker.boot_id != boot_id:
- raise WorkerAlreadyAttachedError(agent_id)
- return current
-
- def bind_worker(
- self, conn: AgentConnection, binding: WorkerBinding, attached: dict[str, Any]
- ) -> None:
- """Mark an opened connection as the worker; `worker_attached` goes first."""
- conn.worker = binding
- conn.idle_report = None
- conn.worker_frames = WorkerFrames(conn.wake)
- conn.worker_frames.push("worker_attached", attached)
-
- def supersede(self, agent_id: str, revision: int) -> None:
- """Evict the agent's workers bound to a revision older than `revision`."""
- for cid in list(self._by_agent.get(agent_id, set())):
- conn = self._by_id.get(cid)
- if (
- conn is not None
- and conn.worker is not None
- and conn.worker.launch_revision < revision
- ):
- self.close(cid, LAUNCH_SUPERSEDED)
-
- def ring_worker(self, agent_id: str, event: str, data: dict[str, Any]) -> bool:
- """Send a doorbell frame to the attached worker. False if none is attached."""
- conn = self.attached_worker(agent_id)
- if conn is None:
- return False
- conn.worker_frames.push(event, data)
- return True
-
- def _cancel_relay(self, relay: PendingRelay) -> None:
- """Tell the worker a relay expired, if it is still on the stream it went to."""
- conn = self._by_id.get(relay.connection_id)
- if (
- conn is not None
- and conn.worker is not None
- and conn.stream_generation == relay.generation
- ):
- conn.worker_frames.push("relay_cancel", {"id": relay.id})
-
- def record_idle_report(self, conn: AgentConnection, report: IdleReport) -> bool:
- """Keep the report unless it is not newer than the one held."""
- held = conn.idle_report
- if held is not None and report.report_seq <= held.report_seq:
- return False
- conn.idle_report = report
- return True
-
- def fresh_idle_report(
- self, agent_id: str, launch_id: str, revision: int
- ) -> IdleReport | None:
- """The worker's idle report, if it can count as evidence right now.
-
- Fresh means: from the live, attached worker bound to this launch at
- this revision, for its current generation, and recent. Anything else
- is absent, and absent counts as busy.
- """
- conn = self.attached_worker(agent_id)
- if conn is None or conn.worker is None:
- return None
- if (
- conn.worker.launch_id != launch_id
- or conn.worker.launch_revision != revision
- ):
- return None
- report = conn.idle_report
- if report is None or report.generation != conn.stream_generation:
- return None
- if time.monotonic() - report.received_monotonic >= IDLE_FRESH_FOR_SECONDS:
- return None
- return report
-
# ------------------------------------------------------------------
# Room slots
# ------------------------------------------------------------------
diff --git a/core/switch_core/bridges/agent/protocol/agent_core.py b/core/switch_core/bridges/agent/protocol/agent_core.py
index 801204ebc..8b83ef216 100644
--- a/core/switch_core/bridges/agent/protocol/agent_core.py
+++ b/core/switch_core/bridges/agent/protocol/agent_core.py
@@ -35,7 +35,6 @@ from switch_core.bridges.agent.protocol.agent_detail import (
list_agent_summaries,
)
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.hosted_workers import hosted_launch_of
from switch_core.bridges.agent.protocol.presence import rooms_occupied
from switch_core.bridges.agent.protocol.statuses import compute_agent_statuses
from switch_core.bridges.agent.protocol.types import (
@@ -71,7 +70,6 @@ from switch_core.db.models import (
Agent,
AgentRuntimeState,
ApiKey,
- HostedLaunch,
Message,
MessageAttachment,
Model,
@@ -93,7 +91,6 @@ from switch_core.db.stores.agent_runtime_state_store import (
AgentRuntimeStateStore,
)
from switch_core.db.stores.budget_store import BudgetStore
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
from switch_core.db.stores.message_store import MessageStore
from switch_core.db.stores.room_group_store import RoomGroupStore
from switch_core.db.stores.room_role_store import RoomRoleStore
@@ -366,7 +363,6 @@ class AgentCore:
addressable_by_agent_ids: list[str] | None = None,
owner_only: bool = True,
registration_path: str = "other",
- reserved_agent_id: str | None = None,
) -> RegistrationResult:
"""Register or re-register an agent.
@@ -444,21 +440,7 @@ class AgentCore:
async with self.session_factory() as session:
await self.agent_store.lock_name(session, name)
- reservation = await session.scalar(
- select(HostedLaunch).where(
- HostedLaunch.tenant_id == tenant_id,
- HostedLaunch.name == name,
- )
- )
- if reservation is not None and reservation.agent_id != reserved_agent_id:
- raise AgentExistsError(
- "A cloud launch already reserves this agent name."
- )
existing = await self.agent_store.get_by_name(session, name)
- if reserved_agent_id is not None and existing is not None:
- raise AgentExistsError(
- "The reserved cloud identity cannot overwrite an agent."
- )
if existing and not overwrite:
raise AgentExistsError(
f"Agent already exists: {name!r}. "
@@ -505,7 +487,6 @@ class AgentCore:
reject_reserved_mention_name(name, kind="Agent name")
agent_id = await self._create_agent(
session=session,
- reserved_agent_id=reserved_agent_id,
name=name,
description=description,
icon_url=validated_icon_url,
@@ -646,7 +627,6 @@ class AgentCore:
self,
*,
session: AsyncSession,
- reserved_agent_id: str | None,
name: str,
description: str,
icon_url: str | None,
@@ -684,7 +664,6 @@ class AgentCore:
)
agent = Agent(
- **({"id": reserved_agent_id} if reserved_agent_id is not None else {}),
name=name,
description=description,
icon_url=icon_url,
@@ -3713,9 +3692,7 @@ class AgentCore:
"""Write a validated profile update to an agent and return its fresh
detail.
- Owner-only, as `require_same_owner`. A cloud agent's launch spec, which
- registers it again, is kept on the same values, as the gateway routes
- for each field do.
+ Owner-only, as `require_same_owner`.
"""
await self.require_same_owner(agent_id, target_agent_id)
async with self.session_factory() as session:
@@ -3726,11 +3703,6 @@ class AgentCore:
await self.agent_store.update(
session, target_agent_id, **update.columns
)
- launch_id = hosted_launch_of(target.metadata_)
- if launch_id is not None:
- await HostedLaunchStore().merge_spec(
- session, launch_id, update.columns
- )
await session.commit()
refreshed = await self.agent_store.get(session, target_agent_id)
diff --git a/core/switch_core/bridges/agent/protocol/hosted_workers.py b/core/switch_core/bridges/agent/protocol/hosted_workers.py
deleted file mode 100644
index d92e37d18..000000000
--- a/core/switch_core/bridges/agent/protocol/hosted_workers.py
+++ /dev/null
@@ -1,572 +0,0 @@
-"""In-memory state of hosted workers: bindings, frame queues, relays, idle reports.
-
-A hosted agent's watcher runs on a cloud worker and attaches with a capability
-the controller minted for the launch's current revision. Everything here is
-per Core boot and lost on restart; loss always reads as busy or retry, never as
-idle or delivered.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import json
-import time
-import uuid
-from collections import deque
-from collections.abc import Callable
-from dataclasses import dataclass, field
-from datetime import datetime
-from typing import TYPE_CHECKING, Any, Literal
-
-if TYPE_CHECKING:
- from switch_core.bridges.agent.protocol.agent_connections import (
- AgentConnection,
- AgentConnectionRegistry,
- )
- from switch_core.db.models import HostedLaunch
-
-#: The protocol revision that carries the hosted worker frames.
-HOSTED_PROTOCOL_REVISION = 7
-
-#: The retained-state layout a worker's volume must have been migrated to
-#: (`hosted-preflight`) before it may attach; sent as
-#: `X-Switch-Worker-State-Version`.
-HOSTED_WORKER_STATE_VERSION = 1
-
-#: Bound on frames that carry a body (a relayed request, a room event).
-FRAME_QUEUE_FRAMES = 64
-FRAME_QUEUE_BYTES = 8 * 1024 * 1024
-
-IDLE_REPORT_EVERY_SECONDS = 30
-IDLE_FRESH_FOR_SECONDS = 75
-
-RELAY_REQUEST_LIMIT_BYTES = 2 * 1024 * 1024
-RELAY_REPLY_LIMIT_BYTES = 1024 * 1024
-#: Room for the reply's envelope around a value at the limit.
-RELAY_REPLY_ENVELOPE_BYTES = 64 * 1024
-RELAY_TIMEOUT_LIMIT_MS = 30_000
-#: How long a resolved relay is remembered, so a second reply is refused
-#: rather than reported unknown.
-RELAY_RESOLVED_RETENTION_SECONDS = 60.0
-
-#: Bound on one Console relay stream's undelivered frames.
-CONSOLE_VIEW_FRAMES = 1000
-CONSOLE_VIEW_BYTES = 4 * 1024 * 1024
-
-#: What a room is told, by notice reason, when a message was not processed.
-NOTICE_MESSAGES = {
- "startup": "I could not start the provider, so I could not process your request. Open this session in Switch Console to check the error and restart it.",
- "delivery": "I could not verify your earlier message after reconnecting, so I did not process it. Please send the message and any attachments again.",
- "conversation": "This saved conversation cannot continue. Send !reset @{name} here, or choose Start a fresh conversation in Switch Console. Your pending messages will be delivered after you make that choice.",
- "capacity": "I am already running as many sessions as my cloud worker allows, so I could not start one for this message. Stop a session in Switch Console, then send the message again.",
- "auto_start_off": "I have no session for this room and I am not set to start one automatically, so I did not process this message. Start a session for this room in Switch Console, then send the message again.",
- "stopped": "My cloud worker was stopped before I processed this message, so I did not process it. Start me again in Switch Console, then send it again.",
- "expired": "I could not process this message in time, so I did not process it. Please send it again.",
- "cancelled": "Processing of this message was cancelled before it ran. Please send it again if it is still needed.",
- "revoked": "My owner's provider connection was removed, so I cannot process messages. Ask my owner to reconnect the provider in Switch.",
- "removed": "My cloud agent was removed before I processed this message, so I did not process it.",
- "upgrade": "My cloud worker is being upgraded and could not process this message. Please send it again in a few minutes.",
- "started_before_stop": "I had already started processing this message before my cloud worker was stopped, so it may have been processed in part. Check the conversation before sending it again.",
- "started_before_expiry": "I had already started processing this message before it expired, so it may have been processed in part. Check the conversation before sending it again.",
- "expired_uncertain": "I could not confirm whether my cloud worker received this message in time, so it may or may not have been processed. Check the conversation before sending it again.",
- "cutover_uncertain": "My cloud worker was upgraded while I was processing this message, so it may have been interrupted. Check the conversation and send it again if needed.",
- "cutover_unrecoverable": "My cloud worker was upgraded before I processed this message, and it could not be carried over, so it was not run. Please send it again.",
- "cutover_interrupted": "An approval I was waiting for here was interrupted by an upgrade of my cloud worker. Ask me again if it is still needed.",
- "cutover_run_now": "My cloud worker has been upgraded, and I will process this message now.",
- "unreachable": "My cloud worker is not connected right now: it may still be starting, or it may be unreachable. I have kept your message and will process it when the worker connects.",
-}
-
-#: The subscription name a watcher pushes health under.
-HEALTH_SUBSCRIPTION = "health"
-
-MUTATING_MESSAGES = frozenset(
- {"command", "place", "forget", "attachment", "attachmentCancel"}
-)
-REFUSED_MESSAGES = frozenset({"room", "approvals", "ensure"})
-READ_ONLY_MESSAGES = frozenset(
- {
- "snapshot",
- "subscribe",
- "unsubscribe",
- "health",
- "watchHealth",
- "list",
- "journal",
- "page",
- }
-)
-
-MessageKind = Literal["mutating", "read_only"]
-
-
-class RelayError(Exception):
- """A relay that could not be answered, with the code Console acts on."""
-
- def __init__(self, code: str, message: str, status: int) -> None:
- super().__init__(message)
- self.code = code
- self.status = status
-
-
-HOSTED_WORKER_ONLY_MESSAGE = (
- "This agent runs on a cloud worker; only its attached worker may do this."
-)
-
-
-class CodedPermissionError(PermissionError):
- """A refusal every front door renders as the same `{code, message}` object."""
-
- def __init__(self, code: str, message: str) -> None:
- super().__init__(json.dumps({"code": code, "message": message}))
- self.detail = {"code": code, "message": message}
-
-
-class WorkerBusyError(RelayError):
- def __init__(self) -> None:
- super().__init__(
- "worker_busy",
- "The worker's frame queue is full. Retry shortly.",
- 503,
- )
-
-
-def classify_message(message: dict[str, Any]) -> MessageKind:
- """Whether a relayed message changes the worker's state.
-
- The message is one element of the host control vocabulary. `room`,
- `approvals` and `ensure` are the watcher's own and are refused; anything
- unrecognised is refused too rather than guessed at.
- """
- if not isinstance(message, dict):
- raise RelayError("refused_message", "A relayed message is an object.", 400)
- if "sessionId" in message:
- request = message.get("request")
- if set(message) != {"sessionId", "request"} or not isinstance(request, dict):
- raise RelayError(
- "refused_message",
- "A session request carries sessionId and request only.",
- 400,
- )
- kind = request.get("type")
- if kind in REFUSED_MESSAGES:
- raise RelayError(
- "refused_message",
- f"Session request {kind!r} is the watcher's own.",
- 400,
- )
- if kind == "command":
- return "mutating"
- if kind == "snapshot":
- return "read_only"
- raise RelayError("refused_message", f"Unknown session request {kind!r}.", 400)
- if len(message) != 1:
- raise RelayError(
- "refused_message", "A relayed message names exactly one request.", 400
- )
- (name,) = message
- if name in REFUSED_MESSAGES:
- raise RelayError("refused_message", f"{name!r} cannot be relayed.", 400)
- if name in MUTATING_MESSAGES:
- return "mutating"
- if name in READ_ONLY_MESSAGES:
- return "read_only"
- raise RelayError("refused_message", f"Unknown relayed message {name!r}.", 400)
-
-
-def hosted_launch_of(metadata: dict[str, Any] | None) -> str | None:
- """The hosted launch an agent runs under, from its metadata; None if local."""
- value = (metadata or {}).get("hosted_launch_id")
- return value if isinstance(value, str) else None
-
-
-def frame_size(data: dict[str, Any]) -> int:
- return len(json.dumps(data, separators=(",", ":")).encode())
-
-
-@dataclass(frozen=True, slots=True)
-class WorkerBinding:
- """What an attach proved: which launch revision and host this stream is."""
-
- launch_id: str
- launch_revision: int
- boot_id: str
- instance_id: str
-
-
-@dataclass(frozen=True, slots=True)
-class IdleReport:
- report_seq: int
- relays_through: int
- busy: bool
- reasons: list[dict[str, Any]]
- sessions: dict[str, int]
- launch_revision: int
- generation: int
- received_monotonic: float
- received_at: datetime
-
-
-class FrameSlot:
- """A reserved place in a worker's frame queue. Filling it cannot fail."""
-
- def __init__(self, queue: WorkerFrames, size: int) -> None:
- self._queue = queue
- self._size = size
- self._open = True
-
- def put(self, event: str, data: dict[str, Any]) -> None:
- if not self._open:
- raise RuntimeError("A frame slot is filled once.")
- self._open = False
- self._queue._fill(self._size, event, data)
-
- def release(self) -> None:
- if self._open:
- self._open = False
- self._queue._unreserve(self._size)
-
-
-class WorkerFrames:
- """The protocol 7 frames queued for one worker stream.
-
- Frames with a body count against the bound, and their place is reserved
- before anything durable is written for them, so a full queue refuses work
- before it takes a sequence number. Doorbells carry no body and are never
- refused.
- """
-
- def __init__(self, wake: asyncio.Event) -> None:
- self._wake = wake
- self._frames: list[tuple[str, dict[str, Any]]] = []
- self._body_frames = 0
- self._body_bytes = 0
- self._sizes: list[int] = []
-
- def reserve(self, size: int) -> FrameSlot:
- if (
- self._body_frames + 1 > FRAME_QUEUE_FRAMES
- or self._body_bytes + size > FRAME_QUEUE_BYTES
- ):
- raise WorkerBusyError()
- self._body_frames += 1
- self._body_bytes += size
- return FrameSlot(self, size)
-
- def push(self, event: str, data: dict[str, Any]) -> None:
- self._frames.append((event, data))
- self._sizes.append(0)
- self._wake.set()
-
- def _fill(self, size: int, event: str, data: dict[str, Any]) -> None:
- self._frames.append((event, data))
- self._sizes.append(size)
- self._wake.set()
-
- def _unreserve(self, size: int) -> None:
- self._body_frames -= 1
- self._body_bytes -= size
-
- def drain(self) -> list[tuple[str, dict[str, Any]]]:
- frames = self._frames
- for size in self._sizes:
- if size:
- self._unreserve(size)
- self._frames = []
- self._sizes = []
- return frames
-
- def __bool__(self) -> bool:
- return bool(self._frames)
-
-
-@dataclass
-class PendingRelay:
- id: str
- tenant_id: str
- agent_id: str
- launch_id: str
- launch_revision: int
- relay_seq: int | None
- core_boot: int
- connection_id: str
- generation: int
- boot_id: str
- deadline: float
- future: asyncio.Future[dict[str, Any]] = field(repr=False)
- timer: asyncio.TimerHandle | None = field(default=None, repr=False)
-
-
-class PendingRelays:
- """Relays dispatched to a worker and not yet answered. Memory only.
-
- Each relay fails `relay_timeout` at its deadline whether or not anyone is
- still waiting for it, and `on_expire` is told so the worker can be sent
- `relay_cancel`.
- """
-
- def __init__(self, on_expire: Callable[[PendingRelay], None]) -> None:
- self._by_id: dict[str, PendingRelay] = {}
- self._on_expire = on_expire
-
- def register(
- self,
- *,
- tenant_id: str,
- agent_id: str,
- binding: WorkerBinding,
- relay_seq: int | None,
- core_boot: int,
- connection_id: str,
- generation: int,
- timeout_ms: int,
- ) -> PendingRelay:
- loop = asyncio.get_running_loop()
- relay = PendingRelay(
- id=str(uuid.uuid4()),
- tenant_id=tenant_id,
- agent_id=agent_id,
- launch_id=binding.launch_id,
- launch_revision=binding.launch_revision,
- relay_seq=relay_seq,
- core_boot=core_boot,
- connection_id=connection_id,
- generation=generation,
- boot_id=binding.boot_id,
- deadline=time.monotonic() + timeout_ms / 1000,
- future=loop.create_future(),
- )
- relay.timer = loop.call_later(timeout_ms / 1000, self._expire, relay)
- self._by_id[relay.id] = relay
- return relay
-
- def get(self, relay_id: str) -> PendingRelay | None:
- return self._by_id.get(relay_id)
-
- def unregister(self, relay: PendingRelay) -> None:
- """Withdraw a relay that was never sent."""
- if relay.timer is not None:
- relay.timer.cancel()
- self._by_id.pop(relay.id, None)
- if not relay.future.done():
- relay.future.cancel()
-
- def resolve(self, relay: PendingRelay, answer: dict[str, Any]) -> None:
- if relay.timer is not None:
- relay.timer.cancel()
- relay.future.set_result(answer)
- self._forget_later(relay.id)
-
- def fail(self, relay: PendingRelay, error: RelayError) -> None:
- if relay.timer is not None:
- relay.timer.cancel()
- if not relay.future.done():
- relay.future.set_exception(error)
- # Nobody may be waiting any more; the failure is still the answer.
- relay.future.exception()
- self._forget_later(relay.id)
-
- def _expire(self, relay: PendingRelay) -> None:
- if relay.future.done():
- return
- self.fail(
- relay,
- RelayError(
- "relay_timeout",
- "The worker did not answer in time. The outcome is unknown; "
- "check the session before retrying.",
- 504,
- ),
- )
- self._on_expire(relay)
-
- def _forget_later(self, relay_id: str) -> None:
- asyncio.get_running_loop().call_later(
- RELAY_RESOLVED_RETENTION_SECONDS, self._by_id.pop, relay_id, None
- )
-
- def fail_connection(self, connection_id: str, generation: int | None) -> None:
- """Fail every open relay dispatched to this connection (or generation)."""
- for relay in list(self._by_id.values()):
- if relay.future.done() or relay.connection_id != connection_id:
- continue
- if generation is not None and relay.generation != generation:
- continue
- self.fail(
- relay,
- RelayError(
- "generation_changed",
- "The worker stream changed before it answered.",
- 409,
- ),
- )
-
- def mutating_pending(self, agent_id: str, launch_id: str) -> bool:
- return any(
- not relay.future.done()
- and relay.relay_seq is not None
- and relay.agent_id == agent_id
- and relay.launch_id == launch_id
- for relay in self._by_id.values()
- )
-
-
-class ConsoleView:
- """One Console relay stream's undelivered frames.
-
- Bounded; an overflow drops what is buffered and says `resync`, so the
- Console rebuilds its view instead of applying a view with holes in it.
- """
-
- def __init__(self, subscriptions: frozenset[str]) -> None:
- self.subscriptions = subscriptions
- self.wake = asyncio.Event()
- self._frames: deque[tuple[str, dict[str, Any], int]] = deque()
- self._bytes = 0
-
- def push(self, event: str, data: dict[str, Any]) -> None:
- size = frame_size(data)
- if (
- len(self._frames) + 1 > CONSOLE_VIEW_FRAMES
- or self._bytes + size > CONSOLE_VIEW_BYTES
- ):
- self._frames.clear()
- self._bytes = 0
- event, data = "resync", {"sessionId": None, "reason": "overflow"}
- size = frame_size(data)
- self._frames.append((event, data, size))
- self._bytes += size
- self.wake.set()
-
- def drain(self) -> list[tuple[str, dict[str, Any]]]:
- frames = [(event, data) for event, data, _ in self._frames]
- self._frames.clear()
- self._bytes = 0
- return frames
-
-
-@dataclass
-class _Subscription:
- views: set[ConsoleView] = field(default_factory=set)
- #: The worker generation this subscription was last asked of.
- subscribed_generation: int | None = None
- #: The last push forwarded, as (generation, seq).
- last: tuple[int, int] | None = None
-
-
-def subscribe_message(subscription: str, on: bool) -> dict[str, Any]:
- """The relayed message that starts or stops a worker subscription."""
- if subscription == HEALTH_SUBSCRIPTION:
- return {"watchHealth": on}
- return {"subscribe": subscription} if on else {"unsubscribe": subscription}
-
-
-class RelayViews:
- """Console live views of hosted workers, one worker subscription each.
-
- A worker subscription is held once per (agent, session) however many
- Console streams watch it, and released with the last of them.
- """
-
- def __init__(self) -> None:
- self._subs: dict[tuple[str, str], _Subscription] = {}
-
- def open(self, agent_id: str, view: ConsoleView) -> None:
- for name in view.subscriptions:
- self._subs.setdefault((agent_id, name), _Subscription()).views.add(view)
-
- def close(self, agent_id: str, view: ConsoleView) -> list[tuple[str, int | None]]:
- """Drop the view; the subscriptions nothing watches any more.
-
- Each comes with the generation it was subscribed at, so the caller
- unsubscribes only a worker that was asked.
- """
- released = []
- for name in view.subscriptions:
- sub = self._subs.get((agent_id, name))
- if sub is None:
- continue
- sub.views.discard(view)
- if not sub.views:
- del self._subs[(agent_id, name)]
- released.append((name, sub.subscribed_generation))
- return released
-
- def unsubscribed(self, agent_id: str, generation: int) -> list[str]:
- """Subscriptions the worker at `generation` has not been asked for yet.
-
- Marked asked as they are returned, so one Console stream subscribes
- for all of them.
- """
- names = []
- for (agent, name), sub in self._subs.items():
- if agent == agent_id and sub.subscribed_generation != generation:
- sub.subscribed_generation = generation
- names.append(name)
- return names
-
- def subscribe_failed(self, agent_id: str, name: str, error: RelayError) -> None:
- """Tell every view of a subscription the worker could not be asked, and ask again later."""
- sub = self._subs.get((agent_id, name))
- if sub is None:
- return
- sub.subscribed_generation = None
- session_id = None if name == HEALTH_SUBSCRIPTION else name
- for view in sub.views:
- view.push(
- "error",
- {"sessionId": session_id, "code": error.code, "message": str(error)},
- )
-
- def deliver(
- self, agent_id: str, generation: int, pushes: list[dict[str, Any]]
- ) -> list[str]:
- """Forward a worker's pushes in order; the subscriptions nobody holds."""
- unsubscribe: list[str] = []
- for push in pushes:
- name = push["subscription"]
- sub = self._subs.get((agent_id, name))
- if sub is None:
- if name not in unsubscribe:
- unsubscribe.append(name)
- continue
- seq = push["seq"]
- session_id = None if name == HEALTH_SUBSCRIPTION else name
- if sub.last is not None and sub.last[0] == generation:
- if seq <= sub.last[1]:
- continue
- if seq != sub.last[1] + 1:
- for view in sub.views:
- view.push("resync", {"sessionId": session_id, "reason": "gap"})
- sub.last = (generation, seq)
- for key in ("event", "failure", "health"):
- if key in push:
- frame = (
- {"health": push[key]}
- if key == "health"
- else {"sessionId": session_id, key: push[key]}
- )
- for view in sub.views:
- view.push(key, frame)
- return unsubscribe
-
-
-def offer_key(boot: int, conn: AgentConnection) -> str:
- """Who holds a wake mailbox offer: this Core boot, the worker's connection and its generation."""
- return f"{boot}:{conn.id}:{conn.stream_generation}"
-
-
-def attached_worker_for(
- registry: AgentConnectionRegistry, launch: HostedLaunch
-) -> AgentConnection | None:
- """The launch's attached worker, if it is bound to the launch's current revision."""
- if launch.agent_id is None:
- return None
- conn = registry.attached_worker(launch.agent_id)
- if (
- conn is None
- or conn.worker is None
- or conn.worker.launch_id != launch.id
- or conn.worker.launch_revision != launch.revision
- ):
- return None
- return conn
diff --git a/core/switch_core/bridges/agent/protocol/stream.py b/core/switch_core/bridges/agent/protocol/stream.py
index fe101188d..6ec4ee2f8 100644
--- a/core/switch_core/bridges/agent/protocol/stream.py
+++ b/core/switch_core/bridges/agent/protocol/stream.py
@@ -223,9 +223,6 @@ async def _event_stream(
resync[0] = True
try:
yield _frame("connection_state", _connection_state(conn))
- # A worker's `worker_attached` comes before any gap or buffered event.
- for event, data in conn.worker_frames.drain():
- yield _frame(event, data)
# A cursor ahead of everything we hold is a cursor from a previous
# life of this process: the buffer is in memory, so a restart resets
@@ -327,11 +324,6 @@ async def _event_stream(
yield _frame("evicted", _eviction(TAKEN_OVER))
return
if conn.closure is not None:
- # A Stop's cancel is sent ahead of the eviction it causes, so
- # the worker can release what it had not started yet.
- for event, data in conn.worker_frames.drain():
- if event == "mailbox_cancel":
- yield _frame(event, data)
yield _frame("evicted", _eviction(conn.closure))
return
if not conn.is_alive(time.monotonic()):
@@ -352,10 +344,6 @@ async def _event_stream(
yield _frame("evicted", _eviction(HEARTBEAT_LAPSED))
return
- if conn.worker_frames:
- for event, data in conn.worker_frames.drain():
- yield _frame(event, data)
-
if conn.session_commands:
relayed = list(conn.session_commands)
conn.session_commands.clear()
@@ -499,7 +487,6 @@ async def _event_stream(
# and the clear would otherwise wait for the keepalive timeout.
if (
conn.session_commands
- or conn.worker_frames
or conn.released_rooms
or outcomes
or resync[0]
diff --git a/core/switch_core/bridges/collaboration/collaboration_core.py b/core/switch_core/bridges/collaboration/collaboration_core.py
index c8ca41e89..13fe69aa5 100644
--- a/core/switch_core/bridges/collaboration/collaboration_core.py
+++ b/core/switch_core/bridges/collaboration/collaboration_core.py
@@ -16,7 +16,6 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.aliases import AliasError, validate_alias_format
from switch_core.attachments import parse_attachment_group
from switch_core.bridges.agent.commands import stop_control_frame
-from switch_core.bridges.agent.protocol.hosted_workers import hosted_launch_of
from switch_core.bridges.collaboration.adapter import (
AgentPresentation,
PlatformAdapter,
@@ -1671,14 +1670,7 @@ class CollaborationCore:
thread_id=target.thread_id,
surface=self._bridge_type,
)
- async with tenant_session(
- self._session_factory, self._bridge_tenant_id
- ) as session:
- agent = await self._agent_store.get(session, target.agent_id)
- hosted = agent is not None and hosted_launch_of(agent.metadata_) is not None
- if not self._connections.relay_session_command(
- target.agent_id, frame, worker_only=hosted
- ):
+ if not self._connections.relay_session_command(target.agent_id, frame):
await tell(
"The agent was not stopped: its controller is not connected to Switch."
)
diff --git a/core/switch_core/clients/agent_consumer.py b/core/switch_core/clients/agent_consumer.py
index 662a951a0..7487f6737 100644
--- a/core/switch_core/clients/agent_consumer.py
+++ b/core/switch_core/clients/agent_consumer.py
@@ -6,6 +6,7 @@ import random
import re
from collections.abc import Mapping
from dataclasses import dataclass, field
+from datetime import UTC, datetime
from typing import TYPE_CHECKING, Literal, NamedTuple
from switch_core.attachments import parse_attachment_group
@@ -17,12 +18,6 @@ from switch_core.bridges.agent.commands import (
)
from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.hosted_workers import (
- NOTICE_MESSAGES,
- attached_worker_for,
- hosted_launch_of,
- offer_key,
-)
from switch_core.bridges.agent.protocol.presence import (
agents_present_in,
rooms_occupied,
@@ -58,7 +53,7 @@ from switch_core.clients.mentions import (
strip_emphasis as _strip_emphasis,
)
from switch_core.clients.room_meta import RoomMeta
-from switch_core.db.models import Agent, HostedLaunch, HostedMachine
+from switch_core.db.models import Agent, HostedMachine
from switch_core.db.session_scope import tenant_session
from switch_core.db.stores.agent_session_store import AgentSessionStore
from switch_core.db.stores.agent_store import AgentStore
@@ -66,18 +61,7 @@ from switch_core.db.stores.budget_store import BudgetStore
from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
from switch_core.db.stores.document_store import DocumentStore
from switch_core.db.stores.external_user_store import ExternalUserStore
-from switch_core.db.stores.hosted_launch_store import (
- HostedLaunchStore,
- ProviderDisconnected,
- is_waking,
-)
from switch_core.db.stores.hosted_machine_store import owner_stopped
-from switch_core.db.stores.hosted_mailbox_store import (
- MAILBOX_LIMIT,
- HostedMailboxStore,
- MailboxEntry,
- MailboxFull,
-)
from switch_core.db.stores.reference_store import ReferenceStore
from switch_core.db.stores.room_role_store import RoomRoleStore
from switch_core.db.stores.room_store import RoomStore
@@ -96,6 +80,7 @@ from switch_core.events import (
TaskFinalise,
TaskUpdate,
)
+from switch_core.gateway.cloud_controllers import wake_controller_machine
from switch_core.gateway.known_agents import known_agent_for
from switch_core.observability.catalogue import OBSERVER_FAILURES
from switch_core.observability.metrics import metrics
@@ -172,91 +157,6 @@ _UNAVAILABLE_MESSAGES = {
# the connector will spin a session up on demand to handle the message.
_STARTING_SESSION_MESSAGE = "Starting a session to handle this — one moment."
-_WAKING_MESSAGE = "Waking up my cloud worker — I'll answer in a minute or two."
-
-_MAILBOX_FULL_MESSAGE = (
- f"My cloud worker already has {MAILBOX_LIMIT} messages waiting, so I did not "
- "queue this one. Please send it again once I have caught up."
-)
-
-
-@dataclass(frozen=True)
-class HostedNote:
- """What addressing a hosted agent did.
-
- `refusal` is set when the event was not taken into the wake mailbox and
- the room must be told why; the event is then not delivered either.
- `deliver` is False for an event the mailbox already holds.
- """
-
- launch: HostedLaunch | None
- machine: HostedMachine | None
- refusal: str | None
- deliver: bool
-
- @property
- def redelivered(self) -> bool:
- """The mailbox already held this event: it was seen here before."""
- return self.refusal is None and not self.deliver
-
-
-_HOSTED_STOPPED_MESSAGE = (
- "My cloud worker is stopped, so I did not process this message. "
- "Start me again in Switch Console, then send it again."
-)
-_HOSTED_REMOVED_MESSAGE = (
- "My cloud worker has been removed, so I cannot process messages."
-)
-_HOSTED_ERROR_MESSAGE = (
- "My cloud worker has a problem, so I did not process this message. "
- "My owner can check it in Switch Console."
-)
-_HOSTED_MACHINE_STOPPED_MESSAGE = (
- "My owner stopped my cloud machine, so I did not process this message. "
- "Ask my owner to start it in Switch Console, then send it again."
-)
-_HOSTED_MACHINE_ERROR_MESSAGE = (
- "My cloud machine has a problem, so I did not process this message. "
- "My owner can check it in Switch Console."
-)
-
-
-def _hosted_unavailable(
- launch: HostedLaunch, machine: HostedMachine | None
-) -> str | None:
- """Why a hosted agent that takes no mail cannot answer, for the room.
-
- The generic unavailable reply offers a terminal command, which means
- nothing for an agent that runs on a cloud worker.
- """
- if launch.desired_state == "deleted":
- return _HOSTED_REMOVED_MESSAGE
- if launch.state == "error":
- return _HOSTED_ERROR_MESSAGE
- if machine is not None and machine.state == "error":
- return _HOSTED_MACHINE_ERROR_MESSAGE
- if machine is not None and owner_stopped(machine):
- return _HOSTED_MACHINE_STOPPED_MESSAGE
- if launch.desired_state == "stopped":
- return _HOSTED_STOPPED_MESSAGE
- return None
-
-
-def _takes_mail(launch: HostedLaunch, machine: HostedMachine | None) -> bool:
- """Whether an addressed event goes into the launch's wake mailbox.
-
- Not after an explicit Stop of the agent or its machine, a delete, or an
- error of the launch or its machine: the unavailable reply tells the room
- instead. An idle-sleeping machine takes mail; addressing it wakes it.
- """
- return (
- launch.desired_state == "running"
- and launch.state != "error"
- and not (machine is not None and machine.state == "error")
- and not (machine is not None and owner_stopped(machine))
- )
-
-
# Fallback (no known-agent connect command) for a session_addressable agent
# that has a session bound to this room but is not reporting as live — the
# usual cause is a session started without the dev-channels flag.
@@ -336,6 +236,41 @@ def _machine_offline_message(machine: str, owner_handle: str | None) -> str:
)
+@dataclass(frozen=True)
+class CloudMachineState:
+ """How a Switch cloud machine stood once a message for its agents tried to wake it."""
+
+ state: str
+ revision: int
+ owner_stopped: bool
+
+ @classmethod
+ def of(cls, machine: HostedMachine) -> CloudMachineState:
+ return cls(
+ state=machine.state,
+ revision=machine.revision,
+ owner_stopped=owner_stopped(machine),
+ )
+
+
+def _cloud_machine_waking_message(machine: str) -> str:
+ """A managed agent whose Switch cloud machine was asleep, and is starting
+ for this message: Switch keeps the message until the machine is back."""
+ return (
+ f"My machine, **{machine}**, was asleep and is starting now. I'll pick "
+ "this up as soon as it's back, usually within a few minutes."
+ )
+
+
+def _cloud_machine_stopped_message(machine: str, owner_handle: str | None) -> str:
+ """A managed agent whose Switch cloud machine its owner stopped: a message
+ does not start it again."""
+ return (
+ f"My machine, **{machine}**, is stopped, so I can't answer. "
+ f"{_owner_ref(owner_handle).capitalize()} can start it again in Switch Console."
+ )
+
+
def _machine_removed_message(machine: str, owner_handle: str | None) -> str:
removed = (
f"my machine, **{machine}**, has been removed from Switch, so nothing "
@@ -401,7 +336,6 @@ class AgentConsumer(Consumer[AgentActor]):
agent_session_store: AgentSessionStore,
room_role_store: RoomRoleStore,
external_user_store: ExternalUserStore,
- hosted_launch_store: HostedLaunchStore,
connections: AgentConnectionRegistry,
frontend_base_url: str | None,
message_telemetry: MessageTelemetry,
@@ -416,9 +350,7 @@ class AgentConsumer(Consumer[AgentActor]):
self._agent_session_store = agent_session_store
self._room_role_store = room_role_store
self._external_user_store = external_user_store
- self._hosted_launch_store = hosted_launch_store
- self._waking_notice_revisions: dict[str, int] = {}
- self._unreachable_notice_revisions: dict[str, int] = {}
+ self._cloud_waking_notices: set[tuple[str, int]] = set()
self._connections = connections
self._message_telemetry = message_telemetry
self._frontend_base_url = (
@@ -664,42 +596,9 @@ class AgentConsumer(Consumer[AgentActor]):
),
)
- hosted: HostedNote | None = None
- if is_addressed:
- hosted = await self._note_hosted_addressed(agent, agent_event)
- launch = None if hosted is None else hosted.launch
- machine = None if hosted is None else hosted.machine
- if hosted is not None and hosted.refusal is not None:
- unavailable = (
- None if self._triggered_by_auto_reply(event) else hosted.refusal
- )
- elif (
- unavailable is not None
- and launch is not None
- and machine is not None
- and is_waking(launch, machine)
- ):
- unavailable = None
- if self._waking_notice_revisions.get(meta.room_id) != machine.revision:
- self._waking_notice_revisions[meta.room_id] = machine.revision
- unavailable = _WAKING_MESSAGE
- elif (
- unavailable is not None
- and launch is not None
- and attached_worker_for(self._connections, launch) is None
- ):
- # Once per room per revision, like the waking notice: the
- # mailbox holds every message until the worker attaches.
- unavailable = None
- if (
- self._unreachable_notice_revisions.get(meta.room_id)
- != launch.revision
- ):
- self._unreachable_notice_revisions[meta.room_id] = launch.revision
- unavailable = NOTICE_MESSAGES["unreachable"]
if is_addressed:
self._report_addressed(
- meta, event, hosted, agent_live=agent_live, has_attachment=False
+ meta, event, agent_live=agent_live, has_attachment=False
)
if refusal is not None:
@@ -726,9 +625,6 @@ class AgentConsumer(Consumer[AgentActor]):
else reply_thread_root,
)
- if hosted is not None and not hosted.deliver:
- return
-
# Serialising runs once per agent per message, so only when it will be seen.
if logger.isEnabledFor(logging.DEBUG):
logger.debug("Enqueuing event %s", agent_event.model_dump_json(indent=2))
@@ -903,7 +799,6 @@ class AgentConsumer(Consumer[AgentActor]):
body: str,
) -> None:
reply_thread_root = thread_id if thread_id is not None else event.event_id
- gated: Agent | None = None
agent_live = False
if is_addressed:
async with self.session_factory() as session:
@@ -917,8 +812,6 @@ class AgentConsumer(Consumer[AgentActor]):
and not self._triggered_by_auto_reply(event)
):
agent_live = await self._is_available(session, agent, meta.room_id)
- if gate.addressed:
- gated = agent
is_addressed = gate.addressed
if gate.refusal is not None:
await self._post_auto_reply(
@@ -954,19 +847,10 @@ class AgentConsumer(Consumer[AgentActor]):
),
)
- hosted: HostedNote | None = None
- if gated is not None:
- hosted = await self._note_hosted_addressed(gated, agent_event)
- if hosted is not None and hosted.refusal is not None:
- await self._post_auto_reply(
- room.room_id, event, hosted.refusal, reply_thread_root
- )
if is_addressed:
self._report_addressed(
- meta, event, hosted, agent_live=agent_live, has_attachment=True
+ meta, event, agent_live=agent_live, has_attachment=True
)
- if hosted is not None and not hosted.deliver:
- return
if logger.isEnabledFor(logging.DEBUG):
logger.debug(
@@ -1003,10 +887,6 @@ class AgentConsumer(Consumer[AgentActor]):
if handled:
return
- async with tenant_session(self.session_factory, self.tenant_id) as session:
- agent = await self._fresh_agent(session)
- await self._note_hosted_addressed(agent, None)
-
self._event_buffer.enqueue(
self.agent.id,
meta.room_id,
@@ -1135,94 +1015,6 @@ class AgentConsumer(Consumer[AgentActor]):
self._room_meta[transport_room_id] = meta
return meta
- async def _note_hosted_addressed(
- self, agent: Agent, event: AgentEvent | None
- ) -> HostedNote | None:
- """Keep a hosted agent's cloud worker awake, waking it if it idled out.
-
- `event`, when the watcher acts on it, is written to the wake mailbox
- in the same transaction, already offered when the agent's worker is
- attached at the launch's revision. None for a command, which is never
- queued. A wake starts the launch's idle-stopped machine; an
- owner-stopped machine is left stopped. None when the agent is not
- hosted or its launch could not be updated.
- """
- launch_id = hosted_launch_of(agent.metadata_)
- if launch_id is None:
- return None
- entry = None if event is None else MailboxEntry.of(event)
- refusal: str | None = None
- written = True
- try:
- async with (
- asyncio.timeout(5),
- tenant_session(self.session_factory, self.tenant_id) as session,
- ):
- launch, machine = await self._hosted_launch_store.note_addressed(
- session, launch_id
- )
- if (
- launch is not None
- and entry is not None
- and not _takes_mail(launch, machine)
- ):
- refusal = _hosted_unavailable(launch, machine)
- written = False
- elif launch is not None and entry is not None:
- worker = attached_worker_for(self._connections, launch)
- try:
- written = await HostedMailboxStore().write(
- session,
- agent_id=agent.id,
- launch_id=launch.id,
- entry=entry,
- offered_to=None
- if worker is None
- else offer_key(self._event_buffer.boot, worker),
- )
- except MailboxFull:
- logger.warning(
- "Wake mailbox of agent %s is full; refused message %s in room %s",
- agent.id,
- entry.message_id,
- entry.room_id,
- )
- refusal = _MAILBOX_FULL_MESSAGE
- await session.commit()
- except ProviderDisconnected:
- logger.warning(
- "Not waking the cloud worker of agent %s: its owner's provider connection is gone",
- agent.id,
- )
- return HostedNote(
- launch=None,
- machine=None,
- refusal=NOTICE_MESSAGES["revoked"],
- deliver=False,
- )
- except Exception:
- logger.error(
- "Could not wake cloud worker for agent %s or write its wake mailbox; "
- "the event is only in the live buffer",
- agent.id,
- exc_info=True,
- )
- return None
- if launch is None:
- logger.error(
- "Agent %s names cloud launch %s, which does not exist",
- agent.id,
- launch_id,
- )
- return None
- self._connections.supersede(agent.id, launch.revision)
- return HostedNote(
- launch=launch,
- machine=machine,
- refusal=refusal,
- deliver=refusal is None and written,
- )
-
async def _reply_when_unavailable_here(
self, session: AsyncSession, agent: Agent, meta: RoomMeta, asker_handle: str
) -> str | None:
@@ -1358,9 +1150,42 @@ class AgentConsumer(Consumer[AgentActor]):
if controllers.is_revoked(binding.controller_id):
return _machine_removed_message(binding.controller_name, owner)
if not controllers.is_live(self.agent.id):
- return _machine_offline_message(binding.controller_name, owner)
+ machine = await self._wake_cloud_machine(binding.controller_id)
+ if machine is None or machine.state == "error":
+ return _machine_offline_message(binding.controller_name, owner)
+ if machine.owner_stopped:
+ return _cloud_machine_stopped_message(binding.controller_name, owner)
+ # Once per room for each start: the next messages just wait with it.
+ notice = (meta.room_id, machine.revision)
+ if notice in self._cloud_waking_notices:
+ return None
+ self._cloud_waking_notices.add(notice)
+ return _cloud_machine_waking_message(binding.controller_name)
return None
+ async def _wake_cloud_machine(self, controller_id: str) -> CloudMachineState | None:
+ """Starts the Switch cloud machine running `controller_id` if it is
+ asleep, and says how it stands; None for a controller on no cloud
+ machine, or when the machine could not be read."""
+ try:
+ async with (
+ asyncio.timeout(5),
+ tenant_session(self.session_factory, self.tenant_id) as session,
+ ):
+ machine = await wake_controller_machine(
+ session, controller_id, datetime.now(UTC)
+ )
+ state = None if machine is None else CloudMachineState.of(machine)
+ await session.commit()
+ return state
+ except Exception:
+ logger.warning(
+ "Could not wake the cloud machine of controller %s",
+ controller_id,
+ exc_info=True,
+ )
+ return None
+
async def owner_handle_in(
self, session: AsyncSession, agent: Agent, bridge_id: str | None
) -> str | None:
@@ -1499,8 +1324,6 @@ class AgentConsumer(Consumer[AgentActor]):
async def on_task_delegate(self, room: RoomRef, event: TaskDelegate) -> None:
if event.performer_agent_id != self.agent.id:
return
- async with tenant_session(self.session_factory, self.tenant_id) as session:
- agent = await self._fresh_agent(session)
meta = await self._resolve_room_meta(room.room_id)
agent_event = (
None
@@ -1519,22 +1342,11 @@ class AgentConsumer(Consumer[AgentActor]):
),
)
)
- hosted = await self._note_hosted_addressed(agent, agent_event)
- launch = None if hosted is None else hosted.launch
- machine = None if hosted is None else hosted.machine
- if hosted is not None and hosted.refusal is not None:
- reply = hosted.refusal
- elif launch is not None and is_waking(launch, machine):
- reply = _WAKING_MESSAGE
- else:
- reply = "Working on it."
await self.actor.send_message(
- room.room_id, reply, format="markdown", metered=False
+ room.room_id, "Working on it.", format="markdown", metered=False
)
if meta is None or agent_event is None:
return
- if hosted is not None and not hosted.deliver:
- return
self._event_buffer.enqueue(self.agent.id, meta.room_id, agent_event)
async def on_task_accept(self, room: RoomRef, event: TaskAccept) -> None:
@@ -1743,7 +1555,6 @@ class AgentConsumer(Consumer[AgentActor]):
self,
meta: RoomMeta,
event: InboundMessage,
- hosted: HostedNote | None,
*,
agent_live: bool,
has_attachment: bool,
@@ -1752,13 +1563,9 @@ class AgentConsumer(Consumer[AgentActor]):
Every message let through the addressing policy and budget, whether or
not the agent could take it, so an offline or stopped agent is still
- asked. `agent_live` is the agent having a live session for the room,
- and false as well when a hosted agent's mailbox refused the message:
- a full mailbox or a lost provider turns it away even with a worker
- attached. Not an auto-reply: Switch's notice that another agent is
- offline or refused is not someone asking this one for something. Not
- a redelivery the mailbox already holds: that request was counted when
- it first arrived. Not the files of a multi-file post delivered without
+ asked. `agent_live` is the agent having a live session for the room.
+ Not an auto-reply: Switch's notice that another agent is offline or
+ refused is not someone asking this one for something. Not the files of a multi-file post delivered without
its first part, which happens when a group times out incomplete and
the rest arrives later: the post is counted with its first part, as
the transport counts it, so it is one request however it is split.
@@ -1769,8 +1576,6 @@ class AgentConsumer(Consumer[AgentActor]):
"""
if self._triggered_by_auto_reply(event):
return
- if hosted is not None and hosted.redelivered:
- return
group = parse_attachment_group(event.content)
if group is not None and group[1] != 0:
return
@@ -1781,7 +1586,7 @@ class AgentConsumer(Consumer[AgentActor]):
sender_transport_user_id=event.sender,
from_platform=PLATFORM_MARKER in event.content,
agent_metadata=self.agent.metadata_,
- agent_live=agent_live and (hosted is None or hosted.refusal is None),
+ agent_live=agent_live,
has_attachment=has_attachment,
)
except Exception:
diff --git a/core/switch_core/config.py b/core/switch_core/config.py
index 59062760a..978d3a2c6 100644
--- a/core/switch_core/config.py
+++ b/core/switch_core/config.py
@@ -221,15 +221,13 @@ class SwitchConfig(BaseSettings):
# OIDC first sign-in alike), read from the users table so it holds across
# replicas. Sign-up is refused once the count reaches this.
gateway_signup_max_per_hour: int = Field(default=20, ge=1)
+ # How many cloud machines the bound tenant may have at once. Cloud machines
+ # run the agents controller, so they need AGENT_MANAGEMENT_ENABLED.
hosted_launch_capacity: int = Field(default=0, ge=0, le=100)
- hosted_sessions_per_agent: int = Field(default=8, ge=1, le=100)
- hosted_agents_per_owner: int = Field(default=3, ge=1, le=100)
hosted_idle_stop_minutes: int = Field(default=30, ge=0, le=1440)
hosted_disk_retention_days: int = Field(default=7, ge=1, le=90)
hosted_controller_config_path: str | None = None
hosted_github_config_path: str | None = None
- hosted_provider_verification_enabled: bool = False
- hosted_claude_verifier_path: str | None = None
# Sets the Secure flag on the gateway's cookies (the switch_auth session
# and the OIDC sign-in cookie), so they are never sent over plain HTTP.
# Only a local stack served over http:// should turn it off.
@@ -721,6 +719,11 @@ class SwitchConfig(BaseSettings):
f"{self.controller_status_interval_seconds!r}."
)
if not self.agent_management_enabled:
+ if self.hosted_launch_capacity > 0:
+ raise ValueError(
+ "HOSTED_LAUNCH_CAPACITY needs AGENT_MANAGEMENT_ENABLED: cloud "
+ "machines run the agents controller."
+ )
return self
if not self.controller_token_secret:
raise ValueError(
@@ -1508,14 +1511,6 @@ class SwitchConfig(BaseSettings):
return {"ssl": context}
-def hosted_configured(config: SwitchConfig) -> bool:
- """Whether this server runs cloud agents: it has a hosted controller or launch capacity."""
- return (
- config.hosted_controller_config_path is not None
- or config.hosted_launch_capacity > 0
- )
-
-
def deprecated_env_names() -> list[str]:
"""Settings read from a name kept only for compatibility, as warnings.
diff --git a/core/switch_core/db/hosted_cutover_gate.py b/core/switch_core/db/hosted_cutover_gate.py
deleted file mode 100644
index a085cde23..000000000
--- a/core/switch_core/db/hosted_cutover_gate.py
+++ /dev/null
@@ -1,200 +0,0 @@
-"""What the drop of the old server-side session tables waits for on a hosted deployment.
-
-Plain SQL on a synchronous connection, so `hosted-cutover-upgrade`,
-`migrations/env.py` and the merge revision `33e037ee949f` evaluate the same
-conditions. `env.py` checks them on the migration connection before the first
-revision of any upgrade plan that includes `b9e4d2a71c05`, while the old
-tables can still be read; the merge revision checks again after the drop, and
-there whatever needs the old tables is gone and skipped.
-"""
-
-from __future__ import annotations
-
-from sqlalchemy import Connection, text
-
-DROP_REVISION = "b9e4d2a71c05"
-
-RUNNING_LAUNCHES = """
- SELECT id FROM hosted_launches
- WHERE state NOT IN ('deleting', 'deleted')
- AND NOT (state = 'stopped' AND desired_state = 'stopped' AND sleeping = false)
- ORDER BY id
-"""
-
-_HOSTED_AGENTS = """
- SELECT DISTINCT ON (tenant_id, agent_id) tenant_id, agent_id, id AS launch_id
- FROM hosted_launches
- WHERE agent_id IS NOT NULL AND state NOT IN ('deleting', 'deleted')
- ORDER BY tenant_id, agent_id, created_at DESC
-"""
-
-RETAINED = """
- JOIN hosted_launches l ON l.tenant_id = {alias}.tenant_id AND l.id = {alias}.launch_id
- AND l.state NOT IN ('deleting', 'deleted')
-"""
-
-_IMPORT_BLOBS = f"""
- SELECT i.tenant_id, i.launch_id, a->>'mxc' AS uri
- FROM hosted_cutover_items i
- {RETAINED.format(alias="i")}
- CROSS JOIN LATERAL jsonb_array_elements(
- COALESCE(i.payload->'payload'->'attachments', '[]'::jsonb)) a
- WHERE i.disposition = 'import'
-"""
-
-_VOLUMES: list[tuple[str, str]] = [
- (
- f"""
- SELECT h.launch_id FROM ({_HOSTED_AGENTS}) h
- WHERE NOT EXISTS (SELECT 1 FROM hosted_cutover_volumes v
- WHERE v.tenant_id = h.tenant_id AND v.launch_id = h.launch_id)
- ORDER BY 1
- """,
- "launch {} has no cutover volume; it was created after `prepare`, so "
- "downgrade to 95fc38e451b6 and run `prepare` again",
- ),
- (
- f"""
- SELECT v.launch_id FROM hosted_cutover_volumes v {RETAINED.format(alias="v")}
- WHERE v.preflight_state = 'pending' ORDER BY 1
- """,
- "the volume of launch {} has no recorded preflight check; run "
- "`--preflight-check` on it and `record` the result",
- ),
- (
- f"""
- SELECT v.launch_id || ': ' || COALESCE(v.blocked_reason, '')
- FROM hosted_cutover_volumes v {RETAINED.format(alias="v")}
- WHERE v.preflight_state = 'blocked' ORDER BY 1
- """,
- "the preflight check blocked on the volume of launch {}; repair the file "
- "it names, check the volume again and `record` the result",
- ),
- (
- f"""
- SELECT DISTINCT i.launch_id FROM hosted_cutover_items i {RETAINED.format(alias="i")}
- WHERE i.disposition IS NULL ORDER BY 1
- """,
- "launch {} has cutover items no manifest decided; `record` its volume",
- ),
- (
- f"""
- SELECT i.launch_id || ' ' || i.room_id || ' ' || i.message_id
- FROM hosted_cutover_items i {RETAINED.format(alias="i")}
- WHERE i.disposition = 'import' AND i.payload IS NULL ORDER BY 1
- """,
- "the import {} has no event to queue",
- ),
- (
- f"""
- SELECT b.launch_id || ' ' || b.uri FROM ({_IMPORT_BLOBS}) b
- WHERE NOT EXISTS (SELECT 1 FROM media_blobs m
- WHERE m.tenant_id = b.tenant_id AND m.uri = b.uri)
- ORDER BY 1
- """,
- "the attachment of an import for launch {} is gone",
- ),
-]
-
-_OLD_TABLES: list[tuple[str, str]] = [
- (
- f"""
- SELECT b.launch_id || ' ' || b.uri FROM ({_IMPORT_BLOBS}) b
- WHERE EXISTS (SELECT 1 FROM media_blobs m
- WHERE m.tenant_id = b.tenant_id AND m.uri = b.uri
- AND m.sdk_session_id IS NOT NULL)
- ORDER BY 1
- """,
- "the attachment of an import for launch {} would be dropped with its "
- "session; `record` the volume again to keep it",
- ),
- (
- f"""
- SELECT latest.command_id FROM (
- SELECT DISTINCT ON (c.tenant_id, s.agent_id, c.command->'origin'->>'roomId',
- c.command->'origin'->>'messageId')
- c.tenant_id, s.agent_id, c.command_id, c.status->>'status' AS status,
- c.command->'origin'->>'roomId' AS room_id,
- c.command->'origin'->>'messageId' AS message_id
- FROM sdk_session_commands c
- JOIN sdk_sessions s ON s.tenant_id = c.tenant_id AND s.id = c.session_id
- JOIN ({_HOSTED_AGENTS}) h ON h.tenant_id = s.tenant_id AND h.agent_id = s.agent_id
- WHERE c.command->'body'->>'type' = 'message.send'
- AND c.command->'origin'->>'roomId' IS NOT NULL
- AND c.command->'origin'->>'messageId' IS NOT NULL
- ORDER BY c.tenant_id, s.agent_id, c.command->'origin'->>'roomId',
- c.command->'origin'->>'messageId', c.accepted_sequence DESC, c.command_id DESC
- ) latest
- WHERE NOT EXISTS (SELECT 1 FROM hosted_cutover_items i
- WHERE i.tenant_id = latest.tenant_id AND i.agent_id = latest.agent_id
- AND i.kind = 'room_message' AND i.room_id = latest.room_id
- AND i.message_id = latest.message_id
- AND i.evidence->'core'->>'command_id' = latest.command_id
- AND i.evidence->'core'->>'status' IS NOT DISTINCT FROM latest.status)
- UNION ALL
- SELECT c.command_id FROM sdk_session_commands c
- JOIN sdk_sessions s ON s.tenant_id = c.tenant_id AND s.id = c.session_id
- JOIN ({_HOSTED_AGENTS}) h ON h.tenant_id = s.tenant_id AND h.agent_id = s.agent_id
- WHERE c.command->'origin'->>'surface' = 'console'
- AND c.command->'origin'->>'roomId' IS NULL
- AND c.status->>'status' = 'accepted'
- AND NOT EXISTS (SELECT 1 FROM hosted_cutover_items i
- WHERE i.tenant_id = c.tenant_id AND i.agent_id = s.agent_id
- AND i.kind = 'console_command'
- AND i.evidence->'core'->>'command_id' = c.command_id)
- UNION ALL
- SELECT p.request_id FROM session_request_posts p
- JOIN sdk_sessions s ON s.tenant_id = p.tenant_id AND s.id = p.session_id
- JOIN ({_HOSTED_AGENTS}) h ON h.tenant_id = s.tenant_id AND h.agent_id = s.agent_id
- WHERE p.removed_at IS NULL AND p.room_id IS NOT NULL
- AND NOT EXISTS (SELECT 1 FROM hosted_cutover_items i
- WHERE i.tenant_id = p.tenant_id AND i.agent_id = s.agent_id
- AND i.kind = 'request_open'
- AND i.evidence->'core'->>'request_id' = p.request_id)
- ORDER BY 1
- """,
- "{} changed in the old session tables after `prepare` captured them, so "
- "the old Core ran again; downgrade to 95fc38e451b6 and run `prepare` again",
- ),
-]
-
-
-def _present(connection: Connection, table: str) -> bool:
- return bool(
- connection.scalar(
- text("SELECT to_regclass(:table) IS NOT NULL"), {"table": table}
- )
- )
-
-
-def cutover_gate_problems(connection: Connection) -> list[str]:
- """Everything that still stops the drop, each naming its launch or item."""
- if not _present(connection, "hosted_launches"):
- return []
- problems = [
- f"launch {launch_id} is not stopped"
- for launch_id in connection.scalars(text(RUNNING_LAUNCHES))
- ]
- if not _present(connection, "hosted_cutover_volumes"):
- return problems + [
- f"launch {row.launch_id} has hosted state the cutover has not "
- "captured; run `just hosted-cutover-upgrade prepare`"
- for row in connection.execute(text(_HOSTED_AGENTS))
- ]
- gate = _VOLUMES + (_OLD_TABLES if _present(connection, "sdk_sessions") else [])
- for query, message in gate:
- problems += [message.format(row) for row in connection.scalars(text(query))]
- return problems
-
-
-def refuse_incomplete_cutover(connection: Connection) -> None:
- """Raise unless the drop of the old session tables may go ahead."""
- problems = cutover_gate_problems(connection)
- if problems:
- raise RuntimeError(
- "Refusing to drop the old session tables before the hosted cutover is "
- "complete: "
- + "; ".join(problems)
- + ". Run `just hosted-cutover-upgrade status` and follow the cutover "
- "steps in docs/hosted-activity-contracts.md."
- )
diff --git a/core/switch_core/db/key_rotation.py b/core/switch_core/db/key_rotation.py
index 49604d06a..148b6e36b 100644
--- a/core/switch_core/db/key_rotation.py
+++ b/core/switch_core/db/key_rotation.py
@@ -22,12 +22,9 @@ from switch_core.db.encrypted_json import reencrypt_stale_values
from switch_core.db.models import (
ApiKey,
CollaborationBridge,
- GitHubIssuedToken,
- HostedLaunch,
HostedMachine,
MessagingInstall,
ProviderConnection,
- ProviderVerification,
ServerConnector,
)
from switch_core.db.session_scope import tenant_session
@@ -43,11 +40,7 @@ _ENCRYPTED_TEXT_COLUMNS: tuple[tuple[type[Any], str], ...] = (
(ApiKey, "encrypted_key"),
(MessagingInstall, "encrypted_bot_token"),
(ProviderConnection, "encrypted_credential"),
- (ProviderVerification, "encrypted_credential"),
- (ProviderVerification, "encrypted_token"),
- (GitHubIssuedToken, "encrypted_token"),
- (HostedMachine, "machine_capability_encrypted"),
- (HostedLaunch, "worker_capability_encrypted"),
+ (HostedMachine, "enrollment_code_encrypted"),
)
diff --git a/core/switch_core/db/models.py b/core/switch_core/db/models.py
index 47084f516..b73879aaf 100644
--- a/core/switch_core/db/models.py
+++ b/core/switch_core/db/models.py
@@ -288,11 +288,11 @@ class ProviderConnection(TenantScoped, Base):
__table_args__ = (
PrimaryKeyConstraint("tenant_id", "user_id", "provider"),
CheckConstraint(
- "provider IN ('claude', 'github', 'codex', 'opencode', 'cursor', 'antigravity')",
+ "provider = 'github'",
name="ck_provider_connections_provider",
),
CheckConstraint(
- "(provider = 'claude' AND kind IN ('api-key', 'setup-token')) OR (provider = 'github' AND kind = 'oauth') OR (provider = 'codex' AND kind IN ('api-key', 'auth-json')) OR (provider = 'cursor' AND kind = 'api-key') OR (provider IN ('opencode', 'antigravity') AND kind = 'auth-json')",
+ "kind = 'oauth'",
name="ck_provider_connections_kind",
),
)
@@ -311,40 +311,8 @@ class ProviderConnection(TenantScoped, Base):
)
-class ProviderVerification(TenantScoped, Base):
- __tablename__ = "provider_verifications"
- __table_args__ = (
- PrimaryKeyConstraint("tenant_id", "id"),
- Index(
- "ix_provider_verification_owner",
- "tenant_id",
- "user_id",
- "provider",
- "created_at",
- ),
- Index("ix_provider_verification_state", "tenant_id", "state"),
- )
-
- id: Mapped[str] = mapped_column(Text, nullable=False)
- user_id: Mapped[str] = mapped_column(
- Text, ForeignKey("users.id", ondelete="CASCADE"), nullable=False
- )
- provider: Mapped[str] = mapped_column(Text, nullable=False)
- kind: Mapped[str] = mapped_column(Text, nullable=False)
- encrypted_credential: Mapped[str | None] = mapped_column(Text)
- encrypted_token: Mapped[str | None] = mapped_column(Text)
- token_hash: Mapped[str] = mapped_column(Text, nullable=False)
- state: Mapped[str] = mapped_column(Text, nullable=False)
- result: Mapped[bool | None] = mapped_column(Boolean)
- instance_id: Mapped[str | None] = mapped_column(Text)
- created_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), nullable=False
- )
- deadline: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
-
-
class HostedMachine(TenantScoped, Base):
- """One user's hosted VM, which runs every one of that user's cloud agents."""
+ """One user's cloud VM, whose agents controller runs that user's cloud agents."""
__tablename__ = "hosted_machines"
__table_args__ = (
@@ -361,7 +329,12 @@ class HostedMachine(TenantScoped, Base):
"stop_reason IS NULL OR stop_reason IN ('idle', 'owner')",
name="ck_hosted_machine_stop_reason",
),
- CheckConstraint("generation >= 1", name="ck_hosted_machine_generation"),
+ ForeignKeyConstraint(
+ ["tenant_id", "controller_id"],
+ ["agent_controllers.tenant_id", "agent_controllers.id"],
+ name="fk_hosted_machines_controller",
+ ondelete="SET NULL (controller_id)",
+ ),
Index(
"uq_hosted_machine_owner",
"tenant_id",
@@ -369,26 +342,10 @@ class HostedMachine(TenantScoped, Base):
unique=True,
postgresql_where=text("state <> 'deleted'"),
),
- Index(
- "uq_hosted_machine_slot",
- "tenant_id",
- "slot_id",
- unique=True,
- postgresql_where=text("state <> 'deleted'"),
- ),
- Index(
- "uq_hosted_machine_generation",
- "tenant_id",
- "slot_id",
- "generation",
- unique=True,
- ),
)
id: Mapped[str] = mapped_column(Text, nullable=False)
owner_id: Mapped[str] = mapped_column(Text, ForeignKey("users.id"), nullable=False)
- slot_id: Mapped[str] = mapped_column(Text, nullable=False)
- generation: Mapped[int] = mapped_column(Integer, nullable=False)
state: Mapped[str] = mapped_column(Text, nullable=False)
desired_state: Mapped[str] = mapped_column(
Text, nullable=False, server_default="running"
@@ -399,12 +356,13 @@ class HostedMachine(TenantScoped, Base):
data_volume_id: Mapped[str | None] = mapped_column(Text)
instance_id: Mapped[str | None] = mapped_column(Text)
retain_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
- machine_capability_hash: Mapped[str | None] = mapped_column(Text)
- machine_capability_encrypted: Mapped[str | None] = mapped_column(Text)
- machine_capability_revision: Mapped[int | None] = mapped_column(Integer)
- agents_version: Mapped[int] = mapped_column(
- Integer, nullable=False, server_default="1"
- )
+ # The agents controller the machine enrolled as, with a one-time code
+ # minted for it; the owner's managed agents placed on it run there.
+ controller_id: Mapped[str | None] = mapped_column(Text)
+ # The code the machine enrolls with, kept for the revision it was issued
+ # at, so every retry of that revision hands over the same one.
+ enrollment_code_encrypted: Mapped[str | None] = mapped_column(Text)
+ enrollment_code_revision: Mapped[int | None] = mapped_column(Integer)
active_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
heartbeat: Mapped[dict | None] = mapped_column(JSONB)
heartbeat_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
@@ -422,285 +380,6 @@ class HostedMachine(TenantScoped, Base):
)
-class HostedLaunch(TenantScoped, Base):
- __tablename__ = "hosted_launches"
- __table_args__ = (
- PrimaryKeyConstraint("tenant_id", "id"),
- UniqueConstraint("tenant_id", "name", name="uq_hosted_launch_name"),
- CheckConstraint(
- "state IN ('queued', 'provisioning', 'ready', 'error', 'stopping', 'stopped', 'deleting', 'deleted')",
- name="ck_hosted_launch_state",
- ),
- CheckConstraint(
- "process_state IS NULL OR process_state IN ('pending', 'starting', 'running', 'stopping', 'stopped', 'restarting', 'crashed', 'failed')",
- name="ck_hosted_launch_process_state",
- ),
- ForeignKeyConstraint(
- ["tenant_id", "machine_id"],
- ["hosted_machines.tenant_id", "hosted_machines.id"],
- name="fk_hosted_launches_machine",
- ),
- )
-
- id: Mapped[str] = mapped_column(Text, nullable=False)
- owner_id: Mapped[str] = mapped_column(Text, ForeignKey("users.id"), nullable=False)
- name: Mapped[str] = mapped_column(Text, nullable=False)
- spec: Mapped[dict] = mapped_column(JSONB, nullable=False)
- state: Mapped[str] = mapped_column(Text, nullable=False, server_default="queued")
- agent_id: Mapped[str | None] = mapped_column(Text, nullable=True)
- desired_state: Mapped[str] = mapped_column(
- Text, nullable=False, server_default="running"
- )
- revision: Mapped[int] = mapped_column(Integer, nullable=False, server_default="1")
- error: Mapped[str | None] = mapped_column(Text, nullable=True)
- error_code: Mapped[str | None] = mapped_column(Text, nullable=True)
- deletion_cleanup: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
- created_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), server_default=func.now(), nullable=False
- )
- updated_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), server_default=func.now(), nullable=False
- )
- active_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), server_default=func.now(), nullable=False
- )
- worker_capability_hash: Mapped[str | None] = mapped_column(Text, nullable=True)
- worker_capability_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
- worker_capability_revision: Mapped[int | None] = mapped_column(
- Integer, nullable=True
- )
- relay_seq: Mapped[int] = mapped_column(
- BigInteger, nullable=False, server_default="0"
- )
- machine_id: Mapped[str | None] = mapped_column(Text)
- repository: Mapped[str | None] = mapped_column(Text)
- process_state: Mapped[str | None] = mapped_column(Text)
- process_restarts: Mapped[int] = mapped_column(
- Integer, nullable=False, server_default="0"
- )
- process_oom_kills: Mapped[int] = mapped_column(
- Integer, nullable=False, server_default="0"
- )
- process_exit: Mapped[dict | None] = mapped_column(JSONB)
- process_reported_at: Mapped[datetime | None] = mapped_column(
- DateTime(timezone=True)
- )
-
-
-class GitHubIssuedToken(TenantScoped, Base):
- __tablename__ = "github_issued_tokens"
- __table_args__ = (
- PrimaryKeyConstraint("tenant_id", "id"),
- ForeignKeyConstraint(
- ["tenant_id", "launch_id"],
- ["hosted_launches.tenant_id", "hosted_launches.id"],
- ),
- Index("ix_github_issued_tokens_owner", "tenant_id", "owner_id"),
- )
- id: Mapped[str] = mapped_column(Text, nullable=False)
- owner_id: Mapped[str] = mapped_column(Text, ForeignKey("users.id"), nullable=False)
- launch_id: Mapped[str] = mapped_column(Text, nullable=False)
- launch_revision: Mapped[int] = mapped_column(Integer, nullable=False)
- encrypted_token: Mapped[str] = mapped_column(Text, nullable=False)
- expires_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), nullable=False
- )
- revoke_requested: Mapped[bool] = mapped_column(Boolean, nullable=False)
- attempts: Mapped[int] = mapped_column(Integer, nullable=False)
- claim_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
-
-
-class HostedOperation(TenantScoped, Base):
- __tablename__ = "hosted_operations"
- __table_args__ = (
- PrimaryKeyConstraint("tenant_id", "id"),
- ForeignKeyConstraint(
- ["tenant_id", "launch_id"],
- ["hosted_launches.tenant_id", "hosted_launches.id"],
- ondelete="CASCADE",
- ),
- CheckConstraint(
- "state IN ('queued', 'claimed', 'applied', 'failed', 'unknown')",
- name="ck_hosted_operation_state",
- ),
- )
- id: Mapped[str] = mapped_column(Text, nullable=False)
- launch_id: Mapped[str] = mapped_column(Text, nullable=False)
- launch_revision: Mapped[int] = mapped_column(Integer, nullable=False)
- session_id: Mapped[str] = mapped_column(Text, nullable=False)
- action: Mapped[str] = mapped_column(Text, nullable=False)
- state: Mapped[str] = mapped_column(Text, nullable=False, server_default="queued")
- error: Mapped[str | None] = mapped_column(Text)
- claimed_by: Mapped[str | None] = mapped_column(Text)
- claimed_boot_id: Mapped[str | None] = mapped_column(Text)
- created_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), server_default=func.now(), nullable=False
- )
- updated_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), server_default=func.now(), nullable=False
- )
-
-
-class HostedWakeMailbox(TenantScoped, Base):
- """An addressed event for a hosted agent, kept until its worker has admitted it."""
-
- __tablename__ = "hosted_wake_mailbox"
- __table_args__ = (
- PrimaryKeyConstraint("tenant_id", "agent_id", "room_id", "message_id"),
- ForeignKeyConstraint(
- ["tenant_id", "launch_id"],
- ["hosted_launches.tenant_id", "hosted_launches.id"],
- ondelete="CASCADE",
- ),
- CheckConstraint(
- "state IN ('pending', 'offered', 'accepted', 'admitted', 'held', 'cancelled', 'cancel_requested', 'refused', 'duplicate', 'expired', 'expired_uncertain')",
- name="ck_hosted_wake_mailbox_state",
- ),
- CheckConstraint(
- "cancel_reason IS NULL OR cancel_reason IN ('stopped', 'expired')",
- name="ck_hosted_wake_mailbox_cancel_reason",
- ),
- CheckConstraint(
- "origin IN ('live', 'cutover')", name="ck_hosted_wake_mailbox_origin"
- ),
- CheckConstraint(
- "notice_owed IS NULL OR notice_owed IN ('stopped', 'expired', 'expired_uncertain', 'started_before_stop', 'started_before_expiry')",
- name="ck_hosted_wake_mailbox_notice_owed",
- ),
- CheckConstraint(
- "notice_dropped IS NULL OR notice_dropped IN ('agent_deleted')",
- name="ck_hosted_wake_mailbox_notice_dropped",
- ),
- Index(
- "ix_hosted_wake_mailbox_agent_state",
- "tenant_id",
- "agent_id",
- "state",
- "addressed_at",
- ),
- Index(
- "ix_hosted_wake_mailbox_notice_owed",
- "tenant_id",
- "addressed_at",
- postgresql_where=text("notice_owed IS NOT NULL"),
- ),
- )
- agent_id: Mapped[str] = mapped_column(Text, nullable=False)
- room_id: Mapped[str] = mapped_column(Text, nullable=False)
- message_id: Mapped[str] = mapped_column(Text, nullable=False)
- launch_id: Mapped[str] = mapped_column(Text, nullable=False)
- thread_id: Mapped[str | None] = mapped_column(Text)
- event: Mapped[dict] = mapped_column(JSONB, nullable=False)
- state: Mapped[str] = mapped_column(Text, nullable=False, server_default="pending")
- cancel_reason: Mapped[str | None] = mapped_column(Text)
- #: The room notice this row's terminal move owes, until one is posted.
- notice_owed: Mapped[str | None] = mapped_column(Text)
- #: Why `notice_owed` will never be posted; the upkeep no longer retries it.
- notice_dropped: Mapped[str | None] = mapped_column(Text)
- ever_offered: Mapped[bool] = mapped_column(
- Boolean, nullable=False, server_default="false"
- )
- offered_to: Mapped[str | None] = mapped_column(Text)
- offered_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
- origin: Mapped[str] = mapped_column(Text, nullable=False, server_default="live")
- addressed_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), nullable=False
- )
- updated_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), nullable=False
- )
- expires_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), nullable=False
- )
-
-
-class HostedCutoverVolume(TenantScoped, Base):
- """A launch's retained worker volume, complete once its preflight manifest is recorded."""
-
- __tablename__ = "hosted_cutover_volumes"
- __table_args__ = (
- PrimaryKeyConstraint("tenant_id", "launch_id"),
- ForeignKeyConstraint(
- ["tenant_id", "launch_id"],
- ["hosted_launches.tenant_id", "hosted_launches.id"],
- ondelete="CASCADE",
- ),
- CheckConstraint(
- "preflight_state IN ('pending', 'blocked', 'complete')",
- name="ck_hosted_cutover_volumes_state",
- ),
- )
- launch_id: Mapped[str] = mapped_column(Text, nullable=False)
- preflight_state: Mapped[str] = mapped_column(
- Text, nullable=False, server_default="pending"
- )
- manifest_sha256: Mapped[str | None] = mapped_column(Text)
- completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
- blocked_reason: Mapped[str | None] = mapped_column(Text)
- imports_queued_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
-
-
-class HostedCutoverItem(TenantScoped, Base):
- """Pre-cutover work for a hosted agent: its evidence, and what was done with it."""
-
- __tablename__ = "hosted_cutover_items"
- __table_args__ = (
- PrimaryKeyConstraint("tenant_id", "id"),
- ForeignKeyConstraint(
- ["tenant_id", "launch_id"],
- ["hosted_launches.tenant_id", "hosted_launches.id"],
- ondelete="CASCADE",
- ),
- CheckConstraint(
- "kind IN ('room_message', 'console_command', 'session', 'request_open', 'reset_pending', 'operation')",
- name="ck_hosted_cutover_items_kind",
- ),
- CheckConstraint(
- "disposition IS NULL OR disposition IN ('ran', 'uncertain', 'unrecoverable', 'import', 'settled_by_host', 'owner_notice', 'interrupted', 'preserved')",
- name="ck_hosted_cutover_items_disposition",
- ),
- CheckConstraint(
- "kind <> 'room_message' OR (room_id IS NOT NULL AND message_id IS NOT NULL)",
- name="ck_hosted_cutover_items_room_message",
- ),
- CheckConstraint(
- "notice_dropped IS NULL OR notice_dropped IN ('agent_deleted')",
- name="ck_hosted_cutover_items_notice_dropped",
- ),
- Index(
- "uq_hosted_cutover_items_room_message",
- "tenant_id",
- "agent_id",
- "room_id",
- "message_id",
- unique=True,
- postgresql_where=text("kind = 'room_message'"),
- ),
- Index("ix_hosted_cutover_items_launch", "tenant_id", "launch_id"),
- )
- id: Mapped[str] = mapped_column(
- Text, nullable=False, server_default=text("gen_random_uuid()::text")
- )
- agent_id: Mapped[str] = mapped_column(Text, nullable=False)
- launch_id: Mapped[str] = mapped_column(Text, nullable=False)
- session_id: Mapped[str | None] = mapped_column(Text)
- kind: Mapped[str] = mapped_column(Text, nullable=False)
- room_id: Mapped[str | None] = mapped_column(Text)
- message_id: Mapped[str | None] = mapped_column(Text)
- thread_id: Mapped[str | None] = mapped_column(Text)
- evidence: Mapped[dict] = mapped_column(JSONB, nullable=False)
- disposition: Mapped[str | None] = mapped_column(Text)
- payload: Mapped[dict | None] = mapped_column(JSONB)
- notice_posted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
- #: Why the notice will never be posted; nothing retries it. Deferred because
- #: `hosted-cutover-upgrade record` loads items at `a3c9e5f71d28`, before the
- #: column exists.
- notice_dropped: Mapped[str | None] = mapped_column(Text, deferred=True)
- created_at: Mapped[datetime] = mapped_column(
- DateTime(timezone=True), server_default=func.now(), nullable=False
- )
-
-
# ── Invitations ────────────────────────────────────────────────────────────────
@@ -3222,6 +2901,11 @@ class AgentControllerEnrollmentCode(TenantScoped, Base):
["agent_controllers.tenant_id", "agent_controllers.id"],
name="fk_agent_controller_enrollment_codes_controller",
),
+ ForeignKeyConstraint(
+ ["tenant_id", "hosted_machine_id"],
+ ["hosted_machines.tenant_id", "hosted_machines.id"],
+ name="fk_agent_controller_enrollment_codes_hosted_machine",
+ ),
Index("ix_agent_controller_enrollment_codes_tenant_id", "tenant_id"),
)
@@ -3235,6 +2919,9 @@ class AgentControllerEnrollmentCode(TenantScoped, Base):
DateTime(timezone=True), nullable=True
)
controller_id: Mapped[str | None] = mapped_column(Text, nullable=True)
+ # Set for a code minted for a Switch cloud machine: what enrolls with it
+ # is that machine's controller.
+ hosted_machine_id: Mapped[str | None] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
@@ -3348,6 +3035,47 @@ class AgentControllerOperation(TenantScoped, Base):
)
+class SealedProviderLogin(TenantScoped, Base):
+ """A provider login for one controller, sealed to that controller's own
+ public key (`AgentController.public_key`) by the owner's client before it
+ reached Switch.
+
+ Switch keeps and relays the ciphertext and can never open it: only the
+ controller holding the private key can. One per controller and provider;
+ `revision` grows with each new one, so the controller can tell a changed
+ login from the one it already applied. Removed with its controller, and
+ when the controller is revoked.
+ """
+
+ __tablename__ = "sealed_provider_logins"
+ __table_args__ = (
+ ForeignKeyConstraint(
+ ["tenant_id", "controller_id"],
+ ["agent_controllers.tenant_id", "agent_controllers.id"],
+ name="fk_sealed_provider_logins_controller",
+ ondelete="CASCADE",
+ ),
+ Index("ix_sealed_provider_logins_tenant_id", "tenant_id"),
+ )
+
+ controller_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ provider: Mapped[str] = mapped_column(Text, primary_key=True)
+ revision: Mapped[int] = mapped_column(Integer, nullable=False)
+ # `{alg, key_id, ephemeral_key, nonce, ciphertext}`; see
+ # `management/sealed_logins.py`.
+ sealed: Mapped[dict] = mapped_column(JSONB, nullable=False)
+ sealed_by: Mapped[str] = mapped_column(Text, ForeignKey("users.id"), nullable=False)
+ created_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True), server_default=func.now(), nullable=False
+ )
+ updated_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True),
+ server_default=func.now(),
+ onupdate=func.now(),
+ nullable=False,
+ )
+
+
# Same reasoning as the notify trigger above: `create_all` has to build the
# row-level-security policies too, or the isolation test would pass against a
# schema that has none. See `db/rls_ddl.py` for the DDL and why it takes this
diff --git a/core/switch_core/db/stores/agent_controller_store.py b/core/switch_core/db/stores/agent_controller_store.py
index 0eadbb757..0868700be 100644
--- a/core/switch_core/db/stores/agent_controller_store.py
+++ b/core/switch_core/db/stores/agent_controller_store.py
@@ -3,10 +3,15 @@ from __future__ import annotations
from datetime import datetime
from typing import Any
-from sqlalchemy import or_, select, update
+from sqlalchemy import delete, func, or_, select, update
+from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
-from switch_core.db.models import AgentController, AgentControllerEnrollmentCode
+from switch_core.db.models import (
+ AgentController,
+ AgentControllerEnrollmentCode,
+ SealedProviderLogin,
+)
class AgentControllerStore:
@@ -100,6 +105,105 @@ class AgentControllerStore:
await session.refresh(controller)
return controller
+ async def set_public_key(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ controller_id: str,
+ public_key: dict[str, Any],
+ ) -> AgentController:
+ """Record the key provider logins are sealed to, and return the row."""
+ controller = await self.get(session, tenant_id, controller_id)
+ if controller is None:
+ raise LookupError(f"No such controller: {controller_id}")
+ controller.public_key = public_key
+ await session.flush()
+ await session.refresh(controller)
+ return controller
+
+ async def sealed_login(
+ self, session: AsyncSession, tenant_id: str, controller_id: str, provider: str
+ ) -> SealedProviderLogin | None:
+ result = await session.execute(
+ select(SealedProviderLogin).where(
+ SealedProviderLogin.tenant_id == tenant_id,
+ SealedProviderLogin.controller_id == controller_id,
+ SealedProviderLogin.provider == provider,
+ )
+ )
+ return result.scalar_one_or_none()
+
+ async def sealed_logins(
+ self, session: AsyncSession, tenant_id: str, controller_id: str
+ ) -> list[SealedProviderLogin]:
+ rows = await session.scalars(
+ select(SealedProviderLogin)
+ .where(
+ SealedProviderLogin.tenant_id == tenant_id,
+ SealedProviderLogin.controller_id == controller_id,
+ )
+ .order_by(SealedProviderLogin.provider)
+ )
+ return list(rows)
+
+ async def put_sealed_login(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ controller_id: str,
+ provider: str,
+ sealed: dict[str, Any],
+ sealed_by: str,
+ ) -> SealedProviderLogin:
+ """Store the controller's login for `provider`, replacing any before it
+ under the next revision, and return the row."""
+ statement = (
+ insert(SealedProviderLogin)
+ .values(
+ tenant_id=tenant_id,
+ controller_id=controller_id,
+ provider=provider,
+ revision=1,
+ sealed=sealed,
+ sealed_by=sealed_by,
+ )
+ .on_conflict_do_update(
+ index_elements=["controller_id", "provider"],
+ set_={
+ "revision": SealedProviderLogin.revision + 1,
+ "sealed": sealed,
+ "sealed_by": sealed_by,
+ "updated_at": func.now(),
+ },
+ )
+ )
+ await session.execute(statement)
+ row = await self.sealed_login(session, tenant_id, controller_id, provider)
+ if row is None:
+ raise LookupError(
+ f"The {provider} login for {controller_id} was not stored"
+ )
+ await session.refresh(row)
+ return row
+
+ async def delete_sealed_logins(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ controller_id: str,
+ provider: str | None,
+ ) -> int:
+ """Delete the controller's login for `provider`, or every one of its
+ logins when `provider` is None. Returns how many went."""
+ statement = delete(SealedProviderLogin).where(
+ SealedProviderLogin.tenant_id == tenant_id,
+ SealedProviderLogin.controller_id == controller_id,
+ )
+ if provider is not None:
+ statement = statement.where(SealedProviderLogin.provider == provider)
+ result = await session.execute(statement)
+ return int(result.rowcount or 0) # type: ignore[attr-defined]
+
async def bump_assignment_revision(
self, session: AsyncSession, tenant_id: str, controller_id: str
) -> int:
@@ -273,9 +377,13 @@ class AgentControllerStore:
owner_id: str,
api_key_id: str,
expires_at: datetime,
+ hosted_machine_id: str | None,
) -> AgentControllerEnrollmentCode:
code = AgentControllerEnrollmentCode(
- owner_id=owner_id, api_key_id=api_key_id, expires_at=expires_at
+ owner_id=owner_id,
+ api_key_id=api_key_id,
+ expires_at=expires_at,
+ hosted_machine_id=hosted_machine_id,
)
session.add(code)
await session.flush()
diff --git a/core/switch_core/db/stores/hosted_launch_store.py b/core/switch_core/db/stores/hosted_launch_store.py
deleted file mode 100644
index 6ea8aefdb..000000000
--- a/core/switch_core/db/stores/hosted_launch_store.py
+++ /dev/null
@@ -1,383 +0,0 @@
-from __future__ import annotations
-
-import secrets
-from dataclasses import dataclass
-from datetime import UTC, datetime, timedelta
-from typing import TYPE_CHECKING, cast
-
-from sqlalchemy import and_, case, exists, literal, or_, select, update
-from sqlalchemy.dialects.postgresql import JSONB
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.db.models import (
- Agent,
- ApprovalRequest,
- HostedLaunch,
- HostedMachine,
- HostedOperation,
- ProviderConnection,
- require_tenant_id,
-)
-from switch_core.db.stores.agent_store import AgentStore
-from switch_core.db.stores.hosted_machine_store import (
- HostedMachineStore,
- capability_hash,
- idle_sleeping,
- lock_launch,
- lock_launches,
- machine_starting,
-)
-from switch_core.db.stores.hosted_mailbox_store import HostedMailboxStore
-from switch_core.keys import Keyring
-
-if TYPE_CHECKING:
- from switch_core.bridges.agent.protocol.agent_connections import (
- AgentConnectionRegistry,
- )
- from switch_core.bridges.agent.protocol.hosted_workers import IdleReport
-
-
-class HostedLaunchConflict(Exception):
- pass
-
-
-@dataclass(frozen=True)
-class IdleEvidence:
- """Whether a launch is busy, why, and the report an idle verdict rests on."""
-
- busy: bool
- reasons: list[str]
- report: IdleReport | None
-
-
-UNCONFIRMED_CLAIM_EXPIRY = timedelta(minutes=5)
-
-
-class HostedLaunchStore:
- async def reserve(
- self,
- session: AsyncSession,
- *,
- request_id: str,
- owner_id: str,
- name: str,
- spec: dict,
- capacity: int,
- owner_capacity: int,
- slots: list[str],
- repository: str | None,
- now: datetime,
- ) -> HostedLaunch:
- """Reserve a launch on the owner's machine, claiming one if needed.
-
- Raises `HostedLaunchConflict` for the name, request and per-owner
- checks, and `HostedMachineConflict` when no machine can take it.
- """
- tenant_id = require_tenant_id()
- await lock_launches(session)
- await AgentStore().lock_name(session, name)
- existing = await session.get(HostedLaunch, (tenant_id, request_id))
- if existing:
- if (
- existing.owner_id != owner_id
- or existing.name != name
- or existing.spec != spec
- ):
- raise HostedLaunchConflict(
- "This launch request was already used for different agent details."
- )
- return existing
- if await session.scalar(
- select(Agent.id).where(Agent.tenant_id == tenant_id, Agent.name == name)
- ):
- raise HostedLaunchConflict("An agent already uses this name.")
- launches = list(
- (
- await session.scalars(
- select(HostedLaunch).where(HostedLaunch.tenant_id == tenant_id)
- )
- ).all()
- )
- if any(launch.name == name for launch in launches):
- raise HostedLaunchConflict(
- "A cloud launch already reserves this agent name."
- )
- if (
- sum(
- launch.owner_id == owner_id and launch.state != "deleted"
- for launch in launches
- )
- >= owner_capacity
- ):
- raise HostedLaunchConflict(
- "Your cloud agent limit has been reached. Remove an agent before creating another."
- )
- machines = HostedMachineStore()
- machine = await machines.claim(
- session, owner_id=owner_id, slots=slots, capacity=capacity, now=now
- )
- launch = HostedLaunch(
- id=request_id,
- owner_id=owner_id,
- name=name,
- spec=spec,
- state="queued",
- agent_id=None,
- machine_id=machine.id,
- repository=repository,
- active_at=now,
- created_at=now,
- updated_at=now,
- )
- session.add(launch)
- machines.bump_agents(machine)
- await session.flush()
- return launch
-
- async def owned(
- self, session: AsyncSession, request_id: str, owner_id: str
- ) -> HostedLaunch | None:
- return cast(
- HostedLaunch | None,
- await session.scalar(
- select(HostedLaunch).where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.id == request_id,
- HostedLaunch.owner_id == owner_id,
- )
- ),
- )
-
- async def merge_spec(
- self, session: AsyncSession, launch_id: str, changes: dict
- ) -> None:
- """Overwrite these keys of a launch's spec in one statement, keeping its other keys."""
- await session.execute(
- update(HostedLaunch)
- .where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.id == launch_id,
- )
- .values(
- spec=HostedLaunch.spec.op("||", return_type=JSONB)(
- literal(changes, JSONB)
- )
- )
- .execution_options(synchronize_session=False)
- )
-
- async def fail_stale_operations(
- self, session: AsyncSession, launch_id: str, revision: int
- ) -> None:
- await session.execute(
- update(HostedOperation)
- .where(
- HostedOperation.tenant_id == require_tenant_id(),
- HostedOperation.launch_id == launch_id,
- HostedOperation.launch_revision != revision,
- HostedOperation.state.in_(["queued", "claimed"]),
- )
- .values(
- state=case(
- (HostedOperation.state == "claimed", "unknown"), else_="failed"
- ),
- error="The worker changed before this operation was confirmed. Inspect the session if the outcome is unknown.",
- updated_at=datetime.now(UTC),
- )
- )
- await session.execute(
- update(HostedOperation)
- .where(
- HostedOperation.tenant_id == require_tenant_id(),
- HostedOperation.launch_id == launch_id,
- HostedOperation.state == "claimed",
- HostedOperation.updated_at
- < datetime.now(UTC) - UNCONFIRMED_CLAIM_EXPIRY,
- )
- .values(
- state="unknown",
- error="The worker did not confirm the outcome. Inspect the session before issuing another operation.",
- updated_at=datetime.now(UTC),
- )
- )
-
- async def locked(
- self, session: AsyncSession, launch_id: str
- ) -> HostedLaunch | None:
- await lock_launch(session, launch_id)
- return await session.get(
- HostedLaunch, (require_tenant_id(), launch_id), populate_existing=True
- )
-
- def issue_worker_capability(self, launch: HostedLaunch, keyring: Keyring) -> str:
- """The worker capability for the launch's current revision.
-
- The same bytes for every call at one revision, so a lost `prepare`
- response costs nothing; a new revision mints new ones and overwrites
- the old, so at most one is ever valid. The caller commits.
- """
- if (
- launch.worker_capability_revision == launch.revision
- and launch.worker_capability_encrypted is not None
- ):
- return keyring.decrypt(launch.worker_capability_encrypted)
- capability = secrets.token_urlsafe(32)
- launch.worker_capability_encrypted = keyring.encrypt(capability)
- launch.worker_capability_hash = capability_hash(capability)
- launch.worker_capability_revision = launch.revision
- return capability
-
- @staticmethod
- def capability_matches(launch: HostedLaunch, capability: str) -> bool:
- """Whether `capability` is the one minted for the launch's current revision.
-
- Launch state and owner membership are the caller's to check alongside.
- """
- return (
- launch.worker_capability_hash is not None
- and launch.worker_capability_revision == launch.revision
- and secrets.compare_digest(
- launch.worker_capability_hash, capability_hash(capability)
- )
- )
-
- async def queued_operation_ids(
- self, session: AsyncSession, launch: HostedLaunch, boot_id: str
- ) -> list[str]:
- """The operations the worker on `boot_id` should claim.
-
- Includes those that boot already claimed without a result: the claim
- reply may have been lost, and the worker runs an operation only after
- journaling its claim, so claiming again never runs it twice.
- """
- return list(
- await session.scalars(
- select(HostedOperation.id)
- .where(
- HostedOperation.tenant_id == require_tenant_id(),
- HostedOperation.launch_id == launch.id,
- HostedOperation.launch_revision == launch.revision,
- or_(
- HostedOperation.state == "queued",
- and_(
- HostedOperation.state == "claimed",
- HostedOperation.claimed_boot_id == boot_id,
- ),
- ),
- )
- .order_by(HostedOperation.created_at)
- )
- )
-
- async def credential_revision(
- self, session: AsyncSession, launch: HostedLaunch
- ) -> str | None:
- """The owner's provider credential revision, as the credential route reports it."""
- connection = await session.get(
- ProviderConnection,
- (
- require_tenant_id(),
- launch.owner_id,
- launch.spec.get("provider", "claude"),
- ),
- populate_existing=True,
- )
- return None if connection is None else str(connection.verified_at)
-
- async def idle_evidence(
- self,
- session: AsyncSession,
- launch: HostedLaunch,
- registry: AgentConnectionRegistry,
- ) -> IdleEvidence:
- """What Core knows about whether the launch's worker is doing anything.
-
- Read under the launch lock. Busy wins on any single signal, and a
- missing or stale report is a signal: the controller stops a VM only on
- evidence.
- """
- reasons: list[str] = []
- report = (
- registry.fresh_idle_report(launch.agent_id, launch.id, launch.revision)
- if launch.agent_id
- else None
- )
- if report is None:
- reasons.append("no_fresh_report")
- else:
- if report.busy:
- reasons.append("report_busy")
- if report.relays_through < launch.relay_seq:
- reasons.append("relays_unacknowledged")
- if launch.agent_id and registry.relays.mutating_pending(
- launch.agent_id, launch.id
- ):
- reasons.append("relay_pending")
- if await session.scalar(
- select(
- exists().where(
- HostedOperation.tenant_id == require_tenant_id(),
- HostedOperation.launch_id == launch.id,
- HostedOperation.launch_revision == launch.revision,
- HostedOperation.state.in_(["queued", "claimed"]),
- )
- )
- ):
- reasons.append("operation_pending")
- if await HostedMailboxStore().busy(session, launch.id):
- reasons.append("mailbox_pending")
- if launch.agent_id and await session.scalar(
- select(
- exists().where(
- ApprovalRequest.tenant_id == require_tenant_id(),
- ApprovalRequest.agent_id == launch.agent_id,
- ApprovalRequest.state == "open",
- )
- )
- ):
- reasons.append("approval_open")
- return IdleEvidence(busy=bool(reasons), reasons=reasons, report=report)
-
- async def note_addressed(
- self, session: AsyncSession, launch_id: str
- ) -> tuple[HostedLaunch | None, HostedMachine | None]:
- """Record that the launch's agent was addressed, waking its machine if idle-stopped.
-
- Takes the machine and launch locks. The caller commits. Raises
- `ProviderDisconnected` rather than wake a machine for a launch whose
- owner has no provider connection: the agent would only fail to start.
- """
- machines = HostedMachineStore()
- launch, machine = await machines.locked_launch(session, launch_id)
- if launch is None or machine is None:
- return launch, machine
- now = datetime.now(UTC)
- if (
- idle_sleeping(machine)
- and machine.state != "error"
- and launch.desired_state == "running"
- and launch.state != "error"
- ):
- if await self.credential_revision(session, launch) is None:
- raise ProviderDisconnected(
- f"launch {launch.id} is asleep and its owner's provider connection is gone"
- )
- machines.start(machine, now)
- launch.active_at = now
- elif launch.desired_state == "running" and machine.desired_state == "running":
- launch.active_at = now
- return launch, machine
-
-
-class ProviderDisconnected(Exception):
- """A launch on a sleeping machine was addressed, but its owner's provider connection is gone."""
-
-
-def is_waking(launch: HostedLaunch, machine: HostedMachine | None) -> bool:
- return (
- machine is not None
- and machine.state != "error"
- and launch.desired_state == "running"
- and launch.state != "error"
- and (machine_starting(machine) or launch.state in {"queued", "provisioning"})
- )
diff --git a/core/switch_core/db/stores/hosted_machine_store.py b/core/switch_core/db/stores/hosted_machine_store.py
index 70c3972ab..d2a780977 100644
--- a/core/switch_core/db/stores/hosted_machine_store.py
+++ b/core/switch_core/db/stores/hosted_machine_store.py
@@ -1,7 +1,5 @@
from __future__ import annotations
-import hashlib
-import secrets
from datetime import datetime, timedelta
from typing import Literal, cast
from uuid import uuid4
@@ -9,8 +7,11 @@ from uuid import uuid4
from sqlalchemy import exists, func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
-from switch_core.db.models import HostedLaunch, HostedMachine, require_tenant_id
-from switch_core.keys import Keyring
+from switch_core.db.models import (
+ AgentDefinition,
+ HostedMachine,
+ require_tenant_id,
+)
MACHINE_CONNECT_TIMEOUT = timedelta(minutes=10)
MACHINE_NEEDS_ATTENTION = (
@@ -19,6 +20,7 @@ MACHINE_NEEDS_ATTENTION = (
MACHINE_NEEDS_ADMIN = (
"Your cloud machine needs attention. Contact your server administrator."
)
+MACHINES_FULL = "Every Switch cloud machine on this server is in use. Try again later."
MACHINE_BEING_REMOVED = (
"Your previous cloud machine is being removed. Try again in a minute."
)
@@ -28,10 +30,6 @@ class HostedMachineConflict(Exception):
"""A machine cannot take the requested change; the message is the 409 detail."""
-def capability_hash(capability: str) -> str:
- return hashlib.sha256(capability.encode()).hexdigest()
-
-
async def _advisory_lock(session: AsyncSession, key: str) -> None:
await session.execute(
text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
@@ -39,21 +37,16 @@ async def _advisory_lock(session: AsyncSession, key: str) -> None:
)
-async def lock_launches(session: AsyncSession) -> None:
- """The tenant-wide lock taken before creating a launch or claiming a machine."""
+async def lock_claims(session: AsyncSession) -> None:
+ """The tenant-wide lock taken before claiming a machine."""
await _advisory_lock(session, f"hosted-launches:{require_tenant_id()}")
async def lock_machine(session: AsyncSession, machine_id: str) -> None:
- """The per-machine lock, taken after the tenant lock and before any launch lock."""
+ """The per-machine lock, taken after the tenant lock."""
await _advisory_lock(session, f"hosted-machine:{require_tenant_id()}:{machine_id}")
-async def lock_launch(session: AsyncSession, launch_id: str) -> None:
- """The per-launch advisory lock every lifecycle, relay and attach step takes."""
- await _advisory_lock(session, f"hosted-launch:{require_tenant_id()}:{launch_id}")
-
-
def idle_sleeping(machine: HostedMachine) -> bool:
return machine.desired_state == "stopped" and machine.stop_reason == "idle"
@@ -151,37 +144,18 @@ class HostedMachineStore:
),
)
- async def locked_launch(
- self, session: AsyncSession, launch_id: str
- ) -> tuple[HostedLaunch | None, HostedMachine | None]:
- """A launch and its machine, locked machine first, then launch."""
- tenant_id = require_tenant_id()
- launch = await session.get(HostedLaunch, (tenant_id, launch_id))
- if launch is None:
- return None, None
- machine_id = launch.machine_id
- if machine_id is not None:
- await lock_machine(session, machine_id)
- await lock_launch(session, launch_id)
- launch = await session.get(
- HostedLaunch, (tenant_id, launch_id), populate_existing=True
- )
- machine = None if machine_id is None else await self.get(session, machine_id)
- return launch, machine
-
async def claim(
self,
session: AsyncSession,
*,
owner_id: str,
- slots: list[str],
capacity: int,
now: datetime,
) -> HostedMachine:
- """The owner's machine, reused or newly placed on a free slot.
+ """The owner's machine, reused or newly claimed within `capacity`.
- The caller holds `lock_launches`, so no other claim in the tenant can
- take the same slot or count towards capacity meanwhile.
+ The caller holds `lock_claims`, so no other claim in the tenant can
+ count towards capacity meanwhile.
"""
tenant_id = require_tenant_id()
machine = await self.live_for_owner(session, owner_id)
@@ -204,30 +178,21 @@ class HostedMachineStore:
await session.flush()
return machine
- used = set(
- await session.scalars(
- select(HostedMachine.slot_id).where(
- HostedMachine.tenant_id == tenant_id,
- HostedMachine.state != "deleted",
- )
- )
- )
- slot_id = next((slot for slot in slots if slot not in used), None)
- if len(used) >= capacity or slot_id is None:
- raise HostedMachineConflict("no machine slot available")
- generation = await session.scalar(
- select(func.max(HostedMachine.generation)).where(
+ live = await session.scalar(
+ select(func.count())
+ .select_from(HostedMachine)
+ .where(
HostedMachine.tenant_id == tenant_id,
- HostedMachine.slot_id == slot_id,
+ HostedMachine.state != "deleted",
)
)
+ if (live or 0) >= capacity:
+ raise HostedMachineConflict(MACHINES_FULL)
machine_id = str(uuid4())
await lock_machine(session, machine_id)
machine = HostedMachine(
id=machine_id,
owner_id=owner_id,
- slot_id=slot_id,
- generation=(generation or 0) + 1,
state="queued",
desired_state="running",
active_at=now,
@@ -262,24 +227,6 @@ class HostedMachineStore:
machine.error_code = None
bump_revision(machine, now)
- def bump_agents(self, machine: HostedMachine) -> None:
- machine.agents_version += 1
-
- async def launches(
- self, session: AsyncSession, machine_id: str
- ) -> list[HostedLaunch]:
- return list(
- await session.scalars(
- select(HostedLaunch)
- .where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.machine_id == machine_id,
- HostedLaunch.state != "deleted",
- )
- .order_by(HostedLaunch.created_at)
- )
- )
-
async def retain_if_empty(
self,
session: AsyncSession,
@@ -288,21 +235,13 @@ class HostedMachineStore:
retention_days: int,
now: datetime,
) -> bool:
- """Retain the machine's disk once no agent is left on it.
+ """Retain the machine's disk once no managed agent is placed on its
+ controller.
Returns whether it did. The caller holds the machine lock and commits.
"""
await session.flush()
- if await session.scalar(
- select(
- exists().where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.machine_id == machine.id,
- HostedLaunch.state != "deleted",
- HostedLaunch.desired_state != "deleted",
- )
- )
- ):
+ if await self.has_agents(session, machine):
return False
machine.desired_state = "retained"
machine.stop_reason = None
@@ -318,57 +257,49 @@ class HostedMachineStore:
retention_days: int,
now: datetime,
) -> bool:
- """Retain an agentless machine, expiring at once if it never had an agent.
-
- Launch rows are never hard-deleted, so a machine no row references has
- no disk worth keeping. The caller holds the machine lock and commits.
+ """Retain an agentless machine, expiring at once if its controller
+ never enrolled: then its disk holds nothing worth keeping. The caller
+ holds the machine lock and commits.
"""
if not await self.retain_if_empty(
session, machine, retention_days=retention_days, now=now
):
return False
- if not await self.ever_hosted(session, machine.id):
+ if not await self.ever_hosted(session, machine):
machine.retain_until = now
return True
- async def ever_hosted(self, session: AsyncSession, machine_id: str) -> bool:
- """Whether any launch row ever referenced the machine, deleted or not."""
- return bool(
+ async def has_agents(self, session: AsyncSession, machine: HostedMachine) -> bool:
+ """Whether a managed agent is still placed on the machine's controller."""
+ return machine.controller_id is not None and bool(
await session.scalar(
select(
exists().where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.machine_id == machine_id,
+ AgentDefinition.tenant_id == require_tenant_id(),
+ AgentDefinition.controller_id == machine.controller_id,
)
)
)
)
- def issue_capability(self, machine: HostedMachine, keyring: Keyring) -> str:
- """The machine capability for the machine's current revision.
-
- The same bytes for every call at one revision; a new revision mints new
- ones and overwrites the old, so at most one is ever valid. The caller
- commits.
- """
- if (
- machine.machine_capability_revision == machine.revision
- and machine.machine_capability_encrypted is not None
- ):
- return keyring.decrypt(machine.machine_capability_encrypted)
- capability = secrets.token_urlsafe(32)
- machine.machine_capability_encrypted = keyring.encrypt(capability)
- machine.machine_capability_hash = capability_hash(capability)
- machine.machine_capability_revision = machine.revision
- return capability
-
- @staticmethod
- def capability_matches(machine: HostedMachine, capability: str) -> bool:
- """Whether `capability` is the one last issued for the machine.
-
- Not tied to the current revision: issuing rotates it, so the stored hash
- is the only valid one. Machine state is the caller's to check alongside.
- """
- return machine.machine_capability_hash is not None and secrets.compare_digest(
- machine.machine_capability_hash, capability_hash(capability)
+ async def managed_agent_ids(
+ self, session: AsyncSession, machine: HostedMachine
+ ) -> list[str]:
+ """The managed agents placed on the machine's controller."""
+ if machine.controller_id is None:
+ return []
+ return list(
+ await session.scalars(
+ select(AgentDefinition.agent_id)
+ .where(
+ AgentDefinition.tenant_id == require_tenant_id(),
+ AgentDefinition.controller_id == machine.controller_id,
+ )
+ .order_by(AgentDefinition.agent_id)
+ )
)
+
+ async def ever_hosted(self, session: AsyncSession, machine: HostedMachine) -> bool:
+ """Whether the machine's disk may hold anything worth keeping: its
+ controller ever enrolled."""
+ return machine.controller_id is not None
diff --git a/core/switch_core/db/stores/hosted_mailbox_store.py b/core/switch_core/db/stores/hosted_mailbox_store.py
deleted file mode 100644
index f1b8cf423..000000000
--- a/core/switch_core/db/stores/hosted_mailbox_store.py
+++ /dev/null
@@ -1,675 +0,0 @@
-"""The wake mailbox: every addressed event for a hosted agent until its worker admits it.
-
-Every transition is a conditional update that only moves a row forward, so a
-retried ack, a reclaim racing an ack or a Stop racing an offer can never
-regress a row. Rows are keyed by `(agent, room, roomInputId)`, the same key
-the watcher dedupes by.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import json
-from collections.abc import Sequence
-from dataclasses import dataclass
-from datetime import UTC, datetime, timedelta
-from typing import Any, Literal
-
-from sqlalchemy import and_, delete, func, or_, select, tuple_, update
-from sqlalchemy.dialects.postgresql import insert
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.bridges.agent.protocol.types import AgentEvent
-from switch_core.db.models import HostedWakeMailbox, require_tenant_id
-
-#: Rows `pending` or `offered` one agent may hold before an insert is refused.
-MAILBOX_LIMIT = 500
-OFFER_LEASE = timedelta(seconds=60)
-MAILBOX_EXPIRY = timedelta(hours=24)
-MAILBOX_RETENTION = timedelta(days=7)
-WAKE_ENTRIES_PER_FRAME = 50
-ACKS_PER_CALL = 200
-#: Owed room notices the upkeep retries in one pass over a tenant.
-NOTICE_RETRIES_PER_PASS = 100
-
-#: Rows the worker has still to settle; these keep the VM awake.
-BUSY_STATES = ("pending", "offered", "accepted")
-TERMINAL_STATES = (
- "admitted",
- "cancelled",
- "refused",
- "duplicate",
- "expired",
- "expired_uncertain",
-)
-
-MailboxOutcome = Literal[
- "journaled", "admitted", "duplicate", "refused", "held", "cancelled"
-]
-
-#: Outcome -> (states it moves a row from, state it moves the row to).
-TRANSITIONS: dict[str, tuple[tuple[str, ...], str]] = {
- "journaled": (("pending", "offered"), "accepted"),
- "duplicate": (("pending", "offered"), "accepted"),
- "admitted": (
- ("pending", "offered", "accepted", "held", "cancel_requested"),
- "admitted",
- ),
- "refused": (("pending", "offered", "accepted"), "refused"),
- "held": (("pending", "offered", "accepted"), "held"),
- "cancelled": (("cancel_requested",), "cancelled"),
-}
-
-
-#: The notice for a tombstone the watcher answers `admitted`, by cancel reason.
-STARTED_BEFORE = {"stopped": "started_before_stop", "expired": "started_before_expiry"}
-
-
-class MailboxFull(Exception):
- """The agent already has `MAILBOX_LIMIT` rows waiting for its worker."""
-
-
-def room_input_id(event: AgentEvent) -> str | None:
- """The watcher's `roomInputId` (`host/room-inbox.ts`): the key it dedupes a room input by.
-
- None for an event the watcher does not act on, which the mailbox does not
- hold either.
- """
- payload = event.payload.model_dump(mode="json")
- if event.type == "message":
- return payload["message_id"] if payload.get("addressed") is True else None
- if event.type == "room_join" and payload.get("listening") is not True:
- return None
- if event.type != "room_join" and not event.type.startswith("task_"):
- return None
- canonical = json.dumps(
- payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False
- )
- return f"{event.type}:{hashlib.sha256(canonical.encode()).hexdigest()}"
-
-
-@dataclass(frozen=True)
-class MailboxEntry:
- """One addressed event as the mailbox holds it and a `wake` frame carries it."""
-
- room_id: str
- message_id: str
- thread_id: str | None
- event: dict[str, Any]
- origin: Literal["live", "cutover"]
-
- @classmethod
- def of(cls, event: AgentEvent) -> MailboxEntry | None:
- message_id = room_input_id(event)
- if message_id is None:
- return None
- payload = event.payload.model_dump(mode="json")
- thread_id = payload.get("thread_id")
- return cls(
- room_id=event.room_id,
- message_id=message_id,
- thread_id=thread_id if isinstance(thread_id, str) else None,
- event={"type": event.type, "payload": payload, "missed": None},
- origin="live",
- )
-
- @classmethod
- def of_row(cls, row: HostedWakeMailbox) -> MailboxEntry:
- return cls(
- room_id=row.room_id,
- message_id=row.message_id,
- thread_id=row.thread_id,
- event=row.event,
- origin="cutover" if row.origin == "cutover" else "live",
- )
-
- def wire(self) -> dict[str, Any]:
- return {
- "room_id": self.room_id,
- "message_id": self.message_id,
- "thread_id": self.thread_id,
- "event": self.event,
- "origin": self.origin,
- }
-
-
-@dataclass(frozen=True)
-class MailboxNotice:
- """A notice Core owes a room for a mailbox row, posted after the commit."""
-
- agent_id: str
- room_id: str
- message_id: str
- thread_id: str | None
- reason: str
-
-
-@dataclass(frozen=True)
-class StopSplit:
- """What an explicit Stop did to the mailbox."""
-
- cancelled: list[MailboxNotice]
- cancel_requested: list[tuple[str, str]]
-
-
-@dataclass(frozen=True)
-class AgentBacklog:
- agent_id: str
- waiting: int
- oldest: datetime | None
- refused: int
-
-
-def by_room(
- notices: Sequence[MailboxNotice],
-) -> list[tuple[MailboxNotice, list[str]]]:
- """Per (agent, room, reason): the latest row's notice, and every message it answers for."""
- groups: dict[tuple[str, str, str], tuple[MailboxNotice, list[str]]] = {}
- for notice in notices:
- key = (notice.agent_id, notice.room_id, notice.reason)
- messages = groups[key][1] if key in groups else []
- messages.append(notice.message_id)
- groups[key] = (notice, messages)
- return list(groups.values())
-
-
-def one_per_room(notices: Sequence[MailboxNotice]) -> list[MailboxNotice]:
- """The latest row's notice per (agent, room, reason): one notice per room, in its thread."""
- return [notice for notice, _ in by_room(notices)]
-
-
-def _notice(row: HostedWakeMailbox, reason: str) -> MailboxNotice:
- return MailboxNotice(
- agent_id=row.agent_id,
- room_id=row.room_id,
- message_id=row.message_id,
- thread_id=row.thread_id,
- reason=reason,
- )
-
-
-def _notices(owed: Sequence[tuple[HostedWakeMailbox, str]]) -> list[MailboxNotice]:
- """Oldest first, so the latest row of a room is the same one on every retry."""
- return [
- _notice(row, reason)
- for row, reason in sorted(
- owed, key=lambda item: (item[0].addressed_at, item[0].message_id)
- )
- ]
-
-
-class HostedMailboxStore:
- async def write(
- self,
- session: AsyncSession,
- *,
- agent_id: str,
- launch_id: str,
- entry: MailboxEntry,
- offered_to: str | None,
- ) -> bool:
- """Insert the row, already offered when a worker is there to take it.
-
- False when the row exists (a redelivered event). Raises `MailboxFull`
- at the limit. Serialised per launch by the launch lock the caller holds.
- """
- tenant_id = require_tenant_id()
- exists = await session.scalar(
- select(HostedWakeMailbox.state).where(
- HostedWakeMailbox.tenant_id == tenant_id,
- HostedWakeMailbox.agent_id == agent_id,
- HostedWakeMailbox.room_id == entry.room_id,
- HostedWakeMailbox.message_id == entry.message_id,
- )
- )
- if exists is not None:
- return False
- waiting = await session.scalar(
- select(func.count()).where(
- HostedWakeMailbox.tenant_id == tenant_id,
- HostedWakeMailbox.agent_id == agent_id,
- HostedWakeMailbox.state.in_(("pending", "offered")),
- )
- )
- if (waiting or 0) >= MAILBOX_LIMIT:
- raise MailboxFull(
- f"agent {agent_id} already has {waiting} mailbox rows waiting"
- )
- now = datetime.now(UTC)
- offered = offered_to is not None
- result = await session.execute(
- insert(HostedWakeMailbox)
- .values(
- tenant_id=tenant_id,
- agent_id=agent_id,
- room_id=entry.room_id,
- message_id=entry.message_id,
- launch_id=launch_id,
- thread_id=entry.thread_id,
- event=entry.event,
- state="offered" if offered else "pending",
- ever_offered=offered,
- offered_to=offered_to,
- offered_until=now + OFFER_LEASE if offered else None,
- origin=entry.origin,
- addressed_at=now,
- updated_at=now,
- expires_at=now + MAILBOX_EXPIRY,
- )
- .on_conflict_do_nothing()
- .returning(HostedWakeMailbox.message_id)
- )
- return result.scalar() is not None
-
- async def pending(
- self, session: AsyncSession, agent_id: str
- ) -> list[HostedWakeMailbox]:
- """The agent's `pending` rows, oldest first."""
- return list(
- await session.scalars(
- select(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.agent_id == agent_id,
- HostedWakeMailbox.state == "pending",
- )
- .order_by(HostedWakeMailbox.addressed_at, HostedWakeMailbox.room_id)
- )
- )
-
- async def mark_offered(
- self,
- session: AsyncSession,
- agent_id: str,
- keys: Sequence[tuple[str, str]],
- offered_to: str,
- ) -> set[tuple[str, str]]:
- """Offer the rows that are still `pending`; the keys that moved."""
- if not keys:
- return set()
- now = datetime.now(UTC)
- result = await session.execute(
- update(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.agent_id == agent_id,
- tuple_(HostedWakeMailbox.room_id, HostedWakeMailbox.message_id).in_(
- list(keys)
- ),
- HostedWakeMailbox.state == "pending",
- )
- .values(
- state="offered",
- ever_offered=True,
- offered_to=offered_to,
- offered_until=now + OFFER_LEASE,
- updated_at=now,
- )
- .returning(HostedWakeMailbox.room_id, HostedWakeMailbox.message_id)
- .execution_options(synchronize_session=False)
- )
- return {(row.room_id, row.message_id) for row in result}
-
- async def reclaim(
- self,
- session: AsyncSession,
- agent_id: str | None,
- live_offers: dict[str, str],
- ) -> int:
- """Return `offered` rows to `pending` unless their lease is held by a live stream.
-
- `live_offers` maps an agent to the offer key of its attached worker;
- a row offered to anything else (another Core boot, an older
- generation) or past its lease is reclaimed. `agent_id` narrows it to
- one agent.
- """
- now = datetime.now(UTC)
- conditions = [
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.state == "offered",
- ]
- if agent_id is not None:
- conditions.append(HostedWakeMailbox.agent_id == agent_id)
- held = [
- and_(
- HostedWakeMailbox.agent_id == agent,
- HostedWakeMailbox.offered_to == offer,
- HostedWakeMailbox.offered_until > now,
- )
- for agent, offer in live_offers.items()
- ]
- if held:
- conditions.append(~or_(*held))
- result = await session.execute(
- update(HostedWakeMailbox)
- .where(*conditions)
- .values(
- state="pending", offered_to=None, offered_until=None, updated_at=now
- )
- .execution_options(synchronize_session=False)
- )
- return int(getattr(result, "rowcount", 0) or 0)
-
- async def agents_with_pending(self, session: AsyncSession) -> list[str]:
- return list(
- await session.scalars(
- select(HostedWakeMailbox.agent_id)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.state == "pending",
- )
- .distinct()
- )
- )
-
- async def cancelled_entries(
- self, session: AsyncSession, agent_id: str
- ) -> list[dict[str, str]]:
- """Every tombstone the worker has still to answer, for `worker_attached`."""
- rows = await session.execute(
- select(
- HostedWakeMailbox.room_id,
- HostedWakeMailbox.message_id,
- HostedWakeMailbox.cancel_reason,
- )
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.agent_id == agent_id,
- HostedWakeMailbox.state == "cancel_requested",
- )
- .order_by(HostedWakeMailbox.addressed_at)
- )
- return [
- {"room_id": room_id, "message_id": message_id, "reason": reason}
- for room_id, message_id, reason in rows
- ]
-
- async def busy(self, session: AsyncSession, launch_id: str) -> bool:
- """Whether a row of the launch waits for its worker (`held` and tombstones do not)."""
- return bool(
- await session.scalar(
- select(
- select(HostedWakeMailbox.message_id)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.launch_id == launch_id,
- HostedWakeMailbox.state.in_(BUSY_STATES),
- )
- .exists()
- )
- )
- )
-
- async def stop(self, session: AsyncSession, launch_id: str) -> StopSplit:
- """An explicit Stop: definite cancels for rows never offered, tombstones for the rest."""
- tenant_id = require_tenant_id()
- now = datetime.now(UTC)
- cancelled = await session.scalars(
- update(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == tenant_id,
- HostedWakeMailbox.launch_id == launch_id,
- HostedWakeMailbox.state == "pending",
- HostedWakeMailbox.ever_offered.is_(False),
- )
- .values(state="cancelled", notice_owed="stopped", updated_at=now)
- .returning(HostedWakeMailbox)
- .execution_options(synchronize_session=False)
- )
- notices = _notices([(row, "stopped") for row in cancelled])
- requested = await session.execute(
- update(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == tenant_id,
- HostedWakeMailbox.launch_id == launch_id,
- HostedWakeMailbox.state.in_(("pending", "offered", "accepted", "held")),
- )
- .values(
- state="cancel_requested",
- cancel_reason="stopped",
- offered_to=None,
- offered_until=None,
- updated_at=now,
- )
- .returning(HostedWakeMailbox.room_id, HostedWakeMailbox.message_id)
- .execution_options(synchronize_session=False)
- )
- return StopSplit(
- cancelled=notices,
- cancel_requested=[(row.room_id, row.message_id) for row in requested],
- )
-
- async def ack(
- self,
- session: AsyncSession,
- agent_id: str,
- acks: Sequence[tuple[str, str, str]],
- ) -> tuple[dict[tuple[str, str], str], list[MailboxNotice]]:
- """Apply `(room, message, outcome)` acks; each row's state after, and the notices owed.
-
- An ack for a row at or past its target, or for no row at all, changes
- nothing and is not an error: acks are retried.
- """
- keys = list({(room, message) for room, message, _ in acks})
- rows = {
- (row.room_id, row.message_id): row
- for row in await session.scalars(
- select(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.agent_id == agent_id,
- tuple_(HostedWakeMailbox.room_id, HostedWakeMailbox.message_id).in_(
- keys
- ),
- )
- .with_for_update()
- .execution_options(populate_existing=True)
- )
- }
- now = datetime.now(UTC)
- owed: list[tuple[HostedWakeMailbox, str]] = []
- for room, message, outcome in acks:
- row = rows.get((room, message))
- if row is None:
- continue
- sources, target = TRANSITIONS[outcome]
- if row.state not in sources:
- continue
- if row.state == "cancel_requested":
- reason = row.cancel_reason or "stopped"
- row.notice_owed = (
- reason if target == "cancelled" else STARTED_BEFORE[reason]
- )
- owed.append((row, row.notice_owed))
- row.state = target
- row.offered_to = None
- row.offered_until = None
- row.updated_at = now
- return {key: row.state for key, row in rows.items()}, _notices(owed)
-
- async def expire(
- self, session: AsyncSession, now: datetime
- ) -> tuple[list[MailboxNotice], int]:
- """Settle rows 24 h old by state; the notices owed and the tombstones made.
-
- `cancel_requested` rows are never touched: a tombstone stays until the
- watcher answers it or the launch goes.
- """
- tenant_id = require_tenant_id()
- due = [
- HostedWakeMailbox.tenant_id == tenant_id,
- HostedWakeMailbox.expires_at <= now,
- ]
- never_offered = await session.scalars(
- update(HostedWakeMailbox)
- .where(
- *due,
- HostedWakeMailbox.state == "pending",
- HostedWakeMailbox.ever_offered.is_(False),
- )
- .values(state="expired", notice_owed="expired", updated_at=now)
- .returning(HostedWakeMailbox)
- .execution_options(synchronize_session=False)
- )
- owed = [(row, "expired") for row in never_offered]
- uncertain = await session.scalars(
- update(HostedWakeMailbox)
- .where(*due, HostedWakeMailbox.state.in_(("pending", "offered")))
- .values(
- state="expired_uncertain",
- notice_owed="expired_uncertain",
- offered_to=None,
- offered_until=None,
- updated_at=now,
- )
- .returning(HostedWakeMailbox)
- .execution_options(synchronize_session=False)
- )
- notices = _notices(owed + [(row, "expired_uncertain") for row in uncertain])
- tombstoned = await session.execute(
- update(HostedWakeMailbox)
- .where(*due, HostedWakeMailbox.state.in_(("accepted", "held")))
- .values(state="cancel_requested", cancel_reason="expired", updated_at=now)
- .execution_options(synchronize_session=False)
- )
- return notices, int(getattr(tombstoned, "rowcount", 0) or 0)
-
- async def owed_notices(
- self, session: AsyncSession, limit: int
- ) -> list[MailboxNotice]:
- """Room notices terminal moves owe and no one has posted yet, oldest first.
-
- Bounded to `limit` (agent, room, reason) groups, each whole, so a
- group's latest row is the one every retry posts for.
- """
- tenant_id = require_tenant_id()
- owed = (
- HostedWakeMailbox.tenant_id == tenant_id,
- HostedWakeMailbox.notice_owed.is_not(None),
- HostedWakeMailbox.notice_dropped.is_(None),
- )
- groups = (
- select(
- HostedWakeMailbox.agent_id,
- HostedWakeMailbox.room_id,
- HostedWakeMailbox.notice_owed,
- )
- .where(*owed)
- .group_by(
- HostedWakeMailbox.agent_id,
- HostedWakeMailbox.room_id,
- HostedWakeMailbox.notice_owed,
- )
- .order_by(func.min(HostedWakeMailbox.addressed_at))
- .limit(limit)
- .subquery()
- )
- rows = await session.scalars(
- select(HostedWakeMailbox)
- .join(
- groups,
- and_(
- HostedWakeMailbox.agent_id == groups.c.agent_id,
- HostedWakeMailbox.room_id == groups.c.room_id,
- HostedWakeMailbox.notice_owed == groups.c.notice_owed,
- ),
- )
- .where(*owed)
- .order_by(HostedWakeMailbox.addressed_at, HostedWakeMailbox.message_id)
- )
- return [
- _notice(row, row.notice_owed) for row in rows if row.notice_owed is not None
- ]
-
- async def notice_posted(
- self,
- session: AsyncSession,
- agent_id: str,
- room_id: str,
- reason: str,
- message_ids: Sequence[str],
- ) -> None:
- """The room has the notice these rows owed under `reason`; they owe nothing more."""
- await session.execute(
- update(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.agent_id == agent_id,
- HostedWakeMailbox.room_id == room_id,
- HostedWakeMailbox.message_id.in_(list(message_ids)),
- HostedWakeMailbox.notice_owed == reason,
- )
- .values(notice_owed=None)
- .execution_options(synchronize_session=False)
- )
-
- async def notice_dropped(
- self,
- session: AsyncSession,
- agent_id: str,
- room_id: str,
- reason: str,
- message_ids: Sequence[str],
- dropped: str,
- ) -> None:
- """The notice these rows owe under `reason` will never be posted, for `dropped`."""
- await session.execute(
- update(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.agent_id == agent_id,
- HostedWakeMailbox.room_id == room_id,
- HostedWakeMailbox.message_id.in_(list(message_ids)),
- HostedWakeMailbox.notice_owed == reason,
- )
- .values(notice_dropped=dropped)
- .execution_options(synchronize_session=False)
- )
-
- async def prune(self, session: AsyncSession, now: datetime) -> int:
- result = await session.execute(
- delete(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.state.in_(TERMINAL_STATES),
- HostedWakeMailbox.updated_at < now - MAILBOX_RETENTION,
- )
- .execution_options(synchronize_session=False)
- )
- return int(getattr(result, "rowcount", 0) or 0)
-
- async def delete_launch(self, session: AsyncSession, launch_id: str) -> None:
- """A removed launch's rows go with it, tombstones included, with no notice."""
- await session.execute(
- delete(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.launch_id == launch_id,
- )
- .execution_options(synchronize_session=False)
- )
-
- async def backlog(
- self, session: AsyncSession, since: datetime
- ) -> list[AgentBacklog]:
- """Per agent: rows waiting, the oldest of them, and watcher refusals since `since`."""
- waiting = func.count().filter(
- HostedWakeMailbox.state.in_(("pending", "offered"))
- )
- oldest = func.min(HostedWakeMailbox.addressed_at).filter(
- HostedWakeMailbox.state.in_(("pending", "offered"))
- )
- refused = func.count().filter(
- HostedWakeMailbox.state == "refused",
- HostedWakeMailbox.updated_at >= since,
- )
- rows = await session.execute(
- select(HostedWakeMailbox.agent_id, waiting, oldest, refused)
- .where(HostedWakeMailbox.tenant_id == require_tenant_id())
- .group_by(HostedWakeMailbox.agent_id)
- )
- return [
- AgentBacklog(agent_id=agent, waiting=count, oldest=first, refused=no)
- for agent, count, first, no in rows
- if count or no
- ]
diff --git a/core/switch_core/db/stores/provider_connection_store.py b/core/switch_core/db/stores/provider_connection_store.py
deleted file mode 100644
index aa58fda07..000000000
--- a/core/switch_core/db/stores/provider_connection_store.py
+++ /dev/null
@@ -1,82 +0,0 @@
-from datetime import datetime
-
-from sqlalchemy import delete, select, text
-from sqlalchemy.dialects.postgresql import insert
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.db.models import ProviderConnection, require_tenant_id
-
-
-class ProviderConnectionBusy(Exception):
- pass
-
-
-class ProviderConnectionStore:
- async def wait_user(self, session: AsyncSession, user_id: str) -> None:
- await session.execute(
- text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
- {"key": f"provider-connection:{require_tenant_id()}:{user_id}"},
- )
-
- async def lock_user(self, session: AsyncSession, user_id: str) -> None:
- acquired = await session.scalar(
- text("SELECT pg_try_advisory_xact_lock(hashtextextended(:key, 0))"),
- {"key": f"provider-connection:{require_tenant_id()}:{user_id}"},
- )
- if not acquired:
- raise ProviderConnectionBusy(
- "A connection change is already in progress. Try again shortly."
- )
-
- async def get(
- self, session: AsyncSession, user_id: str
- ) -> ProviderConnection | None:
- result = await session.execute(
- select(ProviderConnection).where(
- ProviderConnection.tenant_id == require_tenant_id(),
- ProviderConnection.user_id == user_id,
- ProviderConnection.provider == "claude",
- )
- )
-
- return result.scalar_one_or_none()
-
- async def save(
- self,
- session: AsyncSession,
- user_id: str,
- kind: str,
- encrypted: str,
- verified_at: datetime,
- ) -> None:
- values = dict(
- tenant_id=require_tenant_id(),
- user_id=user_id,
- provider="claude",
- kind=kind,
- encrypted_credential=encrypted,
- verified_at=verified_at,
- verification_status="verified",
- )
- await session.execute(
- insert(ProviderConnection)
- .values(**values)
- .on_conflict_do_update(
- index_elements=["tenant_id", "user_id", "provider"],
- set_={
- "kind": kind,
- "encrypted_credential": encrypted,
- "verified_at": verified_at,
- "verification_status": "verified",
- },
- )
- )
-
- async def delete(self, session: AsyncSession, user_id: str) -> None:
- await session.execute(
- delete(ProviderConnection).where(
- ProviderConnection.tenant_id == require_tenant_id(),
- ProviderConnection.user_id == user_id,
- ProviderConnection.provider == "claude",
- )
- )
diff --git a/core/switch_core/gateway/agents.py b/core/switch_core/gateway/agents.py
index 4e83fe3ae..8e0b9474e 100644
--- a/core/switch_core/gateway/agents.py
+++ b/core/switch_core/gateway/agents.py
@@ -22,14 +22,12 @@ from switch_core.bridges.agent.protocol.agent_detail import (
build_agent_summary,
list_agent_summaries,
)
-from switch_core.bridges.agent.protocol.hosted_workers import hosted_launch_of
from switch_core.bridges.agent.protocol.types import (
IntegrationProfile,
TaskProtocolConfig,
)
-from switch_core.db.models import Agent, User
+from switch_core.db.models import User
from switch_core.db.stores.agent_store import AgentStore
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
from switch_core.db.stores.room_store import RoomStore
from switch_core.db.stores.user_store import UserStore
from switch_core.gateway.auth import get_current_user, get_tenant_is_admin
@@ -64,17 +62,6 @@ logger = logging.getLogger(__name__)
router = APIRouter()
-CLOUD_AGENT_DELETE_REFUSED = (
- "This is a cloud agent. Remove it from Switch Console's cloud agents instead."
-)
-
-
-async def _sync_hosted_spec(session: AsyncSession, agent: Agent, changes: dict) -> None:
- """Keep a cloud agent's launch spec, which registers it again, on the agent's values."""
- launch_id = hosted_launch_of(agent.metadata_)
- if launch_id is not None:
- await HostedLaunchStore().merge_spec(session, launch_id, changes)
-
@router.get("")
async def list_agents(
@@ -108,8 +95,6 @@ async def delete_agent_by_name(
status_code=403,
detail="Only the agent's owner or an admin can delete it.",
)
- if hosted_launch_of(agent.metadata_) is not None:
- raise HTTPException(status_code=409, detail=CLOUD_AGENT_DELETE_REFUSED)
# Not held across the delete, which opens its own sessions and calls every bridge.
await session.commit()
try:
@@ -143,8 +128,6 @@ async def delete_agent(
status_code=403,
detail="Only the agent's owner or an admin can delete it.",
)
- if hosted_launch_of(agent.metadata_) is not None:
- raise HTTPException(status_code=409, detail=CLOUD_AGENT_DELETE_REFUSED)
# Not held across the delete, which opens its own sessions and calls every bridge.
await session.commit()
try:
@@ -444,7 +427,6 @@ async def update_agent_icon(
raise HTTPException(status_code=400, detail=str(exc)) from exc
await agent_store.update(session, agent_id, icon_url=icon_url)
- await _sync_hosted_spec(session, agent, {"icon_url": icon_url})
await session.commit()
await session.refresh(agent)
@@ -499,7 +481,6 @@ async def update_agent_display_name(
raise HTTPException(status_code=400, detail=str(exc)) from exc
await agent_store.update(session, agent_id, display_name=display_name)
- await _sync_hosted_spec(session, agent, {"display_name": display_name})
await session.commit()
await session.refresh(agent)
@@ -542,7 +523,6 @@ async def update_agent_description(
raise HTTPException(status_code=400, detail="description must not be blank")
await agent_store.update(session, agent_id, description=description)
- await _sync_hosted_spec(session, agent, {"description": description})
await session.commit()
await session.refresh(agent)
@@ -628,7 +608,6 @@ async def update_addressing_policy(
stored = req.policy.model_dump() if req.policy is not None else None
await agent_store.update(session, agent_id, addressing_policy=stored)
- await _sync_hosted_spec(session, agent, {"addressing_policy": stored})
await session.commit()
logger.info(
diff --git a/core/switch_core/gateway/app.py b/core/switch_core/gateway/app.py
index 44e79d9b4..6b81fdc85 100644
--- a/core/switch_core/gateway/app.py
+++ b/core/switch_core/gateway/app.py
@@ -44,9 +44,7 @@ from switch_core.gateway.documents import router as documents_router
from switch_core.gateway.ecosystem import router as ecosystem_router
from switch_core.gateway.github_connections import router as github_connections_router
from switch_core.gateway.hosted_controller import router as hosted_controller_router
-from switch_core.gateway.hosted_launches import router as hosted_launches_router
from switch_core.gateway.hosted_machines import router as hosted_machines_router
-from switch_core.gateway.hosted_relay import router as hosted_relay_router
from switch_core.gateway.invite_mail import InviteMailer
from switch_core.gateway.messaging_installs import (
router as messaging_installs_router,
@@ -54,12 +52,6 @@ from switch_core.gateway.messaging_installs import (
from switch_core.gateway.oidc_routes import register_oidc_client
from switch_core.gateway.oidc_routes import router as oidc_router
from switch_core.gateway.packages import router as packages_router
-from switch_core.gateway.provider_connections import (
- router as provider_connections_router,
-)
-from switch_core.gateway.provider_verifications import (
- router as provider_verifications_router,
-)
from switch_core.gateway.references import router as references_router
from switch_core.gateway.room_groups import router as room_groups_router
from switch_core.gateway.room_links import router as room_links_router
@@ -69,7 +61,6 @@ from switch_core.gateway.template_runs import router as template_runs_router
from switch_core.gateway.templates import router as templates_router
from switch_core.gateway.tenants import router as tenants_router
from switch_core.keys import Purpose
-from switch_core.providers.claude_verifier import ClaudeVerifier
from switch_core.providers.github import GitHubConnections
from switch_core.providers.hosted import HostedControllerSettings
from switch_core.room_service import RoomService
@@ -139,24 +130,12 @@ def create_gateway_app(
if config.hosted_controller_config_path
else None
)
- if config.hosted_launch_capacity and (
- app.state.hosted_controller_settings is None
- or len(app.state.hosted_controller_settings.machine_slots)
- < config.hosted_launch_capacity
- ):
+ if config.hosted_launch_capacity and app.state.hosted_controller_settings is None:
raise ValueError(
- "Cloud launch capacity requires enough configured machine slots."
+ "HOSTED_LAUNCH_CAPACITY above 0 requires HOSTED_CONTROLLER_CONFIG_PATH."
)
- app.include_router(hosted_launches_router, tags=["hosted-launches"])
- app.include_router(hosted_relay_router, tags=["hosted-launches"])
app.include_router(hosted_machines_router, tags=["hosted-machines"])
- if (
- config.hosted_provider_verification_enabled
- and app.state.hosted_controller_settings is None
- ):
- raise ValueError("Provider verification requires a hosted controller.")
app.include_router(hosted_controller_router, tags=["hosted-controller"])
- app.include_router(provider_verifications_router, tags=["provider-verifications"])
app.state.github_connections = (
GitHubConnections(config.hosted_github_config_path)
if config.hosted_github_config_path
@@ -167,16 +146,6 @@ def create_gateway_app(
tags=["provider-connections"],
)
app.include_router(connection_catalog_router, tags=["provider-connections"])
- app.state.claude_verifier = (
- ClaudeVerifier(config.hosted_claude_verifier_path)
- if config.hosted_claude_verifier_path
- else None
- )
- app.include_router(
- provider_connections_router,
- prefix="/provider-connections",
- tags=["provider-connections"],
- )
# authlib's OIDC client stores transient state/nonce/PKCE in the request
# session across the IdP redirect round-trip; SameSite=Lax lets the cookie
diff --git a/core/switch_core/gateway/auth_routes.py b/core/switch_core/gateway/auth_routes.py
index 1fc547ff2..9cb5c401d 100644
--- a/core/switch_core/gateway/auth_routes.py
+++ b/core/switch_core/gateway/auth_routes.py
@@ -40,8 +40,11 @@ from switch_core.gateway.dependencies import (
get_system_session,
get_user_store,
)
-from switch_core.gateway.hosted_launches import hosted_settings
-from switch_core.gateway.hosted_machines import MachineUnavailable, ensure_machine
+from switch_core.gateway.hosted_machines import (
+ MachineUnavailable,
+ ensure_machine,
+ hosted_settings,
+)
from switch_core.gateway.schemas import (
AuthConfigResponse,
ChangePasswordRequest,
diff --git a/core/switch_core/gateway/cloud_controllers.py b/core/switch_core/gateway/cloud_controllers.py
new file mode 100644
index 000000000..a2013617f
--- /dev/null
+++ b/core/switch_core/gateway/cloud_controllers.py
@@ -0,0 +1,190 @@
+"""Switch cloud machines, which run the agents controller.
+
+A cloud machine boots the same `switch-agent-controller` any machine can
+run, and enrolls it with a one-time code Core mints for the machine and hands
+over when it prepares it. The controller it enrolls as is linked to the
+machine (`controller_id`); the owner's managed agents placed on it run there,
+each as a Linux user of its own, and its status reports are what make the
+machine ready.
+
+Enrollment codes and controllers belong to agent management, which Core does
+not import (`management/wiring.py` installs it here with
+`set_cloud_enrollment`); Core keeps the machine side.
+"""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime, timedelta
+from typing import Any, Protocol
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from switch_core.db.models import (
+ AgentController,
+ HostedMachine,
+ require_tenant_id,
+)
+from switch_core.db.stores.hosted_machine_store import HostedMachineStore, idle_sleeping
+from switch_core.keys import Keyring
+
+# How recent the machine's last status report must be for its silence to count
+# as idle rather than as a machine that stopped reporting.
+FRESH_HEARTBEAT = timedelta(minutes=5)
+
+
+class CloudEnrollment(Protocol):
+ """What agent management does for a cloud machine's controller."""
+
+ async def machine_enrollment_code(
+ self, session: AsyncSession, machine: HostedMachine, now: datetime
+ ) -> str:
+ """A new one-time code the machine's controller enrolls with. The caller commits."""
+ ...
+
+
+_enrollment: CloudEnrollment | None = None
+
+
+def set_cloud_enrollment(enrollment: CloudEnrollment | None) -> None:
+ global _enrollment
+ _enrollment = enrollment
+
+
+class CloudEnrollmentUnavailable(Exception):
+ """Controller machines need agent management, and this server does not run it."""
+
+
+def cloud_enrollment() -> CloudEnrollment:
+ if _enrollment is None:
+ raise CloudEnrollmentUnavailable(
+ "Cloud machines need agent management: set AGENT_MANAGEMENT_ENABLED."
+ )
+ return _enrollment
+
+
+async def live_controller(
+ session: AsyncSession, machine: HostedMachine
+) -> AgentController | None:
+ """The controller the machine enrolled as, unless it was revoked since."""
+ if machine.controller_id is None:
+ return None
+ controller = await session.get(
+ AgentController, machine.controller_id, populate_existing=True
+ )
+ if (
+ controller is None
+ or controller.tenant_id != require_tenant_id()
+ or controller.revoked_at is not None
+ ):
+ return None
+ return controller
+
+
+async def enrollment_code(
+ session: AsyncSession, machine: HostedMachine, keyring: Keyring, now: datetime
+) -> str:
+ """The code the machine enrolls with at its current revision: the same one
+ for every retry of a revision, a new one for a new revision. The caller
+ commits."""
+ if (
+ machine.enrollment_code_revision == machine.revision
+ and machine.enrollment_code_encrypted is not None
+ ):
+ return keyring.decrypt(machine.enrollment_code_encrypted)
+ code = await cloud_enrollment().machine_enrollment_code(session, machine, now)
+ machine.enrollment_code_encrypted = keyring.encrypt(code)
+ machine.enrollment_code_revision = machine.revision
+ return code
+
+
+async def machine_of_controller(
+ session: AsyncSession, controller_id: str
+) -> HostedMachine | None:
+ """The live machine running `controller_id`, locked; None for any other controller."""
+ machine_id = await session.scalar(
+ select(HostedMachine.id).where(
+ HostedMachine.tenant_id == require_tenant_id(),
+ HostedMachine.controller_id == controller_id,
+ HostedMachine.state != "deleted",
+ )
+ )
+ if machine_id is None:
+ return None
+ return await HostedMachineStore().locked(session, machine_id)
+
+
+def _reading(free: Any, total: Any) -> dict[str, int] | None:
+ if isinstance(free, int) and isinstance(total, int) and total > 0:
+ return {"total_bytes": total, "available_bytes": free}
+ return None
+
+
+async def record_controller_status(
+ session: AsyncSession, controller_id: str, machine_reading: dict[str, Any]
+) -> None:
+ """A status report from a cloud machine's controller is its heartbeat: it
+ records the machine's disk and memory, and makes a machine that started
+ at this revision ready. The caller commits."""
+ machine = await machine_of_controller(session, controller_id)
+ if machine is None:
+ return
+ now = datetime.now(UTC)
+ machine.heartbeat = {
+ "disk": _reading(
+ machine_reading.get("disk_free_bytes"),
+ machine_reading.get("disk_total_bytes"),
+ ),
+ "memory": _reading(
+ machine_reading.get("mem_free_bytes"),
+ machine_reading.get("mem_total_bytes"),
+ ),
+ }
+ machine.heartbeat_at = now
+ sessions = machine_reading.get("sessions_running")
+ if isinstance(sessions, int):
+ machine.heartbeat["sessions_running"] = sessions
+ if sessions > 0:
+ machine.active_at = now
+ if (
+ machine.state == "provisioning"
+ and machine.desired_state == "running"
+ and machine.running_observed_at is not None
+ ):
+ machine.state = "ready"
+ machine.error = None
+ machine.error_code = None
+ machine.updated_at = now
+
+
+async def wake_controller_machine(
+ session: AsyncSession, controller_id: str, now: datetime
+) -> HostedMachine | None:
+ """The cloud machine running `controller_id`, started again if it was put
+ to sleep when idle, and kept awake otherwise. A machine its owner stopped
+ is left stopped. None for a controller that runs on no cloud machine. The
+ caller holds no other machine lock, and commits."""
+ machine = await machine_of_controller(session, controller_id)
+ if machine is None:
+ return None
+ store = HostedMachineStore()
+ if idle_sleeping(machine):
+ store.start(machine, now)
+ elif machine.desired_state == "running":
+ machine.active_at = now
+ return machine
+
+
+def controller_machine_idle(
+ machine: HostedMachine, idle_after: timedelta, now: datetime
+) -> bool:
+ """Whether a controller machine has been idle for `idle_after`: its last
+ status report is recent and says no session runs, and neither a session
+ nor a message addressed to its agents has kept it active since."""
+ heartbeat = machine.heartbeat or {}
+ return (
+ machine.heartbeat_at is not None
+ and now - machine.heartbeat_at <= FRESH_HEARTBEAT
+ and heartbeat.get("sessions_running") == 0
+ and now - machine.active_at >= idle_after
+ )
diff --git a/core/switch_core/gateway/github_connections.py b/core/switch_core/gateway/github_connections.py
index 1cd7dd959..496019c03 100644
--- a/core/switch_core/gateway/github_connections.py
+++ b/core/switch_core/gateway/github_connections.py
@@ -1,4 +1,3 @@
-import asyncio
import json
import logging
import secrets
@@ -19,8 +18,6 @@ from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession
from switch_core.config import SwitchConfig
from switch_core.db.engine import create_session_factory
from switch_core.db.models import (
- GitHubIssuedToken,
- HostedLaunch,
ProviderConnection,
Tenant,
User,
@@ -35,17 +32,7 @@ from switch_core.providers.github import (
GitHubError,
GitHubFlow,
)
-from switch_core.providers.github_installation import (
- GitHubInstallationCredentials,
- RepositoryCredential,
-)
-from switch_core.providers.github_revocations import (
- ACCESS_WARNING,
- queue_revocation,
- remember_repository_token,
- revoke_oauth,
- revoke_pending,
-)
+from switch_core.providers.github_revocations import revoke_oauth
from switch_core.providers.github_tasks import finish_shielded
logger = logging.getLogger(__name__)
@@ -347,7 +334,6 @@ async def confirm(
if previous
else None
)
- await queue_revocation(session, (GitHubIssuedToken.owner_id == user.id,))
encrypted = config.keyring.encrypt(json.dumps(flow.credentials))
now = datetime.now(UTC)
await session.execute(
@@ -382,15 +368,7 @@ async def confirm(
if previous_token and previous_token != flow.credentials["access_token"]
else None
)
- remaining = await revoke_pending(
- session, config, (GitHubIssuedToken.owner_id == user.id,)
- )
- messages = [
- message
- for message in (warning, ACCESS_WARNING if remaining else None)
- if message
- ]
- return JSONResponse({"warning": " ".join(messages) or None})
+ return JSONResponse({"warning": warning})
@router.get("")
@@ -514,7 +492,6 @@ async def disconnect(
if row
else None
)
- await queue_revocation(session, (GitHubIssuedToken.owner_id == user.id,))
await session.execute(
delete(ProviderConnection).where(*conditions(user.id)) # nosemgrep
)
@@ -523,69 +500,4 @@ async def disconnect(
if (flow.tenant_id, flow.user_id) == (require_tenant_id(), user.id):
del github.flows[key]
warning = await revoke_oauth(github, token) if token else None
- remaining = await revoke_pending(
- session, config, (GitHubIssuedToken.owner_id == user.id,)
- )
- messages = [
- message
- for message in (warning, ACCESS_WARNING if remaining else None)
- if message
- ]
- return JSONResponse({"warning": " ".join(messages) or None})
-
-
-async def discard_github_credential(
- session: AsyncSession,
- signer: GitHubInstallationCredentials,
- credential: RepositoryCredential,
- config: SwitchConfig,
- launch_id: str,
- original: BaseException,
-) -> None:
- async def cleanup() -> None:
- try:
- await session.rollback()
- except Exception:
- logger.error(
- "Rollback failed while discarding a GitHub token; original failure: %r",
- original,
- exc_info=True,
- )
- try:
- await signer.revoke(credential.token)
- except Exception:
- logger.error(
- "GitHub token revocation failed; original failure: %r",
- original,
- exc_info=True,
- )
- try:
- engine = session.bind
- assert isinstance(engine, AsyncEngine)
- async with tenant_session(
- create_session_factory(engine), require_tenant_id()
- ) as cleanup_session:
- launch = await cleanup_session.get(
- HostedLaunch, (require_tenant_id(), launch_id)
- )
- if launch is None:
- raise RuntimeError("The token's cloud launch is missing")
- record = remember_repository_token(
- cleanup_session, launch, credential, config
- )
- record.revoke_requested = True
- await cleanup_session.commit()
- except Exception as queue_error:
- logger.error(
- "Failed GitHub token revocation could not be queued: error_type=%s",
- type(queue_error).__name__,
- )
-
- try:
- await finish_shielded(cleanup())
- except asyncio.CancelledError:
- logger.warning(
- "GitHub token cleanup finished after another caller cancellation; original failure: %r",
- original,
- )
- raise
+ return JSONResponse({"warning": warning})
diff --git a/core/switch_core/gateway/hosted_controller.py b/core/switch_core/gateway/hosted_controller.py
index 0f1b6cdd7..a83a703f1 100644
--- a/core/switch_core/gateway/hosted_controller.py
+++ b/core/switch_core/gateway/hosted_controller.py
@@ -9,31 +9,27 @@ from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.config import SwitchConfig
from switch_core.db.models import (
- HostedLaunch,
HostedMachine,
- HostedOperation,
TenantMember,
require_tenant_id,
)
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
from switch_core.db.stores.hosted_machine_store import (
MACHINE_CONNECT_TIMEOUT,
HostedMachineStore,
bump_revision,
idle_sleeping,
- lock_launch,
retention_expired,
)
-from switch_core.gateway.dependencies import (
- get_config,
- get_protocol,
- get_session_factory,
+from switch_core.gateway.cloud_controllers import (
+ CloudEnrollmentUnavailable,
+ controller_machine_idle,
+ enrollment_code,
+ live_controller,
)
-from switch_core.gateway.hosted_launches import controller_settings, finish_removal
-from switch_core.providers.github_revocations import revoke_pending
+from switch_core.gateway.dependencies import get_config, get_session_factory
+from switch_core.gateway.hosted_machines import controller_settings
from switch_core.providers.hosted import HostedControllerSettings
from switch_core.tenant_context import tenant_scope
@@ -42,8 +38,6 @@ logger = logging.getLogger(__name__)
router = APIRouter(prefix="/hosted-controller")
QUEUED_TIMEOUT = timedelta(minutes=10)
-DELETING_RESUME_AFTER = timedelta(minutes=5)
-AGENT_STOP_TIMEOUT = timedelta(minutes=10)
async def controller_session(
@@ -69,8 +63,6 @@ async def controller_session(
def machine_item(machine: HostedMachine) -> dict:
return {
"machine_id": machine.id,
- "slot_id": machine.slot_id,
- "generation": machine.generation,
"state": machine.state,
"desired_state": machine.desired_state,
"revision": machine.revision,
@@ -78,10 +70,16 @@ def machine_item(machine: HostedMachine) -> dict:
"retain_until": machine.retain_until.isoformat()
if machine.retain_until
else None,
- "bundle_revision": machine.machine_capability_revision,
+ "bundle_revision": machine.enrollment_code_revision,
}
+def _idle_stops(machine: HostedMachine, idle_minutes: int) -> bool:
+ """Whether an idle machine is put to sleep: a message addressed to one of
+ its agents wakes it again."""
+ return idle_minutes > 0
+
+
def _connect_timed_out(machine: HostedMachine, now: datetime) -> bool:
return (
machine.state == "provisioning"
@@ -102,16 +100,6 @@ def _start_timed_out(machine: HostedMachine, now: datetime) -> bool:
)
-def _stop_timed_out(
- launch: HostedLaunch, machine: HostedMachine, now: datetime
-) -> bool:
- return (
- launch.state == "stopping"
- and now - launch.updated_at > AGENT_STOP_TIMEOUT
- and machine.state == "ready"
- )
-
-
def _errored_unclaimed(
machine: HostedMachine, now: datetime, idle_minutes: int
) -> bool:
@@ -132,90 +120,16 @@ def _needs_sweep(machine: HostedMachine, now: datetime, idle_minutes: int) -> bo
or _errored_unclaimed(machine, now, idle_minutes)
or idle_sleeping(machine)
or (
- idle_minutes > 0
+ _idle_stops(machine, idle_minutes)
and machine.state == "ready"
and machine.desired_state == "running"
)
)
-async def _should_sleep(
- session: AsyncSession,
- machine: HostedMachine,
- protocol: AgentCore,
- idle_after: timedelta,
- now: datetime,
-) -> bool:
- """Whether every counted agent on the machine is provably idle.
-
- Counted agents are the ones meant to run and not in error. A busy one
- renews its own activity, so it keeps the machine awake for another window.
- """
- idle = True
- for candidate in await HostedMachineStore().launches(session, machine.id):
- await lock_launch(session, candidate.id)
- launch = await session.get(
- HostedLaunch, (require_tenant_id(), candidate.id), populate_existing=True
- )
- if (
- launch is None
- or launch.desired_state != "running"
- or launch.state in {"error", "deleted"}
- ):
- continue
- evidence = await HostedLaunchStore().idle_evidence(
- session, launch, protocol.connections
- )
- if evidence.busy:
- launch.active_at = now
- idle = False
- elif (
- now - launch.active_at < idle_after
- or evidence.report is None
- or evidence.report.received_at <= launch.active_at
- ):
- idle = False
- return idle and now - machine.active_at >= idle_after
-
-
-async def _resume_removals(
- session: AsyncSession,
- protocol: AgentCore,
- config: SwitchConfig,
- now: datetime,
-) -> None:
- """Finish removals that were interrupted between their two commits."""
- stalled = list(
- await session.scalars(
- select(HostedLaunch.id).where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.state == "deleting",
- HostedLaunch.updated_at < now - DELETING_RESUME_AFTER,
- )
- )
- )
- for launch_id in stalled:
- launch, machine = await HostedMachineStore().locked_launch(session, launch_id)
- if launch is None or machine is None or launch.state != "deleting":
- await session.commit()
- continue
- try:
- await finish_removal(session, protocol, config, launch, machine, now)
- except Exception:
- logger.error(
- "Cloud launch %s: finishing its interrupted removal failed",
- launch_id,
- exc_info=True,
- )
- await session.rollback()
- continue
- await session.commit()
-
-
async def _sweep(
session: AsyncSession,
machine: HostedMachine,
- protocol: AgentCore,
idle_minutes: int,
retention_days: int,
now: datetime,
@@ -240,7 +154,7 @@ async def _sweep(
machine.desired_state = "deleted"
bump_revision(machine, now)
elif _errored_unclaimed(machine, now, idle_minutes) and not await store.ever_hosted(
- session, machine.id
+ session, machine
):
await store.release_if_empty(
session, machine, retention_days=retention_days, now=now
@@ -250,12 +164,10 @@ async def _sweep(
session, machine, retention_days=retention_days, now=now
)
elif (
- idle_minutes > 0
+ _idle_stops(machine, idle_minutes)
and machine.state == "ready"
and machine.desired_state == "running"
- and await _should_sleep(
- session, machine, protocol, timedelta(minutes=idle_minutes), now
- )
+ and controller_machine_idle(machine, timedelta(minutes=idle_minutes), now)
):
if not await store.release_if_empty(
session, machine, retention_days=retention_days, now=now
@@ -263,68 +175,13 @@ async def _sweep(
store.stop(machine, "idle", now)
-async def _expire_operations(session: AsyncSession) -> None:
- launch_ids = list(
- await session.scalars(
- select(HostedOperation.launch_id)
- .where(
- HostedOperation.tenant_id == require_tenant_id(),
- HostedOperation.state.in_(["queued", "claimed"]),
- )
- .distinct()
- )
- )
- for launch_id in launch_ids:
- launch, _ = await HostedMachineStore().locked_launch(session, launch_id)
- if launch is not None:
- await HostedLaunchStore().fail_stale_operations(
- session, launch.id, launch.revision
- )
- await session.commit()
-
-
-async def _time_out_stopping_launches(session: AsyncSession, now: datetime) -> None:
- launch_ids = list(
- await session.scalars(
- select(HostedLaunch.id)
- .join(
- HostedMachine,
- (HostedMachine.tenant_id == HostedLaunch.tenant_id)
- & (HostedMachine.id == HostedLaunch.machine_id),
- )
- .where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.state == "stopping",
- HostedLaunch.updated_at < now - AGENT_STOP_TIMEOUT,
- HostedMachine.state == "ready",
- )
- )
- )
- for launch_id in launch_ids:
- launch, machine = await HostedMachineStore().locked_launch(session, launch_id)
- if (
- launch is not None
- and machine is not None
- and _stop_timed_out(launch, machine, now)
- ):
- launch.state = "error"
- launch.error_code = "agent_stop_timeout"
- launch.error = (
- "The agent did not stop within 10 minutes. Retry it in Switch Console."
- )
- launch.updated_at = now
- await session.commit()
-
-
@router.get("/machines")
async def machines(
session: Annotated[AsyncSession, Depends(controller_session)],
config: Annotated[SwitchConfig, Depends(get_config)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
) -> dict:
store = HostedMachineStore()
now = datetime.now(UTC)
- await _resume_removals(session, protocol, config, now)
candidates = list(
await session.scalars(
select(HostedMachine)
@@ -343,14 +200,11 @@ async def machines(
await _sweep(
session,
machine,
- protocol,
config.hosted_idle_stop_minutes,
config.hosted_disk_retention_days,
now,
)
await session.commit()
- await _expire_operations(session)
- await _time_out_stopping_launches(session, now)
rows = await session.scalars(
select(HostedMachine)
.where(
@@ -362,7 +216,6 @@ async def machines(
)
response = {"machines": [machine_item(machine) for machine in rows]}
await session.commit()
- await revoke_pending(session, config, ())
return response
@@ -394,18 +247,31 @@ async def prepare(
raise HTTPException(
409, "The cloud machine's owner is no longer a workspace member."
)
- capability = HostedMachineStore().issue_capability(machine, config.keyring)
+ # What the machine boots from at its current revision: the controller it
+ # enrolled as, or, until it has one that is not revoked, a one-time code to
+ # enroll with. Never a long-lived credential: the machine keeps the one it
+ # enrolls with on its own disk.
+ controller = await live_controller(session, machine)
+ try:
+ code = (
+ None
+ if controller is not None
+ else await enrollment_code(session, machine, config.keyring, now)
+ )
+ except CloudEnrollmentUnavailable as error:
+ raise HTTPException(409, str(error)) from None
if machine.state == "queued":
machine.state = "provisioning"
machine.updated_at = now
result = {
"machine_id": machine.id,
- "slot_id": machine.slot_id,
- "generation": machine.generation,
"revision": machine.revision,
- "bundle_revision": machine.machine_capability_revision,
- "machine_capability": capability,
+ "bundle_revision": machine.revision,
"api_endpoint": settings.agent_api_endpoint,
+ "controller": {
+ "id": controller.id if controller is not None else None,
+ "enrollment_code": code,
+ },
}
await session.commit()
return result
@@ -487,14 +353,6 @@ async def observe(
machine.state = body.state
machine.error = None
machine.error_code = None
- if body.state == "deleted":
- for launch in await HostedMachineStore().launches(session, machine.id):
- logger.error(
- "Cloud machine %s was deleted while launch %s (state %s) was still on it.",
- machine.id,
- launch.id,
- launch.state,
- )
if machine.state != previous:
machine.updated_at = now
await session.commit()
diff --git a/core/switch_core/gateway/hosted_launches.py b/core/switch_core/gateway/hosted_launches.py
deleted file mode 100644
index b6b46e08c..000000000
--- a/core/switch_core/gateway/hosted_launches.py
+++ /dev/null
@@ -1,861 +0,0 @@
-import asyncio
-import json
-import logging
-from datetime import UTC, datetime
-from typing import Annotated, Literal, Self, cast
-from uuid import UUID, uuid4
-
-from fastapi import APIRouter, Depends, HTTPException, Request
-from fastapi.responses import JSONResponse
-from pydantic import BaseModel, ConfigDict, Field, model_validator
-from sqlalchemy import delete, select
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.addressing import AddressingPolicy
-from switch_core.agent_display_name import normalise_display_name
-from switch_core.agent_icon import normalise_icon_url
-from switch_core.bridges.agent.api.hosted_worker_routes import (
- post_mailbox_notices,
- post_room_notice,
-)
-from switch_core.bridges.agent.protocol.agent_core import AgentCore, AgentExistsError
-from switch_core.config import SwitchConfig
-from switch_core.db.models import (
- Agent,
- ApiKey,
- GitHubIssuedToken,
- HostedLaunch,
- HostedMachine,
- HostedOperation,
- ProviderConnection,
- User,
- require_tenant_id,
-)
-from switch_core.db.session_scope import tenant_session
-from switch_core.db.stores.agent_store import AgentStore
-from switch_core.db.stores.hosted_launch_store import (
- HostedLaunchConflict,
- HostedLaunchStore,
-)
-from switch_core.db.stores.hosted_machine_store import (
- HostedMachineConflict,
- HostedMachineStore,
- idle_sleeping,
- machine_starting,
- owner_stopped,
-)
-from switch_core.db.stores.hosted_mailbox_store import (
- HostedMailboxStore,
- MailboxNotice,
- one_per_room,
-)
-from switch_core.db.stores.provider_connection_store import (
- ProviderConnectionBusy,
- ProviderConnectionStore,
-)
-from switch_core.gateway.auth import get_current_user
-from switch_core.gateway.dependencies import get_config, get_protocol, get_session
-from switch_core.gateway.github_connections import connection_status
-from switch_core.gateway.github_connections import service as get_github
-from switch_core.gateway.known_agents import KNOWN_AGENTS
-from switch_core.gateway.provider_connections import get_verifier
-from switch_core.providers.claude_verifier import (
- ClaudeVerificationError,
- ClaudeVerifier,
-)
-from switch_core.providers.github import GitHubConnections, repository_writable
-from switch_core.providers.github_revocations import (
- ACCESS_WARNING,
- queue_revocation,
- revoke_pending,
-)
-from switch_core.providers.hosted import HostedControllerSettings
-
-logger = logging.getLogger(__name__)
-
-router = APIRouter(prefix="/hosted-launches")
-
-IDENTITY_FAILED = (
- "Switch could not create this cloud agent's identity. Retry it to try again."
-)
-MACHINE_ERROR = "The cloud machine needs attention. Retry it in Switch Console."
-MACHINE_ERROR_NEEDS_ADMIN = (
- "The cloud machine needs attention. Contact your server administrator."
-)
-MACHINE_STOPPED = "The owner stopped the cloud machine. Start it in Switch Console."
-WORKER_WAKING = "The cloud machine is starting. Try again in a moment."
-
-
-def coded_conflict(code: str, message: str) -> JSONResponse:
- """A 409 whose `detail` is the message and whose `code` names the refusal."""
- return JSONResponse(status_code=409, content={"detail": message, "code": code})
-
-
-def machine_error_detail(machine: HostedMachine) -> str:
- """Why an errored machine refuses work, and whether retrying it can help."""
- if machine.error_code == "machine_needs_attention":
- return MACHINE_ERROR_NEEDS_ADMIN
- return MACHINE_ERROR
-
-
-LAUNCH_DISABLED = "Cloud agent launch is not enabled on this server."
-
-
-def hosted_settings(request: Request) -> HostedControllerSettings | None:
- return cast(
- HostedControllerSettings | None, request.app.state.hosted_controller_settings
- )
-
-
-def controller_settings(request: Request) -> HostedControllerSettings:
- settings = hosted_settings(request)
- if settings is None:
- raise HTTPException(503, LAUNCH_DISABLED)
- return settings
-
-
-def launch_enabled(config: SwitchConfig, settings: HostedControllerSettings) -> bool:
- """Whether the bound tenant may claim cloud machines on this server."""
- return (
- config.hosted_launch_capacity > 0 and settings.tenant_id == require_tenant_id()
- )
-
-
-class LaunchRequest(BaseModel):
- model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
- request_id: UUID
- name: str = Field(pattern=r"^[a-z0-9][a-z0-9._-]{0,127}$")
- description: str = Field(min_length=1, max_length=4096)
- display_name: str | None
- icon_url: str | None
- instructions: str = Field(max_length=32768)
- provider: Literal["claude", "codex", "opencode", "cursor", "antigravity"] = "claude"
- definition: str = Field(max_length=65536)
- installation_id: int = Field(gt=0, strict=True)
- repository_id: int = Field(gt=0, strict=True)
- definition_attributes: dict
- auto_session: bool
- auto_approve: bool
- addressing_policy: AddressingPolicy | None
-
- @model_validator(mode="after")
- def bounded_spec(self) -> Self:
- if len(json.dumps(self.model_dump(mode="json")).encode()) > 32768:
- raise ValueError("Cloud agent configuration must fit within 32 KiB.")
- return self
-
-
-def summary(launch: HostedLaunch, machine: HostedMachine | None) -> dict:
- return {
- "request_id": launch.id,
- "name": launch.name,
- "provider": launch.spec.get("provider", "claude"),
- "state": launch.state,
- "agent_id": launch.agent_id,
- "error": launch.error,
- "error_code": launch.error_code,
- "desired_state": launch.desired_state,
- "revision": launch.revision,
- "sleeping": machine is not None and idle_sleeping(machine),
- "machine_id": launch.machine_id,
- "process_state": launch.process_state,
- "process_restarts": launch.process_restarts,
- "oom_kills": launch.process_oom_kills,
- }
-
-
-async def launch_summary(session: AsyncSession, launch: HostedLaunch) -> dict:
- machine = (
- None
- if launch.machine_id is None
- else await HostedMachineStore().get(session, launch.machine_id)
- )
- return summary(launch, machine)
-
-
-def worktree_path(agent_id: str, repository: str | None) -> str:
- """The agent's worktree on its machine's disk, which is also its `repo_dir`."""
- if repository is None:
- return f"/data/worktrees/{agent_id}/workspace"
- return f"/data/worktrees/{agent_id}/{repository.lower()}"
-
-
-async def locked_owned(
- session: AsyncSession, request_id: str, owner_id: str
-) -> tuple[HostedLaunch, HostedMachine]:
- """The owner's launch and its machine, locked machine first, then launch."""
- launch, machine = await HostedMachineStore().locked_launch(session, request_id)
- if launch is None or launch.owner_id != owner_id:
- raise HTTPException(404, "Cloud launch not found.")
- if machine is None:
- raise RuntimeError(f"Cloud launch {launch.id} has no machine.")
- return launch, machine
-
-
-async def owned_launches(
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> list[dict]:
- rows = list(
- await session.scalars(
- select(HostedLaunch)
- .where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.owner_id == user.id,
- )
- .order_by(HostedLaunch.created_at)
- )
- )
- machine_ids = {row.machine_id for row in rows if row.machine_id is not None}
- machines = {
- machine.id: machine
- for machine in await session.scalars(
- select(HostedMachine).where(
- HostedMachine.tenant_id == require_tenant_id(),
- HostedMachine.id.in_(machine_ids),
- )
- )
- }
- return [
- summary(row, None if row.machine_id is None else machines[row.machine_id])
- for row in rows
- ]
-
-
[email protected]("/{request_id}")
-async def status(
- request_id: UUID,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict:
- launch = await HostedLaunchStore().owned(session, str(request_id), user.id)
- if launch is None:
- raise HTTPException(404, "Cloud launch not found.")
- return await launch_summary(session, launch)
-
-
-class ConfigurationRequest(BaseModel):
- model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
- instructions: str = Field(max_length=32768)
- definition: str = Field(max_length=65536)
- definition_attributes: dict
-
-
-def configuration(spec: dict) -> dict:
- return {
- "description": spec["description"],
- "instructions": spec["instructions"],
- "definition_attributes": spec["definition_attributes"],
- }
-
-
[email protected]("/{request_id}/configuration")
-async def get_configuration(
- request_id: UUID,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict:
- launch = await HostedLaunchStore().owned(session, str(request_id), user.id)
- if launch is None:
- raise HTTPException(404, "Cloud launch not found.")
- return configuration(launch.spec)
-
-
[email protected]("/{request_id}/configuration")
-async def update_configuration(
- request_id: UUID,
- body: ConfigurationRequest,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict:
- """Replace the launch's instructions and definition; the agent's next start runs them.
-
- The revision is not bumped: a running agent keeps its deployment until a
- lifecycle action issues a new revision, which the machine applies from
- this spec.
- """
- launch, _machine = await locked_owned(session, str(request_id), user.id)
- if launch.desired_state == "deleted":
- raise HTTPException(409, "This worker has been removed.")
- if launch.spec.get("provider", "claude") == "claude" and not body.definition:
- raise HTTPException(422, "Claude Code requires an agent definition.")
- changes = body.model_dump(mode="json")
- spec = {**launch.spec, **changes}
- bounded = {key: value for key, value in spec.items() if key != "session_limit"}
- if len(json.dumps(bounded).encode()) > 32768:
- raise HTTPException(422, "Cloud agent configuration must fit within 32 KiB.")
- await HostedLaunchStore().merge_spec(session, launch.id, changes)
- await session.commit()
- return configuration(spec)
-
-
-async def _register(protocol: AgentCore, launch: HostedLaunch, agent_id: str) -> None:
- provider = launch.spec.get("provider", "claude")
- known_type = "claude-code" if provider == "claude" else provider
- known = KNOWN_AGENTS[known_type]
- options = known.parse_options(
- {
- "channels_enabled": True,
- "repo_dir": worktree_path(agent_id, launch.repository),
- "auto_session": launch.spec["auto_session"],
- }
- )
- await protocol.register_agent(
- reserved_agent_id=agent_id,
- name=launch.name,
- description=launch.spec["description"],
- display_name=launch.spec["display_name"],
- icon_url=launch.spec["icon_url"],
- connector_type=known.connector_type,
- integration_profile=known.build_profile(options),
- tools=known.tools,
- models=known.models,
- metadata={
- "known_agent_type": known_type,
- "known_agent_options": options.model_dump(),
- "hosted_launch_id": launch.id,
- },
- owner_id=launch.owner_id,
- )
-
-
-async def _identity_failed(
- session: AsyncSession,
- protocol: AgentCore,
- request_id: str,
- agent_id: str,
- error: str,
-) -> HostedLaunch:
- """Mark the launch `identity_failed`, dropping an identity left half registered."""
- await session.rollback()
- machines = HostedMachineStore()
- launch, machine = await machines.locked_launch(session, request_id)
- assert launch is not None and machine is not None
- if await session.get(Agent, agent_id) is not None:
- try:
- await protocol.delete_agent(agent_id=agent_id)
- except Exception:
- logger.error(
- "Cloud launch %s: could not delete the partly registered agent %s",
- request_id,
- agent_id,
- exc_info=True,
- )
- if (
- launch.agent_id == agent_id
- and await session.get(Agent, agent_id, populate_existing=True) is None
- ):
- launch.agent_id = None
- if launch.desired_state != "deleted":
- launch.state = "error"
- launch.error = error
- launch.error_code = "identity_failed"
- launch.updated_at = datetime.now(UTC)
- machines.bump_agents(machine)
- await session.commit()
- return launch
-
-
-async def register_identity(
- session: AsyncSession, protocol: AgentCore, request_id: str
-) -> HostedLaunch:
- """Register the launch's agent identity, unless it already has one.
-
- Core mints the agent id and records it on the launch before registering,
- so the launch's name reservation admits only this identity. A launch that
- has an id but no agent, left by an interrupted registration, registers
- that id again. Registration runs under the machine and launch locks, so
- only one caller registers. A failure leaves the launch in `error` with
- `identity_failed`, which `retry` registers again. Raises 422 when another
- agent holds the name.
- """
- machines = HostedMachineStore()
- launch, machine = await machines.locked_launch(session, request_id)
- assert launch is not None and machine is not None
- if launch.agent_id is None and launch.desired_state != "deleted":
- launch.agent_id = str(uuid4())
- launch.updated_at = datetime.now(UTC)
- await session.commit()
- launch, machine = await machines.locked_launch(session, request_id)
- assert launch is not None and machine is not None
- agent_id = launch.agent_id
- if (
- agent_id is None
- or launch.desired_state == "deleted"
- or await session.get(Agent, agent_id) is not None
- ):
- await session.commit()
- return launch
- try:
- await _register(protocol, launch, agent_id)
- except AgentExistsError:
- await _identity_failed(
- session, protocol, request_id, agent_id, "An agent already uses this name."
- )
- raise HTTPException(422, "An agent already uses this name.") from None
- except Exception:
- logger.error(
- "Cloud launch %s: registering its agent identity failed",
- request_id,
- exc_info=True,
- )
- return await _identity_failed(
- session, protocol, request_id, agent_id, IDENTITY_FAILED
- )
- agent = await session.get(Agent, agent_id, populate_existing=True)
- if agent is None:
- logger.error(
- "Cloud launch %s: agent %s is missing after registration",
- request_id,
- agent_id,
- )
- return await _identity_failed(
- session, protocol, request_id, agent_id, IDENTITY_FAILED
- )
- if launch.spec["addressing_policy"] is not None:
- agent.addressing_policy = launch.spec["addressing_policy"]
- launch.updated_at = datetime.now(UTC)
- machines.bump_agents(machine)
- await session.commit()
- return launch
-
-
-class LifecycleRequest(BaseModel):
- model_config = ConfigDict(extra="forbid")
- action: Literal["stop", "start", "restart", "remove", "retry"]
- revision: int = Field(ge=1)
-
-
-def ring_mailbox_cancel(
- protocol: AgentCore, agent_id: str, entries: list[tuple[str, str]]
-) -> None:
- protocol.connections.ring_worker(
- agent_id,
- "mailbox_cancel",
- {
- "entries": [
- {"room_id": room_id, "message_id": message_id}
- for room_id, message_id in entries
- ]
- },
- )
-
-
[email protected]("/{request_id}/lifecycle", response_model=None)
-async def lifecycle(
- request_id: UUID,
- body: LifecycleRequest,
- config: Annotated[SwitchConfig, Depends(get_config)],
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict | JSONResponse:
- launch, machine = await locked_owned(session, str(request_id), user.id)
- if launch.revision != body.revision:
- raise HTTPException(
- 409, "The worker changed. Refresh its status before trying again."
- )
- resuming_remove = body.action == "remove" and launch.state == "deleting"
- if launch.desired_state == "deleted" and not resuming_remove:
- raise HTTPException(409, "This worker has been removed.")
- if body.action == "restart" and launch.state != "ready":
- raise HTTPException(409, "Only a ready worker can be restarted.")
- if body.action == "retry" and launch.state != "error":
- raise HTTPException(409, "Only a worker in error can be retried.")
- if body.action in ("start", "restart", "retry") and machine.state == "error":
- return coded_conflict("machine_error", machine_error_detail(machine))
- if body.action in ("restart", "retry") and owner_stopped(machine):
- return coded_conflict("machine_stopped", MACHINE_STOPPED)
- if body.action == "remove":
- return await remove(session, protocol, config, launch, machine)
- machines = HostedMachineStore()
- now = datetime.now(UTC)
- split = None
- if body.action == "stop":
- stopped = (
- launch.state == "stopped"
- or launch.agent_id is None
- or machine.desired_state == "stopped"
- )
- launch.desired_state = "stopped"
- launch.state = "stopped" if stopped else "stopping"
- split = await HostedMailboxStore().stop(session, launch.id)
- else:
- launch.desired_state = "running"
- launch.state = "queued"
- if machine.desired_state == "stopped":
- machines.start(machine, now)
- if body.action == "retry":
- launch.process_restarts = 0
- launch.process_oom_kills = 0
- launch.error = None
- launch.error_code = None
- launch.revision += 1
- launch.updated_at = now
- await HostedLaunchStore().fail_stale_operations(session, launch.id, launch.revision)
- await queue_revocation(session, (GitHubIssuedToken.launch_id == launch.id,))
- machines.bump_agents(machine)
- await session.commit()
- if launch.agent_id:
- if split is not None and split.cancel_requested:
- ring_mailbox_cancel(protocol, launch.agent_id, split.cancel_requested)
- protocol.connections.supersede(launch.agent_id, launch.revision)
- if split is not None:
- await post_mailbox_notices(protocol, split.cancelled)
- remaining = await revoke_pending(
- session, config, (GitHubIssuedToken.launch_id == launch.id,)
- )
- if launch.desired_state == "running":
- launch = await register_identity(session, protocol, launch.id)
- response = await launch_summary(session, launch)
- return {**response, "access_warning": ACCESS_WARNING if remaining else None}
-
-
-async def post_removed_notices(
- protocol: AgentCore, agent: Agent, notices: list[MailboxNotice]
-) -> None:
- """Tell each room its queued messages will not run, while the agent can still post."""
- for notice in one_per_room(notices):
- try:
- await post_room_notice(
- protocol,
- agent,
- notice.room_id,
- notice.message_id,
- notice.thread_id,
- "removed",
- )
- except Exception:
- logger.warning(
- "Cloud agent %s: removed notice for room %s was not posted",
- agent.id,
- notice.room_id,
- exc_info=True,
- )
-
-
-async def remove(
- session: AsyncSession,
- protocol: AgentCore,
- config: SwitchConfig,
- launch: HostedLaunch,
- machine: HostedMachine,
-) -> dict:
- """Remove a launch and its agent identity, synchronously, in any state.
-
- Two commits under the machine and launch locks: the first records the
- removal and what to clean up, and drops the queued mail; the second
- deletes the identity and retains the disk when no agent is left. A crash
- between them leaves the launch `deleting`; `remove` on it again, or the
- controller sweep, finishes the cleanup.
- """
- machines = HostedMachineStore()
- now = datetime.now(UTC)
- cancelled: list[MailboxNotice] = []
- cancel_requested: list[tuple[str, str]] = []
- if launch.desired_state != "deleted":
- launch.desired_state = "deleted"
- launch.revision += 1
- await HostedLaunchStore().fail_stale_operations(
- session, launch.id, launch.revision
- )
- await queue_revocation(session, (GitHubIssuedToken.launch_id == launch.id,))
- mailbox = HostedMailboxStore()
- split = await mailbox.stop(session, launch.id)
- cancelled, cancel_requested = split.cancelled, split.cancel_requested
- await mailbox.delete_launch(session, launch.id)
- await AgentStore().lock_name(session, launch.name)
- agent = await session.get(Agent, launch.agent_id) if launch.agent_id else None
- if agent is not None:
- if (
- agent.owner_id != launch.owner_id
- or (agent.metadata_ or {}).get("hosted_launch_id") != launch.id
- ):
- raise HTTPException(
- 409, "The removed worker identity belongs to another agent."
- )
- launch.deletion_cleanup = {
- "client_id": agent.client_id,
- "key_id": agent.api_key_id,
- }
- launch.state = "deleting"
- launch.error = None
- launch.error_code = None
- launch.updated_at = now
- machines.bump_agents(machine)
- await session.commit()
-
- if launch.agent_id:
- if cancel_requested:
- ring_mailbox_cancel(protocol, launch.agent_id, cancel_requested)
- protocol.connections.supersede(launch.agent_id, launch.revision)
- if agent is not None:
- await post_removed_notices(protocol, agent, cancelled)
-
- launch, machine = await locked_owned(session, launch.id, launch.owner_id)
- await finish_removal(session, protocol, config, launch, machine, now)
- await session.commit()
- remaining = await revoke_pending(
- session, config, (GitHubIssuedToken.launch_id == launch.id,)
- )
- response = summary(launch, machine)
- return {**response, "access_warning": ACCESS_WARNING if remaining else None}
-
-
-async def finish_removal(
- session: AsyncSession,
- protocol: AgentCore,
- config: SwitchConfig,
- launch: HostedLaunch,
- machine: HostedMachine,
- now: datetime,
-) -> None:
- """Delete a `deleting` launch's identity and mark it `deleted`.
-
- The caller holds the machine and launch locks and commits. Safe to run
- again after an interruption.
- """
- machines = HostedMachineStore()
- await AgentStore().lock_name(session, launch.name)
- if launch.agent_id and await session.get(Agent, launch.agent_id) is not None:
- await protocol.delete_agent(agent_id=launch.agent_id)
- if launch.deletion_cleanup:
- client_id = launch.deletion_cleanup["client_id"]
- await protocol.client_lifecycle.stop(client_id)
- await protocol.client_lifecycle.delete_record(session, client_id)
- if launch.agent_id:
- protocol.api_key_cache.invalidate_agent(launch.agent_id)
- await session.execute(
- delete(ApiKey).where(
- ApiKey.tenant_id == require_tenant_id(),
- ApiKey.id == launch.deletion_cleanup["key_id"],
- )
- )
- launch.deletion_cleanup = None
- launch.state = "deleted"
- launch.name = "removed:" + launch.id
- launch.updated_at = now
- machines.bump_agents(machine)
- await machines.retain_if_empty(
- session, machine, retention_days=config.hosted_disk_retention_days, now=now
- )
-
-
-def operation_summary(operation: HostedOperation) -> dict:
- return {
- "id": operation.id,
- "session_id": operation.session_id,
- "action": operation.action,
- "state": operation.state,
- "error": operation.error,
- }
-
-
-class SessionOperationRequest(BaseModel):
- model_config = ConfigDict(extra="forbid")
- id: UUID
- session_id: UUID
- action: Literal["start", "restart"]
-
-
-OPERATION_RERING_SECONDS = 5
-OPERATION_RERINGS = 6
-_doorbells: set[asyncio.Task[None]] = set()
-
-
-async def ring_operation(
- protocol: AgentCore, tenant_id: str, agent_id: str, operation_id: str
-) -> None:
- """Ring the worker for a queued operation until it is claimed or the rings run out."""
- protocol.connections.ring_worker(agent_id, "operation", {"id": operation_id})
- for _ in range(OPERATION_RERINGS):
- await asyncio.sleep(OPERATION_RERING_SECONDS)
- async with tenant_session(protocol.session_factory, tenant_id) as session:
- operation = await session.get(HostedOperation, (tenant_id, operation_id))
- if operation is None or operation.state != "queued":
- return
- protocol.connections.ring_worker(agent_id, "operation", {"id": operation_id})
-
-
[email protected]("/{request_id}/sessions", status_code=202, response_model=None)
-async def session_operation(
- request_id: UUID,
- body: SessionOperationRequest,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict | JSONResponse:
- launch, machine = await locked_owned(session, str(request_id), user.id)
- existing = await session.get(HostedOperation, (require_tenant_id(), str(body.id)))
- if existing:
- if (
- existing.launch_id != launch.id
- or existing.action != body.action
- or existing.session_id != str(body.session_id)
- ):
- raise HTTPException(
- 409, "This operation ID was already used for different details."
- )
- return operation_summary(existing)
- if machine.state == "error":
- return coded_conflict("machine_error", machine_error_detail(machine))
- if owner_stopped(machine):
- return coded_conflict("machine_stopped", MACHINE_STOPPED)
- if (
- idle_sleeping(machine)
- and launch.desired_state == "running"
- and launch.state != "error"
- ):
- now = datetime.now(UTC)
- HostedMachineStore().start(machine, now)
- launch.active_at = now
- await session.commit()
- return coded_conflict("worker_waking", WORKER_WAKING)
- if machine_starting(machine):
- return coded_conflict("worker_waking", WORKER_WAKING)
- if (
- launch.state not in ("ready", "running")
- or launch.desired_state != "running"
- or launch.agent_id is None
- ):
- raise HTTPException(409, "Start the cloud worker and wait until it is ready.")
- pending = await session.scalar(
- select(HostedOperation.id).where(
- HostedOperation.tenant_id == require_tenant_id(),
- HostedOperation.launch_id == launch.id,
- HostedOperation.session_id == str(body.session_id),
- HostedOperation.state.in_(["queued", "claimed"]),
- )
- )
- if pending is not None:
- raise HTTPException(409, "This session already has a pending operation.")
- operation = HostedOperation(
- id=str(body.id),
- launch_id=launch.id,
- launch_revision=launch.revision,
- session_id=str(body.session_id),
- action=body.action,
- )
- session.add(operation)
- await session.commit()
- task = asyncio.create_task(
- ring_operation(protocol, require_tenant_id(), launch.agent_id, operation.id)
- )
- _doorbells.add(task)
- task.add_done_callback(_doorbells.discard)
- return operation_summary(operation)
-
-
[email protected]("/{request_id}/sessions/{operation_id}")
-async def session_operation_status(
- request_id: UUID,
- operation_id: UUID,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict:
- launch = await HostedLaunchStore().owned(session, str(request_id), user.id)
- operation = await session.get(
- HostedOperation, (require_tenant_id(), str(operation_id))
- )
- if launch is None or operation is None or operation.launch_id != launch.id:
- raise HTTPException(404, "Cloud operation not found.")
- return operation_summary(operation)
-
-
[email protected]("", status_code=202)
-async def create(
- body: LaunchRequest,
- user: Annotated[User, Depends(get_current_user)],
- settings: Annotated[HostedControllerSettings, Depends(controller_settings)],
- session: Annotated[AsyncSession, Depends(get_session)],
- config: Annotated[SwitchConfig, Depends(get_config)],
- verifier: Annotated[ClaudeVerifier, Depends(get_verifier)],
- github: Annotated[GitHubConnections, Depends(get_github)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict:
- if not launch_enabled(config, settings):
- raise HTTPException(503, LAUNCH_DISABLED)
- try:
- body.icon_url = normalise_icon_url(body.icon_url)
- body.display_name = normalise_display_name(body.display_name)
- except ValueError as error:
- raise HTTPException(422, str(error)) from None
- spec = body.model_dump(mode="json", exclude={"request_id"})
- spec["session_limit"] = config.hosted_sessions_per_agent
- store = HostedLaunchStore()
- existing = await store.owned(session, str(body.request_id), user.id)
- if existing:
- previous = {"provider": "claude", **existing.spec}
- previous.pop("session_limit", None)
- requested = {
- key: value for key, value in spec.items() if key != "session_limit"
- }
- if previous != requested:
- raise HTTPException(
- 409, "This launch request was already used for different agent details."
- )
- if existing.state != "error" and existing.desired_state != "deleted":
- existing = await register_identity(session, protocol, existing.id)
- return await launch_summary(session, existing)
- connections = ProviderConnectionStore()
- try:
- await connections.lock_user(session, user.id)
- except ProviderConnectionBusy as error:
- raise HTTPException(409, str(error)) from None
- connection = await session.get(
- ProviderConnection, (require_tenant_id(), user.id, body.provider)
- )
- if connection is None:
- raise HTTPException(
- 422, "Connect the selected provider before creating a cloud agent."
- )
- if body.provider == "claude":
- if not body.definition:
- raise HTTPException(422, "Claude Code requires an agent definition.")
- try:
- await verifier.verify(
- connection.kind,
- config.keyring.decrypt(connection.encrypted_credential),
- )
- except ClaudeVerificationError as error:
- raise HTTPException(422, str(error)) from None
- await session.commit()
- access = await connection_status(user.id, session, config, github)
- repository = next(
- (
- repo
- for installation in access.get("installations", [])
- if installation["id"] == body.installation_id
- for repo in installation["repositories"]
- if repo["id"] == body.repository_id
- ),
- None,
- )
- if repository is None:
- raise HTTPException(
- 422, "Your GitHub account no longer has access to the selected repository."
- )
- if not repository_writable(repository):
- raise HTTPException(
- 422,
- "Your GitHub account needs write access to this repository to run a cloud agent.",
- )
- try:
- launch = await store.reserve(
- session,
- request_id=str(body.request_id),
- owner_id=user.id,
- name=body.name,
- spec=spec,
- capacity=config.hosted_launch_capacity,
- owner_capacity=config.hosted_agents_per_owner,
- slots=list(settings.machine_slots),
- repository=repository["name"],
- now=datetime.now(UTC),
- )
- except (HostedLaunchConflict, HostedMachineConflict) as error:
- raise HTTPException(409, str(error)) from None
- await session.commit()
- launch = await register_identity(session, protocol, launch.id)
- return await launch_summary(session, launch)
diff --git a/core/switch_core/gateway/hosted_machines.py b/core/switch_core/gateway/hosted_machines.py
index 05dd9d6af..0c97b37b4 100644
--- a/core/switch_core/gateway/hosted_machines.py
+++ b/core/switch_core/gateway/hosted_machines.py
@@ -1,39 +1,51 @@
from datetime import UTC, datetime
-from typing import Annotated, Literal
+from typing import Annotated, Literal, cast
-from fastapi import APIRouter, Depends, HTTPException
+from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
-from switch_core.bridges.agent.api.hosted_worker_routes import post_mailbox_notices
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.config import SwitchConfig
from switch_core.db.models import HostedMachine, User, require_tenant_id
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
from switch_core.db.stores.hosted_machine_store import (
HostedMachineConflict,
HostedMachineStore,
claim_conflict,
idle_sleeping,
- lock_launches,
+ lock_claims,
owner_stopped,
)
-from switch_core.db.stores.hosted_mailbox_store import HostedMailboxStore, MailboxNotice
from switch_core.gateway.auth import get_current_user
-from switch_core.gateway.dependencies import get_config, get_protocol, get_session
-from switch_core.gateway.hosted_launches import (
- LAUNCH_DISABLED,
- hosted_settings,
- launch_enabled,
- ring_mailbox_cancel,
-)
+from switch_core.gateway.dependencies import get_config, get_session
from switch_core.providers.hosted import HostedControllerSettings
router = APIRouter(prefix="/hosted-machines")
RETIRED_STATES = frozenset({"retained", "deleting", "deleted"})
+MACHINES_DISABLED = "Switch cloud machines are not enabled on this server."
+
+
+def hosted_settings(request: Request) -> HostedControllerSettings | None:
+ return cast(
+ HostedControllerSettings | None, request.app.state.hosted_controller_settings
+ )
+
+
+def controller_settings(request: Request) -> HostedControllerSettings:
+ settings = hosted_settings(request)
+ if settings is None:
+ raise HTTPException(503, MACHINES_DISABLED)
+ return settings
+
+
+def machines_enabled(config: SwitchConfig, settings: HostedControllerSettings) -> bool:
+ """Whether the bound tenant may claim cloud machines on this server."""
+ return (
+ config.hosted_launch_capacity > 0 and settings.tenant_id == require_tenant_id()
+ )
+
def _usage(heartbeat: dict | None, key: str) -> dict | None:
reading = None if heartbeat is None else heartbeat.get(key)
@@ -46,7 +58,9 @@ def _usage(heartbeat: dict | None, key: str) -> dict | None:
async def machine_summary(session: AsyncSession, machine: HostedMachine) -> dict:
- launches = await HostedMachineStore().launches(session, machine.id)
+ """The machine as its owner sees it: `agents` are the managed agents
+ placed on the controller it enrolled as, `controller_id` (null until it
+ has)."""
return {
"machine_id": machine.id,
"state": machine.state,
@@ -65,7 +79,8 @@ async def machine_summary(session: AsyncSession, machine: HostedMachine) -> dict
else machine.heartbeat_at.isoformat(),
"disk": _usage(machine.heartbeat, "disk"),
"memory": _usage(machine.heartbeat, "memory"),
- "agents": [launch.id for launch in launches],
+ "agents": await HostedMachineStore().managed_agent_ids(session, machine),
+ "controller_id": machine.controller_id,
}
@@ -87,10 +102,10 @@ async def ensure_machine(
machines and `HostedMachineConflict` when none can be had. The caller
commits.
"""
- if settings is None or not launch_enabled(config, settings):
- raise MachineUnavailable(LAUNCH_DISABLED)
+ if settings is None or not machines_enabled(config, settings):
+ raise MachineUnavailable(MACHINES_DISABLED)
machines = HostedMachineStore()
- await lock_launches(session)
+ await lock_claims(session)
existing = await machines.live_for_owner(session, owner_id)
if existing is not None:
existing = await machines.locked(session, existing.id)
@@ -102,7 +117,6 @@ async def ensure_machine(
return await machines.claim(
session,
owner_id=owner_id,
- slots=list(settings.machine_slots),
capacity=config.hosted_launch_capacity,
now=datetime.now(UTC),
)
@@ -172,7 +186,6 @@ async def lifecycle(
body: MachineLifecycleRequest,
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
) -> dict:
machines = HostedMachineStore()
await _owned(session, machine_id, user.id)
@@ -192,19 +205,8 @@ async def lifecycle(
):
raise HTTPException(409, f"machine is {machine.desired_state}")
now = datetime.now(UTC)
- cancelled: list[MailboxNotice] = []
- cancel_requested: dict[str, list[tuple[str, str]]] = {}
if body.action == "stop":
machines.stop(machine, "owner", now)
- mailbox = HostedMailboxStore()
- for candidate in await machines.launches(session, machine.id):
- launch = await HostedLaunchStore().locked(session, candidate.id)
- if launch is None or launch.desired_state == "deleted":
- continue
- split = await mailbox.stop(session, launch.id)
- cancelled.extend(split.cancelled)
- if launch.agent_id and split.cancel_requested:
- cancel_requested[launch.agent_id] = split.cancel_requested
elif body.action == "start":
machines.start(machine, now)
else:
@@ -212,7 +214,4 @@ async def lifecycle(
raise HTTPException(409, "Only a machine in error can be retried.")
machines.retry(machine, now)
await session.commit()
- for agent_id, entries in cancel_requested.items():
- ring_mailbox_cancel(protocol, agent_id, entries)
- await post_mailbox_notices(protocol, cancelled)
return {"machine": await machine_summary(session, machine)}
diff --git a/core/switch_core/gateway/hosted_relay.py b/core/switch_core/gateway/hosted_relay.py
deleted file mode 100644
index 08e39f67d..000000000
--- a/core/switch_core/gateway/hosted_relay.py
+++ /dev/null
@@ -1,442 +0,0 @@
-"""Console ⇄ cloud worker relay: one request answered by the worker, and live views.
-
-Owner only. A launch that is not the caller's reads as not found, so its
-existence does not leak. Nothing relayed is written to the database or the log;
-the only durable write is a mutating relay's `relay_seq`.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import json
-import logging
-import time
-from collections.abc import AsyncIterator
-from datetime import UTC, datetime
-from typing import Annotated, Any
-from uuid import UUID
-
-from fastapi import APIRouter, Depends, HTTPException, Query, Request
-from fastapi.responses import JSONResponse, StreamingResponse
-from pydantic import BaseModel, ConfigDict, Field, ValidationError
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnection
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.bridges.agent.protocol.hosted_workers import (
- HEALTH_SUBSCRIPTION,
- RELAY_REQUEST_LIMIT_BYTES,
- RELAY_TIMEOUT_LIMIT_MS,
- ConsoleView,
- PendingRelay,
- RelayError,
- classify_message,
- frame_size,
- subscribe_message,
-)
-from switch_core.bridges.agent.protocol.stream import KEEPALIVE_INTERVAL_SECONDS
-from switch_core.db.models import HostedLaunch, HostedMachine, User, require_tenant_id
-from switch_core.db.session_scope import tenant_session
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore, is_waking
-from switch_core.db.stores.hosted_machine_store import (
- HostedMachineStore,
- idle_sleeping,
- owner_stopped,
-)
-from switch_core.gateway.auth import get_current_user, get_current_user_in_transaction
-from switch_core.gateway.dependencies import get_protocol, get_session
-from switch_core.gateway.hosted_launches import (
- launch_summary,
- machine_error_detail,
- summary,
-)
-
-logger = logging.getLogger(__name__)
-
-router = APIRouter(prefix="/hosted-launches")
-
-#: Room for a relay frame's own fields around the relayed message.
-RELAY_FRAME_ENVELOPE_BYTES = 256
-#: A subscription the stream asks the worker for itself waits this long.
-SUBSCRIPTION_RELAY_TIMEOUT_MS = 30_000
-MAX_STREAM_SUBSCRIPTIONS = 32
-
-_dispatches: set[asyncio.Task[Any]] = set()
-
-
-class RelayRequest(BaseModel):
- model_config = ConfigDict(extra="forbid")
- message: dict[str, Any]
- timeout_ms: int = Field(gt=0, le=RELAY_TIMEOUT_LIMIT_MS)
-
-
-def worker_info(conn: AgentConnection | None) -> dict[str, Any] | None:
- if conn is None or conn.worker is None:
- return None
- return {
- "launch_revision": conn.worker.launch_revision,
- "boot_id": conn.worker.boot_id,
- "generation": conn.stream_generation,
- }
-
-
-def relay_error(error: RelayError, worker: dict[str, Any] | None) -> JSONResponse:
- body: dict[str, Any] = {
- "ok": False,
- "error": {"code": error.code, "message": str(error)},
- "worker": worker,
- }
- if error.code == "worker_sleeping":
- body["wake_available"] = True
- return JSONResponse(status_code=error.status, content=body)
-
-
-def relay_target(
- protocol: AgentCore,
- launch: HostedLaunch,
- machine: HostedMachine | None,
- kind: str,
-) -> AgentConnection:
- """The worker attached for the launch's current revision, or why there is none.
-
- A mutating message to an idle-sleeping machine is the caller's to wake
- first; here it falls through to the launch's own state.
- """
- if machine is not None and machine.state == "error":
- raise RelayError("machine_error", machine_error_detail(machine), 409)
- if machine is not None and owner_stopped(machine):
- raise RelayError("machine_stopped", "The owner stopped the cloud machine.", 409)
- if machine is not None and idle_sleeping(machine) and kind == "read_only":
- raise RelayError("worker_sleeping", "The cloud machine is asleep.", 409)
- if is_waking(launch, machine):
- raise RelayError("worker_waking", "The cloud machine is starting.", 409)
- if launch.desired_state == "stopped":
- raise RelayError("agent_stopped", "The cloud agent is stopped.", 409)
- if launch.state == "error" and launch.error_code == "agent_crashed":
- raise RelayError("agent_crashed", "The cloud agent keeps crashing.", 409)
- conn = (
- protocol.connections.attached_worker(launch.agent_id)
- if launch.agent_id
- else None
- )
- if (
- conn is None
- or conn.worker is None
- or conn.worker.launch_id != launch.id
- or conn.worker.launch_revision != launch.revision
- ):
- raise RelayError(
- "worker_not_attached", "The cloud worker is not connected.", 409
- )
- return conn
-
-
-def relay_frame(
- relay: PendingRelay, message: dict[str, Any], timeout_ms: int
-) -> dict[str, Any]:
- return {
- "id": relay.id,
- "deadline_ms": int(time.time() * 1000) + timeout_ms,
- "relay_seq": relay.relay_seq,
- "message": message,
- }
-
-
-def dispatch_read_only(
- protocol: AgentCore,
- tenant_id: str,
- conn: AgentConnection,
- message: dict[str, Any],
- timeout_ms: int,
-) -> PendingRelay:
- assert conn.worker is not None
- slot = conn.worker_frames.reserve(frame_size(message) + RELAY_FRAME_ENVELOPE_BYTES)
- try:
- relay = protocol.connections.relays.register(
- tenant_id=tenant_id,
- agent_id=conn.agent_id,
- binding=conn.worker,
- relay_seq=None,
- core_boot=protocol.event_buffer.boot,
- connection_id=conn.id,
- generation=conn.stream_generation,
- timeout_ms=timeout_ms,
- )
- except BaseException:
- slot.release()
- raise
- slot.put("relay", relay_frame(relay, message, timeout_ms))
- return relay
-
-
-async def dispatch_mutating(
- protocol: AgentCore,
- tenant_id: str,
- launch_id: str,
- message: dict[str, Any],
- timeout_ms: int,
-) -> PendingRelay:
- """Sequence and send a mutating relay. Runs to completion even if the caller leaves.
-
- The slot is reserved before the sequence is taken, so a full queue refuses
- without consuming one; once committed, the frame always goes out.
- """
- async with tenant_session(protocol.session_factory, tenant_id) as session:
- machines = HostedMachineStore()
- launch, machine = await machines.locked_launch(session, launch_id)
- if launch is None or launch.desired_state == "deleted":
- raise RelayError("worker_not_attached", "The cloud launch is gone.", 409)
- now = datetime.now(UTC)
- if (
- machine is not None
- and idle_sleeping(machine)
- and machine.state != "error"
- and launch.desired_state == "running"
- and launch.state != "error"
- ):
- machines.start(machine, now)
- launch.active_at = now
- await session.commit()
- raise RelayError("worker_waking", "The cloud machine is starting.", 409)
- conn = relay_target(protocol, launch, machine, "mutating")
- assert conn.worker is not None
- slot = conn.worker_frames.reserve(
- frame_size(message) + RELAY_FRAME_ENVELOPE_BYTES
- )
- relay: PendingRelay | None = None
- try:
- launch.relay_seq += 1
- launch.active_at = now
- if machine is not None:
- machine.active_at = now
- relay = protocol.connections.relays.register(
- tenant_id=tenant_id,
- agent_id=conn.agent_id,
- binding=conn.worker,
- relay_seq=launch.relay_seq,
- core_boot=protocol.event_buffer.boot,
- connection_id=conn.id,
- generation=conn.stream_generation,
- timeout_ms=timeout_ms,
- )
- await session.commit()
- except BaseException:
- slot.release()
- if relay is not None:
- protocol.connections.relays.unregister(relay)
- raise
- slot.put("relay", relay_frame(relay, message, timeout_ms))
- return relay
-
-
-async def owned_launch(
- session: AsyncSession, request_id: UUID, user: User
-) -> HostedLaunch:
- launch = await HostedLaunchStore().owned(session, str(request_id), user.id)
- if launch is None or launch.desired_state == "deleted":
- raise HTTPException(404, "Cloud launch not found.")
- return launch
-
-
-async def current_summary(
- protocol: AgentCore, tenant_id: str, launch_id: str
-) -> dict[str, Any] | None:
- """The launch summary as committed, after a mutating dispatch may have woken it."""
- async with tenant_session(protocol.session_factory, tenant_id) as session:
- launch = await session.get(HostedLaunch, (tenant_id, launch_id))
- return None if launch is None else await launch_summary(session, launch)
-
-
-async def request_body(request: Request) -> bytes:
- """The body, read ahead of the caller's user.
-
- `relay` takes the user with its transaction left open, so the body has to
- be in hand before that transaction starts: from the user read to the
- rollback nothing waits on the client.
- """
- return await request.body()
-
-
[email protected]("/{request_id}/relay", response_model=None)
-async def relay(
- request_id: UUID,
- raw: Annotated[bytes, Depends(request_body)],
- user: Annotated[User, Depends(get_current_user_in_transaction)],
- session: Annotated[AsyncSession, Depends(get_session)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> JSONResponse:
- if len(raw) > RELAY_REQUEST_LIMIT_BYTES:
- return relay_error(
- RelayError("too_large", "A relay request is at most 2 MiB.", 413), None
- )
- try:
- body = RelayRequest.model_validate_json(raw)
- except ValidationError as exc:
- raise HTTPException(422, exc.errors(include_input=False)) from exc
- launch = await owned_launch(session, request_id, user)
- machine = (
- None
- if launch.machine_id is None
- else await HostedMachineStore().get(session, launch.machine_id)
- )
- worker = summary(launch, machine)
- tenant_id = require_tenant_id()
- try:
- kind = classify_message(body.message)
- if kind == "read_only":
- conn = relay_target(protocol, launch, machine, kind)
- pending = dispatch_read_only(
- protocol, tenant_id, conn, body.message, body.timeout_ms
- )
- await session.rollback()
- else:
- launch_id = launch.id
- await session.rollback()
- task = asyncio.create_task(
- dispatch_mutating(
- protocol, tenant_id, launch_id, body.message, body.timeout_ms
- )
- )
- _dispatches.add(task)
- task.add_done_callback(_dispatches.discard)
- try:
- pending = await asyncio.shield(task)
- except RelayError as error:
- return relay_error(
- error, await current_summary(protocol, tenant_id, launch_id)
- )
- answer = await asyncio.shield(pending.future)
- except RelayError as error:
- return relay_error(error, worker)
- return JSONResponse(
- content={
- **answer,
- "worker": {
- "launch_revision": pending.launch_revision,
- "boot_id": pending.boot_id,
- "generation": pending.generation,
- },
- }
- )
-
-
-def ask_worker(
- protocol: AgentCore,
- tenant_id: str,
- conn: AgentConnection,
- subscription: str,
- on: bool,
-) -> None:
- """Subscribe or unsubscribe the worker for Console views; the answer is not awaited.
-
- A subscribe the worker's queue cannot take is shown on every view of it and
- asked again on the next pass. An unsubscribe that cannot be sent is not
- lost either: the worker's next push for it is answered with `unsubscribe`.
- """
- try:
- dispatch_read_only(
- protocol,
- tenant_id,
- conn,
- subscribe_message(subscription, on),
- SUBSCRIPTION_RELAY_TIMEOUT_MS,
- )
- except RelayError as error:
- if on:
- protocol.connections.relay_views.subscribe_failed(
- conn.agent_id, subscription, error
- )
- return
- logger.warning(
- "Could not close worker subscription for agent %s: %s; the worker's next push is refused instead",
- conn.agent_id,
- error.code,
- )
-
-
-def stream_frame(event: str, data: dict[str, Any]) -> bytes:
- return (
- f"event: {event}\ndata: {json.dumps(data, separators=(',', ':'))}\n\n".encode()
- )
-
-
-NO_WORKER = {"launch_revision": None, "boot_id": None, "generation": None}
-
-
-def launch_worker(
- protocol: AgentCore, agent_id: str, launch_id: str
-) -> AgentConnection | None:
- conn = protocol.connections.attached_worker(agent_id)
- if conn is None or conn.worker is None or conn.worker.launch_id != launch_id:
- return None
- return conn
-
-
-async def relay_events(
- protocol: AgentCore,
- tenant_id: str,
- agent_id: str,
- launch_id: str,
- view: ConsoleView,
-) -> AsyncIterator[bytes]:
- views = protocol.connections.relay_views
- views.open(agent_id, view)
- sent: dict[str, Any] | None = None
- last_write = time.monotonic()
- try:
- while True:
- conn = launch_worker(protocol, agent_id, launch_id)
- info = worker_info(conn) or NO_WORKER
- if info != sent:
- sent = info
- last_write = time.monotonic()
- yield stream_frame("worker", info)
- if conn is not None:
- for name in views.unsubscribed(agent_id, conn.stream_generation):
- ask_worker(protocol, tenant_id, conn, name, True)
- view.wake.clear()
- for event, data in view.drain():
- last_write = time.monotonic()
- yield stream_frame(event, data)
- try:
- await asyncio.wait_for(view.wake.wait(), timeout=1.0)
- except TimeoutError:
- pass
- if time.monotonic() - last_write >= KEEPALIVE_INTERVAL_SECONDS:
- last_write = time.monotonic()
- yield b": keepalive\n\n"
- finally:
- conn = launch_worker(protocol, agent_id, launch_id)
- for name, generation in views.close(agent_id, view):
- if conn is not None and generation == conn.stream_generation:
- ask_worker(protocol, tenant_id, conn, name, False)
-
-
[email protected]("/{request_id}/relay/stream", response_model=None)
-async def relay_stream(
- request_id: UUID,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
- subscribe: Annotated[list[UUID] | None, Query()] = None,
- watch_health: Annotated[bool, Query(alias="watchHealth")] = False,
-) -> StreamingResponse:
- launch = await owned_launch(session, request_id, user)
- if launch.agent_id is None:
- raise HTTPException(404, "Cloud launch not found.")
- names = {str(session_id) for session_id in subscribe or []}
- if len(names) > MAX_STREAM_SUBSCRIPTIONS:
- raise HTTPException(422, "Too many subscriptions on one relay stream.")
- if watch_health:
- names.add(HEALTH_SUBSCRIPTION)
- tenant_id = require_tenant_id()
- agent_id = launch.agent_id
- launch_id = launch.id
- await session.rollback()
- return StreamingResponse(
- relay_events(
- protocol, tenant_id, agent_id, launch_id, ConsoleView(frozenset(names))
- ),
- media_type="text/event-stream",
- headers={"Cache-Control": "no-store", "X-Accel-Buffering": "no"},
- )
diff --git a/core/switch_core/gateway/provider_connections.py b/core/switch_core/gateway/provider_connections.py
deleted file mode 100644
index 3cc855ee8..000000000
--- a/core/switch_core/gateway/provider_connections.py
+++ /dev/null
@@ -1,322 +0,0 @@
-import json
-from datetime import UTC, datetime
-from typing import Annotated, Literal, cast
-
-from fastapi import APIRouter, Depends, HTTPException, Request, Response
-from sqlalchemy import case, delete, func, select, update
-from sqlalchemy.dialects.postgresql import insert
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.config import SwitchConfig
-from switch_core.db.models import (
- HostedLaunch,
- ProviderConnection,
- ProviderVerification,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.hosted_machine_store import HostedMachineStore
-from switch_core.db.stores.provider_connection_store import (
- ProviderConnectionBusy,
- ProviderConnectionStore,
-)
-from switch_core.gateway.auth import get_current_user
-from switch_core.gateway.dependencies import get_config, get_protocol, get_session
-from switch_core.providers.claude_verifier import (
- ClaudeVerificationError,
- ClaudeVerifier,
-)
-from switch_core.providers.credentials import validate_provider_credential
-from switch_core.providers.verification import ACTIVE, latest, queue, summary
-
-OtherProvider = Literal["codex", "cursor", "opencode", "antigravity"]
-
-router = APIRouter()
-
-
-def get_connection_store() -> ProviderConnectionStore:
- return ProviderConnectionStore()
-
-
-def get_verifier(request: Request) -> ClaudeVerifier:
- verifier = request.app.state.claude_verifier
- if verifier is None:
- raise HTTPException(503, "Claude connections are not enabled on this server.")
- return cast(ClaudeVerifier, verifier)
-
-
-async def get_connection(
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
- store: Annotated[ProviderConnectionStore, Depends(get_connection_store)],
- _verifier: Annotated[ClaudeVerifier, Depends(get_verifier)],
-) -> dict:
- connection = await store.get(session, user.id)
- if connection is None:
- return {"status": "not_connected"}
- return {
- "status": "connected",
- "kind": connection.kind,
- "verified_at": str(connection.verified_at),
- }
-
-
-async def connect_claude(
- request: Request,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
- store: Annotated[ProviderConnectionStore, Depends(get_connection_store)],
- config: Annotated[SwitchConfig, Depends(get_config)],
- verifier: Annotated[ClaudeVerifier, Depends(get_verifier)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict:
- body = bytearray()
- async for chunk in request.stream():
- body.extend(chunk)
- if len(body) > 20 * 1024:
- raise HTTPException(413, "The credential is too long.")
- try:
- payload = json.loads(body)
- except (ValueError, UnicodeError, RecursionError):
- raise HTTPException(400, "Provide a credential and its type.") from None
- if not isinstance(payload, dict) or set(payload) != {"kind", "credential"}:
- raise HTTPException(400, "Provide a credential and its type.")
- kind = payload["kind"]
- credential = payload["credential"]
- if kind not in ("api-key", "setup-token") or not isinstance(credential, str):
- raise HTTPException(400, "Choose an API key or subscription setup token.")
- credential = credential.strip()
- prefix = "sk-ant-api" if kind == "api-key" else "sk-ant-oat"
- if (
- not credential.startswith(prefix)
- or len(credential) > 16 * 1024
- or any(not 33 <= ord(c) <= 126 for c in credential)
- ):
- raise HTTPException(
- 400, "The credential format does not match the selected type."
- )
- try:
- await store.lock_user(session, user.id)
- except ProviderConnectionBusy as error:
- raise HTTPException(409, str(error)) from None
- try:
- await verifier.verify(kind, credential)
- except ClaudeVerificationError as error:
- raise HTTPException(422, str(error)) from None
- now = datetime.now(UTC)
- await store.save(session, user.id, kind, config.keyring.encrypt(credential), now)
- await bump_machine_agents(session, user.id)
- await session.commit()
- await ring_credential_change(
- session, protocol.connections, user.id, "claude", str(now)
- )
- return {"status": "connected", "kind": kind, "verified_at": str(now)}
-
-
-async def ring_credential_change(
- session: AsyncSession,
- registry: AgentConnectionRegistry,
- user_id: str,
- provider: str,
- revision: str | None,
-) -> None:
- """Tell the owner's attached workers on this provider to fetch a new credential.
-
- A doorbell only: the frame carries the revision, never the secret. A lost
- one is caught up by the next idle-report response. A disconnect rings with
- no revision; the worker's fetch then reads `revoked`, and it stops every
- host but stays attached.
- """
- agent_ids = await session.scalars(
- select(HostedLaunch.agent_id).where(
- HostedLaunch.tenant_id == require_tenant_id(),
- HostedLaunch.owner_id == user_id,
- HostedLaunch.desired_state == "running",
- HostedLaunch.agent_id.is_not(None),
- func.coalesce(HostedLaunch.spec["provider"].astext, "claude") == provider,
- )
- )
- for agent_id in agent_ids:
- assert agent_id is not None
- registry.ring_worker(agent_id, "credential", {"revision": revision})
-
-
-async def bump_machine_agents(session: AsyncSession, user_id: str) -> None:
- """Make the owner's machine re-read its agent list, which carries the credential kind."""
- machines = HostedMachineStore()
- live = await machines.live_for_owner(session, user_id)
- if live is None:
- return
- machine = await machines.locked(session, live.id)
- if machine is not None:
- machines.bump_agents(machine)
-
-
[email protected]("/claude", status_code=204)
-async def disconnect_claude(
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
- store: Annotated[ProviderConnectionStore, Depends(get_connection_store)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> Response:
- try:
- await store.lock_user(session, user.id)
- except ProviderConnectionBusy as error:
- raise HTTPException(409, str(error)) from None
- await store.delete(session, user.id)
- await bump_machine_agents(session, user.id)
- await session.commit()
- await ring_credential_change(session, protocol.connections, user.id, "claude", None)
- return Response(status_code=204)
-
-
[email protected]("/{provider}")
-async def get_other_connection(
- provider: OtherProvider,
- config: Annotated[SwitchConfig, Depends(get_config)],
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
-) -> dict:
- if config.hosted_provider_verification_enabled:
- job = await latest(session, user.id, provider)
- if (
- job
- and job.state not in ("cancelled", "succeeded")
- and not (job.state == "finishing" and job.result is True)
- ):
- return summary(job)
- row = await session.get(
- ProviderConnection,
- (require_tenant_id(), user.id, provider),
- )
- if row is None:
- return {"status": "not_connected"}
- return {
- "status": "connected"
- if row.verification_status == "verified"
- else "configured",
- "kind": row.kind,
- "verified_at": str(row.verified_at),
- }
-
-
[email protected]("/{provider}")
-async def connect_other_provider(
- provider: OtherProvider,
- request: Request,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
- config: Annotated[SwitchConfig, Depends(get_config)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict:
- body = bytearray()
- async for chunk in request.stream():
- body.extend(chunk)
- if len(body) > 20 * 1024:
- raise HTTPException(413, "The credential is too long.")
- try:
- payload = json.loads(body)
- if (
- not isinstance(payload, dict)
- or set(payload) != {"kind", "credential"}
- or not isinstance(payload["kind"], str)
- or not isinstance(payload["credential"], str)
- ):
- raise ValueError("Provide a credential and its type.")
- credential = validate_provider_credential(
- provider, payload["kind"], payload["credential"]
- )
- except (ValueError, UnicodeError, RecursionError):
- raise HTTPException(
- 400, "Provide a valid credential and supported type."
- ) from None
- store = ProviderConnectionStore()
- try:
- await store.lock_user(session, user.id)
- except ProviderConnectionBusy as error:
- raise HTTPException(409, str(error)) from None
- if config.hosted_provider_verification_enabled:
- return await queue(
- session,
- user.id,
- provider,
- payload["kind"],
- credential,
- config.keyring,
- )
- now = datetime.now(UTC)
- values = {
- "tenant_id": require_tenant_id(),
- "user_id": user.id,
- "provider": provider,
- "kind": payload["kind"],
- "encrypted_credential": config.keyring.encrypt(credential),
- "verified_at": now,
- "verification_status": "configured",
- }
- await session.execute(
- insert(ProviderConnection) # nosemgrep
- .values(**values)
- .on_conflict_do_update(
- index_elements=["tenant_id", "user_id", "provider"],
- set_={
- key: value
- for key, value in values.items()
- if key not in {"tenant_id", "user_id", "provider"}
- },
- )
- )
- await bump_machine_agents(session, user.id)
- await session.commit()
- await ring_credential_change(
- session, protocol.connections, user.id, provider, str(now)
- )
- return {"status": "configured", "kind": payload["kind"], "verified_at": str(now)}
-
-
[email protected]("/{provider}", status_code=204)
-async def disconnect_other_provider(
- provider: OtherProvider,
- user: Annotated[User, Depends(get_current_user)],
- session: Annotated[AsyncSession, Depends(get_session)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> Response:
- try:
- await ProviderConnectionStore().lock_user(session, user.id)
- except ProviderConnectionBusy as error:
- raise HTTPException(409, str(error)) from None
- await session.execute(
- delete(ProviderConnection).where(
- ProviderConnection.tenant_id == require_tenant_id(),
- ProviderConnection.user_id == user.id,
- ProviderConnection.provider == provider,
- )
- )
- await session.execute(
- update(ProviderVerification)
- .where(
- ProviderVerification.tenant_id == require_tenant_id(),
- ProviderVerification.user_id == user.id,
- ProviderVerification.provider == provider,
- )
- .values(
- state="cancelled",
- encrypted_credential=None,
- encrypted_token=None,
- instance_id=case(
- (
- ProviderVerification.state.in_(ACTIVE),
- ProviderVerification.instance_id,
- ),
- else_=None,
- ),
- )
- )
- await bump_machine_agents(session, user.id)
- await session.commit()
- await ring_credential_change(session, protocol.connections, user.id, provider, None)
- return Response(status_code=204)
diff --git a/core/switch_core/gateway/provider_verifications.py b/core/switch_core/gateway/provider_verifications.py
deleted file mode 100644
index 518bf467a..000000000
--- a/core/switch_core/gateway/provider_verifications.py
+++ /dev/null
@@ -1,253 +0,0 @@
-import hashlib
-import json
-import secrets
-from collections.abc import AsyncIterator
-from datetime import UTC, datetime
-from typing import Annotated
-from uuid import UUID
-
-from fastapi import APIRouter, Depends, HTTPException, Request, Response
-from pydantic import BaseModel, ConfigDict, Field
-from sqlalchemy import and_, or_, select
-from sqlalchemy.dialects.postgresql import insert
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.config import SwitchConfig
-from switch_core.db.models import (
- ProviderConnection,
- ProviderVerification,
- TenantMember,
- require_tenant_id,
-)
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.gateway.dependencies import (
- get_config,
- get_protocol,
- get_session_factory,
-)
-from switch_core.gateway.hosted_controller import controller_session
-from switch_core.gateway.hosted_launches import controller_settings
-from switch_core.gateway.provider_connections import (
- bump_machine_agents,
- ring_credential_change,
-)
-from switch_core.providers.credentials import validate_provider_credential
-from switch_core.providers.hosted import HostedControllerSettings
-from switch_core.providers.verification import ACTIVE, latest
-from switch_core.tenant_context import tenant_scope
-
-router = APIRouter(prefix="/provider-verifications")
-
-
-async def get_job(session: AsyncSession, job_id: UUID) -> ProviderVerification:
- job = await session.get(ProviderVerification, (require_tenant_id(), str(job_id)))
- if job is None:
- raise HTTPException(404, "Connection check not found.")
- return job
-
-
-async def worker_session(
- job_id: UUID,
- request: Request,
- settings: Annotated[HostedControllerSettings, Depends(controller_settings)],
- factory: Annotated[async_sessionmaker[AsyncSession], Depends(get_session_factory)],
-) -> AsyncIterator[AsyncSession]:
- token = request.headers.get("authorization", "").removeprefix("Bearer ")
- with tenant_scope(settings.tenant_id):
- async with factory() as session:
- job = await get_job(session, job_id)
- if (
- not secrets.compare_digest(
- job.token_hash, hashlib.sha256(token.encode()).hexdigest()
- )
- or job.state not in ACTIVE
- or job.deadline <= datetime.now(UTC)
- ):
- raise HTTPException(403, "Connection check is expired or unauthorized.")
- if (
- await session.get(TenantMember, (require_tenant_id(), job.user_id))
- is None
- ):
- raise HTTPException(
- 403, "Connection check owner is no longer a member."
- )
- # Not held while the route reads the worker's body, at the worker's pace.
- await session.commit()
- yield session
-
-
-async def pending(
- session: Annotated[AsyncSession, Depends(controller_session)],
-) -> list[dict]:
- jobs = await session.scalars(
- select(ProviderVerification)
- .where(
- ProviderVerification.tenant_id == require_tenant_id(),
- or_(
- ProviderVerification.state.in_(ACTIVE),
- and_(
- ProviderVerification.state == "cancelled",
- ProviderVerification.instance_id.is_not(None),
- ),
- ),
- )
- .order_by(ProviderVerification.created_at)
- )
- return [
- {
- "id": job.id,
- "state": job.state,
- "result": job.result,
- "instance_id": job.instance_id,
- "deadline": job.deadline.timestamp(),
- }
- for job in jobs
- ]
-
-
[email protected]("/{job_id}/prepare")
-async def prepare(
- job_id: UUID,
- response: Response,
- session: Annotated[AsyncSession, Depends(controller_session)],
- config: Annotated[SwitchConfig, Depends(get_config)],
-) -> dict:
- job = await get_job(session, job_id)
- if (
- job.state not in ACTIVE
- or job.deadline <= datetime.now(UTC)
- or not job.encrypted_token
- ):
- raise HTTPException(409, "Connection check is no longer pending.")
- response.headers["Cache-Control"] = "no-store"
- return {"token": config.keyring.decrypt(job.encrypted_token)}
-
-
-class Observation(BaseModel):
- model_config = ConfigDict(extra="forbid")
- instance_id: str | None = Field(pattern=r"^i-[0-9a-f]+$")
- terminated: bool
-
-
[email protected]("/{job_id}/observe")
-async def observe(
- job_id: UUID,
- body: Observation,
- session: Annotated[AsyncSession, Depends(controller_session)],
-) -> dict:
- job = await get_job(session, job_id)
- await ProviderConnectionStore().wait_user(session, job.user_id)
- await session.refresh(job, with_for_update=True)
- if job.state in ("succeeded", "failed"):
- return {"state": job.state}
- if job.instance_id and body.instance_id != job.instance_id:
- raise HTTPException(409, "Verification instance does not match.")
- job.instance_id = body.instance_id
- if body.terminated:
- if job.state != "cancelled":
- job.state = "succeeded" if job.result is True else "failed"
- if job.state == "cancelled":
- job.instance_id = None
- job.encrypted_credential = None
- job.encrypted_token = None
- elif job.state == "queued":
- job.state = "running"
- await session.commit()
- return {"state": job.state}
-
-
[email protected]("/{job_id}/credential")
-async def credential(
- job_id: UUID,
- response: Response,
- session: Annotated[AsyncSession, Depends(worker_session)],
- config: Annotated[SwitchConfig, Depends(get_config)],
-) -> dict:
- job = await get_job(session, job_id)
- if job.result is not None or not job.encrypted_credential:
- raise HTTPException(409, "Connection check already finished.")
- response.headers["Cache-Control"] = "no-store"
- return {
- "provider": job.provider,
- "kind": job.kind,
- "credential": config.keyring.decrypt(job.encrypted_credential),
- }
-
-
[email protected]("/{job_id}/result")
-async def result(
- job_id: UUID,
- request: Request,
- session: Annotated[AsyncSession, Depends(worker_session)],
- config: Annotated[SwitchConfig, Depends(get_config)],
- protocol: Annotated[AgentCore, Depends(get_protocol)],
-) -> dict:
- raw = bytearray()
- async for chunk in request.stream():
- raw.extend(chunk)
- if len(raw) > 20 * 1024:
- raise HTTPException(413, "Verification result is too large.")
- job = await get_job(session, job_id)
- await ProviderConnectionStore().wait_user(session, job.user_id)
- await session.refresh(job, with_for_update=True)
- if job.state not in ACTIVE or job.deadline <= datetime.now(UTC):
- raise HTTPException(409, "Connection check is no longer pending.")
- if job.result is not None:
- return {"received": True}
- try:
- value = json.loads(raw)
- if (
- not isinstance(value, dict)
- or set(value) - {"succeeded", "credential"}
- or type(value.get("succeeded")) is not bool
- ):
- raise ValueError()
- updated = value.get("credential")
- if updated is not None:
- if not isinstance(updated, str):
- raise ValueError()
- updated = validate_provider_credential(job.provider, job.kind, updated)
- except (ValueError, UnicodeError, RecursionError):
- raise HTTPException(400, "Invalid verification result.") from None
- job.result = value["succeeded"]
- job.state = "finishing"
- if job.result and updated is not None:
- job.encrypted_credential = config.keyring.encrypt(updated)
- current = await latest(session, job.user_id, job.provider)
- member = await session.get(TenantMember, (require_tenant_id(), job.user_id))
- if not current or current.id != job.id or not member:
- raise HTTPException(409, "Connection check is no longer current.")
- verified_at = datetime.now(UTC)
- if job.result is True:
- values = dict(
- tenant_id=require_tenant_id(),
- user_id=job.user_id,
- provider=job.provider,
- kind=job.kind,
- encrypted_credential=job.encrypted_credential,
- verified_at=verified_at,
- verification_status="verified",
- )
- await session.execute(
- insert(ProviderConnection) # nosemgrep
- .values(**values)
- .on_conflict_do_update(
- index_elements=["tenant_id", "user_id", "provider"],
- set_={
- key: value
- for key, value in values.items()
- if key not in {"tenant_id", "user_id", "provider"}
- },
- )
- )
- await bump_machine_agents(session, job.user_id)
- job.encrypted_credential = None
- job.encrypted_token = None
- await session.commit()
- if job.result is True:
- await ring_credential_change(
- session, protocol.connections, job.user_id, job.provider, str(verified_at)
- )
- return {"received": True}
diff --git a/core/switch_core/gateway/tenants.py b/core/switch_core/gateway/tenants.py
index c1491eb9a..d752490b3 100644
--- a/core/switch_core/gateway/tenants.py
+++ b/core/switch_core/gateway/tenants.py
@@ -21,7 +21,6 @@ from switch_core.clients.client_lifecycle_service import (
from switch_core.config import SwitchConfig
from switch_core.db.audit import AuditAction, list_audit_events, record_audit_event
from switch_core.db.models import (
- GitHubIssuedToken,
Invitation,
ProviderConnection,
Tenant,
@@ -115,12 +114,7 @@ from switch_core.gateway.schemas import (
TenantMembershipResponse,
UsageTotalResponse,
)
-from switch_core.providers.github_revocations import (
- ACCESS_WARNING,
- queue_revocation,
- revoke_oauth,
- revoke_pending,
-)
+from switch_core.providers.github_revocations import revoke_oauth
from switch_core.telemetry import emit_safely
from switch_core.telemetry.ages import age_hours
from switch_core.tenant_context import current_tenant_id
@@ -1456,7 +1450,6 @@ async def remove_member(
if github_row
else None
)
- await queue_revocation(session, (GitHubIssuedToken.owner_id == user_id,))
await session.execute(
delete(ProviderConnection).where(
ProviderConnection.tenant_id == tenant_id,
@@ -1501,12 +1494,4 @@ async def remove_member(
warning = "GitHub could not revoke the old sign-in. Revoke it in your GitHub settings."
else:
warning = await revoke_oauth(github, github_token)
- remaining = await revoke_pending(
- session, config, (GitHubIssuedToken.owner_id == user_id,)
- )
- messages = [
- message
- for message in (warning, ACCESS_WARNING if remaining else None)
- if message
- ]
- return {"ok": True, "warning": " ".join(messages) or None}
+ return {"ok": True, "warning": warning}
diff --git a/core/switch_core/hosted_cutover_upgrade.py b/core/switch_core/hosted_cutover_upgrade.py
deleted file mode 100644
index d01959c4b..000000000
--- a/core/switch_core/hosted_cutover_upgrade.py
+++ /dev/null
@@ -1,336 +0,0 @@
-"""Upgrade a database that ran hosted agents on the old server-side session tables.
-
-`b9e4d2a71c05` drops those tables, and what they hold for a hosted agent is
-only safe to drop once every retained worker volume has been checked and its
-manifest merged with Core's capture. So the upgrade runs in steps, each
-refusing until the one before it is done:
-
-- `prepare`: with every hosted launch stopped, run the cutover-manifest
- revision, which captures the tables and makes a volume row per launch.
-- `record <launch-id> <check.json>`: apply what `hosted-bootstrap.mjs
- --preflight-check` answered for that launch's stopped volume: a manifest
- decides every item and keeps each import's attachment through the drop, a
- blocked check marks the volume blocked with its reason.
-- `status`: what is still missing.
-- `upgrade` (the default): check everything the drop depends on, upgrade to
- head, then queue each volume's imports into the wake mailbox.
-
-A database without those tables upgrades exactly as `switch-migrate` would.
-"""
-
-from __future__ import annotations
-
-import argparse
-import asyncio
-import json
-import logging
-import sys
-from pathlib import Path
-from typing import Any
-
-from alembic import command as alembic_command
-from alembic.config import Config as AlembicConfig
-from pydantic import BaseModel, ConfigDict, TypeAdapter
-from sqlalchemy import text
-from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine, create_async_engine
-from sqlalchemy.pool import NullPool
-
-from switch_core.bridges.agent.hosted_cutover import (
- CutoverManifest,
- apply_manifest,
- queue_imports,
- record_blocked,
-)
-from switch_core.config import SwitchConfig
-from switch_core.db.boot_lock import boot_lock
-from switch_core.db.engine import create_session_factory
-from switch_core.db.hosted_cutover_gate import (
- RETAINED,
- RUNNING_LAUNCHES,
- cutover_gate_problems,
-)
-from switch_core.db.session_scope import tenant_session
-from switch_core.db.stores.hosted_launch_store import lock_launch
-from switch_core.logging_config import configure_logging
-from switch_core.main import _migrate_and_grant
-from switch_core.version import switch_core_version
-
-logger = logging.getLogger(__name__)
-
-MANIFEST_REVISION = "a3c9e5f71d28"
-
-_PIN_IMPORT_BLOBS = """
- UPDATE media_blobs m SET sdk_session_id = NULL
- FROM hosted_cutover_items i
- CROSS JOIN LATERAL jsonb_array_elements(
- COALESCE(i.payload->'payload'->'attachments', '[]'::jsonb)) a
- WHERE i.tenant_id = :tenant AND i.launch_id = :launch AND i.disposition = 'import'
- AND m.tenant_id = i.tenant_id AND m.uri = a->>'mxc'
- AND m.sdk_session_id IS NOT NULL
-"""
-
-
-class CutoverRefused(RuntimeError):
- """A step of the cutover cannot run yet; the message says what is missing."""
-
-
-class PreflightBlocked(BaseModel):
- model_config = ConfigDict(extra="forbid")
- step: str
- file: str
- line: int | None
- error: str
-
-
-class ManifestCheck(BaseModel):
- model_config = ConfigDict(extra="forbid")
- manifest: CutoverManifest
-
-
-class BlockedCheck(BaseModel):
- model_config = ConfigDict(extra="forbid")
- blocked: PreflightBlocked
-
-
-#: What `hosted-bootstrap.mjs --preflight-check` printed: a manifest, or why not.
-PreflightCheck = ManifestCheck | BlockedCheck
-PREFLIGHT_CHECK: TypeAdapter[PreflightCheck] = TypeAdapter(PreflightCheck)
-
-
-def _owner_engine(config: SwitchConfig) -> AsyncEngine:
- return create_async_engine(
- config.owner_database_url or config.database_url, poolclass=NullPool
- )
-
-
-def _alembic_config() -> AlembicConfig:
- return AlembicConfig(str(Path(__file__).resolve().parent.parent / "alembic.ini"))
-
-
-async def _session_tables_present(conn: AsyncConnection) -> bool:
- return bool(
- await conn.scalar(
- text(
- "SELECT to_regclass('sdk_sessions') IS NOT NULL "
- "AND to_regclass('hosted_launches') IS NOT NULL"
- )
- )
- )
-
-
-async def _versions(conn: AsyncConnection) -> list[str]:
- return list(await conn.scalars(text("SELECT version_num FROM alembic_version")))
-
-
-async def running_launches(config: SwitchConfig) -> list[str] | None:
- """Launches not yet stopped for good; None when the old session tables are gone."""
- engine = _owner_engine(config)
- try:
- async with engine.connect() as conn:
- if not await _session_tables_present(conn):
- return None
- return list(await conn.scalars(text(RUNNING_LAUNCHES)))
- finally:
- await engine.dispose()
-
-
-def _refuse_running(running: list[str]) -> None:
- if running:
- raise CutoverRefused(
- f"{len(running)} hosted launch(es) are not stopped: {', '.join(running)}. "
- "Stop each one with an explicit Stop (desired state stopped, not "
- "sleeping), wait for state stopped, then run this again."
- )
-
-
-async def _unprepared(conn: AsyncConnection) -> str | None:
- """Why the database is not at the point where volumes are recorded, if it is not."""
- if not await _session_tables_present(conn):
- return (
- "the old session tables are gone; the cutover is past the point where "
- "a volume can be recorded"
- )
- versions = await _versions(conn)
- if versions != [MANIFEST_REVISION]:
- return (
- f"the database is at {', '.join(versions) or 'no revision'}, not "
- f"{MANIFEST_REVISION}; run `hosted-cutover-upgrade prepare` first"
- )
- return None
-
-
-async def prepare(config: SwitchConfig) -> None:
- """Capture the old session tables at the cutover-manifest revision, launches stopped."""
- running = await running_launches(config)
- if running is None:
- raise CutoverRefused(
- "This database has no old session tables to capture; run "
- "`hosted-cutover-upgrade` to upgrade it."
- )
- _refuse_running(running)
- async with boot_lock(config):
- await asyncio.to_thread(
- alembic_command.upgrade, _alembic_config(), MANIFEST_REVISION
- )
- logger.warning(
- "Captured the old session tables at %s; check and record every volume next",
- MANIFEST_REVISION,
- )
-
-
-async def record(config: SwitchConfig, launch_id: str, check: PreflightCheck) -> None:
- """Apply one volume's preflight check before the drop."""
- engine = _owner_engine(config)
- try:
- async with engine.connect() as conn:
- unprepared = await _unprepared(conn)
- if unprepared is not None:
- raise CutoverRefused(f"Cannot record launch {launch_id}: {unprepared}.")
- _refuse_running(await running_launches(config) or [])
- row = (
- await conn.execute(
- text(
- "SELECT tenant_id, agent_id FROM hosted_launches "
- "WHERE id = :launch AND agent_id IS NOT NULL"
- ),
- {"launch": launch_id},
- )
- ).one_or_none()
- if row is None:
- raise CutoverRefused(
- f"There is no hosted launch {launch_id} with an agent."
- )
- tenant_id, agent_id = row
- async with tenant_session(create_session_factory(engine), tenant_id) as session:
- if isinstance(check, BlockedCheck):
- blocked = check.blocked
- where = (
- blocked.file
- if blocked.line is None
- else f"{blocked.file}:{blocked.line}"
- )
- await record_blocked(
- session, launch_id, f"{blocked.step} at {where}: {blocked.error}"
- )
- else:
- await apply_manifest(
- session,
- agent_id=agent_id,
- launch_id=launch_id,
- manifest=check.manifest,
- )
- await session.execute(
- text(_PIN_IMPORT_BLOBS), {"tenant": tenant_id, "launch": launch_id}
- )
- await session.commit()
- finally:
- await engine.dispose()
-
-
-async def cutover_problems(config: SwitchConfig) -> list[str]:
- """Everything that still stops the drop, each naming its launch or item."""
- engine = _owner_engine(config)
- try:
- async with engine.connect() as conn:
- if not await _session_tables_present(conn):
- return []
- unprepared = await _unprepared(conn)
- problems = await conn.run_sync(cutover_gate_problems)
- return problems if unprepared is None else [*problems, unprepared]
- finally:
- await engine.dispose()
-
-
-async def status(config: SwitchConfig) -> list[dict[str, Any]]:
- engine = _owner_engine(config)
- try:
- async with engine.connect() as conn:
- if not await conn.scalar(
- text("SELECT to_regclass('hosted_cutover_volumes') IS NOT NULL")
- ):
- return []
- rows = await conn.execute(
- text(
- "SELECT v.launch_id, l.state, v.preflight_state, v.manifest_sha256, "
- "v.blocked_reason, v.imports_queued_at, "
- "(SELECT count(*) FROM hosted_cutover_items i "
- " WHERE i.tenant_id = v.tenant_id AND i.launch_id = v.launch_id "
- " AND i.disposition IS NULL) AS undecided "
- "FROM hosted_cutover_volumes v JOIN hosted_launches l "
- "ON l.tenant_id = v.tenant_id AND l.id = v.launch_id "
- "ORDER BY v.launch_id"
- )
- )
- return [dict(row._mapping) for row in rows]
- finally:
- await engine.dispose()
-
-
-async def queue_all_imports(config: SwitchConfig) -> None:
- """Queue the imports of every recorded volume not yet queued, once the mailbox exists."""
- engine = _owner_engine(config)
- try:
- async with engine.connect() as conn:
- volumes = (
- await conn.execute(
- text( # nosemgrep
- "SELECT v.tenant_id, v.launch_id FROM hosted_cutover_volumes v "
- + RETAINED.format(alias="v")
- + "WHERE v.preflight_state = 'complete' "
- "AND v.imports_queued_at IS NULL ORDER BY v.launch_id"
- )
- )
- ).all()
- factory = create_session_factory(engine)
- for tenant_id, launch_id in volumes:
- async with tenant_session(factory, tenant_id) as session:
- await lock_launch(session, launch_id)
- queued = await queue_imports(session, launch_id)
- await session.commit()
- logger.warning(
- "Queued %d cutover import(s) for launch %s", queued, launch_id
- )
- finally:
- await engine.dispose()
-
-
-async def upgrade(config: SwitchConfig) -> None:
- problems = await cutover_problems(config)
- if problems:
- raise CutoverRefused(
- "Refusing to drop the old session tables:\n- " + "\n- ".join(problems)
- )
- await _migrate_and_grant(config)
- await queue_all_imports(config)
-
-
-def main() -> None:
- parser = argparse.ArgumentParser(prog="switch-hosted-cutover-upgrade")
- steps = parser.add_subparsers(dest="step")
- steps.add_parser("prepare")
- recording = steps.add_parser("record")
- recording.add_argument("launch_id")
- recording.add_argument("check", type=Path)
- steps.add_parser("status")
- steps.add_parser("upgrade")
- args = parser.parse_args()
-
- config = SwitchConfig()
- configure_logging(config, switch_core_version())
- if args.step == "prepare":
- asyncio.run(prepare(config))
- elif args.step == "record":
- check = PREFLIGHT_CHECK.validate_json(args.check.read_bytes())
- asyncio.run(record(config, args.launch_id, check))
- elif args.step == "status":
- for volume in asyncio.run(status(config)):
- print(json.dumps(volume, default=str))
- problems = asyncio.run(cutover_problems(config))
- for problem in problems:
- print(f"blocking: {problem}", file=sys.stderr)
- else:
- asyncio.run(upgrade(config))
-
-
-if __name__ == "__main__":
- main()
diff --git a/core/switch_core/main.py b/core/switch_core/main.py
index cb9217a59..590733cbe 100644
--- a/core/switch_core/main.py
+++ b/core/switch_core/main.py
@@ -8,7 +8,6 @@ import signal
import time
from collections.abc import AsyncIterator, Coroutine
from contextlib import asynccontextmanager
-from functools import partial
from pathlib import Path
import httpx
@@ -27,7 +26,6 @@ from sqlalchemy.ext.asyncio import (
from sqlalchemy.pool import NullPool
from switch_core.bridges.agent.app import create_agent_bridge_app
-from switch_core.bridges.agent.hosted_mailbox import mailbox_upkeep
from switch_core.bridges.agent.protocol.agent_connections import (
HEARTBEAT_TTL_SECONDS,
AgentConnectionRegistry,
@@ -135,7 +133,6 @@ from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
from switch_core.db.stores.document_store import DocumentStore
from switch_core.db.stores.external_user_store import ExternalUserStore
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
from switch_core.db.stores.invitation_store import InvitationStore
from switch_core.db.stores.join_domain_store import JoinDomainStore
from switch_core.db.stores.media_store import MediaStore
@@ -649,7 +646,6 @@ async def run(config: SwitchConfig) -> None:
agent_session_store=agent_session_store,
room_role_store=room_role_store,
external_user_store=external_user_store,
- hosted_launch_store=HostedLaunchStore(),
connections=connections,
frontend_base_url=config.frontend_base_url,
message_telemetry=message_telemetry,
@@ -987,9 +983,7 @@ async def run(config: SwitchConfig) -> None:
asyncio.create_task(connector_lifecycle.start_all())
sweep_task = asyncio.create_task(_runtime_state_sweep_loop(protocol))
session_activity_task = asyncio.create_task(
- session_activity_maintenance_loop(
- session_factory, partial(mailbox_upkeep, protocol)
- )
+ session_activity_maintenance_loop(session_factory)
)
connection_sweep_task = asyncio.create_task(
_connection_sweep_loop(protocol, observability.lag)
diff --git a/core/switch_core/management/controller_routes.py b/core/switch_core/management/controller_routes.py
index 57f1db18b..0bd89f8a6 100644
--- a/core/switch_core/management/controller_routes.py
+++ b/core/switch_core/management/controller_routes.py
@@ -227,6 +227,20 @@ async def update_controller_info(
)
+ "/v1/management/controllers/{controller_id}/provider-credentials/{provider}"
+)
+async def get_sealed_login(
+ provider: str, principal: PathController, management: Management, session: Session
+) -> JSONResponse:
+ """The provider login given to this machine, sealed to its own key, or
+ 404 `provider_login_missing`."""
+ return JSONResponse(
+ await management.own_sealed_login(session, principal, provider),
+ headers={"Cache-Control": "no-store"},
+ )
+
+
@router.post("/v1/management/controllers/{controller_id}/credential/rotate")
async def rotate_credential(
principal: PathController, management: Management, session: Session
diff --git a/core/switch_core/management/gateway_routes.py b/core/switch_core/management/gateway_routes.py
index 1eb2319b4..2989afd44 100644
--- a/core/switch_core/management/gateway_routes.py
+++ b/core/switch_core/management/gateway_routes.py
@@ -32,6 +32,7 @@ from switch_core.management.schemas import (
UpdateControllerRequest,
wire_time,
)
+from switch_core.management.sealed_logins import PutSealedLoginRequest
from switch_core.management.service import ManagementService, placement_from
router = APIRouter(route_class=ManagementRoute, tags=["agent management"])
@@ -115,6 +116,48 @@ async def revoke_controller(
return {"ok": True}
[email protected]("/controllers/{controller_id}/provider-logins")
+async def list_sealed_logins(
+ controller_id: str, session: Session, user: CurrentUser, management: Management
+) -> list[dict[str, Any]]:
+ """The provider logins given to this machine: which, and which revision;
+ never what they hold."""
+ return await management.list_sealed_logins(
+ session, require_tenant_id(), user.id, controller_id
+ )
+
+
[email protected]("/controllers/{controller_id}/provider-logins/{provider}")
+async def give_sealed_login(
+ controller_id: str,
+ provider: str,
+ body: PutSealedLoginRequest,
+ session: Session,
+ user: CurrentUser,
+ management: Management,
+) -> dict[str, Any]:
+ """Give this machine a provider login, sealed to its `public_key` by the
+ caller's client. Returns the login and the `provider.login` operation
+ whose result says whether the provider signs in with it."""
+ return await management.give_sealed_login(
+ session, require_tenant_id(), user.id, controller_id, provider, body.sealed
+ )
+
+
[email protected]("/controllers/{controller_id}/provider-logins/{provider}")
+async def withdraw_sealed_login(
+ controller_id: str,
+ provider: str,
+ session: Session,
+ user: CurrentUser,
+ management: Management,
+) -> dict[str, bool]:
+ await management.withdraw_sealed_login(
+ session, require_tenant_id(), user.id, controller_id, provider
+ )
+ return {"ok": True}
+
+
@router.get("/advanced-config")
async def get_advanced_config(user: CurrentUser) -> dict[str, Any]:
"""Each provider's advanced-configuration fields, which a definition's
diff --git a/core/switch_core/management/placement.py b/core/switch_core/management/placement.py
index 49c564833..553e1f873 100644
--- a/core/switch_core/management/placement.py
+++ b/core/switch_core/management/placement.py
@@ -85,26 +85,31 @@ def placement_refusal(
leases: ProcessLeases,
now: datetime,
interval_seconds: int,
+ waking: bool,
) -> tuple[str, str] | None:
- """The reason code and message refusing this placement, or None to allow it."""
+ """The reason code and message refusing this placement, or None to allow
+ it. A `waking` controller is a Switch cloud machine that is starting: it is
+ judged by its last status report, since it is offline until it is up."""
state = controller_state(controller, leases)
if state == "revoked":
return reason_codes.CONTROLLER_REVOKED, "the controller has been revoked"
- if state == "unknown":
- return (
- reason_codes.CONTROLLER_OFFLINE,
- "the controller has never connected to Switch",
- )
- if state == "offline":
- return (
- reason_codes.CONTROLLER_OFFLINE,
- "the controller is not connected to Switch",
- )
- if not status_is_fresh(controller, now=now, interval_seconds=interval_seconds):
- return (
- reason_codes.CONTROLLER_OFFLINE,
- "the controller has not reported status recently",
- )
+ # A machine that is starting has no live connection to judge it by yet.
+ if not (waking and controller.status is not None):
+ if state == "unknown":
+ return (
+ reason_codes.CONTROLLER_OFFLINE,
+ "the controller has never connected to Switch",
+ )
+ if state == "offline":
+ return (
+ reason_codes.CONTROLLER_OFFLINE,
+ "the controller is not connected to Switch",
+ )
+ if not status_is_fresh(controller, now=now, interval_seconds=interval_seconds):
+ return (
+ reason_codes.CONTROLLER_OFFLINE,
+ "the controller has not reported status recently",
+ )
assert controller.status is not None
providers = controller.status.get("providers")
entry = next(
@@ -141,6 +146,7 @@ def require_placement(
leases: ProcessLeases,
now: datetime,
interval_seconds: int,
+ waking: bool,
) -> None:
refusal = placement_refusal(
controller,
@@ -148,6 +154,7 @@ def require_placement(
leases=leases,
now=now,
interval_seconds=interval_seconds,
+ waking=waking,
)
if refusal is not None:
code, message = refusal
diff --git a/core/switch_core/management/schemas.py b/core/switch_core/management/schemas.py
index 5e76a099c..ffc955776 100644
--- a/core/switch_core/management/schemas.py
+++ b/core/switch_core/management/schemas.py
@@ -19,10 +19,16 @@ from typing import Any, Literal, Self
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
-from switch_core.db.models import Agent, AgentController, AgentControllerOperation
+from switch_core.db.models import (
+ Agent,
+ AgentController,
+ AgentControllerOperation,
+ SealedProviderLogin,
+)
from switch_core.db.models import AgentDefinition as AgentDefinitionRow
from switch_core.management.advanced_config import validate_advanced_config
from switch_core.management.process_lease import ProcessLeases
+from switch_core.management.sealed_logins import PublicKey, key_id
Provider = Literal["claude", "codex", "opencode", "antigravity", "cursor"]
ControllerKind = Literal["console", "daemon", "ec2"]
@@ -77,11 +83,6 @@ class Platform(_StatusBody):
os_version: str
-class PublicKey(_StatusBody):
- alg: str
- key: str
-
-
class DefinitionV1(_GatewayBody):
"""The v1 agent definition, as a person's client submits it."""
@@ -383,10 +384,21 @@ class UpdateControllerRequest(_ControllerDetailsChange):
class ControllerInfoRequest(_ControllerDetailsChange):
- """The controller's own change (`switch-agent-controller set-info`)."""
+ """The controller's own change (`switch-agent-controller set-info`), and
+ the key its provider logins are sealed to, which a controller enrolled
+ before it made one registers once."""
model_config = ConfigDict(extra="ignore")
+ public_key: PublicKey | None = None
+
+ @field_validator("public_key")
+ @classmethod
+ def _key_is_not_cleared(cls, value: PublicKey | None) -> PublicKey:
+ if value is None:
+ raise ValueError("a controller's public key cannot be cleared")
+ return value
+
class CreateManagedAgentRequest(_GatewayBody):
name: str
@@ -533,11 +545,30 @@ def controller_view(
"status": controller.status,
"assignment_revision": controller.assignment_revision,
"workspaces_dir": workspaces_dir_of(controller),
+ "public_key": public_key_view(controller),
"created_at": wire_time(controller.created_at),
"revoked_at": wire_time_or_none(controller.revoked_at),
}
+def public_key_view(controller: AgentController) -> dict[str, str] | None:
+ """The key the owner's client seals this machine's provider logins to,
+ with the id a sealed login names it by; null for one that has none."""
+ if controller.public_key is None:
+ return None
+ return {**controller.public_key, "key_id": key_id(controller.public_key["key"])}
+
+
+def sealed_login_view(row: SealedProviderLogin) -> dict[str, Any]:
+ """What the owner sees of a sealed login: never the ciphertext."""
+ return {
+ "provider": row.provider,
+ "revision": row.revision,
+ "key_id": row.sealed["key_id"],
+ "updated_at": wire_time(row.updated_at),
+ }
+
+
def agent_status_from(
controller: AgentController | None, agent_id: str
) -> dict[str, Any] | None:
diff --git a/core/switch_core/management/sealed_logins.py b/core/switch_core/management/sealed_logins.py
new file mode 100644
index 000000000..d053cce0e
--- /dev/null
+++ b/core/switch_core/management/sealed_logins.py
@@ -0,0 +1,106 @@
+"""Provider logins sealed to a controller's own key.
+
+A controller makes an X25519 keypair when it enrolls and gives Switch only the
+public half (`AgentController.public_key`). The owner's client seals a login
+to that key before sending it, so Switch stores and relays ciphertext it
+cannot open, and only the controller holding the private key can:
+
+- an ephemeral X25519 keypair; the shared secret with the controller's key;
+- `HKDF-SHA256(shared, salt = ephemeral public key || controller public key,
+ info = "switch provider login v1")` gives a 32-byte key;
+- `AES-256-GCM(key, nonce, plaintext, aad)` with a random 12-byte nonce and
+ `aad = "switch-provider-login-v1\\n<controller id>\\n<provider>"`, so a
+ sealed login opens only for the controller and provider it was sealed for;
+- the plaintext is `{"kind": ..., "credential": ...}`.
+
+Switch checks the envelope's shape and that it names the controller's current
+key (`key_id`, the first 16 hex digits of the key's SHA-256); it never sees
+inside. The same construction is in `@switch-console/agent-providers`
+(`sealed-login.ts`), which seals in Console and opens in the controller.
+"""
+
+from __future__ import annotations
+
+import base64
+import binascii
+import hashlib
+from typing import Literal
+
+from pydantic import BaseModel, ConfigDict, field_validator
+
+PUBLIC_KEY_ALG = "X25519"
+SEALED_LOGIN_ALG = "X25519-HKDF-SHA256-A256GCM"
+# A provider auth file is at most 16 KiB; sealing adds the
+# JSON around it and the tag.
+MAX_CIPHERTEXT_BYTES = 20 * 1024
+
+
+def _decoded(value: str, what: str) -> bytes:
+ try:
+ return base64.b64decode(value, validate=True)
+ except (binascii.Error, ValueError) as error:
+ raise ValueError(f"{what} must be base64") from error
+
+
+def key_id(public_key: str) -> str:
+ """The id a sealed login names its controller's key by."""
+ return hashlib.sha256(_decoded(public_key, "key")).hexdigest()[:16]
+
+
+class PublicKey(BaseModel):
+ """The key a controller's provider logins are sealed to."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ alg: Literal["X25519"]
+ key: str
+
+ @field_validator("key")
+ @classmethod
+ def _is_a_key(cls, value: str) -> str:
+ if len(_decoded(value, "key")) != 32:
+ raise ValueError("an X25519 public key is 32 bytes")
+ return value
+
+
+class SealedLogin(BaseModel):
+ """A provider login as the owner's client sealed it."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ alg: Literal["X25519-HKDF-SHA256-A256GCM"]
+ key_id: str
+ ephemeral_key: str
+ nonce: str
+ ciphertext: str
+
+ @field_validator("ephemeral_key")
+ @classmethod
+ def _is_a_key(cls, value: str) -> str:
+ if len(_decoded(value, "ephemeral_key")) != 32:
+ raise ValueError("an X25519 public key is 32 bytes")
+ return value
+
+ @field_validator("nonce")
+ @classmethod
+ def _is_a_nonce(cls, value: str) -> str:
+ if len(_decoded(value, "nonce")) != 12:
+ raise ValueError("the nonce is 12 bytes")
+ return value
+
+ @field_validator("ciphertext")
+ @classmethod
+ def _fits(cls, value: str) -> str:
+ size = len(_decoded(value, "ciphertext"))
+ if size <= 16 or size > MAX_CIPHERTEXT_BYTES:
+ raise ValueError(
+ f"the ciphertext must be more than its tag and at most "
+ f"{MAX_CIPHERTEXT_BYTES} bytes"
+ )
+ return value
+
+
+class PutSealedLoginRequest(BaseModel):
+ model_config = ConfigDict(extra="forbid")
+
+ sealed: SealedLogin
diff --git a/core/switch_core/management/service.py b/core/switch_core/management/service.py
index a92ab7ee4..677532f54 100644
--- a/core/switch_core/management/service.py
+++ b/core/switch_core/management/service.py
@@ -49,6 +49,7 @@ from switch_core.db.models import (
AgentController,
AgentControllerOperation,
ApiKey,
+ HostedMachine,
)
from switch_core.db.models import AgentDefinition as AgentDefinitionRow
from switch_core.db.stores.agent_controller_operation_store import (
@@ -58,7 +59,12 @@ from switch_core.db.stores.agent_controller_store import AgentControllerStore
from switch_core.db.stores.agent_definition_store import AgentDefinitionStore
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
+from switch_core.db.stores.hosted_machine_store import HostedMachineStore
from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.gateway.cloud_controllers import (
+ record_controller_status,
+ wake_controller_machine,
+)
from switch_core.gateway.known_agents import KNOWN_AGENTS, KnownAgent
from switch_core.management import reason_codes, tokens
from switch_core.management.errors import ManagementError, not_found
@@ -75,15 +81,16 @@ from switch_core.management.schemas import (
ControllerDescription,
CreateManagedAgentRequest,
DefinitionV1,
- PublicKey,
StatusReport,
assignment_entry,
controller_view,
managed_agent_view,
operation_view,
operation_wire,
+ sealed_login_view,
workspaces_dir_of,
)
+from switch_core.management.sealed_logins import PublicKey, SealedLogin, key_id
logger = logging.getLogger(__name__)
@@ -95,6 +102,9 @@ OPERATION_LIST_LIMIT = 200
V1_OPERATION_KINDS = frozenset({"agent.restart", "provider.recheck"})
+# What a Switch cloud machine's controller is called, whatever enrolled it.
+CLOUD_MACHINE_NAME = "Switch cloud"
+
@dataclass(frozen=True)
class ManagementSettings:
@@ -217,11 +227,36 @@ class ManagementService:
)
expires_at = self.now() + tokens.ENROLLMENT_CODE_LIFETIME
await self.controllers.create_enrollment_code(
- session, owner_id=owner_id, api_key_id=key.id, expires_at=expires_at
+ session,
+ owner_id=owner_id,
+ api_key_id=key.id,
+ expires_at=expires_at,
+ hosted_machine_id=None,
)
await session.commit()
return code, expires_at
+ async def machine_enrollment_code(
+ self, session: AsyncSession, machine: HostedMachine, now: datetime
+ ) -> str:
+ """A code for a Switch cloud machine's controller to enroll with: the
+ owner's, valid long enough for the machine to boot, and binding what
+ enrolls with it to the machine. The caller commits."""
+ key, code = await self._new_hash_only_key(
+ session,
+ owner_id=machine.owner_id,
+ key_type=CONTROLLER_ENROLLMENT_KEY_TYPE,
+ label="cloud machine enrollment code",
+ )
+ await self.controllers.create_enrollment_code(
+ session,
+ owner_id=machine.owner_id,
+ api_key_id=key.id,
+ expires_at=now + tokens.MACHINE_ENROLLMENT_CODE_LIFETIME,
+ hosted_machine_id=machine.id,
+ )
+ return code
+
async def enroll(
self,
session: AsyncSession,
@@ -246,6 +281,15 @@ class ManagementService:
)
if consumed is None:
raise invalid
+ machine = (
+ None
+ if consumed.hosted_machine_id is None
+ else await HostedMachineStore().locked(session, consumed.hosted_machine_id)
+ )
+ if consumed.hosted_machine_id is not None and (
+ machine is None or machine.state == "deleted"
+ ):
+ raise invalid
controller_key, credential = await self._new_hash_only_key(
session,
owner_id=consumed.owner_id,
@@ -255,9 +299,11 @@ class ManagementService:
controller = await self.controllers.create(
session,
owner_id=consumed.owner_id,
- name=description.name,
+ name=CLOUD_MACHINE_NAME if machine is not None else description.name,
description=description.description,
- kind=description.kind,
+ # A Switch cloud machine's controller is the cloud's, whatever the
+ # program that enrolled says it is.
+ kind="ec2" if machine is not None else description.kind,
platform=description.platform.model_dump(),
version=description.version,
public_key=public_key.model_dump() if public_key is not None else None,
@@ -267,7 +313,34 @@ class ManagementService:
session, tenant_id, consumed.id, controller.id
)
await self.api_keys.delete(session, key.id)
+ replaced: str | None = None
+ moved: list[AgentDefinitionRow] = []
+ revisions: dict[str, int] = {}
+ if machine is not None:
+ replaced = machine.controller_id
+ machine.controller_id = controller.id
+ if replaced is not None and replaced != controller.id:
+ await self._revoke(session, tenant_id, replaced)
+ moved = await self._move_placed(
+ session, tenant_id, replaced, controller.id
+ )
+ revisions = await self._bump_and_collect(
+ session, tenant_id, {replaced, controller.id}
+ )
await session.commit()
+ if replaced is not None and replaced != controller.id:
+ self._announce_revoked(replaced)
+ for row in moved:
+ await self._bind(session, tenant_id, row)
+ self._nudge(revisions)
+ logger.info(
+ "Cloud machine %s enrolled again; revoked its previous controller %s "
+ "and moved its %d agent(s) to %s",
+ consumed.hosted_machine_id,
+ replaced,
+ len(moved),
+ controller.id,
+ )
logger.info(
"Enrolled agent controller %s (%s) by code for user %s",
controller.id,
@@ -381,6 +454,10 @@ class ManagementService:
)
controller = await self._principal_controller(session, principal)
revision = controller.assignment_revision
+ if stored and controller.kind == "ec2":
+ await record_controller_status(
+ session, controller.id, report.machine.model_dump(mode="json")
+ )
await session.commit()
return {
"assignment_revision": revision,
@@ -568,13 +645,42 @@ class ManagementService:
self,
session: AsyncSession,
principal: ControllerPrincipal,
- changes: dict[str, str | None],
+ changes: dict[str, Any],
) -> dict[str, Any]:
"""The controller renames itself or changes its description, with the
- same limits and effects as its owner's change."""
- await self._principal_controller(session, principal)
+ same limits and effects as its owner's change, and registers the key its
+ provider logins are sealed to.
+
+ A key is registered once: replacing it would let whoever holds the
+ controller's credential have the next login sealed to a key of their
+ choosing, so a different one is refused and the machine is enrolled
+ again instead. The same key again is not a change."""
+ controller = await self._principal_controller(session, principal)
+ details = dict(changes)
+ key = details.pop("public_key", None)
+ if isinstance(key, PublicKey):
+ wanted = key.model_dump()
+ if controller.public_key is None:
+ await self.controllers.set_public_key(
+ session, principal.tenant_id, controller.id, wanted
+ )
+ logger.info("Agent controller %s registered its key", controller.id)
+ elif controller.public_key != wanted:
+ raise ManagementError(
+ 409,
+ reason_codes.VALIDATION_ERROR,
+ "This controller already has a different public key. Enroll "
+ "the machine again to give it a new one.",
+ )
+ if not details:
+ leases = await self.leases(session)
+ view = controller_view(
+ controller, self.state_of(controller, leases), leases
+ )
+ await session.commit()
+ return view
return await self._update_details(
- session, principal.tenant_id, principal.controller_id, changes
+ session, principal.tenant_id, principal.controller_id, details
)
async def _update_details(
@@ -612,6 +718,19 @@ class ManagementService:
)
if controller.revoked_at is not None:
return
+ await self._revoke(session, tenant_id, controller_id)
+ await session.commit()
+ logger.info("Revoked agent controller %s", controller_id)
+ self._announce_revoked(controller_id)
+
+ async def _revoke(
+ self, session: AsyncSession, tenant_id: str, controller_id: str
+ ) -> None:
+ """Mark revoked, delete the credential, cancel open operations and
+ drop sealed logins. The caller commits, then announces it."""
+ controller = await self.controllers.get(session, tenant_id, controller_id)
+ if controller is None or controller.revoked_at is not None:
+ return
key_id = controller.api_key_id
await self.controllers.mark_revoked(
session, tenant_id, controller_id, self.now()
@@ -621,11 +740,187 @@ class ManagementService:
await self.operations.cancel_open(
session, tenant_id, controller_id=controller_id, agent_id=None
)
- await session.commit()
- logger.info("Revoked agent controller %s", controller_id)
+ await self.controllers.delete_sealed_logins(
+ session, tenant_id, controller_id, None
+ )
+
+ async def _move_placed(
+ self, session: AsyncSession, tenant_id: str, source: str, target: str
+ ) -> list[AgentDefinitionRow]:
+ """Place every agent of `source` on `target`, as they were. The caller
+ commits, then binds them."""
+ moved = []
+ for row, _agent in await self.definitions.list_for_controller(
+ session, tenant_id, source
+ ):
+ moved.append(
+ await self.definitions.update(
+ session,
+ tenant_id,
+ row.agent_id,
+ controller_id=target,
+ desired_state=row.desired_state,
+ definition=row.definition,
+ )
+ )
+ return moved
+
+ def _announce_revoked(self, controller_id: str) -> None:
self.notifier.credential_revoked(controller_id)
self.presence.revoke_controller(controller_id)
+ # ── Provider logins sealed to a controller ────────────────────────────────
+
+ async def list_sealed_logins(
+ self, session: AsyncSession, tenant_id: str, owner_id: str, controller_id: str
+ ) -> list[dict[str, Any]]:
+ await self.owned_controller(session, tenant_id, owner_id, controller_id)
+ rows = await self.controllers.sealed_logins(session, tenant_id, controller_id)
+ return [sealed_login_view(row) for row in rows]
+
+ async def give_sealed_login(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ owner_id: str,
+ controller_id: str,
+ provider: str,
+ sealed: SealedLogin,
+ ) -> dict[str, Any]:
+ """Give the owner's machine a provider login the owner's client sealed
+ to it, and have the controller take it up now: a `provider.login`
+ operation, whose result says whether the provider signs in with it."""
+ controller = await self._sealing_target(
+ session, tenant_id, owner_id, controller_id, provider
+ )
+ public_key = controller.public_key
+ if public_key is None:
+ raise ManagementError(
+ 409,
+ reason_codes.VALIDATION_ERROR,
+ "This machine has no key to seal a login to yet. Update its "
+ "controller, which registers one when it next runs.",
+ )
+ if sealed.key_id != key_id(public_key["key"]):
+ raise ManagementError(
+ 409,
+ reason_codes.VALIDATION_ERROR,
+ "The login was sealed to a key this machine no longer has. Seal "
+ "it again to the key the machine shows now.",
+ )
+ row = await self.controllers.put_sealed_login(
+ session,
+ tenant_id,
+ controller_id,
+ provider,
+ sealed.model_dump(),
+ owner_id,
+ )
+ login = sealed_login_view(row)
+ operation = await self.operations.create(
+ session,
+ controller_id=controller_id,
+ agent_id=None,
+ kind="provider.login",
+ params={"provider": provider, "method": "sealed", "revision": row.revision},
+ created_by=owner_id,
+ )
+ view = operation_view(operation)
+ await session.commit()
+ logger.info(
+ "Gave agent controller %s a sealed %s login (revision %s)",
+ controller_id,
+ provider,
+ row.revision,
+ )
+ self.notifier.operation_pending(
+ controller_id,
+ operation_id=operation.id,
+ kind="provider.login",
+ agent_id=None,
+ )
+ return {"login": login, "operation": view}
+
+ async def withdraw_sealed_login(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ owner_id: str,
+ controller_id: str,
+ provider: str,
+ ) -> None:
+ """Take the provider login back, and have the controller check the
+ provider again, which stops using it."""
+ await self._sealing_target(
+ session, tenant_id, owner_id, controller_id, provider
+ )
+ removed = await self.controllers.delete_sealed_logins(
+ session, tenant_id, controller_id, provider
+ )
+ if not removed:
+ raise not_found("Sealed login")
+ operation = await self.operations.create(
+ session,
+ controller_id=controller_id,
+ agent_id=None,
+ kind="provider.recheck",
+ params={"provider": provider},
+ created_by=owner_id,
+ )
+ await session.commit()
+ logger.info(
+ "Withdrew agent controller %s's sealed %s login", controller_id, provider
+ )
+ self.notifier.operation_pending(
+ controller_id,
+ operation_id=operation.id,
+ kind="provider.recheck",
+ agent_id=None,
+ )
+
+ async def _sealing_target(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ owner_id: str,
+ controller_id: str,
+ provider: str,
+ ) -> AgentController:
+ if provider not in PROVIDER_KNOWN_AGENT_TYPES:
+ raise ManagementError(
+ 422,
+ reason_codes.VALIDATION_ERROR,
+ f"Unknown provider {provider!r}; one of "
+ f"{', '.join(sorted(PROVIDER_KNOWN_AGENT_TYPES))}.",
+ )
+ controller = await self.owned_controller(
+ session, tenant_id, owner_id, controller_id
+ )
+ if is_revoked(controller):
+ raise ManagementError(
+ 409, reason_codes.CONTROLLER_REVOKED, "The controller has been revoked."
+ )
+ return controller
+
+ async def own_sealed_login(
+ self, session: AsyncSession, principal: ControllerPrincipal, provider: str
+ ) -> dict[str, Any]:
+ """The controller's sealed login for `provider`, as it opens it."""
+ row = await self.controllers.sealed_login(
+ session, principal.tenant_id, principal.controller_id, provider
+ )
+ if row is None:
+ raise ManagementError(
+ 404,
+ reason_codes.PROVIDER_LOGIN_MISSING,
+ f"No {provider} login has been given to this machine.",
+ )
+ return {
+ "provider": row.provider,
+ "revision": row.revision,
+ "sealed": row.sealed,
+ }
+
# ── Managed agents, owner side ────────────────────────────────────────────
async def _view(
@@ -713,12 +1008,21 @@ class ManagementService:
session, tenant_id, owner_id, controller_id
)
if check_placement:
+ # Placing an agent on a sleeping Switch cloud machine wakes it.
+ machine = (
+ await wake_controller_machine(session, controller.id, self.now())
+ if controller.kind == "ec2"
+ else None
+ )
require_placement(
controller,
provider,
leases=await self.leases(session),
now=self.now(),
interval_seconds=self.settings.status_interval_seconds,
+ waking=machine is not None
+ and machine.desired_state == "running"
+ and machine.state != "ready",
)
return controller
diff --git a/core/switch_core/management/tokens.py b/core/switch_core/management/tokens.py
index 88c522d16..d47c72deb 100644
--- a/core/switch_core/management/tokens.py
+++ b/core/switch_core/management/tokens.py
@@ -30,6 +30,9 @@ ACCESS_TOKEN_AUDIENCE = "switch-controller"
ACCESS_TOKEN_ALGORITHM = "HS256"
ACCESS_TOKEN_LIFETIME = timedelta(hours=1)
ENROLLMENT_CODE_LIFETIME = timedelta(minutes=10)
+# A Switch cloud machine's code is handed over before the machine boots, and
+# used once it has mounted its disk and installed its controller.
+MACHINE_ENROLLMENT_CODE_LIFETIME = timedelta(minutes=30)
_REQUIRED_CLAIMS = ["cid", "tid", "oid", "iat", "exp", "aud"]
diff --git a/core/switch_core/management/wiring.py b/core/switch_core/management/wiring.py
index f6d24f669..7e2e61fda 100644
--- a/core/switch_core/management/wiring.py
+++ b/core/switch_core/management/wiring.py
@@ -43,6 +43,7 @@ from switch_core.db.stores.agent_definition_store import AgentDefinitionStore
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.gateway.cloud_controllers import set_cloud_enrollment
from switch_core.management.agent_operations import ManagementAgentOperations
from switch_core.management.auth import ManagementAuthenticator
from switch_core.management.bindings import load_bindings
@@ -159,6 +160,8 @@ def build_management(
presence=presence,
clock=clock,
)
+ # Switch cloud machines that run the controller enroll with codes it mints.
+ set_cloud_enrollment(service)
authenticator = ManagementAuthenticator(
session_factory=session_factory,
controllers=controllers,
diff --git a/core/switch_core/migrations/env.py b/core/switch_core/migrations/env.py
index 213e6d011..eea50e3cb 100644
--- a/core/switch_core/migrations/env.py
+++ b/core/switch_core/migrations/env.py
@@ -2,7 +2,6 @@ import asyncio
from logging.config import fileConfig
from alembic import context
-from alembic.runtime.migration import MigrationContext, RevisionStep
from sqlalchemy import pool
from sqlalchemy.ext.asyncio import async_engine_from_config
@@ -12,7 +11,6 @@ from switch_core.db import (
)
from switch_core.db.base import Base
from switch_core.db.engine import migration_connect_args
-from switch_core.db.hosted_cutover_gate import DROP_REVISION, refuse_incomplete_cutover
from switch_core.logging_config import logging_is_configured
config = context.config
@@ -50,31 +48,8 @@ def run_migrations_offline() -> None:
context.run_migrations()
-def _gate_session_table_drop(migration_context: MigrationContext) -> None:
- """Refuse, before any revision runs, a plan that drops the old session tables
- while the hosted cutover is incomplete. Only here can the gate still read
- those tables: alembic orders `b9e4d2a71c05` freely against the hosted branch,
- and no revision of ours is guaranteed to run before it."""
- plan = migration_context._migrations_fn
- assert plan is not None
-
- def gated(heads, context): # type: ignore[no-untyped-def]
- steps = list(plan(heads, context))
- if any(
- isinstance(step, RevisionStep)
- and step.is_upgrade
- and step.revision.revision == DROP_REVISION
- for step in steps
- ):
- refuse_incomplete_cutover(context.connection)
- return steps
-
- migration_context._migrations_fn = gated
-
-
def do_run_migrations(connection): # type: ignore[no-untyped-def]
context.configure(connection=connection, target_metadata=target_metadata)
- _gate_session_table_drop(context.get_context())
with context.begin_transaction():
context.run_migrations()
diff --git a/core/switch_core/migrations/versions/a8c3e5f1d27b_cloud_machines_without_slots.py b/core/switch_core/migrations/versions/a8c3e5f1d27b_cloud_machines_without_slots.py
new file mode 100644
index 000000000..ecc652787
--- /dev/null
+++ b/core/switch_core/migrations/versions/a8c3e5f1d27b_cloud_machines_without_slots.py
@@ -0,0 +1,56 @@
+"""cloud machines without slots: drop the slot and generation of a machine
+
+A cloud machine no longer borrows one of a fixed pool of operator slots: the
+machine id alone names it and its resources, and capacity is
+`HOSTED_LAUNCH_CAPACITY`.
+
+Revision ID: a8c3e5f1d27b
+Revises: f5d2b8e6c3a1
+Create Date: 2026-10-09 12:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "a8c3e5f1d27b"
+down_revision: str | None = "f5d2b8e6c3a1"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+ op.drop_index("uq_hosted_machine_generation", table_name="hosted_machines")
+ op.drop_index("uq_hosted_machine_slot", table_name="hosted_machines")
+ op.drop_constraint("ck_hosted_machine_generation", "hosted_machines", type_="check")
+ op.drop_column("hosted_machines", "generation")
+ op.drop_column("hosted_machines", "slot_id")
+
+
+def downgrade() -> None:
+ op.add_column("hosted_machines", sa.Column("slot_id", sa.Text(), nullable=True))
+ op.add_column(
+ "hosted_machines", sa.Column("generation", sa.Integer(), nullable=True)
+ )
+ # Every machine gets a slot of its own: its id.
+ op.execute("UPDATE hosted_machines SET slot_id = id, generation = 1")
+ op.alter_column("hosted_machines", "slot_id", nullable=False)
+ op.alter_column("hosted_machines", "generation", nullable=False)
+ op.create_check_constraint(
+ "ck_hosted_machine_generation", "hosted_machines", "generation >= 1"
+ )
+ op.create_index(
+ "uq_hosted_machine_slot",
+ "hosted_machines",
+ ["tenant_id", "slot_id"],
+ unique=True,
+ postgresql_where=sa.text("state <> 'deleted'"),
+ )
+ op.create_index(
+ "uq_hosted_machine_generation",
+ "hosted_machines",
+ ["tenant_id", "slot_id", "generation"],
+ unique=True,
+ )
diff --git a/core/switch_core/migrations/versions/d41c7a9e2b58_sealed_provider_logins.py b/core/switch_core/migrations/versions/d41c7a9e2b58_sealed_provider_logins.py
new file mode 100644
index 000000000..c0de03194
--- /dev/null
+++ b/core/switch_core/migrations/versions/d41c7a9e2b58_sealed_provider_logins.py
@@ -0,0 +1,78 @@
+"""agent management: provider logins sealed to a controller
+
+`sealed_provider_logins` holds, per controller and provider, a login the
+owner's client sealed to that controller's own public key before sending it:
+Switch keeps and relays the ciphertext and cannot open it. Tenant-scoped,
+removed with its controller.
+
+The row-level-security DDL is a verbatim copy of `switch_core/db/rls_ddl.py`
+as it stood when this migration was written, copied rather than imported for
+the reason `265ed188ad6f` gives.
+
+Revision ID: d41c7a9e2b58
+Revises: e4b7d2a91c06
+Create Date: 2026-10-08 18:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects import postgresql
+
+revision: str = "d41c7a9e2b58"
+down_revision: str | None = "e4b7d2a91c06"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+TABLE = "sealed_provider_logins"
+_PREDICATE = '"tenant_id" = (SELECT require_tenant_id())'
+
+
+def upgrade() -> None:
+ op.create_table(
+ TABLE,
+ sa.Column("tenant_id", sa.Text(), nullable=False),
+ sa.Column("controller_id", sa.Text(), nullable=False),
+ sa.Column("provider", sa.Text(), nullable=False),
+ sa.Column("revision", sa.Integer(), nullable=False),
+ sa.Column("sealed", postgresql.JSONB(), nullable=False),
+ sa.Column("sealed_by", sa.Text(), nullable=False),
+ sa.Column(
+ "created_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.text("now()"),
+ nullable=False,
+ ),
+ sa.Column(
+ "updated_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.text("now()"),
+ nullable=False,
+ ),
+ sa.PrimaryKeyConstraint("controller_id", "provider"),
+ sa.ForeignKeyConstraint(
+ ["tenant_id"], ["tenants.id"], name="fk_sealed_provider_logins_tenant"
+ ),
+ sa.ForeignKeyConstraint(["sealed_by"], ["users.id"]),
+ sa.ForeignKeyConstraint(
+ ["tenant_id", "controller_id"],
+ ["agent_controllers.tenant_id", "agent_controllers.id"],
+ name="fk_sealed_provider_logins_controller",
+ ondelete="CASCADE",
+ ),
+ )
+ op.create_index("ix_sealed_provider_logins_tenant_id", TABLE, ["tenant_id"])
+ op.execute(f'ALTER TABLE "{TABLE}" ENABLE ROW LEVEL SECURITY')
+ op.execute(
+ f'CREATE POLICY tenant_isolation ON "{TABLE}"\n'
+ f" FOR ALL\n"
+ f" USING ({_PREDICATE})\n"
+ f" WITH CHECK ({_PREDICATE})"
+ )
+
+
+def downgrade() -> None:
+ op.drop_index("ix_sealed_provider_logins_tenant_id", table_name=TABLE)
+ op.drop_table(TABLE)
diff --git a/core/switch_core/migrations/versions/e7a2c4b9d013_hosted_machines_run_the_controller.py b/core/switch_core/migrations/versions/e7a2c4b9d013_hosted_machines_run_the_controller.py
new file mode 100644
index 000000000..383065d23
--- /dev/null
+++ b/core/switch_core/migrations/versions/e7a2c4b9d013_hosted_machines_run_the_controller.py
@@ -0,0 +1,78 @@
+"""hosted machines: run the agents controller, enrolled with a one-time code
+
+`hosted_machines.runtime` says what a machine runs: `worker` (every existing
+machine, unchanged) or `controller`, the agents controller, enrolled with a
+one-time code Core mints for the machine. `controller_id` is the controller
+it enrolled as; `enrollment_code_encrypted`/`_revision` keep the code handed
+over for a revision, so a retried prepare hands over the same one.
+`agent_controller_enrollment_codes.hosted_machine_id` names the machine a
+code was minted for.
+
+Revision ID: e7a2c4b9d013
+Revises: d41c7a9e2b58
+Create Date: 2026-10-08 20:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "e7a2c4b9d013"
+down_revision: str | None = "d41c7a9e2b58"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+ op.add_column(
+ "hosted_machines",
+ sa.Column("runtime", sa.Text(), server_default="worker", nullable=False),
+ )
+ op.create_check_constraint(
+ "ck_hosted_machine_runtime",
+ "hosted_machines",
+ "runtime IN ('worker', 'controller')",
+ )
+ op.add_column("hosted_machines", sa.Column("controller_id", sa.Text()))
+ op.create_foreign_key(
+ "fk_hosted_machines_controller",
+ "hosted_machines",
+ "agent_controllers",
+ ["tenant_id", "controller_id"],
+ ["tenant_id", "id"],
+ ondelete="SET NULL (controller_id)",
+ )
+ op.add_column("hosted_machines", sa.Column("enrollment_code_encrypted", sa.Text()))
+ op.add_column(
+ "hosted_machines", sa.Column("enrollment_code_revision", sa.Integer())
+ )
+ op.add_column(
+ "agent_controller_enrollment_codes",
+ sa.Column("hosted_machine_id", sa.Text(), nullable=True),
+ )
+ op.create_foreign_key(
+ "fk_agent_controller_enrollment_codes_hosted_machine",
+ "agent_controller_enrollment_codes",
+ "hosted_machines",
+ ["tenant_id", "hosted_machine_id"],
+ ["tenant_id", "id"],
+ )
+
+
+def downgrade() -> None:
+ op.drop_constraint(
+ "fk_agent_controller_enrollment_codes_hosted_machine",
+ "agent_controller_enrollment_codes",
+ type_="foreignkey",
+ )
+ op.drop_column("agent_controller_enrollment_codes", "hosted_machine_id")
+ op.drop_column("hosted_machines", "enrollment_code_revision")
+ op.drop_column("hosted_machines", "enrollment_code_encrypted")
+ op.drop_constraint(
+ "fk_hosted_machines_controller", "hosted_machines", type_="foreignkey"
+ )
+ op.drop_column("hosted_machines", "controller_id")
+ op.drop_constraint("ck_hosted_machine_runtime", "hosted_machines", type_="check")
+ op.drop_column("hosted_machines", "runtime")
diff --git a/core/switch_core/migrations/versions/f5d2b8e6c3a1_remove_the_hosted_worker.py b/core/switch_core/migrations/versions/f5d2b8e6c3a1_remove_the_hosted_worker.py
new file mode 100644
index 000000000..e5102b3c9
--- /dev/null
+++ b/core/switch_core/migrations/versions/f5d2b8e6c3a1_remove_the_hosted_worker.py
@@ -0,0 +1,87 @@
+"""remove the hosted worker: cloud machines only run the agents controller
+
+Drops what only the hosted worker used: cloud launches and their operations,
+wake mailbox, cutover records and GitHub tokens, provider verifications, the
+agent providers' credentials held for workers (GitHub connections stay), and
+the worker columns of `hosted_machines`.
+
+Refuses to run while a cloud agent of the hosted worker is not removed, or a
+machine that runs the hosted worker is not deleted: nothing here can move them
+onto the agents controller.
+
+Revision ID: f5d2b8e6c3a1
+Revises: e7a2c4b9d013
+Create Date: 2026-10-09 09:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "f5d2b8e6c3a1"
+down_revision: str | None = "e7a2c4b9d013"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+ connection = op.get_bind()
+ launches = connection.scalar(
+ sa.text("SELECT count(*) FROM hosted_launches WHERE state <> 'deleted'")
+ )
+ machines = connection.scalar(
+ sa.text(
+ "SELECT count(*) FROM hosted_machines "
+ "WHERE runtime = 'worker' AND state <> 'deleted'"
+ )
+ )
+ if launches or machines:
+ raise RuntimeError(
+ f"remove the hosted worker: {launches} cloud agent(s) of the hosted "
+ f"worker are not removed and {machines} cloud machine(s) that run it "
+ "are not deleted. Remove those cloud agents in Switch Console, wait "
+ "until their machines are deleted, then upgrade again."
+ )
+
+ op.drop_table("hosted_wake_mailbox")
+ op.drop_table("hosted_cutover_items")
+ op.drop_table("hosted_cutover_volumes")
+ op.drop_table("hosted_operations")
+ op.drop_table("github_issued_tokens")
+ op.drop_table("hosted_launches")
+ op.drop_table("provider_verifications")
+
+ op.execute("DELETE FROM provider_connections WHERE provider <> 'github'")
+ op.drop_constraint(
+ "ck_provider_connections_provider", "provider_connections", type_="check"
+ )
+ op.drop_constraint(
+ "ck_provider_connections_kind", "provider_connections", type_="check"
+ )
+ op.create_check_constraint(
+ "ck_provider_connections_provider",
+ "provider_connections",
+ "provider = 'github'",
+ )
+ op.create_check_constraint(
+ "ck_provider_connections_kind", "provider_connections", "kind = 'oauth'"
+ )
+
+ op.drop_constraint("ck_hosted_machine_runtime", "hosted_machines", type_="check")
+ for column in (
+ "runtime",
+ "machine_capability_hash",
+ "machine_capability_encrypted",
+ "machine_capability_revision",
+ "agents_version",
+ ):
+ op.drop_column("hosted_machines", column)
+
+
+def downgrade() -> None:
+ raise RuntimeError(
+ "remove the hosted worker cannot be undone: the hosted worker's tables "
+ "and the credentials held for it are gone."
+ )
diff --git a/core/switch_core/providers/claude_verifier.py b/core/switch_core/providers/claude_verifier.py
deleted file mode 100644
index 2ff1e00e8..000000000
--- a/core/switch_core/providers/claude_verifier.py
+++ /dev/null
@@ -1,103 +0,0 @@
-from __future__ import annotations
-
-import asyncio
-import json
-import os
-import signal
-import tempfile
-from pathlib import Path
-
-
-class ClaudeVerificationError(Exception):
- pass
-
-
-class ClaudeVerifier:
- def __init__(self, executable: str):
- if not Path(executable).is_absolute() or not os.access(executable, os.X_OK):
- raise ValueError("Claude verification requires an absolute executable path")
- self.executable = executable
- self.slots = asyncio.Semaphore(2)
-
- async def verify(self, kind: str, credential: str) -> None:
- if self.slots.locked():
- raise ClaudeVerificationError(
- "Verification is busy. Please try again shortly."
- )
- async with self.slots:
- with tempfile.TemporaryDirectory(prefix="switch-claude-check-") as home:
- variable = (
- "ANTHROPIC_API_KEY"
- if kind == "api-key"
- else "CLAUDE_CODE_OAUTH_TOKEN"
- )
- env = {
- "PATH": "/usr/local/bin:/usr/bin:/bin",
- "HOME": home,
- "CLAUDE_CONFIG_DIR": home,
- "DISABLE_TELEMETRY": "1",
- "DISABLE_ERROR_REPORTING": "1",
- "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
- variable: credential,
- }
- try:
- process = await asyncio.create_subprocess_exec(
- self.executable,
- "--print",
- "Reply with OK.",
- "--model",
- "haiku",
- "--output-format",
- "json",
- "--max-turns",
- "1",
- "--tools",
- "",
- "--setting-sources",
- "",
- "--strict-mcp-config",
- "--mcp-config",
- '{"mcpServers":{}}',
- "--disable-slash-commands",
- "--no-session-persistence",
- cwd=home,
- env=env,
- start_new_session=True,
- stdin=asyncio.subprocess.DEVNULL,
- stdout=asyncio.subprocess.PIPE,
- stderr=asyncio.subprocess.DEVNULL,
- )
- except OSError:
- raise ClaudeVerificationError(
- "Claude verification could not start. Contact your server administrator."
- ) from None
- try:
- output, _ = await asyncio.wait_for(
- process.communicate(), timeout=25
- )
- except TimeoutError:
- raise ClaudeVerificationError(
- "Claude verification timed out. Please try again."
- ) from None
- finally:
- try:
- os.killpg(process.pid, signal.SIGKILL)
- except ProcessLookupError:
- pass
- await process.wait()
- try:
- result = json.loads(output)
- except (ValueError, UnicodeError):
- raise ClaudeVerificationError(
- "Claude did not return a verification result. Please try again."
- ) from None
- if (
- process.returncode != 0
- or not isinstance(result, dict)
- or result.get("type") != "result"
- or result.get("subtype") != "success"
- or result.get("is_error") is not False
- ):
- raise ClaudeVerificationError(
- "Claude could not complete the check. Check your credential, plan or API billing, then try again."
- )
diff --git a/core/switch_core/providers/credentials.py b/core/switch_core/providers/credentials.py
deleted file mode 100644
index cfd70437d..000000000
--- a/core/switch_core/providers/credentials.py
+++ /dev/null
@@ -1,88 +0,0 @@
-import json
-from typing import Literal
-
-from pydantic import BaseModel, Field, HttpUrl, TypeAdapter, field_validator
-
-
-class OpenCodeApiCredential(BaseModel):
- type: Literal["api"]
- key: str = Field(min_length=1)
-
-
-class OpenCodeOAuthCredential(BaseModel):
- type: Literal["oauth"]
- access: str = Field(min_length=1)
- refresh: str = Field(min_length=1)
- expires: int
-
-
-class OpenCodeConsoleAccount(BaseModel):
- id: str = Field(min_length=1)
- email: str = Field(min_length=1)
- url: str
- access_token: str = Field(min_length=1)
- refresh_token: str = Field(min_length=1)
- token_expiry: int | None
- time_created: int
- time_updated: int
-
- @field_validator("url")
- @classmethod
- def https_url(cls, value: str) -> str:
- if TypeAdapter(HttpUrl).validate_python(value).scheme != "https":
- raise ValueError("An HTTPS account URL is required.")
- return value
-
-
-class OpenCodeConsoleCredential(BaseModel):
- format: Literal["switch-opencode-console-v1"]
- account: OpenCodeConsoleAccount
- organization: str = Field(min_length=1)
-
-
-PROVIDER_KINDS = {
- "claude": {"api-key", "setup-token"},
- "codex": {"api-key", "auth-json"},
- "cursor": {"api-key"},
- "opencode": {"auth-json"},
- "antigravity": {"auth-json"},
-}
-
-
-def validate_provider_credential(provider: str, kind: str, credential: str) -> str:
- if kind not in PROVIDER_KINDS.get(provider, set()):
- raise ValueError("Choose a supported credential type for this provider.")
- credential = credential.strip()
- if not credential or len(credential.encode()) > 16384:
- raise ValueError("The credential must be nonempty and smaller than 16 KiB.")
- if kind == "auth-json":
- try:
- value = json.loads(credential)
- except (ValueError, RecursionError):
- raise ValueError(
- "Choose the provider's JSON authentication file."
- ) from None
- if not isinstance(value, dict) or not value:
- raise ValueError("The authentication file must contain a JSON object.")
- if provider == "opencode":
- try:
- if value.get("format") == "switch-opencode-console-v1":
- parsed = OpenCodeConsoleCredential.model_validate(value)
- value = parsed.model_dump(mode="json")
- else:
- login: OpenCodeApiCredential | OpenCodeOAuthCredential = (
- TypeAdapter(
- OpenCodeApiCredential | OpenCodeOAuthCredential
- ).validate_python(value.get("opencode"))
- )
- value = {"opencode": login.model_dump()}
- except ValueError:
- raise ValueError(
- "Sign in to OpenCode and choose its authentication file."
- ) from None
- return json.dumps(value, separators=(",", ":"))
- if any(not 33 <= ord(character) <= 126 for character in credential):
- raise ValueError(
- "The API key must not contain whitespace or control characters."
- )
- return credential
diff --git a/core/switch_core/providers/github.py b/core/switch_core/providers/github.py
index 4f2576c1d..c09738059 100644
--- a/core/switch_core/providers/github.py
+++ b/core/switch_core/providers/github.py
@@ -37,13 +37,6 @@ def rate_limited(response: httpx.Response) -> bool:
)
-def repository_writable(repository: dict) -> bool:
- permissions = repository.get("permissions")
- return isinstance(permissions, dict) and any(
- permissions.get(key) is True for key in ("push", "maintain", "admin")
- )
-
-
@dataclass
class GitHubFlow:
tenant_id: str
diff --git a/core/switch_core/providers/github_installation.py b/core/switch_core/providers/github_installation.py
deleted file mode 100644
index 4c6149155..000000000
--- a/core/switch_core/providers/github_installation.py
+++ /dev/null
@@ -1,180 +0,0 @@
-from __future__ import annotations
-
-import asyncio
-import logging
-import time
-from dataclasses import dataclass, field
-from datetime import datetime
-from pathlib import Path
-
-import httpx
-import jwt
-from cryptography.hazmat.primitives.asymmetric.rsa import RSAPrivateKey
-from cryptography.hazmat.primitives.serialization import load_pem_private_key
-
-from switch_core.providers.github import (
- GitHubConnections,
- GitHubError,
- GitHubUnavailableError,
- rate_limited,
- repository_writable,
-)
-
-logger = logging.getLogger(__name__)
-
-
-@dataclass(frozen=True)
-class RepositoryCredential:
- token: str = field(repr=False)
- expires_at: datetime
- repository_id: int
- repository_name: str
-
-
-class GitHubInstallationCredentials:
- def __init__(self, client_id: str, private_key_path: str):
- if not client_id:
- raise ValueError("A GitHub App client ID is required.")
- try:
- key = load_pem_private_key(
- Path(private_key_path).read_bytes(), password=None
- )
- except (ValueError, TypeError, OSError):
- raise ValueError(
- "The GitHub App signing key could not be loaded."
- ) from None
- if not isinstance(key, RSAPrivateKey) or key.key_size < 2048:
- raise ValueError(
- "The GitHub App signing key must be an RSA key of at least 2048 bits."
- )
- self._key = key
- self._client_id = client_id
-
- @staticmethod
- async def revoke(token: str) -> None:
- try:
- async with httpx.AsyncClient(timeout=20, follow_redirects=False) as client:
- response = await client.delete(
- "https://api.github.com/installation/token",
- headers={
- "Authorization": f"Bearer {token}",
- "Accept": "application/vnd.github+json",
- },
- )
- if response.status_code not in (204, 401, 404, 422):
- raise GitHubError("Could not revoke the GitHub repository credential.")
- except httpx.HTTPError:
- raise GitHubError(
- "Could not reach GitHub to revoke the repository credential."
- ) from None
-
- async def issue(
- self,
- github: GitHubConnections,
- user_token: str,
- installation_id: int,
- repository_id: int,
- ) -> RepositoryCredential:
- if (
- type(installation_id) is not int
- or installation_id <= 0
- or type(repository_id) is not int
- or repository_id <= 0
- ):
- raise GitHubError("Choose a valid GitHub installation and repository.")
- installations = await github.repositories(user_token)
- repository = next(
- (
- repo
- for installation in installations
- if installation["id"] == installation_id
- for repo in installation["repositories"]
- if repo["id"] == repository_id
- ),
- None,
- )
- if repository is None:
- raise GitHubError(
- "Your GitHub account no longer has access to the selected repository."
- )
- if not repository_writable(repository):
- raise GitHubError(
- "Your GitHub account needs write access to this repository to run a cloud agent."
- )
- now = int(time.time())
- assertion = jwt.encode(
- {"iat": now - 60, "exp": now + 540, "iss": self._client_id},
- self._key,
- algorithm="RS256",
- )
- token = None
- try:
- async with httpx.AsyncClient(timeout=20, follow_redirects=False) as client:
- response = await client.post(
- f"https://api.github.com/app/installations/{installation_id}/access_tokens",
- headers={
- "Authorization": f"Bearer {assertion}",
- "Accept": "application/vnd.github+json",
- "X-GitHub-Api-Version": "2022-11-28",
- },
- json={
- "repository_ids": [repository_id],
- "permissions": {"contents": "write", "pull_requests": "write"},
- },
- )
- if rate_limited(response) or response.status_code >= 500:
- raise GitHubUnavailableError(
- "GitHub is temporarily unavailable or rate limited. Please retry."
- )
- if response.status_code != 201:
- raise GitHubError(
- "Could not authorize the worker for this repository. Check the GitHub App installation and signing key."
- )
- result = response.json()
- token = result["token"]
- expires_at = datetime.fromisoformat(
- result["expires_at"].replace("Z", "+00:00")
- )
- permissions = result["permissions"]
- if (
- not isinstance(token, str)
- or not token
- or len(token) > 16384
- or any(not 33 <= ord(char) <= 126 for char in token)
- or expires_at.tzinfo is None
- or not time.time() + 60 < expires_at.timestamp() <= time.time() + 3660
- or not isinstance(permissions, dict)
- or permissions.get("contents") != "write"
- or permissions.get("pull_requests") != "write"
- or set(permissions) - {"contents", "pull_requests", "metadata"}
- or permissions.get("metadata", "read") != "read"
- or [repo["id"] for repo in result["repositories"]] != [repository_id]
- ):
- raise GitHubError(
- "GitHub returned an invalid repository credential or scope."
- )
- return RepositoryCredential(
- token, expires_at, repository_id, repository["name"]
- )
- except (
- GitHubError,
- httpx.HTTPError,
- ValueError,
- KeyError,
- TypeError,
- AttributeError,
- ) as error:
- if isinstance(token, str) and token:
- try:
- async with asyncio.timeout(8):
- await self.revoke(token)
- except Exception as cleanup_error:
- logger.error(
- "Rejected GitHub token could not be revoked: error_type=%s",
- type(cleanup_error).__name__,
- )
- if isinstance(error, GitHubError):
- raise
- raise GitHubError(
- "Could not obtain a scoped GitHub repository credential. Please retry."
- ) from None
diff --git a/core/switch_core/providers/github_revocations.py b/core/switch_core/providers/github_revocations.py
index 31a7a4ad3..a9efb9518 100644
--- a/core/switch_core/providers/github_revocations.py
+++ b/core/switch_core/providers/github_revocations.py
@@ -1,191 +1,9 @@
import asyncio
import logging
-from datetime import UTC, datetime, timedelta
-from uuid import uuid4
-from sqlalchemy import delete, exists, select, text, update
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.config import SwitchConfig
-from switch_core.db.models import (
- GitHubIssuedToken,
- HostedLaunch,
- ProviderConnection,
- TenantMember,
- require_tenant_id,
-)
from switch_core.providers.github import GitHubConnections
-from switch_core.providers.github_installation import (
- GitHubInstallationCredentials,
- RepositoryCredential,
-)
logger = logging.getLogger(__name__)
-ACCESS_WARNING = "Some GitHub repository access may remain for up to 1 hour."
-
-
-def remember_repository_token(
- session: AsyncSession,
- launch: HostedLaunch,
- credential: RepositoryCredential,
- config: SwitchConfig,
-) -> GitHubIssuedToken:
- record = GitHubIssuedToken(
- tenant_id=require_tenant_id(),
- id=str(uuid4()),
- owner_id=launch.owner_id,
- launch_id=launch.id,
- launch_revision=launch.revision,
- encrypted_token=config.keyring.encrypt(credential.token),
- expires_at=credential.expires_at,
- revoke_requested=False,
- attempts=0,
- )
- session.add(record)
- return record
-
-
-async def queue_revocation(session: AsyncSession, conditions: tuple) -> None:
- await session.execute(
- update(GitHubIssuedToken)
- .where(GitHubIssuedToken.tenant_id == require_tenant_id(), *conditions)
- .values(revoke_requested=True)
- )
-
-
-async def revoke_pending(
- session: AsyncSession, config: SwitchConfig, conditions: tuple
-) -> bool:
- assert not session.in_transaction()
- try:
- return await _revoke_pending(session, config, conditions)
- except Exception as error:
- logger.error(
- "Committed access change has pending GitHub cleanup: error_type=%s",
- type(error).__name__,
- )
- try:
- await session.rollback()
- except Exception as rollback_error:
- logger.error(
- "GitHub cleanup rollback failed: error_type=%s",
- type(rollback_error).__name__,
- )
- return True
-
-
-async def _revoke_pending(
- session: AsyncSession, config: SwitchConfig, conditions: tuple
-) -> bool:
- """Run one bounded batch after the caller has committed its access change."""
- assert not session.in_transaction()
- tenant_id = require_tenant_id()
- await session.execute(text("SET LOCAL lock_timeout = '2s'"))
- await session.execute(
- text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
- {"key": f"github-revocation:{tenant_id}"},
- )
- await session.execute(
- delete(GitHubIssuedToken).where(
- GitHubIssuedToken.tenant_id == tenant_id,
- GitHubIssuedToken.expires_at <= datetime.now(UTC),
- )
- )
- valid_launch = exists(
- select(HostedLaunch.id).where(
- HostedLaunch.tenant_id == tenant_id,
- HostedLaunch.id == GitHubIssuedToken.launch_id,
- HostedLaunch.revision == GitHubIssuedToken.launch_revision,
- HostedLaunch.desired_state == "running",
- HostedLaunch.state != "error",
- )
- )
- member = exists(
- select(TenantMember.user_id).where(
- TenantMember.tenant_id == tenant_id,
- TenantMember.user_id == GitHubIssuedToken.owner_id,
- )
- )
- connected = exists(
- select(ProviderConnection.user_id).where(
- ProviderConnection.tenant_id == tenant_id,
- ProviderConnection.user_id == GitHubIssuedToken.owner_id,
- ProviderConnection.provider == "github",
- )
- )
- await queue_revocation(session, (~(valid_launch & member & connected),))
- deadline = datetime.now(UTC) + timedelta(seconds=90)
- rows = list(
- await session.scalars(
- select(GitHubIssuedToken)
- .where(
- GitHubIssuedToken.tenant_id == tenant_id,
- GitHubIssuedToken.revoke_requested.is_(True),
- (
- GitHubIssuedToken.claim_until.is_(None)
- | (GitHubIssuedToken.claim_until <= datetime.now(UTC))
- ),
- *conditions,
- )
- .order_by(GitHubIssuedToken.attempts, GitHubIssuedToken.expires_at)
- .limit(8)
- )
- )
- for row in rows:
- row.attempts += 1
- row.claim_until = deadline
- await session.commit()
-
- async def revoke(row: GitHubIssuedToken) -> str | None:
- try:
- async with asyncio.timeout(8):
- await GitHubInstallationCredentials.revoke(
- config.keyring.decrypt(row.encrypted_token)
- )
- return row.id
- except Exception as error:
- logger.error(
- "GitHub repository token revocation failed: tenant=%s launch=%s token_record=%s error_type=%s",
- tenant_id,
- row.launch_id,
- row.id,
- type(error).__name__,
- )
- return None
-
- claimed_ids = [row.id for row in rows]
- completed = [
- key for key in await asyncio.gather(*(revoke(row) for row in rows)) if key
- ]
- if completed:
- await session.execute(
- delete(GitHubIssuedToken).where(
- GitHubIssuedToken.tenant_id == tenant_id,
- GitHubIssuedToken.id.in_(completed),
- GitHubIssuedToken.claim_until == deadline,
- )
- )
- if claimed_ids:
- await session.execute(
- update(GitHubIssuedToken)
- .where(
- GitHubIssuedToken.tenant_id == tenant_id,
- GitHubIssuedToken.id.in_(claimed_ids),
- GitHubIssuedToken.claim_until == deadline,
- )
- .values(claim_until=None)
- )
- remaining = await session.scalar(
- select(GitHubIssuedToken.id)
- .where(
- GitHubIssuedToken.tenant_id == tenant_id,
- GitHubIssuedToken.revoke_requested.is_(True),
- *conditions,
- )
- .limit(1)
- )
- await session.commit()
- return remaining is not None
async def revoke_oauth(github: GitHubConnections, token: str) -> str | None:
diff --git a/core/switch_core/providers/hosted.py b/core/switch_core/providers/hosted.py
index f6cf2c29b..7a291e7c5 100644
--- a/core/switch_core/providers/hosted.py
+++ b/core/switch_core/providers/hosted.py
@@ -1,5 +1,3 @@
-from pathlib import Path
-from typing import Annotated
from urllib.parse import urlsplit
from pydantic import (
@@ -7,7 +5,6 @@ from pydantic import (
ConfigDict,
Field,
SecretStr,
- StringConstraints,
field_validator,
)
@@ -17,26 +14,8 @@ class HostedControllerSettings(BaseModel):
tenant_id: str = Field(min_length=1)
token: SecretStr = Field(min_length=32)
- machine_slots: list[
- Annotated[str, StringConstraints(pattern=r"^[a-z0-9][a-z0-9-]{2,39}$")]
- ] = Field(min_length=1, max_length=100)
- github_private_key_path: Path
agent_api_endpoint: str
- @field_validator("machine_slots")
- @classmethod
- def unique_slots(cls, value: list[str]) -> list[str]:
- if len(set(value)) != len(value):
- raise ValueError("Cloud machine slots must be unique.")
- return value
-
- @field_validator("github_private_key_path")
- @classmethod
- def absolute_key_path(cls, value: Path) -> Path:
- if not value.is_absolute():
- raise ValueError("The GitHub signing key requires an absolute path.")
- return value
-
@field_validator("agent_api_endpoint")
@classmethod
def https_endpoint(cls, value: str) -> str:
diff --git a/core/switch_core/providers/verification.py b/core/switch_core/providers/verification.py
deleted file mode 100644
index 9f241cb36..000000000
--- a/core/switch_core/providers/verification.py
+++ /dev/null
@@ -1,106 +0,0 @@
-import hashlib
-import secrets
-from datetime import UTC, datetime, timedelta
-from uuid import uuid4
-
-from fastapi import HTTPException
-from sqlalchemy import func, select, text
-from sqlalchemy.ext.asyncio import AsyncSession
-
-from switch_core.db.models import ProviderVerification, require_tenant_id
-from switch_core.keys import Keyring
-
-ACTIVE = ("queued", "running", "finishing")
-FAILURE = "The provider could not complete the connection check. Sign in again and retry. If it still fails, check your plan or API billing."
-
-
-def summary(job: ProviderVerification) -> dict:
- state = job.state
- if state in ACTIVE and job.deadline <= datetime.now(UTC):
- state = "failed"
- return {
- "status": "verifying"
- if state in ACTIVE
- else "connected"
- if state == "succeeded"
- else "failed",
- "verification_id": job.id,
- "kind": job.kind,
- "created_at": str(job.created_at),
- "error": FAILURE if state in ("failed", "cancelled") else None,
- }
-
-
-async def latest(
- session: AsyncSession, user_id: str, provider: str
-) -> ProviderVerification | None:
- job: ProviderVerification | None = await session.scalar(
- select(ProviderVerification)
- .where(
- ProviderVerification.tenant_id == require_tenant_id(),
- ProviderVerification.user_id == user_id,
- ProviderVerification.provider == provider,
- )
- .order_by(ProviderVerification.created_at.desc())
- .limit(1)
- )
- return job
-
-
-async def queue(
- session: AsyncSession,
- user_id: str,
- provider: str,
- kind: str,
- credential: str,
- keyring: Keyring,
-) -> dict:
- await session.execute(
- text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
- {"key": f"provider-verification:{require_tenant_id()}"},
- )
- previous = await latest(session, user_id, provider)
- now = datetime.now(UTC)
- if previous and previous.state in ACTIVE:
- if previous.deadline > now:
- if (
- previous.kind != kind
- or not previous.encrypted_credential
- or keyring.decrypt(previous.encrypted_credential) != credential
- ):
- raise HTTPException(
- 409,
- "A connection check is already running. Wait for it to finish before changing the credential.",
- )
- return summary(previous)
- previous.state = "failed"
- previous.encrypted_credential = None
- previous.encrypted_token = None
- await session.flush()
- count = await session.scalar(
- select(func.count())
- .select_from(ProviderVerification)
- .where(
- ProviderVerification.tenant_id == require_tenant_id(),
- ProviderVerification.state.in_(ACTIVE),
- )
- )
- if count and count >= 2:
- raise HTTPException(429, "Connection checks are busy. Please retry shortly.")
- token = secrets.token_urlsafe(32)
- job = ProviderVerification(
- tenant_id=require_tenant_id(),
- id=str(uuid4()),
- user_id=user_id,
- provider=provider,
- kind=kind,
- encrypted_credential=keyring.encrypt(credential),
- encrypted_token=keyring.encrypt(token),
- token_hash=hashlib.sha256(token.encode()).hexdigest(),
- state="queued",
- created_at=now,
- deadline=now + timedelta(minutes=10),
- )
- session.add(job)
- await session.commit()
- return summary(job)
diff --git a/core/switch_core/session_activity/maintenance.py b/core/switch_core/session_activity/maintenance.py
index f5a4902da..c34518b19 100644
--- a/core/switch_core/session_activity/maintenance.py
+++ b/core/switch_core/session_activity/maintenance.py
@@ -13,8 +13,7 @@ from __future__ import annotations
import asyncio
import logging
-from collections.abc import Awaitable, Callable
-from datetime import UTC, datetime, timedelta
+from datetime import timedelta
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -27,12 +26,6 @@ logger = logging.getLogger(__name__)
EXPIRY_INTERVAL_SECONDS = 5.0
PRUNE_EVERY_SECONDS = 3600.0
ACTIVITY_RETENTION = timedelta(days=7)
-#: How often the hosted wake mailbox is reclaimed, expired, pruned and re-offered.
-MAILBOX_EVERY_SECONDS = 30.0
-
-#: One wake mailbox pass over the bound tenant; given the time of the previous
-#: pass, for its metrics.
-MailboxUpkeep = Callable[[datetime], Awaitable[None]]
async def maintain_once(
@@ -59,40 +52,13 @@ async def maintain_once(
)
-async def _mailbox_pass(
- session_factory: async_sessionmaker[AsyncSession],
- mailbox_upkeep: MailboxUpkeep,
- since: datetime,
-) -> datetime:
- started = datetime.now(UTC)
- try:
- tenant_ids = await all_tenant_ids(session_factory)
- except Exception:
- logger.exception("Wake mailbox upkeep could not list tenants")
- return started
- for tenant_id in tenant_ids:
- with tenant_scope(tenant_id):
- try:
- await mailbox_upkeep(since)
- except Exception:
- logger.exception("Wake mailbox upkeep failed for tenant %s", tenant_id)
- return started
-
-
-async def maintenance_loop(
- session_factory: async_sessionmaker[AsyncSession], mailbox_upkeep: MailboxUpkeep
-) -> None:
- """Session-activity upkeep every 5 s; the wake mailbox at start and every 30 s."""
+async def maintenance_loop(session_factory: async_sessionmaker[AsyncSession]) -> None:
+ """Session-activity upkeep every 5 s."""
with no_tenant():
- mailbox_since = await _mailbox_pass(
- session_factory, mailbox_upkeep, datetime.now(UTC)
- )
- since_mailbox = 0.0
since_prune = PRUNE_EVERY_SECONDS
while True:
await asyncio.sleep(EXPIRY_INTERVAL_SECONDS)
since_prune += EXPIRY_INTERVAL_SECONDS
- since_mailbox += EXPIRY_INTERVAL_SECONDS
prune = since_prune >= PRUNE_EVERY_SECONDS
try:
await maintain_once(session_factory, prune=prune)
@@ -101,8 +67,3 @@ async def maintenance_loop(
else:
if prune:
since_prune = 0.0
- if since_mailbox >= MAILBOX_EVERY_SECONDS:
- mailbox_since = await _mailbox_pass(
- session_factory, mailbox_upkeep, mailbox_since
- )
- since_mailbox = 0.0
diff --git a/core/switch_core/telemetry/catalogue.py b/core/switch_core/telemetry/catalogue.py
index 900bff29f..c82d96521 100644
--- a/core/switch_core/telemetry/catalogue.py
+++ b/core/switch_core/telemetry/catalogue.py
@@ -407,12 +407,11 @@ CATALOGUE: Mapping[str, Mapping[str, PropertyType]] = {
# A message an agent was asked to act on: addressed to it and let through
# its addressing policy and budget, whether or not the agent was there to
# take it. One per agent addressed, so a message naming two agents is two
- # of these and one `room_message_sent`; a message a hosted agent's mailbox
- # already holds is not counted again, nor a multi-file post delivered in
- # two pieces. `agent_live` is whether the agent had a live session for the
- # room when it arrived: false for one that was offline, still starting, or
- # whose cloud worker refused the message, and always for a session_passive
- # agent, which reads its messages later rather than live.
+ # of these and one `room_message_sent`; a multi-file post delivered in two
+ # pieces is not counted again. `agent_live` is whether the agent had a live
+ # session for the room when it arrived: false for one that was offline or
+ # still starting, and always for a session_passive agent, which reads its
+ # messages later rather than live.
"agent_message_received": {
"sender_kind": SENDER_KIND,
"known_agent_type": KNOWN_AGENT_TYPE,
diff --git a/core/tests/benchmarks/server.py b/core/tests/benchmarks/server.py
index 3949daabb..09b0d76d2 100644
--- a/core/tests/benchmarks/server.py
+++ b/core/tests/benchmarks/server.py
@@ -54,7 +54,6 @@ from switch_core.db.models import (
User,
)
from switch_core.db.session_scope import tenant_session
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
from switch_core.db.stores.tenant_store import TenantStore
from switch_core.main import (
_connection_sweep_loop,
@@ -532,7 +531,6 @@ async def _serve(
agent_session_store=session_env.agent_session_store,
room_role_store=session_env.room_role_store,
external_user_store=session_env.external_user_store,
- hosted_launch_store=HostedLaunchStore(),
connections=connections,
frontend_base_url=config.frontend_base_url,
message_telemetry=message_telemetry,
diff --git a/core/tests/integration/conftest.py b/core/tests/integration/conftest.py
index 64c2ad0b4..7f12f13c8 100644
--- a/core/tests/integration/conftest.py
+++ b/core/tests/integration/conftest.py
@@ -71,7 +71,6 @@ from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
from switch_core.db.stores.document_store import DocumentStore
from switch_core.db.stores.external_user_store import ExternalUserStore
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
from switch_core.db.stores.media_store import MediaStore
from switch_core.db.stores.message_store import MessageStore
from switch_core.db.stores.package_store import PackageStore
@@ -638,7 +637,6 @@ async def harness(
agent_session_store=session_env.agent_session_store,
room_role_store=session_env.room_role_store,
external_user_store=session_env.external_user_store,
- hosted_launch_store=HostedLaunchStore(),
connections=connections,
frontend_base_url=config.frontend_base_url,
message_telemetry=message_telemetry,
diff --git a/core/tests/switch_core/bridges/agent/api/test_delete_agent_route.py b/core/tests/switch_core/bridges/agent/api/test_delete_agent_route.py
index 664fd7d7f..cf8223110 100644
--- a/core/tests/switch_core/bridges/agent/api/test_delete_agent_route.py
+++ b/core/tests/switch_core/bridges/agent/api/test_delete_agent_route.py
@@ -1,31 +1,14 @@
-"""An agent cannot delete itself through the agent API when it is a cloud agent."""
+"""An agent can delete itself through the agent API."""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import AsyncMock
-import pytest
-from fastapi import HTTPException
-
from switch_core.bridges.agent.api.handlers import delete_agent
-async def test_a_cloud_agent_cannot_delete_itself() -> None:
- agent = SimpleNamespace(
- id="agent-1", name="cloud", metadata_={"hosted_launch_id": "launch-1"}
- )
- protocol = AsyncMock()
-
- with pytest.raises(HTTPException) as refused:
- await delete_agent("agent-1", agent, protocol)
-
- assert refused.value.status_code == 409
- assert "cloud agent" in refused.value.detail
- protocol.delete_agent.assert_not_awaited()
-
-
-async def test_a_local_agent_can_delete_itself() -> None:
+async def test_an_agent_can_delete_itself() -> None:
agent = SimpleNamespace(id="agent-1", name="local", metadata_={})
protocol = AsyncMock()
diff --git a/core/tests/switch_core/bridges/agent/operations/test_connect_to_room_eviction.py b/core/tests/switch_core/bridges/agent/operations/test_connect_to_room_eviction.py
index e98a2136a..414a0bb33 100644
--- a/core/tests/switch_core/bridges/agent/operations/test_connect_to_room_eviction.py
+++ b/core/tests/switch_core/bridges/agent/operations/test_connect_to_room_eviction.py
@@ -13,21 +13,12 @@ session connects while the first still holds the slot.
from __future__ import annotations
-import json
from contextlib import asynccontextmanager
from types import SimpleNamespace
from typing import Any
-import httpx
import pytest
-from fastapi import FastAPI
-from fastmcp import Client
-from fastmcp.exceptions import ToolError
-
-from switch_core.bridges.agent.api.operations import router as operations_router
-from switch_core.bridges.agent.auth import get_agent_from_scope
-from switch_core.bridges.agent.dependencies import get_protocol, get_session_factory
-from switch_core.bridges.agent.mcp import server as mcp_server
+
from switch_core.bridges.agent.operations import definitions
from switch_core.bridges.agent.operations.callctx import (
CallContext,
@@ -41,9 +32,6 @@ from switch_core.bridges.agent.protocol.agent_connections import (
ClientDeclaration,
)
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.hosted_workers import (
- HOSTED_WORKER_ONLY_MESSAGE,
-)
AGENT = "agent-1"
ROOM = "room-1"
@@ -266,56 +254,3 @@ async def test_a_caller_with_no_live_connection_still_connects(harness: Any) ->
assert result["warning"] is None
assert harness.store.bindings == [(AGENT, ROOM, "mcp-transport-session")]
-
-
-HOSTED_WORKER_ONLY = {
- "code": "hosted_worker_only",
- "message": HOSTED_WORKER_ONLY_MESSAGE,
-}
-
-
-def hosted(monkeypatch: pytest.MonkeyPatch):
- registry = AgentConnectionRegistry()
- protocol = _protocol(
- registry, _RecordingSessionStore(), {"hosted_launch_id": "launch-1"}
- )
- init_operations_protocol(protocol)
- monkeypatch.setattr(definitions, "build_room_instructions", lambda *a, **kw: "")
- yield SimpleNamespace(registry=registry, protocol=protocol)
- init_operations_protocol(None) # type: ignore[arg-type]
-
-
-async def test_a_hosted_agents_non_worker_is_refused_with_a_coded_detail_over_http(
- hosted: Any,
-) -> None:
- _open(hosted.registry, "local-console")
- app = FastAPI()
- app.include_router(operations_router)
- app.dependency_overrides[get_agent_from_scope] = lambda: SimpleNamespace(id=AGENT)
- app.dependency_overrides[get_protocol] = lambda: hosted.protocol
- app.dependency_overrides[get_session_factory] = lambda: None
- async with httpx.AsyncClient(
- transport=httpx.ASGITransport(app=app), base_url="https://switch.example.com"
- ) as client:
- response = await client.post(
- f"/agents/{AGENT}/ops/connect_to_room",
- json={"room_id": ROOM},
- headers={"x-switch-connection-id": "local-console"},
- )
- assert response.status_code == 403
- assert response.json()["detail"] == HOSTED_WORKER_ONLY
-
-
-async def test_a_hosted_agents_non_worker_is_refused_with_a_coded_detail_over_mcp(
- hosted: Any, monkeypatch: pytest.MonkeyPatch
-) -> None:
- monkeypatch.setattr(
- mcp_server,
- "get_http_request",
- lambda: SimpleNamespace(scope={"agent_id": AGENT}),
- )
- async with Client(mcp_server.mcp) as client:
- with pytest.raises(ToolError) as refused:
- await client.call_tool("connect_to_room", {"room_id": ROOM})
- assert json.loads(str(refused.value)) == HOSTED_WORKER_ONLY
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_controller_presence.py b/core/tests/switch_core/bridges/agent/protocol/test_controller_presence.py
index 6a9710c78..46d74c021 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_controller_presence.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_controller_presence.py
@@ -39,6 +39,7 @@ from switch_core.bridges.agent.protocol.types import AgentStatus
from switch_core.clients.agent_consumer import (
_STARTING_SESSION_MESSAGE,
AgentConsumer,
+ CloudMachineState,
_GateOutcome,
)
from switch_core.db.models import (
@@ -200,9 +201,7 @@ class TestTheRegistryAsksTheController:
holder = registry.controllers.holder_id(binding)
assert not registry.is_live("agent")
- assert not registry.relay_session_command(
- "agent", {"origin": {}}, worker_only=False
- )
+ assert not registry.relay_session_command("agent", {"origin": {}})
assert holder not in registry.live_connection_ids()
conn = _go_live(registry, ("agent", {ROOM}))
@@ -217,9 +216,7 @@ class TestTheRegistryAsksTheController:
assert holder in registry.live_connection_ids()
assert registry.live_connection_count() == 1
assert rooms_occupied("agent", registry) == {ROOM}
- assert registry.relay_session_command(
- "agent", {"origin": {"roomId": ROOM}}, worker_only=False
- )
+ assert registry.relay_session_command("agent", {"origin": {"roomId": ROOM}})
assert conn.session_commands == [("agent", {"origin": {"roomId": ROOM}})]
_lapse(conn)
@@ -353,10 +350,17 @@ async def _no_session() -> AsyncIterator[object]:
yield object()
-def _client(registry: AgentConnectionRegistry, agent_id: str) -> SimpleNamespace:
+def _client(
+ registry: AgentConnectionRegistry,
+ agent_id: str,
+ cloud_machine: CloudMachineState | None = None,
+) -> SimpleNamespace:
async def _unavailable_reply(*_args: Any, **_kwargs: Any) -> str:
return "offline"
+ async def _wake_cloud_machine(_controller_id: str) -> CloudMachineState | None:
+ return cloud_machine
+
async def owner_handle_in(*_args: Any) -> str | None:
return "owner"
@@ -366,6 +370,8 @@ def _client(registry: AgentConnectionRegistry, agent_id: str) -> SimpleNamespace
_agent_session_store=_NoRows(),
_unavailable_reply=_unavailable_reply,
owner_handle_in=owner_handle_in,
+ _wake_cloud_machine=_wake_cloud_machine,
+ _cloud_waking_notices=set(),
)
ns._is_available = AgentConsumer._is_available.__get__(ns)
ns._reply_when_unavailable_here = (
@@ -464,9 +470,7 @@ class TestTheAgentClientsReplies:
}
assert not registry.is_live("auto")
assert rooms_occupied("auto", registry) == set()
- assert not registry.relay_session_command(
- "auto", {"origin": {"roomId": ROOM}}, worker_only=False
- )
+ assert not registry.relay_session_command("auto", {"origin": {"roomId": ROOM}})
registry.controllers.bind(
Binding(
@@ -484,6 +488,43 @@ class TestTheAgentClientsReplies:
is None
)
+ async def test_a_sleeping_cloud_machine_is_woken_and_says_so_once(self) -> None:
+ registry = AgentConnectionRegistry()
+ _bind(registry, "auto")
+ meta = SimpleNamespace(room_id=ROOM, name="Room", bridge_id=None)
+ agent = _agent("auto", "auto_session")
+ client = _client(
+ registry,
+ "auto",
+ CloudMachineState(state="stopped", revision=4, owner_stopped=False),
+ )
+
+ first = await client._reply_when_unavailable_here(object(), agent, meta, "u")
+ again = await client._reply_when_unavailable_here(object(), agent, meta, "u")
+
+ assert first == (
+ "My machine, **machine**, was asleep and is starting now. I'll pick "
+ "this up as soon as it's back, usually within a few minutes."
+ )
+ assert again is None
+
+ async def test_a_cloud_machine_its_owner_stopped_stays_stopped(self) -> None:
+ registry = AgentConnectionRegistry()
+ _bind(registry, "auto")
+ meta = SimpleNamespace(room_id=ROOM, name="Room", bridge_id=None)
+ client = _client(
+ registry,
+ "auto",
+ CloudMachineState(state="stopped", revision=4, owner_stopped=True),
+ )
+ reply = await client._reply_when_unavailable_here(
+ object(), _agent("auto", "auto_session"), meta, "u"
+ )
+ assert reply == (
+ "My machine, **machine**, is stopped, so I can't answer. My owner "
+ "(@owner) can start it again in Switch Console."
+ )
+
async def test_an_offline_or_removed_machine_is_named_not_console(
self,
) -> None:
@@ -557,9 +598,6 @@ def _addressing_client(
async def _gate_addressed(*_args: Any) -> _GateOutcome:
return _GateOutcome(addressed=True, refusal=None)
- async def _not_hosted(_agent: object, _event: object) -> None:
- return None
-
ns = _client(registry, agent_id)
ns.agent = SimpleNamespace(
id=agent_id,
@@ -572,7 +610,6 @@ def _addressing_client(
ns._compute_addressed = _addressed
ns._fresh_agent = _fresh_agent
ns._gate_addressed = _gate_addressed
- ns._note_hosted_addressed = _not_hosted
ns._report_addressed = lambda *_a, **_k: None
ns._triggered_by_auto_reply = AgentConsumer._triggered_by_auto_reply
ns.actor = SimpleNamespace(send_message=sent)
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_room_released.py b/core/tests/switch_core/bridges/agent/protocol/test_room_released.py
index 3c98bd96d..aa2a7f240 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_room_released.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_room_released.py
@@ -9,7 +9,6 @@ import pytest
from switch_core.bridges.agent.protocol.agent_connections import (
HEARTBEAT_LAPSED,
- LAUNCH_SUPERSEDED,
PROTOCOL_VERSION,
ROOM_RELEASED_PROTOCOL_REVISION,
AgentConnection,
@@ -18,8 +17,6 @@ from switch_core.bridges.agent.protocol.agent_connections import (
Released,
)
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.hosted_workers import WorkerBinding
-from switch_core.bridges.agent.protocol.presence import agents_present_in
from switch_core.bridges.agent.protocol.stream import KEEPALIVE, event_frames
AGENT = "agent-1"
@@ -244,40 +241,6 @@ async def test_a_client_that_declared_nothing_is_not_sent_room_released() -> Non
) in frames
-def test_a_closed_hosted_worker_leaves_no_session_in_its_rooms() -> None:
- """A sleeping or stopped hosted agent is not present where it last worked.
-
- Otherwise the agent client reads it as available and the room gets no
- waking notice, and no unavailable reply after an explicit Stop.
- """
- registry = AgentConnectionRegistry()
- worker = _open(
- registry, "worker", speaks=ROOM_RELEASED_PROTOCOL_REVISION, scope="all"
- )
- registry.bind_worker(worker, WorkerBinding("launch-1", 1, "boot-a", "i-a"), {})
- registry.replace_placements(worker, {"session-1": ROOM_A})
- assert agents_present_in([AGENT], ROOM_A, registry) == {AGENT}
-
- registry.supersede(AGENT, 2)
-
- assert registry.placements(AGENT) == {}
- assert agents_present_in([AGENT], ROOM_A, registry) == set()
-
-
-def test_closing_a_worker_keeps_placements_made_elsewhere() -> None:
- registry = AgentConnectionRegistry()
- worker = _open(
- registry, "worker", speaks=ROOM_RELEASED_PROTOCOL_REVISION, scope="all"
- )
- registry.bind_worker(worker, WorkerBinding("launch-1", 1, "boot-a", "i-a"), {})
- registry.replace_placements(worker, {"session-1": ROOM_A})
- registry.place_session(AGENT, "session-2", ROOM_B, "mcp-transport")
-
- registry.close(worker.id, LAUNCH_SUPERSEDED)
-
- assert registry.placements(AGENT) == {"session-2": ROOM_B}
-
-
def test_closing_a_local_connection_keeps_its_placements() -> None:
registry = AgentConnectionRegistry()
conn = _open(registry, "local", speaks=ROOM_RELEASED_PROTOCOL_REVISION, scope="all")
diff --git a/core/tests/switch_core/bridges/agent/test_control_commands.py b/core/tests/switch_core/bridges/agent/test_control_commands.py
index 5c4f30348..52025963d 100644
--- a/core/tests/switch_core/bridges/agent/test_control_commands.py
+++ b/core/tests/switch_core/bridges/agent/test_control_commands.py
@@ -9,13 +9,11 @@ import pytest
import switch_core.bridges.agent.commands as commands
from switch_core.bridges.agent.commands import (
_check_control_target,
- _cmd_interrupt,
_cmd_reset,
)
from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
from switch_core.bridges.agent.protocol.controller_presence import Binding
from switch_core.bridges.agent.protocol.types import AgentStatus
-from switch_core.db.models import HostedLaunch, HostedMachine
from switch_core.events import CommandEvent
@@ -47,8 +45,6 @@ def _client(
enqueue: list[Any],
placed: str | None = None,
relayed: list[Any] | None = None,
- hosted: bool = False,
- woke: list[Any] | None = None,
) -> SimpleNamespace:
"""Minimal AgentConsumer stand-in for _dispatch_control_command."""
@@ -56,7 +52,7 @@ def _client(
id="agent-1",
tenant_id="tenant-1",
name="cc",
- metadata_={"hosted_launch_id": "launch-1"} if hosted else {},
+ metadata_={},
integration_profile={
"command_capabilities": {
"reset": command_level,
@@ -65,13 +61,7 @@ def _client(
},
)
- async def _note_hosted_addressed(noted: Any, event: Any) -> None:
- assert event is None
- assert woke is not None
- woke.append(noted.id)
-
- def _relay(_agent: str, frame: Any, *, worker_only: bool) -> bool:
- assert worker_only is hosted
+ def _relay(_agent: str, frame: Any) -> bool:
return relayed is not None and (relayed.append(frame) or True)
async def _fresh_agent(_session: Any) -> SimpleNamespace:
@@ -100,7 +90,6 @@ def _client(
_resolve_room_meta=_resolve_room_meta,
session_factory=_session_factory,
tenant_id="tenant-1",
- _note_hosted_addressed=_note_hosted_addressed,
_agent_session_store=SimpleNamespace(),
_room_role_store=SimpleNamespace(agent_room_role=_agent_room_role),
# Presence unions the heartbeat rows with the live connections
@@ -432,208 +421,3 @@ def test_an_interrupt_names_the_current_turn() -> None:
requester_name="louisa",
)
assert frame["body"] == {"type": "turn.interrupt", "turnId": "current"}
-
-
-def _patch_launch(
- monkeypatch: pytest.MonkeyPatch, sleeping: bool, errored: bool = False
-) -> None:
- launch = SimpleNamespace(desired_state="running", machine_id="machine-1")
- machine = SimpleNamespace(
- state="error" if errored else "stopped" if sleeping else "ready",
- desired_state="stopped" if sleeping else "running",
- stop_reason="idle" if sleeping else None,
- )
-
- class _Session:
- async def get(self, model: Any, key: Any) -> Any:
- if model is HostedMachine:
- assert key == ("tenant-1", "machine-1")
- return machine
- assert model is HostedLaunch
- assert key == ("tenant-1", "launch-1")
- return launch
-
- @asynccontextmanager
- async def _tenant_session(_factory: Any, tenant_id: str) -> Any:
- assert tenant_id == "tenant-1"
- yield _Session()
-
- monkeypatch.setattr(commands, "tenant_session", _tenant_session)
-
-
-async def _no_surface(*_a: Any) -> str:
- return "slack"
-
-
-async def test_reset_of_a_sleeping_hosted_agent_wakes_it_and_queues_nothing(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- reply = _Reply()
- woke: list[Any] = []
- enqueue: list[Any] = []
- monkeypatch.setattr(commands, "_reply", reply)
- _patch_launch(monkeypatch, sleeping=True)
-
- await _cmd_reset(
- _client(
- reply,
- command_level="session_dependent",
- enqueue=enqueue,
- hosted=True,
- woke=woke,
- ),
- SimpleNamespace(room_id="!m:server"),
- _event(),
- False,
- )
-
- assert woke == ["agent-1"]
- assert enqueue == []
- assert reply.bodies == [
- "The cloud worker is waking up. The reset was not queued. Wait until the "
- "agent is back (usually about a minute), then send !reset @cc again to "
- "start a fresh conversation."
- ]
-
-
-async def test_reset_whose_worker_went_away_after_placement_still_wakes(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- reply = _Reply()
- woke: list[Any] = []
- monkeypatch.setattr(commands, "_reply", reply)
- monkeypatch.setattr(commands, "_room_surface", _no_surface)
- _patch_launch(monkeypatch, sleeping=True)
-
- await _cmd_reset(
- _client(
- reply,
- command_level="session_dependent",
- enqueue=[],
- placed="session-1",
- hosted=True,
- woke=woke,
- ),
- SimpleNamespace(room_id="!m:server"),
- _event(),
- False,
- )
-
- assert woke == ["agent-1"]
- assert "waking up" in reply.bodies[-1]
-
-
[email protected]("sleeping", [True, False])
-async def test_reset_of_a_hosted_agent_whose_machine_is_in_error_does_not_wake_it(
- monkeypatch: pytest.MonkeyPatch, sleeping: bool
-) -> None:
- reply = _Reply()
- woke: list[Any] = []
- enqueue: list[Any] = []
- monkeypatch.setattr(commands, "_reply", reply)
- _patch_launch(monkeypatch, sleeping=sleeping, errored=True)
-
- await _cmd_reset(
- _client(
- reply,
- command_level="session_dependent",
- enqueue=enqueue,
- hosted=True,
- woke=woke,
- ),
- SimpleNamespace(room_id="!m:server"),
- _event(),
- False,
- )
-
- assert woke == []
- assert enqueue == []
- assert reply.bodies == [
- "@cc's cloud machine has a problem, so the reset was not sent. Its owner "
- "can check it in Switch Console."
- ]
-
-
-async def test_interrupt_of_a_sleeping_hosted_agent_does_not_wake_it(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- reply = _Reply()
- woke: list[Any] = []
- monkeypatch.setattr(commands, "_reply", reply)
- _patch_launch(monkeypatch, sleeping=True)
-
- await _cmd_interrupt(
- _client(
- reply,
- command_level="session_dependent",
- enqueue=[],
- hosted=True,
- woke=woke,
- ),
- SimpleNamespace(room_id="!m:server"),
- _event("interrupt"),
- False,
- )
-
- assert woke == []
- assert "asleep" in reply.bodies[-1]
-
-
-async def test_an_awake_hosted_agent_relays_to_its_worker_only(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- reply = _Reply()
- relayed: list[Any] = []
- monkeypatch.setattr(commands, "_reply", reply)
- monkeypatch.setattr(commands, "_room_surface", _no_surface)
-
- await _cmd_reset(
- _client(
- reply,
- command_level="session_dependent",
- enqueue=[],
- placed="session-1",
- relayed=relayed,
- hosted=True,
- woke=[],
- ),
- SimpleNamespace(room_id="!m:server"),
- _event(),
- False,
- )
-
- assert len(relayed) == 1
- assert reply.bodies[-1].startswith("Resetting my session")
-
-
-async def test_an_awake_hosted_agent_with_no_worker_is_told_so(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- reply = _Reply()
- woke: list[Any] = []
- monkeypatch.setattr(commands, "_reply", reply)
- monkeypatch.setattr(commands, "_room_surface", _no_surface)
- _patch_launch(monkeypatch, sleeping=False)
-
- await _cmd_reset(
- _client(
- reply,
- command_level="session_dependent",
- enqueue=[],
- placed="session-1",
- hosted=True,
- woke=woke,
- ),
- SimpleNamespace(room_id="!m:server"),
- _event(),
- False,
- )
-
- assert woke == []
- assert "controller is not connected" in reply.bodies[-1]
diff --git a/core/tests/switch_core/clients/test_agent_consumer_attachment_groups.py b/core/tests/switch_core/clients/test_agent_consumer_attachment_groups.py
index 1f8f041c4..cc8813d66 100644
--- a/core/tests/switch_core/clients/test_agent_consumer_attachment_groups.py
+++ b/core/tests/switch_core/clients/test_agent_consumer_attachment_groups.py
@@ -4,7 +4,6 @@ import asyncio
from contextlib import asynccontextmanager
from types import SimpleNamespace
from typing import Any
-from unittest.mock import AsyncMock
import pytest
@@ -115,7 +114,6 @@ def _fake_client() -> SimpleNamespace:
_is_available=_is_available,
availability_checks=0,
queue=queue,
- _note_hosted_addressed=AsyncMock(return_value=None),
)
ns._emit_media = AgentConsumer._emit_media.__get__(ns)
ns._report_addressed = AgentConsumer._report_addressed.__get__(ns)
diff --git a/core/tests/switch_core/clients/test_agent_consumer_reports_addressed.py b/core/tests/switch_core/clients/test_agent_consumer_reports_addressed.py
index 9b265ac44..01467be1a 100644
--- a/core/tests/switch_core/clients/test_agent_consumer_reports_addressed.py
+++ b/core/tests/switch_core/clients/test_agent_consumer_reports_addressed.py
@@ -19,7 +19,6 @@ from switch_core.clients.admin_messages import PLATFORM_MARKER
from switch_core.clients.agent_consumer import (
AUTO_REPLY_FLAG,
AgentConsumer,
- HostedNote,
_GateOutcome,
)
from switch_core.clients.room_meta import RoomMeta
@@ -51,10 +50,9 @@ def _consumer(
addressed: bool,
allowed: bool = True,
live: bool = True,
- hosted: HostedNote | None = None,
) -> SimpleNamespace:
- """An AgentConsumer stand-in with addressing, the policy gate, whether the
- agent is live in the room and a hosted agent's mailbox fixed by the test."""
+ """An AgentConsumer stand-in with addressing, the policy gate and whether
+ the agent is live in the room fixed by the test."""
meta = RoomMeta(
room_id="room-1", name="Room", bridge_id=None, channel_type="channel_public"
)
@@ -81,9 +79,6 @@ def _consumer(
) -> str | None:
return None
- async def _note_hosted_addressed(_a: object, _e: object) -> HostedNote | None:
- return hosted
-
ns = SimpleNamespace(
agent=SimpleNamespace(
id="agent-1", name="agent-a", metadata_={"known_agent_type": "codex"}
@@ -97,7 +92,6 @@ def _consumer(
_gate_addressed=_gate_addressed,
_is_available=_is_available,
_reply_when_unavailable_here=_reply_when_unavailable_here,
- _note_hosted_addressed=_note_hosted_addressed,
_post_auto_reply=_ignore_post,
_sender_handle=lambda _e: "@alice",
_triggered_by_auto_reply=AgentConsumer._triggered_by_auto_reply,
@@ -189,66 +183,6 @@ async def test_a_message_to_an_offline_agent_is_reported_as_not_live() -> None:
assert report["agent_live"] is False
-async def test_a_message_a_hosted_agents_mailbox_refused_is_reported_as_not_live() -> (
- None
-):
- """A stopped or broken cloud worker is an agent that was asked and was not
- there, the same as any other offline agent — not a message nobody sent."""
- consumer = _consumer(
- addressed=True,
- live=False,
- hosted=HostedNote(
- launch=None, machine=None, refusal="worker stopped", deliver=False
- ),
- )
-
- await _deliver(consumer, _message())
-
- [report] = consumer._message_telemetry.addressed
- assert report["agent_live"] is False
-
-
-async def test_a_message_a_live_hosted_worker_could_not_take_is_not_live() -> None:
- """A full mailbox or a lost provider refuses the message even with a
- worker attached, so the agent could not act on it."""
- consumer = _consumer(
- addressed=True,
- live=True,
- hosted=HostedNote(
- launch=None, machine=None, refusal="mailbox full", deliver=False
- ),
- )
-
- await _deliver(consumer, _message())
-
- [report] = consumer._message_telemetry.addressed
- assert report["agent_live"] is False
-
-
-async def test_a_message_held_for_a_waking_hosted_agent_is_reported() -> None:
- consumer = _consumer(
- addressed=True,
- live=False,
- hosted=HostedNote(launch=None, machine=None, refusal=None, deliver=True),
- )
-
- await _deliver(consumer, _message())
-
- assert len(consumer._message_telemetry.addressed) == 1
-
-
-async def test_a_redelivery_the_mailbox_already_holds_is_not_counted_again() -> None:
- """The same message seen a second time is one request, not two."""
- consumer = _consumer(
- addressed=True,
- hosted=HostedNote(launch=None, machine=None, refusal=None, deliver=False),
- )
-
- await _deliver(consumer, _message())
-
- assert consumer._message_telemetry.addressed == []
-
-
async def test_a_telemetry_failure_does_not_cost_the_agent_the_message(
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
diff --git a/core/tests/switch_core/clients/test_no_session_reply_threading.py b/core/tests/switch_core/clients/test_no_session_reply_threading.py
index dcbb6149b..c4acb6ed7 100644
--- a/core/tests/switch_core/clients/test_no_session_reply_threading.py
+++ b/core/tests/switch_core/clients/test_no_session_reply_threading.py
@@ -2,26 +2,15 @@ from __future__ import annotations
from contextlib import asynccontextmanager
from types import SimpleNamespace
-from unittest.mock import AsyncMock
import pytest
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.bridges.agent.protocol.hosted_workers import NOTICE_MESSAGES
from switch_core.clients.admin_messages import PLATFORM_MARKER
from switch_core.clients.agent_consumer import (
- _HOSTED_ERROR_MESSAGE,
- _HOSTED_MACHINE_STOPPED_MESSAGE,
- _HOSTED_REMOVED_MESSAGE,
- _HOSTED_STOPPED_MESSAGE,
_STARTING_SESSION_MESSAGE,
- _WAKING_MESSAGE,
AUTO_REPLY_FLAG,
AgentConsumer,
- HostedNote,
_GateOutcome,
- _hosted_unavailable,
- _takes_mail,
)
from switch_core.transport import InboundMessage, RoomRef
@@ -95,7 +84,6 @@ def _fake_self(
_gate_addressed=_gate_addressed,
_is_available=_is_available,
_reply_when_unavailable_here=_reply_when_unavailable_here,
- _note_hosted_addressed=AsyncMock(return_value=None),
_triggered_by_auto_reply=AgentConsumer._triggered_by_auto_reply,
actor=SimpleNamespace(send_message=send_message),
_event_buffer=SimpleNamespace(enqueue=lambda *a, **k: None),
@@ -311,130 +299,3 @@ async def test_a_kickoff_that_wants_the_channel_gets_its_reply_at_top_level() ->
assert len(send_message.calls) == 1
assert send_message.calls[0]["thread_root_id"] is None
assert send_message.calls[0]["body"].startswith("@dantas.abel ")
-
-
-RUNNING_MACHINE = {"desired_state": "running", "state": "ready", "stop_reason": None}
-
-
-def _hosted(
- send_message: _Recorder, machine: dict[str, object], **launch: object
-) -> SimpleNamespace:
- fake = _fake_self(
- send_message, unavailable_reply="cd /data/workspace && claude ..."
- )
- hosted_launch = SimpleNamespace(
- id="launch-1", agent_id="agent-1", revision=8, **launch
- )
- hosted_machine = SimpleNamespace(revision=3, **machine)
- refusal = (
- None
- if _takes_mail(hosted_launch, hosted_machine) # type: ignore[arg-type]
- else _hosted_unavailable(hosted_launch, hosted_machine) # type: ignore[arg-type]
- )
- fake._note_hosted_addressed = AsyncMock(
- return_value=HostedNote(
- launch=hosted_launch, # type: ignore[arg-type]
- machine=hosted_machine, # type: ignore[arg-type]
- refusal=refusal,
- deliver=refusal is None,
- )
- )
- fake._waking_notice_revisions = {}
- fake._unreachable_notice_revisions = {}
- fake._connections = AgentConnectionRegistry()
- return fake
-
-
- ("launch", "machine", "notice"),
- [
- (
- {"desired_state": "stopped", "state": "stopped"},
- RUNNING_MACHINE,
- _HOSTED_STOPPED_MESSAGE,
- ),
- (
- {"desired_state": "deleted", "state": "deleting"},
- RUNNING_MACHINE,
- _HOSTED_REMOVED_MESSAGE,
- ),
- (
- {"desired_state": "running", "state": "error"},
- RUNNING_MACHINE,
- _HOSTED_ERROR_MESSAGE,
- ),
- (
- {"desired_state": "running", "state": "ready"},
- {"desired_state": "stopped", "state": "stopped", "stop_reason": "owner"},
- _HOSTED_MACHINE_STOPPED_MESSAGE,
- ),
- (
- {"desired_state": "running", "state": "stopped"},
- {"desired_state": "running", "state": "stopped", "stop_reason": None},
- _WAKING_MESSAGE,
- ),
- (
- {"desired_state": "running", "state": "provisioning"},
- RUNNING_MACHINE,
- _WAKING_MESSAGE,
- ),
- (
- {"desired_state": "running", "state": "ready"},
- RUNNING_MACHINE,
- NOTICE_MESSAGES["unreachable"],
- ),
- ],
-)
-async def test_an_unavailable_hosted_agent_says_what_its_worker_is_doing(
- launch: dict[str, object], machine: dict[str, object], notice: str
-) -> None:
- # Not the local terminal command: the agent runs on a cloud worker.
- send_message = _Recorder()
- await AgentConsumer.on_message(
- _hosted(send_message, machine, **launch),
- RoomRef(room_id="!matrix:server"),
- _event(None),
- )
-
- assert [call["body"] for call in send_message.calls] == [f"@louisa {notice}"]
-
-
-async def test_an_unconnected_hosted_worker_is_announced_once_per_room_and_revision() -> (
- None
-):
- send_message = _Recorder()
- fake = _hosted(
- send_message, RUNNING_MACHINE, desired_state="running", state="ready"
- )
- room = RoomRef(room_id="!matrix:server")
-
- await AgentConsumer.on_message(fake, room, _event(None))
- await AgentConsumer.on_message(fake, room, _event(None))
-
- assert [call["body"] for call in send_message.calls] == [
- f"@louisa {NOTICE_MESSAGES['unreachable']}"
- ]
-
-
-async def test_a_connected_hosted_worker_is_not_called_unreachable() -> None:
- send_message = _Recorder()
- fake = _hosted(
- send_message, RUNNING_MACHINE, desired_state="running", state="ready"
- )
- fake._connections = SimpleNamespace(
- attached_worker=lambda _agent_id: SimpleNamespace(
- worker=SimpleNamespace(launch_id="launch-1", launch_revision=8)
- ),
- supersede=lambda *_: None,
- )
-
- await AgentConsumer.on_message(
- fake, RoomRef(room_id="!matrix:server"), _event(None)
- )
-
- assert [call["body"] for call in send_message.calls] == [
- "@louisa cd /data/workspace && claude ..."
- ]
diff --git a/core/tests/switch_core/db/stores/test_agent_management_stores.py b/core/tests/switch_core/db/stores/test_agent_management_stores.py
index dd48158b1..b039ff5e8 100644
--- a/core/tests/switch_core/db/stores/test_agent_management_stores.py
+++ b/core/tests/switch_core/db/stores/test_agent_management_stores.py
@@ -235,12 +235,14 @@ class TestEnrollmentCodes:
owner_id=owner.id,
api_key_id=live.id,
expires_at=NOW + timedelta(minutes=10),
+ hosted_machine_id=None,
)
await CONTROLLERS.create_enrollment_code(
session,
owner_id=owner.id,
api_key_id=stale.id,
expires_at=NOW - timedelta(seconds=1),
+ hosted_machine_id=None,
)
first = await CONTROLLERS.consume_enrollment_code(
session, TENANT_ZERO_ID, live.id, NOW
diff --git a/core/tests/switch_core/db/stores/test_hosted_launch_store.py b/core/tests/switch_core/db/stores/test_hosted_launch_store.py
deleted file mode 100644
index 90f88acfb..000000000
--- a/core/tests/switch_core/db/stores/test_hosted_launch_store.py
+++ /dev/null
@@ -1,340 +0,0 @@
-from datetime import UTC, datetime, timedelta
-
-import pytest
-
-from switch_core.db.models import (
- HostedLaunch,
- HostedMachine,
- ProviderConnection,
- Tenant,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.hosted_launch_store import (
- HostedLaunchConflict,
- HostedLaunchStore,
- ProviderDisconnected,
- is_waking,
-)
-from switch_core.db.stores.hosted_machine_store import HostedMachineStore
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.keys import Keyring
-from switch_core.tenant_context import tenant_scope
-
-TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
-
-SLOTS = ["slot-a", "slot-b"]
-
-
-async def launches(session_factory):
- async with session_factory() as session:
- session.add_all(
- [
- User(
- id="launch-owner",
- name="Owner",
- role="user",
- password_hash="unused",
- ),
- User(
- id="launch-other",
- name="Other",
- role="user",
- password_hash="unused",
- ),
- Tenant(id="launch-other-tenant", slug="launch-other", name="Other"),
- ]
- )
- await session.flush()
- await ProviderConnectionStore().save(
- session,
- "launch-owner",
- "setup-token",
- TEST_KEYRING.encrypt("SYNTHETIC-CLAUDE"),
- datetime.now(UTC),
- )
- await session.commit()
- return HostedLaunchStore(), session_factory
-
-
-async def reserve(store, factory, request_id, name, owner_capacity=3):
- async with factory() as session:
- row = await store.reserve(
- session,
- request_id=request_id,
- owner_id="launch-owner",
- name=name,
- spec={"repository_id": 123},
- capacity=1,
- owner_capacity=owner_capacity,
- slots=SLOTS,
- repository="example-org/example-repo",
- now=datetime.now(UTC),
- )
- await session.commit()
- return row.id
-
-
-async def load(factory, launch_id):
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), launch_id))
- assert launch is not None and launch.machine_id is not None
- machine = await session.get(
- HostedMachine, (require_tenant_id(), launch.machine_id)
- )
- assert machine is not None
- return launch, machine
-
-
-async def test_duplicate_request_survives_new_session_without_second_launch(launches):
- store, factory = launches
- assert await reserve(store, factory, "request-1", "helper") == "request-1"
- assert await reserve(store, factory, "request-1", "helper") == "request-1"
- with pytest.raises(HostedLaunchConflict, match="different"):
- await reserve(store, factory, "request-1", "different")
- with pytest.raises(HostedLaunchConflict, match="name"):
- await reserve(store, factory, "request-2", "helper")
- _, machine = await load(factory, "request-1")
- assert machine.agents_version == 2
-
-
-async def test_reserve_places_the_launch_on_a_new_machine(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- launch, machine = await load(factory, "request-1")
- assert (launch.state, launch.agent_id, launch.repository) == (
- "queued",
- None,
- "example-org/example-repo",
- )
- assert (machine.owner_id, machine.slot_id, machine.state, machine.generation) == (
- "launch-owner",
- "slot-a",
- "queued",
- 1,
- )
-
-
-async def test_an_owner_s_agents_share_one_machine(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "first")
- await reserve(store, factory, "request-2", "second")
- first, machine = await load(factory, "request-1")
- second, _ = await load(factory, "request-2")
- assert first.machine_id == second.machine_id == machine.id
- assert machine.agents_version == 3
-
-
-async def test_owner_limit_counts_live_launches(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "first", owner_capacity=1)
- with pytest.raises(HostedLaunchConflict, match="Remove an agent"):
- await reserve(store, factory, "request-2", "second", owner_capacity=1)
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), "request-1"))
- launch.state = "deleted"
- launch.desired_state = "deleted"
- await session.commit()
- assert (
- await reserve(store, factory, "request-2", "second", owner_capacity=1)
- == "request-2"
- )
-
-
-async def test_owner_and_tenant_scope(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- async with factory() as session:
- assert await store.owned(session, "request-1", "launch-other") is None
- assert await store.owned(session, "request-1", "launch-owner") is not None
- with tenant_scope("launch-other-tenant"):
- async with factory() as session:
- assert await store.owned(session, "request-1", "launch-owner") is None
- assert await reserve(store, factory, "request-1", "helper") == "request-1"
-
-
-async def address(store, factory, launch_id, machine_state=None, **state):
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), launch_id))
- if launch is not None:
- for key, value in state.items():
- setattr(launch, key, value)
- launch.active_at = datetime.now(UTC) - timedelta(hours=1)
- machine = await session.get(
- HostedMachine, (require_tenant_id(), launch.machine_id)
- )
- for key, value in (machine_state or {}).items():
- setattr(machine, key, value)
- machine.active_at = datetime.now(UTC) - timedelta(hours=1)
- await session.commit()
- result = await store.note_addressed(session, launch_id)
- await session.commit()
- return result
-
-
-IDLE_SLEEPING = {
- "state": "stopped",
- "desired_state": "stopped",
- "stop_reason": "idle",
- "revision": 2,
-}
-
-
-async def test_addressing_a_launch_on_a_sleeping_machine_wakes_the_machine(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- launch, machine = await address(
- store, factory, "request-1", machine_state=IDLE_SLEEPING, state="ready"
- )
- assert (machine.desired_state, machine.stop_reason, machine.revision) == (
- "running",
- None,
- 3,
- )
- assert (launch.desired_state, launch.state, launch.revision) == (
- "running",
- "ready",
- 1,
- )
- assert datetime.now(UTC) - launch.active_at < timedelta(minutes=1)
- assert datetime.now(UTC) - machine.active_at < timedelta(minutes=1)
- assert is_waking(launch, machine)
-
-
-async def test_addressing_without_provider_does_not_wake_the_machine(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- async with factory() as session:
- connection = await session.get(
- ProviderConnection, (require_tenant_id(), "launch-owner", "claude")
- )
- await session.delete(connection)
- await session.commit()
- with pytest.raises(ProviderDisconnected):
- await address(
- store, factory, "request-1", machine_state=IDLE_SLEEPING, state="ready"
- )
- launch, machine = await load(factory, "request-1")
- assert (machine.desired_state, machine.revision) == ("stopped", 2)
- assert launch.revision == 1
-
-
-async def test_addressing_a_ready_launch_only_marks_it_active(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- ready, machine = await address(
- store, factory, "request-1", machine_state={"state": "ready"}, state="ready"
- )
- assert (ready.desired_state, ready.state, ready.revision) == ("running", "ready", 1)
- assert machine.revision == 1
- assert datetime.now(UTC) - ready.active_at < timedelta(minutes=1)
- assert not is_waking(ready, machine)
-
-
-async def test_addressing_never_wakes_a_machine_its_owner_stopped(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- launch, machine = await address(
- store,
- factory,
- "request-1",
- machine_state={**IDLE_SLEEPING, "stop_reason": "owner"},
- state="ready",
- )
- assert (machine.desired_state, machine.stop_reason, machine.revision) == (
- "stopped",
- "owner",
- 2,
- )
- assert datetime.now(UTC) - launch.active_at > timedelta(minutes=59)
-
-
-async def test_addressing_never_wakes_an_errored_idle_sleeping_machine(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- launch, machine = await address(
- store,
- factory,
- "request-1",
- machine_state={**IDLE_SLEEPING, "state": "error"},
- state="ready",
- )
- assert (
- machine.state,
- machine.desired_state,
- machine.stop_reason,
- machine.revision,
- ) == ("error", "stopped", "idle", 2)
- assert not is_waking(launch, machine)
-
-
[email protected]("desired", ["stopped", "deleted"])
-async def test_addressing_never_wakes_for_a_launch_its_owner_stopped(launches, desired):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- launch, machine = await address(
- store,
- factory,
- "request-1",
- machine_state=IDLE_SLEEPING,
- desired_state=desired,
- state="stopped",
- )
- assert (machine.desired_state, machine.revision) == ("stopped", 2)
- assert (launch.desired_state, launch.revision) == (desired, 1)
- assert datetime.now(UTC) - launch.active_at > timedelta(minutes=59)
-
-
-async def test_addressing_a_missing_launch_returns_none(launches):
- store, factory = launches
- assert await address(store, factory, "no-such-launch") == (None, None)
-
-
-async def test_addressing_does_not_wake_for_an_errored_launch(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- launch, machine = await address(
- store,
- factory,
- "request-1",
- machine_state=IDLE_SLEEPING,
- state="error",
- error="Agent crashed",
- )
- assert (machine.desired_state, machine.revision) == ("stopped", 2)
- assert (launch.state, launch.revision, launch.error) == (
- "error",
- 1,
- "Agent crashed",
- )
- assert not is_waking(launch, machine)
-
-
-async def test_a_queued_launch_is_waking_until_it_is_ready(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- async with factory() as session:
- launch, machine = await HostedMachineStore().locked_launch(session, "request-1")
- assert launch is not None and machine is not None
- assert is_waking(launch, machine)
- machine.state = "ready"
- assert is_waking(launch, machine)
- launch.state = "ready"
- assert not is_waking(launch, machine)
- assert not is_waking(launch, None)
-
-
-async def test_a_queued_launch_on_an_errored_machine_is_not_waking(launches):
- store, factory = launches
- await reserve(store, factory, "request-1", "helper")
- async with factory() as session:
- launch, machine = await HostedMachineStore().locked_launch(session, "request-1")
- assert launch is not None and machine is not None
- assert launch.state == "queued"
- machine.state = "error"
- assert not is_waking(launch, machine)
- launch.state = "provisioning"
- assert not is_waking(launch, machine)
diff --git a/core/tests/switch_core/db/stores/test_hosted_machine_store.py b/core/tests/switch_core/db/stores/test_hosted_machine_store.py
index 059bf2669..af05c4dea 100644
--- a/core/tests/switch_core/db/stores/test_hosted_machine_store.py
+++ b/core/tests/switch_core/db/stores/test_hosted_machine_store.py
@@ -3,23 +3,29 @@ from datetime import UTC, datetime, timedelta
import pytest
-from switch_core.db.models import HostedLaunch, HostedMachine, User, require_tenant_id
+from switch_core.db.models import (
+ AgentDefinition,
+ HostedMachine,
+ User,
+ require_tenant_id,
+)
from switch_core.db.stores.hosted_machine_store import (
MACHINE_BEING_REMOVED,
MACHINE_NEEDS_ADMIN,
MACHINE_NEEDS_ATTENTION,
+ MACHINES_FULL,
HostedMachineConflict,
HostedMachineStore,
idle_sleeping,
- lock_launches,
+ lock_claims,
machine_starting,
owner_stopped,
)
-from switch_core.keys import Keyring
-from tests.switch_core.hosted_machine_helpers import seed_launch, seed_machine
-
-SLOTS = ["slot-a", "slot-b"]
-TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
+from tests.switch_core.hosted_machine_helpers import (
+ link_controller,
+ place_managed_agent,
+ seed_machine,
+)
@pytest.fixture
@@ -41,13 +47,12 @@ async def factory(session_factory):
return session_factory
-async def claim(factory, owner_id, slots=SLOTS, capacity=2):
+async def claim(factory, owner_id, capacity=2):
async with factory() as session:
- await lock_launches(session)
+ await lock_claims(session)
machine = await HostedMachineStore().claim(
session,
owner_id=owner_id,
- slots=slots,
capacity=capacity,
now=datetime.now(UTC),
)
@@ -73,13 +78,11 @@ async def test_claim_creates_a_queued_machine(factory):
machine = await claim(factory, "owner-a")
assert (
machine.owner_id,
- machine.slot_id,
- machine.generation,
machine.state,
machine.desired_state,
machine.revision,
- machine.agents_version,
- ) == ("owner-a", "slot-a", 1, "queued", "running", 1, 1)
+ machine.controller_id,
+ ) == ("owner-a", "queued", "running", 1, None)
assert machine_starting(machine)
@@ -95,12 +98,10 @@ async def test_reusing_a_queued_machine_keeps_its_queued_clock(factory):
queued = await seed(
factory,
owner_id="owner-a",
- slot_id="slot-a",
state="queued",
desired_state="running",
stop_reason=None,
revision=1,
- generation=1,
)
long_ago = datetime.now(UTC) - timedelta(minutes=11)
async with factory() as session:
@@ -131,12 +132,10 @@ async def test_a_retained_machine_is_reused_on_the_same_disk(factory):
retained = await seed(
factory,
owner_id="owner-a",
- slot_id="slot-b",
state="retained",
desired_state="retained",
stop_reason=None,
revision=4,
- generation=2,
)
async with factory() as session:
row = await HostedMachineStore().get(session, retained.id)
@@ -145,12 +144,10 @@ async def test_a_retained_machine_is_reused_on_the_same_disk(factory):
machine = await claim(factory, "owner-a")
assert (
machine.id,
- machine.slot_id,
- machine.generation,
machine.desired_state,
machine.retain_until,
machine.revision,
- ) == (retained.id, "slot-b", 2, "running", None, 5)
+ ) == (retained.id, "running", None, 5)
assert machine_starting(machine)
@@ -158,12 +155,10 @@ async def test_a_stopped_machine_is_started(factory):
stopped = await seed(
factory,
owner_id="owner-a",
- slot_id="slot-a",
state="stopped",
desired_state="stopped",
stop_reason="owner",
revision=3,
- generation=1,
)
assert owner_stopped(stopped) and not idle_sleeping(stopped)
machine = await claim(factory, "owner-a")
@@ -213,12 +208,10 @@ async def test_a_machine_that_cannot_take_agents_refuses_the_claim(
seeded = await seed(
factory,
owner_id="owner-a",
- slot_id="slot-a",
state=state,
desired_state=desired_state,
stop_reason=None,
revision=2,
- generation=1,
)
async with factory() as session:
row = await HostedMachineStore().get(session, seeded.id)
@@ -238,12 +231,10 @@ async def test_a_retained_machine_in_error_still_retaining_refuses_as_in_error(
seeded = await seed(
factory,
owner_id="owner-a",
- slot_id="slot-a",
state="error",
desired_state="retained",
stop_reason=None,
revision=2,
- generation=1,
)
async with factory() as session:
row = await HostedMachineStore().get(session, seeded.id)
@@ -255,41 +246,30 @@ async def test_a_retained_machine_in_error_still_retaining_refuses_as_in_error(
assert str(raised.value) == MACHINE_NEEDS_ATTENTION
-async def test_no_free_slot_refuses_the_claim(factory):
- await claim(factory, "owner-b", slots=["slot-a"])
- with pytest.raises(HostedMachineConflict) as raised:
- await claim(factory, "owner-a", slots=["slot-a"], capacity=5)
- assert str(raised.value) == "no machine slot available"
-
-
async def test_capacity_caps_live_machines(factory):
await claim(factory, "owner-b", capacity=1)
- with pytest.raises(HostedMachineConflict, match="no machine slot available"):
+ with pytest.raises(HostedMachineConflict) as raised:
await claim(factory, "owner-a", capacity=1)
+ assert str(raised.value) == MACHINES_FULL
-async def test_a_reused_slot_gets_the_next_generation(factory):
- for generation in (1, 3):
- await seed(
- factory,
- owner_id="owner-a",
- slot_id="slot-a",
- state="deleted",
- desired_state="deleted",
- stop_reason=None,
- revision=1,
- generation=generation,
- )
- machine = await claim(factory, "owner-a")
- assert (machine.slot_id, machine.generation) == ("slot-a", 4)
- other = await claim(factory, "owner-b")
- assert (other.slot_id, other.generation) == ("slot-b", 1)
+async def test_deleted_machines_do_not_count_towards_capacity(factory):
+ await seed(
+ factory,
+ owner_id="owner-b",
+ state="deleted",
+ desired_state="deleted",
+ stop_reason=None,
+ revision=1,
+ )
+ machine = await claim(factory, "owner-a", capacity=1)
+ assert machine.state == "queued"
-async def test_concurrent_claims_cannot_exceed_slots(factory):
+async def test_concurrent_claims_cannot_exceed_capacity(factory):
results = await asyncio.gather(
- claim(factory, "owner-a", slots=["slot-a"]),
- claim(factory, "owner-b", slots=["slot-a"]),
+ claim(factory, "owner-a", capacity=1),
+ claim(factory, "owner-b", capacity=1),
return_exceptions=True,
)
assert sum(isinstance(value, HostedMachine) for value in results) == 1
@@ -333,12 +313,10 @@ async def test_starting_a_ready_machine_again_waits_for_it_to_reconnect(factory)
machine = await seed(
factory,
owner_id="owner-a",
- slot_id="slot-a",
state="ready",
desired_state="running",
stop_reason=None,
revision=2,
- generation=1,
)
store = HostedMachineStore()
now = datetime.now(UTC)
@@ -365,12 +343,10 @@ async def test_reusing_a_retained_machine_still_ready_waits_for_it_to_reconnect(
retained = await seed(
factory,
owner_id="owner-a",
- slot_id="slot-a",
state="ready",
desired_state="retained",
stop_reason=None,
revision=3,
- generation=1,
)
machine = await claim(factory, "owner-a")
assert (machine.id, machine.state, machine.desired_state, machine.revision) == (
@@ -382,111 +358,167 @@ async def test_reusing_a_retained_machine_still_ready_waits_for_it_to_reconnect(
assert machine_starting(machine)
-async def test_retain_if_empty_retains_only_once_no_agent_is_left(factory):
+async def ready_machine(session, owner_id="owner-a"):
+ return await seed_machine(
+ session,
+ owner_id=owner_id,
+ state="ready",
+ desired_state="running",
+ stop_reason=None,
+ revision=2,
+ )
+
+
+async def test_retain_if_empty_retains_only_once_no_agent_is_placed(factory):
store = HostedMachineStore()
async with factory() as session:
- machine = await seed_machine(
- session,
- owner_id="owner-a",
- slot_id="slot-a",
- state="ready",
- desired_state="running",
- stop_reason=None,
- revision=2,
- generation=1,
+ machine = await ready_machine(session)
+ controller = await link_controller(session, machine)
+ definition = await place_managed_agent(
+ session, owner_id="owner-a", controller_id=controller.id, name="helper"
+ )
+ now = datetime.now(UTC)
+ assert await store.has_agents(session, machine)
+ assert not await store.retain_if_empty(
+ session, machine, retention_days=7, now=now
)
- launch = await seed_launch(
+ assert (machine.desired_state, machine.revision) == ("running", 2)
+ definition.controller_id = None
+ assert await store.retain_if_empty(session, machine, retention_days=7, now=now)
+ assert (
+ machine.desired_state,
+ machine.stop_reason,
+ machine.revision,
+ machine.retain_until,
+ ) == ("retained", None, 3, now + timedelta(days=7))
+ await session.commit()
+
+
+async def test_only_agents_on_the_machine_s_own_controller_count(factory):
+ store = HostedMachineStore()
+ async with factory() as session:
+ machine = await ready_machine(session)
+ other = await seed_machine(
session,
- machine=machine,
- request_id="request-1",
- name="helper",
+ owner_id="owner-b",
state="ready",
desired_state="running",
+ stop_reason=None,
revision=1,
- agent_id=None,
- spec={},
)
- now = datetime.now(UTC)
- assert [row.id for row in await store.launches(session, machine.id)] == [
- "request-1"
+ mine = await link_controller(session, machine)
+ theirs = await link_controller(session, other)
+ await place_managed_agent(
+ session, owner_id="owner-b", controller_id=theirs.id, name="theirs"
+ )
+ assert not await store.has_agents(session, machine)
+ assert await store.managed_agent_ids(session, machine) == []
+ placed = [
+ await place_managed_agent(
+ session, owner_id="owner-a", controller_id=mine.id, name=name
+ )
+ for name in ("reviewer", "builder")
]
- assert not await store.retain_if_empty(
- session, machine, retention_days=7, now=now
+ assert await store.has_agents(session, machine)
+ assert await store.managed_agent_ids(session, machine) == sorted(
+ definition.agent_id for definition in placed
)
- assert machine.revision == 2
- launch.desired_state = "deleted"
- assert await store.retain_if_empty(session, machine, retention_days=7, now=now)
+
+
+async def test_a_machine_without_a_controller_has_no_agents(factory):
+ store = HostedMachineStore()
+ async with factory() as session:
+ machine = await ready_machine(session)
+ assert not await store.has_agents(session, machine)
+ assert await store.managed_agent_ids(session, machine) == []
+ assert not await store.ever_hosted(session, machine)
+
+
+async def test_release_if_empty_expires_a_machine_whose_controller_never_enrolled(
+ factory,
+):
+ store = HostedMachineStore()
+ async with factory() as session:
+ machine = await ready_machine(session)
+ now = datetime.now(UTC)
+ assert await store.release_if_empty(session, machine, retention_days=7, now=now)
assert (machine.desired_state, machine.revision, machine.retain_until) == (
"retained",
3,
+ now,
+ )
+
+
+async def test_release_if_empty_keeps_the_disk_of_a_machine_that_enrolled(factory):
+ store = HostedMachineStore()
+ async with factory() as session:
+ machine = await ready_machine(session)
+ await link_controller(session, machine)
+ assert await store.ever_hosted(session, machine)
+ now = datetime.now(UTC)
+ assert await store.release_if_empty(session, machine, retention_days=7, now=now)
+ assert (machine.desired_state, machine.retain_until) == (
+ "retained",
now + timedelta(days=7),
)
- launch.state = "deleted"
- assert await store.launches(session, machine.id) == []
- await session.commit()
-async def test_machine_capability_is_stable_per_revision_and_rotates(factory):
+async def test_release_if_empty_leaves_a_machine_with_agents(factory):
store = HostedMachineStore()
- machine = await claim(factory, "owner-a")
- assert not store.capability_matches(machine, "anything")
- first = store.issue_capability(machine, TEST_KEYRING)
- assert store.issue_capability(machine, TEST_KEYRING) == first
- assert machine.machine_capability_revision == 1
- assert store.capability_matches(machine, first)
- assert not store.capability_matches(machine, first + "x")
- store.stop(machine, "owner", datetime.now(UTC))
- assert store.capability_matches(machine, first)
- second = store.issue_capability(machine, TEST_KEYRING)
- assert second != first
- assert machine.machine_capability_revision == 2
- assert store.capability_matches(machine, second)
- assert not store.capability_matches(machine, first)
-
-
-async def test_locked_launch_returns_the_launch_and_its_machine(factory):
+ async with factory() as session:
+ machine = await ready_machine(session)
+ controller = await link_controller(session, machine)
+ await place_managed_agent(
+ session, owner_id="owner-a", controller_id=controller.id, name="helper"
+ )
+ assert not await store.release_if_empty(
+ session, machine, retention_days=7, now=datetime.now(UTC)
+ )
+ assert (machine.desired_state, machine.revision, machine.retain_until) == (
+ "running",
+ 2,
+ None,
+ )
+
+
+async def test_removing_the_agent_takes_it_off_the_machine(factory):
+ store = HostedMachineStore()
+ async with factory() as session:
+ machine = await ready_machine(session)
+ controller = await link_controller(session, machine)
+ definition = await place_managed_agent(
+ session, owner_id="owner-a", controller_id=controller.id, name="helper"
+ )
+ await session.commit()
+ async with factory() as session:
+ row = await session.get(AgentDefinition, definition.id)
+ await session.delete(row)
+ await session.commit()
+ async with factory() as session:
+ machine = await store.get(session, machine.id)
+ assert machine is not None
+ assert not await store.has_agents(session, machine)
+ assert await store.ever_hosted(session, machine)
+
+
+async def test_owned_and_live_for_owner(factory):
store = HostedMachineStore()
async with factory() as session:
- machine = await seed_machine(
+ machine = await ready_machine(session)
+ await seed_machine(
session,
owner_id="owner-a",
- slot_id="slot-a",
- state="ready",
- desired_state="running",
+ state="deleted",
+ desired_state="deleted",
stop_reason=None,
revision=1,
- generation=1,
- )
- await seed_launch(
- session,
- machine=machine,
- request_id="request-1",
- name="helper",
- state="ready",
- desired_state="running",
- revision=1,
- agent_id=None,
- spec={},
- )
- session.add(
- HostedLaunch(
- id="legacy",
- owner_id="owner-a",
- name="removed:legacy",
- spec={},
- state="deleted",
- desired_state="deleted",
- )
)
await session.commit()
async with factory() as session:
- launch, locked = await store.locked_launch(session, "request-1")
- assert launch is not None and locked is not None
- assert (launch.id, locked.id) == ("request-1", machine.id)
- legacy, none = await store.locked_launch(session, "legacy")
- assert legacy is not None and none is None
- assert await store.locked_launch(session, "missing") == (None, None)
- assert await store.owned(session, machine.id, "owner-a") is not None
+ assert (await store.owned(session, machine.id, "owner-a")).id == machine.id
assert await store.owned(session, machine.id, "owner-b") is None
assert (await store.live_for_owner(session, "owner-a")).id == machine.id
+ assert await store.live_for_owner(session, "owner-b") is None
+ assert (await store.locked(session, machine.id)).id == machine.id
+ assert await store.locked(session, "missing") is None
assert await session.get(HostedMachine, (require_tenant_id(), machine.id))
diff --git a/core/tests/switch_core/db/stores/test_hosted_mailbox_store.py b/core/tests/switch_core/db/stores/test_hosted_mailbox_store.py
deleted file mode 100644
index 9fbb8ae11..000000000
--- a/core/tests/switch_core/db/stores/test_hosted_mailbox_store.py
+++ /dev/null
@@ -1,566 +0,0 @@
-from datetime import UTC, datetime, timedelta
-from typing import Any
-
-import pytest
-from sqlalchemy import select, update
-
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.bridges.agent.protocol.types import (
- AgentEvent,
- MessagePayload,
- RoomJoinPayload,
- TaskDelegatePayload,
-)
-from switch_core.db.models import (
- HostedLaunch,
- HostedWakeMailbox,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
-from switch_core.db.stores.hosted_mailbox_store import (
- MAILBOX_LIMIT,
- HostedMailboxStore,
- MailboxEntry,
- MailboxFull,
- MailboxNotice,
- one_per_room,
- room_input_id,
-)
-
-AGENT = "00000000-0000-4000-8000-00000000000a"
-LAUNCH = "mailbox-launch"
-
-
-async def mailbox(session_factory):
- async with session_factory() as session:
- session.add(
- User(
- id="mailbox-owner",
- name="Owner",
- role="user",
- password_hash="unused",
- )
- )
- await session.flush()
- session.add(
- HostedLaunch(
- id=LAUNCH,
- owner_id="mailbox-owner",
- name="mailbox-helper",
- spec={},
- state="ready",
- agent_id=AGENT,
- )
- )
- await session.commit()
- return HostedMailboxStore(), session_factory
-
-
-def message(message_id: str, room: str = "room-1", thread: str | None = None):
- return AgentEvent(
- type="message",
- room_id=room,
- bridge_id=None,
- channel_type=None,
- payload=MessagePayload(
- addressed=True,
- sender="@ana:example.invalid",
- sender_name="Ana",
- message_id=message_id,
- body="hello",
- timestamp=1700000000000,
- thread_id=thread,
- ),
- )
-
-
-def entry(message_id: str, room: str = "room-1", thread: str | None = None):
- made = MailboxEntry.of(message(message_id, room, thread))
- assert made is not None
- return made
-
-
-async def write(store, factory, message_id, *, room="room-1", offered_to=None):
- async with factory() as session:
- written = await store.write(
- session,
- agent_id=AGENT,
- launch_id=LAUNCH,
- entry=entry(message_id, room),
- offered_to=offered_to,
- )
- await session.commit()
- return written
-
-
-async def force(factory, message_id: str, **values: Any) -> None:
- async with factory() as session:
- await session.execute(
- update(HostedWakeMailbox)
- .where(
- HostedWakeMailbox.tenant_id == require_tenant_id(),
- HostedWakeMailbox.message_id == message_id,
- )
- .values(**values)
- )
- await session.commit()
-
-
-async def states(factory) -> dict[str, str]:
- async with factory() as session:
- rows = await session.execute(
- select(HostedWakeMailbox.message_id, HostedWakeMailbox.state).where(
- HostedWakeMailbox.tenant_id == require_tenant_id()
- )
- )
- return dict(rows.tuples().all())
-
-
-async def ack(store, factory, *acks):
- async with factory() as session:
- result = await store.ack(session, AGENT, list(acks))
- await session.commit()
- return result
-
-
-def test_room_input_id_matches_the_watcher():
- """Vectors computed with `roomInputId` in `host/room-inbox.ts`."""
- task = AgentEvent(
- type="task_delegate",
- room_id="room-1",
- bridge_id=None,
- channel_type=None,
- payload=TaskDelegatePayload(
- task_id="task-1",
- requester_agent_id="agent-a",
- performer_agent_id="agent-b",
- summary="Résumé — «ship it»",
- description="line one\nline two",
- ),
- )
- join = AgentEvent(
- type="room_join",
- room_id="room-1",
- bridge_id=None,
- channel_type=None,
- payload=RoomJoinPayload(
- member="@ana:example.invalid",
- member_name="Ana",
- timestamp=1700000000000,
- listening=True,
- ),
- )
- assert room_input_id(task) == (
- "task_delegate:b35d00e95c27b151af1d551878a35b85c72caf0abab076f7c19f2774cc8b50f0"
- )
- assert room_input_id(join) == (
- "room_join:250b6dd87c453eda5e85209a4de2719caf3838021effb5e0304a0d029abf8768"
- )
- assert room_input_id(message("$m1")) == "$m1"
- unaddressed = message("$m2")
- unaddressed.payload.addressed = False
- assert room_input_id(unaddressed) is None
- assert MailboxEntry.of(unaddressed) is None
-
-
-def test_entry_carries_thread_and_handoff_shape():
- made = entry("$m1", thread="$root")
- assert made.thread_id == "$root"
- assert made.event["type"] == "message"
- assert made.event["missed"] is None
- assert made.event["payload"]["message_id"] == "$m1"
- assert made.wire()["message_id"] == "$m1"
-
-
-async def test_write_dedupes_and_offers_when_a_worker_takes_it(mailbox):
- store, factory = mailbox
- assert await write(store, factory, "$m1") is True
- assert await write(store, factory, "$m1") is False
- assert await write(store, factory, "$m2", offered_to="1:conn:7") is True
- async with factory() as session:
- offered = await session.get(
- HostedWakeMailbox, (require_tenant_id(), AGENT, "room-1", "$m2")
- )
- assert offered is not None
- assert (offered.state, offered.ever_offered, offered.offered_to) == (
- "offered",
- True,
- "1:conn:7",
- )
- assert offered.offered_until is not None
- assert offered.expires_at - offered.addressed_at == timedelta(hours=24)
- assert await states(factory) == {"$m1": "pending", "$m2": "offered"}
-
-
-async def test_limit_counts_only_rows_waiting_for_the_worker(mailbox):
- store, factory = mailbox
- for index in range(MAILBOX_LIMIT):
- assert await write(store, factory, f"$m{index}") is True
- with pytest.raises(MailboxFull):
- await write(store, factory, "$over")
- assert await write(store, factory, "$m0") is False
- await force(factory, "$m0", state="accepted")
- assert await write(store, factory, "$over") is True
- with pytest.raises(MailboxFull):
- await write(store, factory, "$over-again")
-
-
-async def test_offer_moves_only_pending_rows(mailbox):
- store, factory = mailbox
- await write(store, factory, "$m1")
- await write(store, factory, "$m2")
- await ack(store, factory, ("room-1", "$m2", "journaled"))
- async with factory() as session:
- moved = await store.mark_offered(
- session, AGENT, [("room-1", "$m1"), ("room-1", "$m2")], "1:conn:1"
- )
- await session.commit()
- assert moved == {("room-1", "$m1")}
- assert await states(factory) == {"$m1": "offered", "$m2": "accepted"}
-
-
-async def test_offer_ack_race_never_regresses(mailbox):
- store, factory = mailbox
- await write(store, factory, "$m1", offered_to="1:old:1")
- await force(factory, "$m1", offered_until=datetime.now(UTC) - timedelta(seconds=1))
- async with factory() as session:
- assert await store.reclaim(session, AGENT, {}) == 1
- await session.commit()
- await ack(store, factory, ("room-1", "$m1", "journaled"))
- async with factory() as session:
- assert await store.pending(session, AGENT) == []
- assert (
- await store.mark_offered(session, AGENT, [("room-1", "$m1")], "x") == set()
- )
- await session.commit()
- assert await states(factory) == {"$m1": "accepted"}
-
-
-async def test_reclaim_keeps_only_live_unexpired_leases(mailbox):
- store, factory = mailbox
- await write(store, factory, "$live", offered_to="2:conn:5")
- await write(store, factory, "$foreign-boot", offered_to="1:conn:5")
- await write(store, factory, "$old-generation", offered_to="2:conn:4")
- await write(store, factory, "$lapsed", offered_to="2:conn:5")
- await force(
- factory, "$lapsed", offered_until=datetime.now(UTC) - timedelta(seconds=1)
- )
- async with factory() as session:
- assert await store.reclaim(session, None, {AGENT: "2:conn:5"}) == 3
- await session.commit()
- assert await states(factory) == {
- "$live": "offered",
- "$foreign-boot": "pending",
- "$old-generation": "pending",
- "$lapsed": "pending",
- }
- async with factory() as session:
- row = await session.get(
- HostedWakeMailbox, (require_tenant_id(), AGENT, "room-1", "$lapsed")
- )
- assert row is not None
- assert (row.offered_to, row.ever_offered) == (None, True)
- assert await store.agents_with_pending(session) == [AGENT]
-
-
-async def test_ack_moves_forward_only(mailbox):
- store, factory = mailbox
- for name in ("$a", "$b", "$c", "$d"):
- await write(store, factory, name, offered_to="1:conn:1")
- after, notices = await ack(
- store,
- factory,
- ("room-1", "$a", "journaled"),
- ("room-1", "$a", "admitted"),
- ("room-1", "$b", "duplicate"),
- ("room-1", "$c", "refused"),
- ("room-1", "$d", "journaled"),
- ("room-1", "$d", "held"),
- ("room-1", "$unknown", "journaled"),
- )
- assert notices == []
- assert after == {
- ("room-1", "$a"): "admitted",
- ("room-1", "$b"): "accepted",
- ("room-1", "$c"): "refused",
- ("room-1", "$d"): "held",
- }
- await ack(
- store,
- factory,
- ("room-1", "$a", "journaled"),
- ("room-1", "$c", "journaled"),
- ("room-1", "$d", "journaled"),
- ("room-1", "$b", "cancelled"),
- )
- assert await states(factory) == {
- "$a": "admitted",
- "$b": "accepted",
- "$c": "refused",
- "$d": "held",
- }
- await ack(store, factory, ("room-1", "$d", "admitted"))
- assert (await states(factory))["$d"] == "admitted"
-
-
-async def test_stop_cancels_never_offered_rows_and_tombstones_the_rest(mailbox):
- store, factory = mailbox
- await write(store, factory, "$never", room="room-1")
- await write(store, factory, "$never-2", room="room-2")
- await write(store, factory, "$reclaimed", offered_to="1:conn:1")
- await force(factory, "$reclaimed", state="pending", offered_to=None)
- await write(store, factory, "$offered", offered_to="1:conn:1")
- await write(store, factory, "$accepted", offered_to="1:conn:1")
- await write(store, factory, "$held", offered_to="1:conn:1")
- await write(store, factory, "$admitted", offered_to="1:conn:1")
- await ack(
- store,
- factory,
- ("room-1", "$accepted", "journaled"),
- ("room-1", "$held", "journaled"),
- ("room-1", "$held", "held"),
- ("room-1", "$admitted", "journaled"),
- ("room-1", "$admitted", "admitted"),
- )
- async with factory() as session:
- split = await store.stop(session, LAUNCH)
- await session.commit()
- assert [(n.room_id, n.message_id, n.reason) for n in split.cancelled] == [
- ("room-1", "$never", "stopped"),
- ("room-2", "$never-2", "stopped"),
- ]
- assert sorted(split.cancel_requested) == [
- ("room-1", "$accepted"),
- ("room-1", "$held"),
- ("room-1", "$offered"),
- ("room-1", "$reclaimed"),
- ]
- assert await states(factory) == {
- "$never": "cancelled",
- "$never-2": "cancelled",
- "$reclaimed": "cancel_requested",
- "$offered": "cancel_requested",
- "$accepted": "cancel_requested",
- "$held": "cancel_requested",
- "$admitted": "admitted",
- }
- async with factory() as session:
- cancelled = await store.cancelled_entries(session, AGENT)
- assert {(e["message_id"], e["reason"]) for e in cancelled} == {
- ("$reclaimed", "stopped"),
- ("$offered", "stopped"),
- ("$accepted", "stopped"),
- ("$held", "stopped"),
- }
-
- # A tombstone is settled only by the watcher, never by a late ack.
- await ack(store, factory, ("room-1", "$offered", "journaled"))
- assert (await states(factory))["$offered"] == "cancel_requested"
-
- _, notices = await ack(
- store,
- factory,
- ("room-1", "$offered", "cancelled"),
- ("room-1", "$accepted", "admitted"),
- )
- assert {(n.message_id, n.reason) for n in notices} == {
- ("$offered", "stopped"),
- ("$accepted", "started_before_stop"),
- }
- _, again = await ack(store, factory, ("room-1", "$offered", "cancelled"))
- assert again == []
-
-
-async def test_stop_after_stop_leaves_settled_rows(mailbox):
- store, factory = mailbox
- await write(store, factory, "$m1")
- async with factory() as session:
- await store.stop(session, LAUNCH)
- second = await store.stop(session, LAUNCH)
- await session.commit()
- assert second.cancelled == [] and second.cancel_requested == []
- assert await states(factory) == {"$m1": "cancelled"}
-
-
-async def test_expiry_by_state(mailbox):
- store, factory = mailbox
- for name in ("$never", "$reclaimed", "$offered", "$accepted", "$held", "$stop"):
- await write(
- store, factory, name, offered_to=None if name == "$never" else "1:c:1"
- )
- await force(factory, "$reclaimed", state="pending", offered_to=None)
- await ack(
- store,
- factory,
- ("room-1", "$accepted", "journaled"),
- ("room-1", "$held", "journaled"),
- ("room-1", "$held", "held"),
- )
- await force(factory, "$stop", state="cancel_requested", cancel_reason="stopped")
- async with factory() as session:
- await session.execute(
- update(HostedWakeMailbox)
- .where(HostedWakeMailbox.tenant_id == require_tenant_id())
- .values(expires_at=datetime.now(UTC) - timedelta(seconds=1))
- )
- await session.commit()
- async with factory() as session:
- notices, tombstoned = await store.expire(session, datetime.now(UTC))
- await session.commit()
- assert {(n.message_id, n.reason) for n in notices} == {
- ("$never", "expired"),
- ("$reclaimed", "expired_uncertain"),
- ("$offered", "expired_uncertain"),
- }
- assert tombstoned == 2
- assert await states(factory) == {
- "$never": "expired",
- "$reclaimed": "expired_uncertain",
- "$offered": "expired_uncertain",
- "$accepted": "cancel_requested",
- "$held": "cancel_requested",
- "$stop": "cancel_requested",
- }
- async with factory() as session:
- reasons = {
- e["message_id"]: e["reason"]
- for e in await store.cancelled_entries(session, AGENT)
- }
- assert reasons == {"$accepted": "expired", "$held": "expired", "$stop": "stopped"}
- _, settled = await ack(
- store,
- factory,
- ("room-1", "$held", "cancelled"),
- ("room-1", "$accepted", "admitted"),
- )
- assert {(n.message_id, n.reason) for n in settled} == {
- ("$held", "expired"),
- ("$accepted", "started_before_expiry"),
- }
- async with factory() as session:
- owed = await store.owed_notices(session, 100)
- assert {(n.message_id, n.reason) for n in owed} == {
- ("$never", "expired"),
- ("$reclaimed", "expired_uncertain"),
- ("$offered", "expired_uncertain"),
- ("$held", "expired"),
- ("$accepted", "started_before_expiry"),
- }
- async with factory() as session:
- await store.notice_posted(
- session, AGENT, "room-1", "expired_uncertain", ["$reclaimed", "$offered"]
- )
- await session.commit()
- remaining = await store.owed_notices(session, 100)
- assert {n.message_id for n in remaining} == {"$never", "$held", "$accepted"}
-
-
-async def test_tombstone_survives_expiry_and_prune(mailbox):
- store, factory = mailbox
- await write(store, factory, "$tomb", offered_to="1:c:1")
- await write(store, factory, "$done", offered_to="1:c:1")
- await write(store, factory, "$recent", offered_to="1:c:1")
- await ack(store, factory, ("room-1", "$done", "admitted"))
- await ack(store, factory, ("room-1", "$recent", "admitted"))
- old = datetime.now(UTC) - timedelta(days=8)
- await force(
- factory,
- "$tomb",
- state="cancel_requested",
- cancel_reason="stopped",
- addressed_at=old,
- updated_at=old,
- expires_at=old + timedelta(hours=24),
- )
- await force(factory, "$done", updated_at=old)
- async with factory() as session:
- notices, tombstoned = await store.expire(session, datetime.now(UTC))
- pruned = await store.prune(session, datetime.now(UTC))
- await session.commit()
- assert (notices, tombstoned, pruned) == ([], 0, 1)
- assert await states(factory) == {"$tomb": "cancel_requested", "$recent": "admitted"}
- async with factory() as session:
- assert [
- e["message_id"] for e in await store.cancelled_entries(session, AGENT)
- ] == ["$tomb"]
-
-
-async def test_busy_counts_rows_waiting_for_the_worker(mailbox):
- store, factory = mailbox
- registry = AgentConnectionRegistry()
-
- async def evidence() -> tuple[bool, list[str]]:
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), LAUNCH))
- assert launch is not None
- found = await HostedLaunchStore().idle_evidence(session, launch, registry)
- return await store.busy(session, LAUNCH), found.reasons
-
- assert await evidence() == (False, ["no_fresh_report"])
- await write(store, factory, "$m1")
- for state in ("pending", "offered", "accepted"):
- await force(factory, "$m1", state=state)
- assert await evidence() == (True, ["no_fresh_report", "mailbox_pending"])
- for state in ("held", "cancel_requested", "admitted"):
- await force(factory, "$m1", state=state)
- assert await evidence() == (False, ["no_fresh_report"])
-
-
-async def test_backlog_reports_waiting_and_refusals(mailbox):
- store, factory = mailbox
- since = datetime.now(UTC) - timedelta(seconds=1)
- await write(store, factory, "$m1")
- await write(store, factory, "$m2", offered_to="1:c:1")
- await write(store, factory, "$m3", offered_to="1:c:1")
- await ack(store, factory, ("room-1", "$m3", "refused"))
- async with factory() as session:
- (agent,) = await store.backlog(session, since)
- assert (agent.agent_id, agent.waiting, agent.refused) == (AGENT, 2, 1)
- assert agent.oldest is not None
-
-
-async def test_delete_launch_drops_rows_without_notices(mailbox):
- store, factory = mailbox
- await write(store, factory, "$m1")
- await force(factory, "$m1", state="cancel_requested", cancel_reason="stopped")
- async with factory() as session:
- await store.delete_launch(session, LAUNCH)
- await session.commit()
- assert await states(factory) == {}
-
-
-async def test_rows_go_with_their_launch(mailbox):
- store, factory = mailbox
- await write(store, factory, "$m1")
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), LAUNCH))
- await session.delete(launch)
- await session.commit()
- assert await states(factory) == {}
-
-
-def test_one_notice_per_room_and_reason():
- notices = [
- MailboxNotice(
- agent_id=AGENT,
- room_id=room,
- message_id=message_id,
- thread_id=None,
- reason=reason,
- )
- for room, message_id, reason in (
- ("room-1", "$a", "stopped"),
- ("room-1", "$b", "stopped"),
- ("room-2", "$c", "stopped"),
- ("room-1", "$d", "expired"),
- )
- ]
- kept = one_per_room(notices)
- assert [(n.room_id, n.message_id) for n in kept] == [
- ("room-1", "$b"),
- ("room-2", "$c"),
- ("room-1", "$d"),
- ]
diff --git a/core/tests/switch_core/fixtures/agent_controllers/controller_info_request.json b/core/tests/switch_core/fixtures/agent_controllers/controller_info_request.json
index 773fb5901..3151527f2 100644
--- a/core/tests/switch_core/fixtures/agent_controllers/controller_info_request.json
+++ b/core/tests/switch_core/fixtures/agent_controllers/controller_info_request.json
@@ -1,4 +1,8 @@
{
"name": "build-box",
- "description": "The build box in the office"
+ "description": "The build box in the office",
+ "public_key": {
+ "alg": "X25519",
+ "key": "bWFjaGluZS1wdWJsaWMta2V5LTMyLWJ5dGVzLi4uLi4="
+ }
}
diff --git a/core/tests/switch_core/fixtures/agent_controllers/controller_info_response.json b/core/tests/switch_core/fixtures/agent_controllers/controller_info_response.json
index 01d5cd4fd..41e63efc5 100644
--- a/core/tests/switch_core/fixtures/agent_controllers/controller_info_response.json
+++ b/core/tests/switch_core/fixtures/agent_controllers/controller_info_response.json
@@ -62,6 +62,11 @@
},
"assignment_revision": 0,
"workspaces_dir": "/home/user/.local/state/switch/agent-controller/workspaces",
+ "public_key": {
+ "alg": "X25519",
+ "key": "bWFjaGluZS1wdWJsaWMta2V5LTMyLWJ5dGVzLi4uLi4=",
+ "key_id": "0123456789abcdef"
+ },
"created_at": "2026-10-01T12:00:00Z",
"revoked_at": null
}
diff --git a/core/tests/switch_core/fixtures/hosted_machines/agents_response.json b/core/tests/switch_core/fixtures/hosted_machines/agents_response.json
deleted file mode 100644
index 607301e90..000000000
--- a/core/tests/switch_core/fixtures/hosted_machines/agents_response.json
+++ /dev/null
@@ -1,52 +0,0 @@
-{
- "machine_id": "3f1c2b4a-0000-4000-8000-000000000001",
- "revision": 1,
- "desired_state": "running",
- "agents_version": 3,
- "agents": [
- {
- "launch_id": "3f1c2b4a-0000-4000-8000-0000000000c1",
- "agent_id": "3f1c2b4a-0000-4000-8000-0000000000a1",
- "name": "reviewer",
- "revision": 1,
- "desired_state": "running",
- "provider": "claude",
- "provider_credential_kind": "setup-token",
- "worker_capability": "wcap-test-00000000000000000000000000000000",
- "switch_credentials": {
- "env": {
- "SWITCH_API_ENDPOINT": "https://switch.example.test/agent-api",
- "SWITCH_API_TOKEN": "test-token-placeholder",
- "SWITCH_AGENT_ID": "3f1c2b4a-0000-4000-8000-0000000000a1"
- }
- },
- "repository": "example-org/example-repo",
- "spec": {
- "name": "reviewer",
- "icon_url": null,
- "provider": "claude",
- "definition": "---\nname: reviewer\ndescription: Reviews pull requests\n---\nYou review pull requests for correctness and clarity.",
- "description": "Reviews pull requests",
- "auto_approve": false,
- "auto_session": true,
- "display_name": null,
- "instructions": "Review each pull request you are asked about and post your findings in the room.",
- "repository_id": 456,
- "session_limit": 8,
- "installation_id": 123,
- "addressing_policy": null,
- "definition_attributes": {
- "model": "sonnet"
- }
- },
- "skills": [
- {
- "slug": "github",
- "files": {
- "SKILL.md": "---\nname: github\ndescription: How to use GitHub from this cloud agent. Load before running git or gh against the granted repository \u2014 cloning, fetching, pushing, branches, pull requests or reviews.\n---\n\n# GitHub\n\nYour owner granted this agent access to one GitHub repository. It is already\ncloned into your workspace, and `git` and `gh` are already signed in for it.\n\n## Access\n\n- `gh` on your `PATH` is a wrapper that fetches a fresh, short-lived token for\n every call. The `git` credential helper does the same for HTTPS pushes and\n fetches to github.com. Both refresh automatically; there is nothing to renew.\n- The token reaches only the granted repository, with exactly these\n permissions:\n - **Contents** read/write: clone, fetch, branch, commit, push.\n - **Pull requests** read/write: create, list, view, diff, comment, review.\n - **Metadata** read.\n- Nothing else is granted. Issues, Actions, check runs, commit statuses and\n workflow files are out of reach: a push that adds or changes a file under\n `.github/workflows/` is rejected, and API operations requiring additional permissions are unavailable.\n Other repositories, organization settings and your owner's personal account\n are out of reach too. That is the scope, not a bug \u2014 tell the user what you\n could not do rather than retrying or looking for other credentials.\n- Never print, echo, log or persist a token. Do not run `gh auth token`,\n `gh auth login` or `gh auth setup-git`, do not set `GH_TOKEN` or\n `GITHUB_TOKEN`, do not put a token in a remote URL, `.git/config`, a file, a\n commit or a message. The wrapper and helper already supply it.\n\n## Everyday commands\n\nRun these from the workspace; `gh` picks the repository from its `origin`.\n\n```sh\ngit switch -c <branch> # work on a branch, not the default one\ngit push -u origin <branch>\n\ngh pr create --base <default-branch> --head <branch> --title \"...\" --body-file <path>\ngh pr list --json number,title,state,headRefName,url\ngh pr view <number> --json number,title,body,state,author,baseRefName,headRefName,url\ngh pr diff <number>\ngh pr review <number> --comment --body \"...\" # or --approve / --request-changes\ngh pr comment <number> --body \"...\"\n\ngh api repos/{owner}/{repo}/issues/<number>/comments # conversation on a pull request\ngh api repos/{owner}/{repo}/pulls/<number>/comments # inline review comments\ngh api repos/{owner}/{repo}/pulls/<number>/reviews\n```\n\nPass `--json` with only the fields you need to `gh pr view` and `gh pr list`.\nPlain `gh pr view` also asks for checks, projects and other data this token\ncannot read, and fails; so do fields such as `statusCheckRollup`,\n`projectItems` or `closingIssuesReferences`. Do not use `gh pr checks`, `gh issue`,\n`gh run` or `gh workflow`; they need permissions the token does not have.\n\nUse `--body-file <path>` for long text instead of shell-quoting it. `gh` never\nprompts in this environment, so pass every value it would ask for as a flag.\n\n## Etiquette\n\n- Push branches and open pull requests; do not push to the default branch or\n force-push a branch someone else is working on unless asked.\n- Link the pull request in the room when you open one, and keep its\n description short: what changed and how you checked it.\n"
- }
- }
- ]
- }
- ]
-}
diff --git a/core/tests/switch_core/fixtures/hosted_machines/agents_response_unavailable.json b/core/tests/switch_core/fixtures/hosted_machines/agents_response_unavailable.json
deleted file mode 100644
index bf4e7b9d5..000000000
--- a/core/tests/switch_core/fixtures/hosted_machines/agents_response_unavailable.json
+++ /dev/null
@@ -1,16 +0,0 @@
-{
- "machine_id": "3f1c2b4a-0000-4000-8000-000000000001",
- "revision": 2,
- "desired_state": "stopped",
- "agents_version": 3,
- "agents": [
- {
- "launch_id": "3f1c2b4a-0000-4000-8000-0000000000c1",
- "agent_id": "3f1c2b4a-0000-4000-8000-0000000000a1",
- "name": "reviewer",
- "revision": 1,
- "desired_state": "running",
- "unavailable": "agent_key_missing"
- }
- ]
-}
diff --git a/core/tests/switch_core/fixtures/hosted_machines/heartbeat_request.json b/core/tests/switch_core/fixtures/hosted_machines/heartbeat_request.json
deleted file mode 100644
index 52f2a1cd5..000000000
--- a/core/tests/switch_core/fixtures/hosted_machines/heartbeat_request.json
+++ /dev/null
@@ -1,32 +0,0 @@
-{
- "boot_id": "00000000-0000-4000-8000-00000000b001",
- "instance_id": "i-0123456789abcdef0",
- "supervisor_version": "2.0.0",
- "runtime_fingerprint": "sha256:0000",
- "disk": {
- "path": "/data",
- "total_bytes": 214748364800,
- "used_bytes": 10737418240,
- "available_bytes": 204010946560
- },
- "memory": {
- "total_bytes": 17179869184,
- "available_bytes": 12884901888
- },
- "agents": [
- {
- "launch_id": "3f1c2b4a-0000-4000-8000-0000000000c1",
- "agent_id": "3f1c2b4a-0000-4000-8000-0000000000a1",
- "revision": 1,
- "process_state": "crashed",
- "restarts": 5,
- "oom_kills": 1,
- "exit": {
- "code": null,
- "signal": 9,
- "result": "start-limit-hit"
- },
- "since": "2026-01-01T00:00:00Z"
- }
- ]
-}
diff --git a/core/tests/switch_core/fixtures/hosted_machines/heartbeat_response.json b/core/tests/switch_core/fixtures/hosted_machines/heartbeat_response.json
deleted file mode 100644
index 1b1d8321f..000000000
--- a/core/tests/switch_core/fixtures/hosted_machines/heartbeat_response.json
+++ /dev/null
@@ -1,5 +0,0 @@
-{
- "agents_version": 3,
- "machine_desired_state": "running",
- "heartbeat_every_s": 15
-}
diff --git a/core/tests/switch_core/fixtures/hosted_machines/launch_summary.json b/core/tests/switch_core/fixtures/hosted_machines/launch_summary.json
deleted file mode 100644
index dc5780112..000000000
--- a/core/tests/switch_core/fixtures/hosted_machines/launch_summary.json
+++ /dev/null
@@ -1,16 +0,0 @@
-{
- "request_id": "3f1c2b4a-0000-4000-8000-0000000000c1",
- "name": "reviewer",
- "provider": "claude",
- "state": "error",
- "agent_id": "3f1c2b4a-0000-4000-8000-0000000000a1",
- "error": "The agent crashed 5 times in 10 minutes. Retry it in Switch Console.",
- "error_code": "agent_crashed",
- "desired_state": "running",
- "revision": 1,
- "sleeping": false,
- "machine_id": "3f1c2b4a-0000-4000-8000-000000000001",
- "process_state": "crashed",
- "process_restarts": 5,
- "oom_kills": 1
-}
diff --git a/core/tests/switch_core/fixtures/hosted_machines/machine_summary_sleeping.json b/core/tests/switch_core/fixtures/hosted_machines/machine_summary_sleeping.json
index f030bd740..1894b1c6d 100644
--- a/core/tests/switch_core/fixtures/hosted_machines/machine_summary_sleeping.json
+++ b/core/tests/switch_core/fixtures/hosted_machines/machine_summary_sleeping.json
@@ -20,5 +20,6 @@
},
"agents": [
"3f1c2b4a-0000-4000-8000-0000000000c1"
- ]
+ ],
+ "controller_id": "3f1c2b4a-0000-4000-8000-0000000000e1"
}
diff --git a/core/tests/switch_core/fixtures/hosted_machines/machines_response.json b/core/tests/switch_core/fixtures/hosted_machines/machines_response.json
index 6c5a22780..5f88bc3ae 100644
--- a/core/tests/switch_core/fixtures/hosted_machines/machines_response.json
+++ b/core/tests/switch_core/fixtures/hosted_machines/machines_response.json
@@ -2,8 +2,6 @@
"machines": [
{
"machine_id": "3f1c2b4a-0000-4000-8000-000000000001",
- "slot_id": "slot-a",
- "generation": 1,
"state": "ready",
"desired_state": "running",
"revision": 1,
diff --git a/core/tests/switch_core/fixtures/hosted_machines/prepare_controller_response.json b/core/tests/switch_core/fixtures/hosted_machines/prepare_controller_response.json
new file mode 100644
index 000000000..e4034c590
--- /dev/null
+++ b/core/tests/switch_core/fixtures/hosted_machines/prepare_controller_response.json
@@ -0,0 +1,10 @@
+{
+ "machine_id": "3f1c2b4a-0000-4000-8000-000000000001",
+ "revision": 1,
+ "bundle_revision": 1,
+ "api_endpoint": "https://switch.example.test/agent-api",
+ "controller": {
+ "id": null,
+ "enrollment_code": "swce_SyntheticEnrollmentCode0000"
+ }
+}
diff --git a/core/tests/switch_core/fixtures/hosted_machines/prepare_response.json b/core/tests/switch_core/fixtures/hosted_machines/prepare_response.json
deleted file mode 100644
index 315638bab..000000000
--- a/core/tests/switch_core/fixtures/hosted_machines/prepare_response.json
+++ /dev/null
@@ -1,9 +0,0 @@
-{
- "machine_id": "3f1c2b4a-0000-4000-8000-000000000001",
- "slot_id": "slot-a",
- "generation": 1,
- "revision": 1,
- "bundle_revision": 1,
- "machine_capability": "mcap-test-00000000000000000000000000000000",
- "api_endpoint": "https://switch.example.test/agent-api"
-}
diff --git a/core/tests/switch_core/gateway/test_agent_delete_route.py b/core/tests/switch_core/gateway/test_agent_delete_route.py
index 519d02f1c..37415ee4d 100644
--- a/core/tests/switch_core/gateway/test_agent_delete_route.py
+++ b/core/tests/switch_core/gateway/test_agent_delete_route.py
@@ -1,23 +1,13 @@
-"""The gateway routes that delete an agent refuse a cloud agent.
-
-A cloud agent's identity belongs to its hosted launch, and only removing the
-launch cleans up both; deleting the identity alone would strand the launch.
-"""
+"""The gateway route that deletes an agent."""
from __future__ import annotations
from unittest.mock import AsyncMock
-import pytest
-from fastapi import HTTPException
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.db.stores.agent_store import AgentStore
-from switch_core.gateway.agents import (
- CLOUD_AGENT_DELETE_REFUSED,
- delete_agent,
- delete_agent_by_name,
-)
+from switch_core.gateway.agents import delete_agent
from tests.switch_core.gateway.agent_route_harness import (
add_agent,
add_user,
@@ -27,42 +17,6 @@ from tests.switch_core.gateway.agent_route_harness import (
_AGENT_STORE = AgentStore()
[email protected]("by_name", [False, True])
-async def test_a_cloud_agent_is_not_deleted(
- session_factory: async_sessionmaker[AsyncSession], by_name: bool
-) -> None:
- async with session_factory() as session:
- owner = await add_user(session, name="owner")
- agent = await add_agent(session, name="cloud", owner_id=owner.id)
- agent.metadata_ = {"hosted_launch_id": "launch-1"}
- await session.flush()
- protocol = AsyncMock()
-
- with pytest.raises(HTTPException) as refused:
- if by_name:
- await delete_agent_by_name(
- agent.name,
- session,
- _AGENT_STORE,
- protocol,
- owner,
- await is_admin(session, owner),
- )
- else:
- await delete_agent(
- agent.id,
- session,
- _AGENT_STORE,
- protocol,
- owner,
- await is_admin(session, owner),
- )
-
- assert refused.value.status_code == 409
- assert refused.value.detail == CLOUD_AGENT_DELETE_REFUSED
- protocol.delete_agent.assert_not_awaited()
-
-
async def test_a_local_agent_is_deleted(
session_factory: async_sessionmaker[AsyncSession],
) -> None:
diff --git a/core/tests/switch_core/gateway/test_cloud_controller_machines.py b/core/tests/switch_core/gateway/test_cloud_controller_machines.py
new file mode 100644
index 000000000..a30c82db9
--- /dev/null
+++ b/core/tests/switch_core/gateway/test_cloud_controller_machines.py
@@ -0,0 +1,541 @@
+"""Switch cloud machines that run the agents controller, end to end against Postgres.
+
+The hosted controller prepares a cloud machine; Core hands it a
+one-time enrollment code (the same one on every retry of a revision); the
+controller enrolls with it and becomes the machine's, as a Switch cloud
+controller; its status reports make the machine ready; and the managed agents
+placed on it are the machine's agents.
+"""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Iterator
+from datetime import UTC, datetime, timedelta
+from pathlib import Path
+from types import SimpleNamespace
+
+import httpx
+import pytest
+from fastapi import FastAPI
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import (
+ AgentController,
+ HostedMachine,
+ User,
+ require_tenant_id,
+)
+from switch_core.db.stores.hosted_machine_store import (
+ HostedMachineStore,
+ bump_revision,
+ lock_claims,
+)
+from switch_core.gateway.auth import get_current_user
+from switch_core.gateway.cloud_controllers import (
+ set_cloud_enrollment,
+ wake_controller_machine,
+)
+from switch_core.gateway.dependencies import (
+ get_config,
+ get_session,
+ get_session_factory,
+)
+from switch_core.gateway.hosted_controller import router as controller_router
+from switch_core.gateway.hosted_machines import router as machines_router
+from switch_core.management.placement import placement_refusal
+from switch_core.management.process_lease import ProcessLeases
+from switch_core.providers.hosted import HostedControllerSettings
+from tests.switch_core.management.harness import (
+ TEST_KEYRING,
+ Harness,
+ add_member,
+ bearer,
+ build_harness,
+ cookies_for,
+ create_managed_agent,
+ platform,
+ report_status,
+)
+
+TOKEN = "SYNTHETIC-CLOUD-CONTROLLER-CREDENTIAL-0123456789"
+HEADERS = {"Authorization": f"Bearer {TOKEN}"}
+API_ENDPOINT = "https://switch.example.test/api/agent"
+FIXTURES = Path(__file__).parent.parent / "fixtures" / "hosted_machines"
+
+
[email protected](autouse=True)
+def _no_enrollment_left_installed() -> Iterator[None]:
+ """Building management installs it process-wide; take it away again."""
+ set_cloud_enrollment(None)
+ yield
+ set_cloud_enrollment(None)
+
+
+def harness(session_factory: async_sessionmaker[AsyncSession]) -> Harness:
+ return build_harness(session_factory)
+
+
+def _hosted_app(
+ session_factory: async_sessionmaker[AsyncSession], owner: User
+) -> httpx.AsyncClient:
+ app = FastAPI()
+ app.state.hosted_controller_settings = HostedControllerSettings(
+ tenant_id=require_tenant_id(),
+ token=TOKEN,
+ agent_api_endpoint=API_ENDPOINT,
+ )
+ app.include_router(controller_router)
+ app.include_router(machines_router)
+ config = SimpleNamespace(
+ keyring=TEST_KEYRING,
+ hosted_idle_stop_minutes=30,
+ hosted_disk_retention_days=7,
+ hosted_launch_capacity=1,
+ )
+
+ async def session():
+ async with session_factory() as opened:
+ yield opened
+
+ async def current_user():
+ return owner
+
+ app.dependency_overrides[get_session] = session
+ app.dependency_overrides[get_session_factory] = lambda: session_factory
+ app.dependency_overrides[get_config] = lambda: config
+ app.dependency_overrides[get_current_user] = current_user
+ return httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=app), base_url="https://switch.example.test"
+ )
+
+
+async def _claim(
+ session_factory: async_sessionmaker[AsyncSession], owner: User
+) -> HostedMachine:
+ async with session_factory() as session:
+ await lock_claims(session)
+ machine = await HostedMachineStore().claim(
+ session,
+ owner_id=owner.id,
+ capacity=1,
+ now=datetime.now(UTC),
+ )
+ await session.commit()
+ return machine
+
+
+async def _machine(
+ session_factory: async_sessionmaker[AsyncSession], machine_id: str
+) -> HostedMachine:
+ async with session_factory() as session:
+ machine = await session.get(HostedMachine, (require_tenant_id(), machine_id))
+ assert machine is not None
+ return machine
+
+
+async def _update(
+ session_factory: async_sessionmaker[AsyncSession], machine_id: str, **values
+) -> None:
+ async with session_factory() as session:
+ machine = await HostedMachineStore().locked(session, machine_id)
+ assert machine is not None
+ for key, value in values.items():
+ setattr(machine, key, value)
+ await session.commit()
+
+
+async def _new_revision(
+ session_factory: async_sessionmaker[AsyncSession], machine_id: str
+) -> None:
+ async with session_factory() as session:
+ machine = await HostedMachineStore().locked(session, machine_id)
+ assert machine is not None
+ bump_revision(machine, datetime.now(UTC))
+ await session.commit()
+
+
+async def _prepare(hosted: httpx.AsyncClient, machine_id: str) -> dict:
+ response = await hosted.post(
+ f"/hosted-controller/machines/{machine_id}/prepare", headers=HEADERS
+ )
+ assert response.status_code == 200, response.text
+ return dict(response.json())
+
+
+async def _enroll(client: httpx.AsyncClient, code: str) -> dict:
+ response = await client.post(
+ "/v1/management/controllers/enroll",
+ json={
+ "proof": {"kind": "enrollment_code", "code": code},
+ "controller": {
+ "kind": "daemon",
+ "name": "ip-10-0-0-1",
+ "platform": platform(),
+ "version": "0.1.1",
+ },
+ },
+ )
+ assert response.status_code == 201, response.text
+ return dict(response.json())
+
+
+async def _connected(
+ client: httpx.AsyncClient, harness: Harness, owner: User, enrolled: dict
+) -> SimpleNamespace:
+ """The enrolled controller, with a token, connected and reporting as a running one."""
+ token = await client.post(
+ f"/v1/management/controllers/{enrolled['controller_id']}/token",
+ json={"credential": enrolled["credential"]},
+ )
+ assert token.status_code == 200, token.text
+ controller = SimpleNamespace(
+ controller_id=enrolled["controller_id"],
+ credential=enrolled["credential"],
+ access_token=token.json()["access_token"],
+ owner=owner,
+ harness=harness,
+ headers=bearer(token.json()["access_token"]),
+ )
+ await report_status(client, controller, 1) # type: ignore[arg-type]
+ return controller
+
+
+class TestControllerMachines:
+ async def test_a_machine_enrolls_its_controller_with_a_code_core_hands_over(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ machine = await _claim(harness.session_factory, owner)
+ async with _hosted_app(harness.session_factory, owner) as hosted:
+ first = await _prepare(hosted, machine.id)
+ retried = await _prepare(hosted, machine.id)
+ assert first["bundle_revision"] == first["revision"]
+ assert first["api_endpoint"] == API_ENDPOINT
+ code = first["controller"]["enrollment_code"]
+ assert code.startswith("swce_")
+ assert first["controller"]["id"] is None
+ # A retry of the same revision hands over the same code.
+ assert retried["controller"] == first["controller"]
+ assert "machine_capability" not in first
+ # The shape the hosted controller is tested against.
+ fixture = json.loads(
+ (FIXTURES / "prepare_controller_response.json").read_text()
+ )
+ assert set(first) == set(fixture)
+ assert set(first["controller"]) == set(fixture["controller"])
+
+ async with harness.client() as client:
+ enrolled = await _enroll(client, code)
+ again = await client.post(
+ "/v1/management/controllers/enroll",
+ json={
+ "proof": {"kind": "enrollment_code", "code": code},
+ "controller": {
+ "kind": "daemon",
+ "name": "x",
+ "platform": platform(),
+ "version": "0.1.1",
+ },
+ },
+ )
+ assert again.status_code == 401, again.text
+ async with harness.session_factory() as session:
+ controller = await session.get(
+ AgentController, enrolled["controller_id"]
+ )
+ assert controller is not None
+ assert controller.kind == "ec2"
+ assert controller.name == "Switch cloud"
+ assert (
+ await _machine(harness.session_factory, machine.id)
+ ).controller_id == (enrolled["controller_id"])
+
+ await _new_revision(harness.session_factory, machine.id)
+ enrolled_since = await _prepare(hosted, machine.id)
+ assert enrolled_since["controller"] == {
+ "id": enrolled["controller_id"],
+ "enrollment_code": None,
+ }
+
+ async def test_its_status_reports_make_it_ready_and_its_agents_are_the_machines(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ machine = await _claim(harness.session_factory, owner)
+ async with _hosted_app(harness.session_factory, owner) as hosted:
+ code = (await _prepare(hosted, machine.id))["controller"]["enrollment_code"]
+ async with harness.client() as client:
+ enrolled = await _enroll(client, code)
+ token = await client.post(
+ f"/v1/management/controllers/{enrolled['controller_id']}/token",
+ json={"credential": enrolled["credential"]},
+ )
+ assert token.status_code == 200, token.text
+ controller = SimpleNamespace(
+ controller_id=enrolled["controller_id"],
+ credential=enrolled["credential"],
+ access_token=token.json()["access_token"],
+ owner=owner,
+ harness=harness,
+ headers=bearer(token.json()["access_token"]),
+ )
+ # Not ready before the machine was seen running at this revision.
+ await report_status(client, controller, 1) # type: ignore[arg-type]
+ assert (await _machine(harness.session_factory, machine.id)).state == (
+ "provisioning"
+ )
+ await _update(
+ harness.session_factory,
+ machine.id,
+ running_observed_at=datetime.now(UTC),
+ )
+ await report_status(client, controller, 2) # type: ignore[arg-type]
+ ready = await _machine(harness.session_factory, machine.id)
+ assert ready.state == "ready"
+ assert ready.heartbeat is not None
+ assert ready.heartbeat["disk"]["total_bytes"] > 0
+
+ created = await create_managed_agent(
+ client,
+ owner,
+ name="reviewer",
+ controller_id=enrolled["controller_id"],
+ )
+ assert created.status_code == 201, created.text
+ summary = await hosted.get(f"/hosted-machines/{machine.id}")
+ assert summary.status_code == 200, summary.text
+ assert summary.json()["controller_id"] == enrolled["controller_id"]
+ assert summary.json()["agents"] == [created.json()["agent_id"]]
+
+ async def test_a_revoked_controller_is_replaced_by_enrolling_again(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ machine = await _claim(harness.session_factory, owner)
+ async with _hosted_app(harness.session_factory, owner) as hosted:
+ code = (await _prepare(hosted, machine.id))["controller"]["enrollment_code"]
+ async with harness.client() as client:
+ first = await _enroll(client, code)
+ await _connected(client, harness, owner, first)
+ created = await create_managed_agent(
+ client, owner, name="reviewer", controller_id=first["controller_id"]
+ )
+ assert created.status_code == 201, created.text
+ revoked = await client.delete(
+ f"/gateway/management/controllers/{first['controller_id']}",
+ cookies=cookies_for(owner),
+ )
+ assert revoked.status_code == 200, revoked.text
+ await _new_revision(harness.session_factory, machine.id)
+ prepared = await _prepare(hosted, machine.id)
+ assert prepared["controller"]["id"] is None
+ second_code = prepared["controller"]["enrollment_code"]
+ assert second_code not in {None, code}
+ second = await _enroll(client, second_code)
+ agent = await client.get(
+ f"/gateway/management/agents/{created.json()['agent_id']}",
+ cookies=cookies_for(owner),
+ )
+ assert agent.status_code == 200, agent.text
+ assert agent.json()["controller_id"] == second["controller_id"]
+ assert (
+ await _machine(harness.session_factory, machine.id)
+ ).controller_id == (second["controller_id"])
+ summary = await hosted.get(f"/hosted-machines/{machine.id}")
+ assert summary.json()["agents"] == [created.json()["agent_id"]]
+
+ async def test_an_agentless_machine_is_retained_only_once_its_agents_are_gone(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ machine = await _claim(harness.session_factory, owner)
+ async with _hosted_app(harness.session_factory, owner) as hosted:
+ code = (await _prepare(hosted, machine.id))["controller"]["enrollment_code"]
+ async with harness.client() as client:
+ enrolled = await _enroll(client, code)
+ await _connected(client, harness, owner, enrolled)
+ created = await create_managed_agent(
+ client,
+ owner,
+ name="reviewer",
+ controller_id=enrolled["controller_id"],
+ desired_state="stopped",
+ )
+ assert created.status_code == 201, created.text
+ store = HostedMachineStore()
+ async with harness.session_factory() as session:
+ locked = await store.locked(session, machine.id)
+ assert locked is not None
+ assert await store.has_agents(session, locked)
+ assert not await store.retain_if_empty(
+ session, locked, retention_days=7, now=datetime.now(UTC)
+ )
+ assert await store.ever_hosted(session, locked)
+ await session.commit()
+ async with harness.client() as client:
+ deleted = await client.delete(
+ f"/gateway/management/agents/{created.json()['agent_id']}",
+ cookies=cookies_for(owner),
+ )
+ assert deleted.status_code == 200, deleted.text
+ async with harness.session_factory() as session:
+ locked = await store.locked(session, machine.id)
+ assert locked is not None
+ assert not await store.has_agents(session, locked)
+ now = datetime.now(UTC)
+ assert await store.retain_if_empty(
+ session, locked, retention_days=7, now=now
+ )
+ assert (locked.desired_state, locked.retain_until) == (
+ "retained",
+ now + timedelta(days=7),
+ )
+
+ @pytest.mark.parametrize(
+ ("heartbeat_age", "sessions", "sleeps"),
+ [
+ (timedelta(minutes=1), 0, True),
+ (timedelta(minutes=1), 2, False),
+ # A machine that stopped reporting is not known to be idle.
+ (timedelta(hours=1), 0, False),
+ ],
+ )
+ async def test_a_controller_machine_sleeps_once_its_reports_say_it_is_idle(
+ self, harness: Harness, heartbeat_age, sessions, sleeps
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ machine = await _claim(harness.session_factory, owner)
+ now = datetime.now(UTC)
+ await _update(
+ harness.session_factory,
+ machine.id,
+ state="ready",
+ active_at=now - timedelta(hours=2),
+ heartbeat={"disk": None, "memory": None, "sessions_running": sessions},
+ heartbeat_at=now - heartbeat_age,
+ )
+ async with _hosted_app(harness.session_factory, owner) as hosted:
+ listed = await hosted.get("/hosted-controller/machines", headers=HEADERS)
+ assert listed.status_code == 200, listed.text
+ after = await _machine(harness.session_factory, machine.id)
+ if sleeps:
+ # With no agent on it, an idle machine is released rather than stopped.
+ assert after.desired_state == "retained"
+ else:
+ assert after.desired_state == "running"
+
+ async def test_an_idle_machine_with_agents_is_put_to_sleep(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ machine = await _claim(harness.session_factory, owner)
+ async with _hosted_app(harness.session_factory, owner) as hosted:
+ code = (await _prepare(hosted, machine.id))["controller"]["enrollment_code"]
+ async with harness.client() as client:
+ enrolled = await _enroll(client, code)
+ await _connected(client, harness, owner, enrolled)
+ created = await create_managed_agent(
+ client,
+ owner,
+ name="reviewer",
+ controller_id=enrolled["controller_id"],
+ )
+ assert created.status_code == 201, created.text
+ now = datetime.now(UTC)
+ await _update(
+ harness.session_factory,
+ machine.id,
+ state="ready",
+ active_at=now - timedelta(hours=2),
+ heartbeat={"disk": None, "memory": None, "sessions_running": 0},
+ heartbeat_at=now,
+ )
+ listed = await hosted.get("/hosted-controller/machines", headers=HEADERS)
+ assert listed.status_code == 200, listed.text
+ after = await _machine(harness.session_factory, machine.id)
+ assert (after.desired_state, after.stop_reason) == ("stopped", "idle")
+
+ async def test_a_message_or_a_placement_wakes_a_sleeping_machine(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ machine = await _claim(harness.session_factory, owner)
+ async with _hosted_app(harness.session_factory, owner) as hosted:
+ code = (await _prepare(hosted, machine.id))["controller"]["enrollment_code"]
+ async with harness.client() as client:
+ enrolled = await _enroll(client, code)
+ await _update(
+ harness.session_factory,
+ machine.id,
+ state="stopped",
+ desired_state="stopped",
+ stop_reason="idle",
+ )
+ before = await _machine(harness.session_factory, machine.id)
+ async with harness.session_factory() as session:
+ woken = await wake_controller_machine(
+ session, enrolled["controller_id"], datetime.now(UTC)
+ )
+ await session.commit()
+ assert woken is not None
+ after = await _machine(harness.session_factory, machine.id)
+ assert (after.desired_state, after.stop_reason) == ("running", None)
+ assert after.revision == before.revision + 1
+
+ await _update(
+ harness.session_factory,
+ machine.id,
+ desired_state="stopped",
+ stop_reason="owner",
+ )
+ async with harness.session_factory() as session:
+ await wake_controller_machine(
+ session, enrolled["controller_id"], datetime.now(UTC)
+ )
+ await session.commit()
+ # A machine its owner stopped stays stopped.
+ assert (await _machine(harness.session_factory, machine.id)).desired_state == (
+ "stopped"
+ )
+
+
+async def test_a_controller_machine_needs_agent_management(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ owner = await add_member(session_factory, "ada")
+ machine = await _claim(session_factory, owner)
+ async with _hosted_app(session_factory, owner) as hosted:
+ response = await hosted.post(
+ f"/hosted-controller/machines/{machine.id}/prepare", headers=HEADERS
+ )
+ assert response.status_code == 409, response.text
+ assert "AGENT_MANAGEMENT_ENABLED" in response.json()["detail"]
+
+
+def test_a_waking_machine_is_judged_by_its_last_report() -> None:
+ now = datetime.now(UTC)
+ controller = AgentController(
+ owner_id="owner",
+ name="Switch cloud",
+ kind="ec2",
+ api_key_id="key",
+ connected_at=now - timedelta(hours=3),
+ disconnected_at=now - timedelta(hours=2),
+ last_seen_at=now - timedelta(hours=2),
+ status={"providers": [{"provider": "claude", "installed": True, "auth": "ok"}]},
+ )
+ leases = ProcessLeases(read_at=now, leases={})
+ asleep = placement_refusal(
+ controller, "claude", leases=leases, now=now, interval_seconds=60, waking=False
+ )
+ waking = placement_refusal(
+ controller, "claude", leases=leases, now=now, interval_seconds=60, waking=True
+ )
+ not_installed = placement_refusal(
+ controller, "codex", leases=leases, now=now, interval_seconds=60, waking=True
+ )
+ assert asleep is not None and asleep[0] == "controller_offline"
+ assert waking is None
+ assert not_installed is not None and not_installed[0] == "provider_not_installed"
diff --git a/core/tests/switch_core/gateway/test_github_connections.py b/core/tests/switch_core/gateway/test_github_connections.py
index 7e17bc293..94e660dec 100644
--- a/core/tests/switch_core/gateway/test_github_connections.py
+++ b/core/tests/switch_core/gateway/test_github_connections.py
@@ -2,11 +2,9 @@ import asyncio
import json
import secrets
import time
-from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from unittest.mock import AsyncMock
from urllib.parse import parse_qs, urlsplit
-from uuid import uuid4
import httpx
import pytest
@@ -14,8 +12,6 @@ from fastapi import FastAPI
from sqlalchemy import select, text
from switch_core.db.models import (
- GitHubIssuedToken,
- HostedLaunch,
ProviderConnection,
User,
require_tenant_id,
@@ -433,34 +429,12 @@ async def test_oauth_revocation_deletes_only_the_selected_token(
assert request.call_args.kwargs["json"] == {"access_token": "SYNTHETIC-OLD-ACCESS"}
-async def test_relink_queues_existing_installation_tokens(github_app, monkeypatch):
- client, github, identity, factory = github_app
+async def test_relink_revokes_the_previous_sign_in(github_app):
+ client, github, _, _ = github_app
first = await authorize(client)
await relay(client, first)
assert (await complete(client, first)).status_code == 204
assert (await confirm(client, first)).status_code == 200
- async with factory() as session:
- launch = HostedLaunch(
- id=str(uuid4()), owner_id=identity["user"].id, name="relink-worker", spec={}
- )
- session.add(launch)
- await session.flush()
- issued = GitHubIssuedToken(
- id=str(uuid4()),
- owner_id=launch.owner_id,
- launch_id=launch.id,
- launch_revision=1,
- encrypted_token=TEST_KEYRING.encrypt("SYNTHETIC-INSTALLATION"),
- expires_at=datetime.now(UTC) + timedelta(hours=1),
- revoke_requested=False,
- attempts=0,
- )
- session.add(issued)
- await session.commit()
- monkeypatch.setattr(
- "switch_core.gateway.github_connections.revoke_pending",
- AsyncMock(return_value=True),
- )
github.exchange.return_value = {
**github.exchange.return_value,
"access_token": "SYNTHETIC-RELINKED-ACCESS",
@@ -470,8 +444,5 @@ async def test_relink_queues_existing_installation_tokens(github_app, monkeypatc
assert (await complete(client, second)).status_code == 204
response = await confirm(client, second)
assert response.status_code == 200
- assert response.json()["warning"]
+ assert response.json()["warning"] is None
github.revoke.assert_awaited_once_with("SYNTHETIC-ACCESS")
- async with factory() as session:
- row = await session.get(GitHubIssuedToken, (require_tenant_id(), issued.id))
- assert row.revoke_requested
diff --git a/core/tests/switch_core/gateway/test_held_connections.py b/core/tests/switch_core/gateway/test_held_connections.py
index 855cd1b73..45537d598 100644
--- a/core/tests/switch_core/gateway/test_held_connections.py
+++ b/core/tests/switch_core/gateway/test_held_connections.py
@@ -57,7 +57,7 @@ from switch_core.gateway.auth import (
)
from switch_core.gateway.auth_routes import router as auth_router
from switch_core.gateway.collaborations import router as bridges_router
-from switch_core.gateway.hosted_launches import hosted_settings
+from switch_core.gateway.hosted_machines import hosted_settings
from switch_core.gateway.rooms import router as rooms_router
from switch_core.keys import Keyring
from switch_core.room_service import RoomService
diff --git a/core/tests/switch_core/gateway/test_hosted_controller.py b/core/tests/switch_core/gateway/test_hosted_controller.py
index ceb9dd507..ffcf8d035 100644
--- a/core/tests/switch_core/gateway/test_hosted_controller.py
+++ b/core/tests/switch_core/gateway/test_hosted_controller.py
@@ -1,86 +1,46 @@
-import asyncio
+"""The hosted-controller routes: listing and sweeping cloud machines, preparing
+one to boot (the controller it enrolled as, or a one-time code to enroll with),
+and recording what the operator controller observed of it."""
+
import json
-import logging
-import time
+from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from pathlib import Path
from types import SimpleNamespace
-from unittest.mock import AsyncMock, Mock
from uuid import uuid4
import httpx
import pytest
from fastapi import FastAPI
-from sqlalchemy import select, text
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-from switch_core.bridges.agent.api.hosted_cutover_routes import (
- router as hosted_cutover_router,
-)
-from switch_core.bridges.agent.api.hosted_machine_routes import (
- router as hosted_machine_router,
-)
-from switch_core.bridges.agent.api.hosted_routes import router as worker_router
-from switch_core.bridges.agent.api.hosted_worker_routes import (
- router as hosted_worker_router,
-)
-from switch_core.bridges.agent.auth import get_agent_from_scope
-from switch_core.bridges.agent.dependencies import get_config as get_worker_config
-from switch_core.bridges.agent.dependencies import get_protocol as get_worker_protocol
-from switch_core.bridges.agent.dependencies import get_session as get_worker_session
-from switch_core.bridges.agent.dependencies import (
- get_session_factory as get_worker_session_factory,
-)
-from switch_core.bridges.agent.protocol.agent_connections import (
- AgentConnection,
- AgentConnectionRegistry,
- ClientDeclaration,
-)
-from switch_core.bridges.agent.protocol.agent_core import AgentExistsError
-from switch_core.bridges.agent.protocol.hosted_workers import IdleReport, WorkerBinding
from switch_core.db.models import (
- Agent,
- ApiKey,
- GitHubIssuedToken,
- HostedLaunch,
+ AgentController,
HostedMachine,
- HostedOperation,
- ProviderConnection,
TenantMember,
User,
require_tenant_id,
)
-from switch_core.db.stores.client_store import ClientStore
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
-from switch_core.db.stores.hosted_machine_store import (
- HostedMachineStore,
- lock_launch,
- lock_launches,
+from switch_core.db.stores.hosted_machine_store import HostedMachineStore, lock_claims
+from switch_core.gateway.auth import get_current_user
+from switch_core.gateway.cloud_controllers import (
+ record_controller_status,
+ set_cloud_enrollment,
)
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.gateway.auth import get_current_user, get_current_user_in_transaction
from switch_core.gateway.dependencies import (
get_config,
- get_protocol,
get_session,
get_session_factory,
)
-from switch_core.gateway.hosted_controller import DELETING_RESUME_AFTER, router
-from switch_core.gateway.hosted_launches import router as launch_router
-from switch_core.gateway.hosted_relay import router as relay_router
-from switch_core.gateway.known_agents import KNOWN_AGENTS
+from switch_core.gateway.hosted_controller import router
+from switch_core.gateway.hosted_machines import router as machines_router
from switch_core.keys import Keyring
-from switch_core.providers.github_installation import (
- GitHubInstallationCredentials,
- RepositoryCredential,
-)
-from switch_core.providers.github_revocations import queue_revocation, revoke_pending
from switch_core.providers.hosted import HostedControllerSettings
-from tests.switch_core.bridges.agent.protocol.registration_harness import (
- PROFILE,
- make_owner,
- make_service,
+from tests.switch_core.hosted_machine_helpers import (
+ link_controller,
+ place_managed_agent,
+ seed_machine,
)
-from tests.switch_core.hosted_machine_helpers import seed_launch, seed_machine
TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
@@ -89,8 +49,6 @@ HEADERS = {"Authorization": "Bearer " + TOKEN}
FIXTURES = Path(__file__).parent.parent / "fixtures" / "hosted_machines"
MACHINE_ITEM_KEYS = {
"machine_id",
- "slot_id",
- "generation",
"state",
"desired_state",
"revision",
@@ -104,196 +62,102 @@ def fixture(name: str) -> dict:
return json.loads((FIXTURES / name).read_text())
-async def register_hosted_agent(
- service, *, owner: str, request_id: str, name: str, spec: dict
-) -> str:
- """Register a launch's agent identity the way launch creation does."""
- known = KNOWN_AGENTS["claude-code"]
- options = known.parse_options(
- {
- "channels_enabled": True,
- "repo_dir": "/data/worktrees/agent/workspace",
- "auto_session": spec["auto_session"],
- }
- )
- result = await service.register_agent(
- name=name,
- description=spec["description"],
- display_name=spec["display_name"],
- icon_url=spec["icon_url"],
- connector_type=known.connector_type,
- integration_profile=known.build_profile(options),
- tools=known.tools,
- models=known.models,
- metadata={
- "known_agent_type": "claude-code",
- "known_agent_options": options.model_dump(),
- "hosted_launch_id": request_id,
- },
- owner_id=owner,
- )
- return result.agent_id
-
-
-SPEC = {
- "description": "Cloud helper",
- "display_name": None,
- "icon_url": None,
- "instructions": "Help with the repository.",
- "auto_session": True,
- "auto_approve": False,
- "addressing_policy": None,
- "definition_attributes": {},
- "installation_id": 123,
- "repository_id": 456,
-}
+class FakeEnrollment:
+ """Agent management's side of enrollment: mints a fresh code per call."""
+
+ def __init__(self) -> None:
+ self.minted: list[tuple[str, int]] = []
+
+ async def machine_enrollment_code(
+ self, session: AsyncSession, machine: HostedMachine, now: datetime
+ ) -> str:
+ self.minted.append((machine.id, machine.revision))
+ return f"swce_synthetic-{len(self.minted)}"
+
+
+@dataclass
+class ControllerApp:
+ client: httpx.AsyncClient
+ factory: async_sessionmaker[AsyncSession]
+ machine_id: str
+ owner_id: str
+ config: SimpleNamespace
+ settings: HostedControllerSettings
+ enrollment: FakeEnrollment
@pytest.fixture
-async def controller_app(session_factory, monkeypatch, tmp_path):
- """A queued machine with one queued launch whose agent is registered."""
- owner = await make_owner(session_factory)
- request_id = str(uuid4())
+async def controller_app(session_factory):
+ """A member's queued cloud machine, behind the hosted-controller and the
+ owner's hosted-machines routes."""
async with session_factory() as session:
- session.add(
- TenantMember(tenant_id=require_tenant_id(), user_id=owner, role="member")
+ owner = User(
+ name="owner", email="[email protected]", role="user", password_hash="x"
)
+ session.add(owner)
+ await session.flush()
session.add(
- ProviderConnection(
- user_id=owner,
- provider="github",
- kind="oauth",
- encrypted_credential=TEST_KEYRING.encrypt(
- json.dumps(
- {
- "access_token": "SYNTHETIC-GITHUB",
- "expires_at": (
- datetime.now(UTC) + timedelta(hours=1)
- ).timestamp(),
- }
- ),
- ),
- verified_at=datetime.now(UTC),
- )
+ TenantMember(tenant_id=require_tenant_id(), user_id=owner.id, role="member")
)
- await ProviderConnectionStore().save(
- session,
- owner,
- "setup-token",
- TEST_KEYRING.encrypt("SYNTHETIC-CLAUDE"),
- datetime.now(UTC),
- )
- await session.commit()
- service = make_service(session_factory)
- service.connections = AgentConnectionRegistry()
- service.event_buffer = SimpleNamespace(boot=1, remove=Mock())
- service.config.hosted_idle_stop_minutes = 0
- service.config.hosted_disk_retention_days = 7
- agent_id = await register_hosted_agent(
- service, owner=owner, request_id=request_id, name="cloud-helper", spec=SPEC
- )
- async with session_factory() as session:
machine = await seed_machine(
session,
- owner_id=owner,
- slot_id="slot-a",
+ owner_id=owner.id,
state="queued",
desired_state="running",
stop_reason=None,
revision=1,
- generation=1,
- )
- await seed_launch(
- session,
- machine=machine,
- request_id=request_id,
- name="cloud-helper",
- state="queued",
- desired_state="running",
- revision=1,
- agent_id=agent_id,
- spec=SPEC,
)
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- launch.repository = "example/project"
await session.commit()
settings = HostedControllerSettings(
tenant_id=require_tenant_id(),
token=TOKEN,
- machine_slots=["slot-a", "slot-b"],
- github_private_key_path="/tmp/synthetic-signing-key.pem",
agent_api_endpoint="https://switch.example.com/api/agent",
)
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", AsyncMock())
+ config = SimpleNamespace(
+ keyring=TEST_KEYRING,
+ hosted_idle_stop_minutes=0,
+ hosted_disk_retention_days=7,
+ hosted_launch_capacity=2,
+ )
app = FastAPI()
app.state.hosted_controller_settings = settings
- app.state.github_connections = SimpleNamespace(client_id="synthetic-app")
app.include_router(router)
- app.include_router(worker_router)
- app.include_router(hosted_machine_router)
- app.include_router(hosted_worker_router, prefix="/agents")
- app.include_router(hosted_cutover_router, prefix="/agents")
- app.include_router(launch_router)
- app.include_router(relay_router)
- settings_path = tmp_path / "controller.json"
- settings_path.write_text(
- json.dumps({**settings.model_dump(mode="json"), "token": TOKEN})
- )
- service.config.hosted_controller_config_path = str(settings_path)
- service.config.hosted_github_config_path = "/tmp/synthetic-github.json"
+ app.include_router(machines_router)
- async def worker_session():
- async with session_factory() as session:
- yield session
+ async def session():
+ async with session_factory() as opened:
+ yield opened
async def current_user():
- async with session_factory() as session:
- return await session.get(User, owner)
-
- app.dependency_overrides[get_current_user] = current_user
- app.dependency_overrides[get_current_user_in_transaction] = current_user
- app.dependency_overrides[get_session] = worker_session
-
- async def worker_agent():
- async with session_factory() as session:
- return await session.get(Agent, agent_id)
+ return owner
- app.dependency_overrides[get_agent_from_scope] = worker_agent
- app.dependency_overrides[get_worker_config] = lambda: service.config
- app.dependency_overrides[get_worker_session] = worker_session
- app.dependency_overrides[get_worker_session_factory] = lambda: session_factory
+ app.dependency_overrides[get_session] = session
app.dependency_overrides[get_session_factory] = lambda: session_factory
- app.dependency_overrides[get_config] = lambda: service.config
- app.dependency_overrides[get_protocol] = lambda: service
- app.dependency_overrides[get_worker_protocol] = lambda: service
- issue = AsyncMock(
- return_value=RepositoryCredential(
- "SYNTHETIC-REPOSITORY",
- datetime.now(UTC) + timedelta(hours=1),
- 456,
- "example/project",
- )
- )
- monkeypatch.setattr(
- "switch_core.bridges.agent.api.hosted_routes.GitHubConnections",
- lambda _: app.state.github_connections,
- )
- monkeypatch.setattr(
- "switch_core.bridges.agent.api.hosted_routes.GitHubInstallationCredentials",
- lambda *_: SimpleNamespace(issue=issue, revoke=AsyncMock()),
- )
- async with httpx.AsyncClient(
- transport=httpx.ASGITransport(app=app), base_url="https://switch.example.com"
- ) as client:
- yield client, request_id, agent_id, service, session_factory, settings
+ app.dependency_overrides[get_config] = lambda: config
+ app.dependency_overrides[get_current_user] = current_user
+ enrollment = FakeEnrollment()
+ set_cloud_enrollment(enrollment)
+ try:
+ async with httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=app),
+ base_url="https://switch.example.com",
+ ) as client:
+ yield ControllerApp(
+ client=client,
+ factory=session_factory,
+ machine_id=machine.id,
+ owner_id=owner.id,
+ config=config,
+ settings=settings,
+ enrollment=enrollment,
+ )
+ finally:
+ set_cloud_enrollment(None)
-async def machine_of(factory, request_id: str) -> HostedMachine:
+async def machine_of(factory, machine_id: str) -> HostedMachine:
async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- machine = await session.get(
- HostedMachine, (require_tenant_id(), launch.machine_id)
- )
+ machine = await session.get(HostedMachine, (require_tenant_id(), machine_id))
assert machine is not None
return machine
@@ -301,66 +165,39 @@ async def machine_of(factory, request_id: str) -> HostedMachine:
async def update_machine(factory, machine_id: str, **values) -> None:
async with factory() as session:
machine = await session.get(HostedMachine, (require_tenant_id(), machine_id))
+ assert machine is not None
for key, value in values.items():
setattr(machine, key, value)
await session.commit()
-async def update_launch(factory, request_id: str, **values) -> None:
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- for key, value in values.items():
- setattr(launch, key, value)
+async def enrolled(app: ControllerApp, *, agents: int) -> tuple[str, list[str]]:
+ """Link the machine to the controller it enrolled as, with `agents`
+ managed agents placed on it; the controller and agent ids."""
+ async with app.factory() as session:
+ machine = await session.get(
+ HostedMachine, (require_tenant_id(), app.machine_id)
+ )
+ assert machine is not None
+ controller = await link_controller(session, machine)
+ placed = [
+ await place_managed_agent(
+ session,
+ owner_id=app.owner_id,
+ controller_id=controller.id,
+ name=f"agent-{index}",
+ )
+ for index in range(agents)
+ ]
await session.commit()
+ return controller.id, [definition.agent_id for definition in placed]
-def attach_worker(
- service,
- agent_id: str,
- request_id: str,
- *,
- revision: int = 1,
- boot_id: str = "boot-a",
- connection_id: str | None = None,
-) -> AgentConnection:
- conn = service.connections.open(
- agent_id=agent_id,
- connection_id=connection_id or str(uuid4()),
- scope="all",
- delivery_filter="all",
- spawn_capable=True,
- cursor=0,
- declaration=ClientDeclaration(speaks=7, accepts=1),
- expected_generation=None,
- )
- service.connections.bind_worker(
- conn, WorkerBinding(request_id, revision, boot_id, "instance-a"), {}
- )
- return conn
-
-
-def fence(conn: AgentConnection) -> dict:
- return {"connection_id": conn.id, "generation": conn.stream_generation}
-
-
-def report_idle(
- service, conn: AgentConnection, *, busy: bool = False, seq: int = 1
-) -> None:
- assert conn.worker is not None
- service.connections.record_idle_report(
- conn,
- IdleReport(
- report_seq=seq,
- relays_through=0,
- busy=busy,
- reasons=[],
- sessions={},
- launch_revision=conn.worker.launch_revision,
- generation=conn.stream_generation,
- received_monotonic=time.monotonic(),
- received_at=datetime.now(UTC),
- ),
- )
+async def report_status(app: ControllerApp, controller_id: str, **reading) -> None:
+ """A status report from the machine's controller."""
+ async with app.factory() as session:
+ await record_controller_status(session, controller_id, reading)
+ await session.commit()
async def list_machines(client) -> list[dict]:
@@ -369,6 +206,12 @@ async def list_machines(client) -> list[dict]:
return response.json()["machines"]
+async def listed(client, machine_id: str) -> dict:
+ return {item["machine_id"]: item for item in await list_machines(client)}[
+ machine_id
+ ]
+
+
async def observe(client, machine_id: str, **body) -> dict:
response = await client.post(
f"/hosted-controller/machines/{machine_id}/observation",
@@ -379,67 +222,68 @@ async def observe(client, machine_id: str, **body) -> dict:
return response.json()
+async def prepare(client, machine_id: str) -> httpx.Response:
+ return await client.post(
+ f"/hosted-controller/machines/{machine_id}/prepare", headers=HEADERS, json={}
+ )
+
+
async def test_controller_requires_its_own_credential(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
for method, path, body in (
("GET", "/hosted-controller/machines", None),
- ("POST", f"/hosted-controller/machines/{machine.id}/prepare", {}),
+ ("POST", f"/hosted-controller/machines/{app.machine_id}/prepare", {}),
(
"POST",
- f"/hosted-controller/machines/{machine.id}/observation",
+ f"/hosted-controller/machines/{app.machine_id}/observation",
{"state": "running", "revision": 1},
),
):
for token in (None, "Bearer wrong"):
- response = await client.request(
+ response = await app.client.request(
method,
path,
headers={} if token is None else {"Authorization": token},
json=body,
)
assert response.status_code == 401
- assert (await machine_of(factory, request_id)).state == "queued"
-
-
-async def test_old_per_launch_controller_routes_are_gone(controller_app):
- client, request_id, *_ = controller_app
- assert (await client.get("/hosted-controller", headers=HEADERS)).status_code in {
- 404,
- 405,
+ saved = await machine_of(app.factory, app.machine_id)
+ assert (saved.state, saved.enrollment_code_revision) == ("queued", None)
+ assert app.enrollment.minted == []
+
+
+async def test_routes_are_unavailable_without_controller_settings(controller_app):
+ app = controller_app
+ app.client._transport.app.state.hosted_controller_settings = None
+ response = await app.client.get("/hosted-controller/machines", headers=HEADERS)
+ assert response.status_code == 503
+ assert response.json() == {
+ "detail": "Switch cloud machines are not enabled on this server."
}
- for suffix in ("prepare", "observation"):
- response = await client.post(
- f"/hosted-controller/{request_id}/{suffix}", headers=HEADERS, json={}
- )
- assert response.status_code in {404, 405}
async def test_machines_lists_every_live_machine_with_the_contract_keys(
controller_app,
):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- async with factory() as session:
+ app = controller_app
+ async with app.factory() as session:
gone = await seed_machine(
session,
- owner_id=machine.owner_id,
- slot_id="slot-b",
+ owner_id=app.owner_id,
state="deleted",
desired_state="deleted",
stop_reason=None,
revision=3,
- generation=1,
)
await session.commit()
- listed = await list_machines(client)
- assert [item["machine_id"] for item in listed] == [machine.id]
- assert gone.id not in {item["machine_id"] for item in listed}
- assert set(listed[0]) == MACHINE_ITEM_KEYS
- assert listed[0] == {
- "machine_id": machine.id,
- "slot_id": "slot-a",
- "generation": 1,
+ items = await list_machines(app.client)
+ assert [item["machine_id"] for item in items] == [app.machine_id]
+ assert gone.id not in {item["machine_id"] for item in items}
+ assert set(items[0]) == MACHINE_ITEM_KEYS
+ [contract] = fixture("machines_response.json")["machines"]
+ assert set(contract) == MACHINE_ITEM_KEYS
+ assert items[0] == {
+ "machine_id": app.machine_id,
"state": "queued",
"desired_state": "running",
"revision": 1,
@@ -450,68 +294,81 @@ async def test_machines_lists_every_live_machine_with_the_contract_keys(
async def test_queued_machine_times_out_with_an_actionable_error(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
await update_machine(
- factory, machine.id, updated_at=datetime.now(UTC) - timedelta(minutes=11)
+ app.factory,
+ app.machine_id,
+ updated_at=datetime.now(UTC) - timedelta(minutes=11),
)
- [item] = await list_machines(client)
+ [item] = await list_machines(app.client)
assert item["state"] == "error"
- saved = await machine_of(factory, request_id)
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.error_code == "machine_connect_timeout"
assert "10 minutes" in saved.error
assert "Retry" in saved.error
async def test_recent_queued_machine_is_left_alone(controller_app):
- client, request_id, *_ = controller_app
- [item] = await list_machines(client)
+ [item] = await list_machines(controller_app.client)
assert item["state"] == "queued"
async def test_running_machine_that_never_connects_times_out(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="provisioning",
running_observed_at=datetime.now(UTC) - timedelta(minutes=11),
)
- [item] = await list_machines(client)
+ [item] = await list_machines(app.client)
assert item["state"] == "error"
- saved = await machine_of(factory, request_id)
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.error_code == "machine_connect_timeout"
- assert "10 minutes" in saved.error
+ assert "did not connect to Switch within 10 minutes" in saved.error
+
+
+async def test_a_status_report_since_the_running_observation_stops_the_timeout(
+ controller_app,
+):
+ app = controller_app
+ observed = datetime.now(UTC) - timedelta(minutes=11)
+ await update_machine(
+ app.factory,
+ app.machine_id,
+ state="provisioning",
+ running_observed_at=observed,
+ heartbeat_at=observed + timedelta(minutes=1),
+ )
+ [item] = await list_machines(app.client)
+ assert item["state"] == "provisioning"
async def test_connect_timeout_counts_from_the_running_observation(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="provisioning",
updated_at=datetime.now(UTC) - timedelta(minutes=30),
running_observed_at=datetime.now(UTC) - timedelta(minutes=5),
)
- [item] = await list_machines(client)
+ [item] = await list_machines(app.client)
assert item["state"] == "provisioning"
async def test_provisioning_machine_never_observed_running_times_out(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
before = datetime.now(UTC)
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="provisioning",
updated_at=before - timedelta(minutes=11),
)
- [item] = await list_machines(client)
+ [item] = await list_machines(app.client)
assert item["state"] == "error"
- saved = await machine_of(factory, request_id)
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.error_code == "machine_connect_timeout"
assert (
saved.error
@@ -521,278 +378,182 @@ async def test_provisioning_machine_never_observed_running_times_out(controller_
async def test_recent_provisioning_machine_is_left_alone(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="provisioning",
updated_at=datetime.now(UTC) - timedelta(minutes=5),
)
- [item] = await list_machines(client)
+ [item] = await list_machines(app.client)
assert item["state"] == "provisioning"
-async def _stopping_launch(controller_app, *, machine_state: str, minutes: int) -> None:
- _, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_machine(factory, machine.id, state=machine_state)
- await update_launch(
- factory,
- request_id,
- state="stopping",
- desired_state="stopped",
- updated_at=datetime.now(UTC) - timedelta(minutes=minutes),
- )
-
-
-async def test_launch_that_never_stops_on_a_ready_machine_times_out(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- before = datetime.now(UTC)
- await _stopping_launch(controller_app, machine_state="ready", minutes=11)
- await list_machines(client)
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch.state == "error"
- assert launch.error_code == "agent_stop_timeout"
- assert (
- launch.error
- == "The agent did not stop within 10 minutes. Retry it in Switch Console."
- )
- assert launch.updated_at >= before
- retried = await client.post(
- f"/hosted-launches/{request_id}/lifecycle",
- json={"action": "retry", "revision": launch.revision},
- )
- assert retried.status_code == 200, retried.text
-
-
- ("machine_state", "minutes"),
- [("stopping", 11), ("provisioning", 11), ("ready", 5)],
- ids=["machine-stopping", "machine-provisioning", "recent"],
-)
-async def test_stopping_launch_is_left_alone_unless_it_should_have_stopped(
- controller_app, machine_state, minutes
-):
- client, request_id, _, _, factory, _ = controller_app
- await _stopping_launch(controller_app, machine_state=machine_state, minutes=minutes)
- await list_machines(client)
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch.state == "stopping"
- assert launch.error_code is None
-
-
async def test_retention_sweep_deletes_an_expired_retained_machine(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
expired = datetime.now(UTC) - timedelta(seconds=1)
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="retained",
desired_state="retained",
retain_until=expired,
revision=3,
)
- [item] = await list_machines(client)
+ [item] = await list_machines(app.client)
assert item["desired_state"] == "deleted"
assert item["revision"] == 4
assert item["retain_until"] == expired.isoformat()
async def test_retention_sweep_keeps_a_machine_inside_its_window(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
until = datetime.now(UTC) + timedelta(days=3)
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="retained",
desired_state="retained",
retain_until=until,
revision=3,
)
- [item] = await list_machines(client)
+ [item] = await list_machines(app.client)
assert item["desired_state"] == "retained"
assert item["revision"] == 3
assert item["retain_until"] == until.isoformat()
-async def test_machines_sweep_expires_unconfirmed_and_superseded_operations(
- controller_app,
-):
- client, request_id, _, _, factory, _ = controller_app
- await update_launch(factory, request_id, revision=2)
- now = datetime.now(UTC)
- operations = {
- "expired_claim": (2, "claimed", now - timedelta(minutes=6)),
- "fresh_claim": (2, "claimed", now),
- "current_queued": (2, "queued", now),
- "old_queued": (1, "queued", now),
- "old_claim": (1, "claimed", now),
- }
- ids = {name: str(uuid4()) for name in operations}
- async with factory() as session:
- for name, (revision, state, updated_at) in operations.items():
- session.add(
- HostedOperation(
- id=ids[name],
- launch_id=request_id,
- launch_revision=revision,
- session_id=str(uuid4()),
- action="start",
- state=state,
- updated_at=updated_at,
- )
- )
- await session.commit()
- await list_machines(client)
- async with factory() as session:
- states = {
- name: (
- await session.get(HostedOperation, (require_tenant_id(), ids[name]))
- ).state
- for name in operations
- }
- assert states == {
- "expired_claim": "unknown",
- "fresh_claim": "claimed",
- "current_queued": "queued",
- "old_queued": "failed",
- "old_claim": "unknown",
- }
-
-
-async def test_prepare_is_idempotent_per_revision_and_rotates_on_a_new_one(
- controller_app,
-):
- client, request_id, _, _, factory, settings = controller_app
- machine = await machine_of(factory, request_id)
- path = f"/hosted-controller/machines/{machine.id}/prepare"
- first = await client.post(path, headers=HEADERS, json={})
- again = await client.post(path, headers=HEADERS, json={})
+async def test_prepare_hands_over_one_enrollment_code_per_revision(controller_app):
+ app = controller_app
+ first = await prepare(app.client, app.machine_id)
+ again = await prepare(app.client, app.machine_id)
assert first.status_code == again.status_code == 200, first.text
assert first.headers["cache-control"] == "no-store"
assert first.json() == again.json()
body = first.json()
- assert set(body) == {
- "machine_id",
- "slot_id",
- "generation",
- "revision",
- "bundle_revision",
- "machine_capability",
- "api_endpoint",
- }
- assert body["machine_id"] == machine.id
- assert body["slot_id"] == "slot-a"
- assert body["generation"] == 1
+ contract = fixture("prepare_controller_response.json")
+ assert set(body) == set(contract)
+ assert set(body["controller"]) == set(contract["controller"])
+ assert body["machine_id"] == app.machine_id
assert body["revision"] == body["bundle_revision"] == 1
- assert body["api_endpoint"] == settings.agent_api_endpoint
- saved = await machine_of(factory, request_id)
+ assert body["api_endpoint"] == app.settings.agent_api_endpoint
+ assert body["controller"] == {"id": None, "enrollment_code": "swce_synthetic-1"}
+ assert app.enrollment.minted == [(app.machine_id, 1)]
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.state == "provisioning"
- assert HostedMachineStore.capability_matches(saved, body["machine_capability"])
- [item] = await list_machines(client)
+ assert saved.enrollment_code_revision == 1
+ assert "swce_synthetic-1" not in (saved.enrollment_code_encrypted or "")
+ [item] = await list_machines(app.client)
assert item["bundle_revision"] == 1
- assert body["machine_capability"] not in json.dumps(item)
+ assert "swce_" not in json.dumps(item)
- await update_machine(factory, machine.id, revision=2)
- rotated = (await client.post(path, headers=HEADERS, json={})).json()
+ await update_machine(app.factory, app.machine_id, revision=2)
+ rotated = (await prepare(app.client, app.machine_id)).json()
assert rotated["revision"] == rotated["bundle_revision"] == 2
- assert rotated["machine_capability"] != body["machine_capability"]
- saved = await machine_of(factory, request_id)
- assert not HostedMachineStore.capability_matches(saved, body["machine_capability"])
- assert HostedMachineStore.capability_matches(saved, rotated["machine_capability"])
+ assert rotated["controller"] == {"id": None, "enrollment_code": "swce_synthetic-2"}
+ assert app.enrollment.minted == [(app.machine_id, 1), (app.machine_id, 2)]
+ assert (await machine_of(app.factory, app.machine_id)).enrollment_code_revision == 2
-async def test_prepare_does_not_touch_launches_or_agents(controller_app):
- client, request_id, agent_id, service, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- started = list(service.client_lifecycle.started)
- response = await client.post(
- f"/hosted-controller/machines/{machine.id}/prepare", headers=HEADERS, json={}
- )
+async def test_prepare_names_the_controller_the_machine_enrolled_as(controller_app):
+ app = controller_app
+ controller_id, _ = await enrolled(app, agents=0)
+ response = await prepare(app.client, app.machine_id)
assert response.status_code == 200, response.text
- assert service.client_lifecycle.started == started
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch.state == "queued"
- assert launch.worker_capability_hash is None
- assert await session.scalar(select(GitHubIssuedToken)) is None
+ assert response.json()["controller"] == {
+ "id": controller_id,
+ "enrollment_code": None,
+ }
+ assert app.enrollment.minted == []
+
+
+async def test_prepare_mints_a_code_again_once_the_controller_is_revoked(
+ controller_app,
+):
+ app = controller_app
+ controller_id, _ = await enrolled(app, agents=0)
+ async with app.factory() as session:
+ controller = await session.get(AgentController, controller_id)
+ controller.revoked_at = datetime.now(UTC)
+ await session.commit()
+ response = await prepare(app.client, app.machine_id)
+ assert response.status_code == 200, response.text
+ assert response.json()["controller"] == {
+ "id": None,
+ "enrollment_code": "swce_synthetic-1",
+ }
+
+
+async def test_prepare_leaves_a_machine_past_queued_in_its_state(controller_app):
+ app = controller_app
+ await update_machine(app.factory, app.machine_id, state="stopped")
+ assert (await prepare(app.client, app.machine_id)).status_code == 200
+ assert (await machine_of(app.factory, app.machine_id)).state == "stopped"
async def test_prepare_refuses_unknown_and_deleted_machines(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- missing = await client.post(
- f"/hosted-controller/machines/{uuid4()}/prepare", headers=HEADERS, json={}
- )
+ app = controller_app
+ missing = await prepare(app.client, str(uuid4()))
assert missing.status_code == 404
- machine = await machine_of(factory, request_id)
- await update_machine(factory, machine.id, desired_state="deleted")
- deleted = await client.post(
- f"/hosted-controller/machines/{machine.id}/prepare", headers=HEADERS, json={}
- )
- assert deleted.status_code == 409
- assert deleted.json() == {"detail": "machine is deleted"}
- assert (await machine_of(factory, request_id)).machine_capability_hash is None
+ for values in (
+ {"desired_state": "deleted"},
+ {"state": "deleted", "desired_state": "deleted"},
+ ):
+ await update_machine(app.factory, app.machine_id, **values)
+ deleted = await prepare(app.client, app.machine_id)
+ assert deleted.status_code == 409
+ assert deleted.json() == {"detail": "machine is deleted"}
+ assert (
+ await machine_of(app.factory, app.machine_id)
+ ).enrollment_code_revision is None
+ assert app.enrollment.minted == []
async def test_prepare_for_a_departed_owner_errors_the_machine(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- async with factory() as session:
+ app = controller_app
+ async with app.factory() as session:
await session.delete(
- await session.get(TenantMember, (require_tenant_id(), machine.owner_id))
+ await session.get(TenantMember, (require_tenant_id(), app.owner_id))
)
await session.commit()
- response = await client.post(
- f"/hosted-controller/machines/{machine.id}/prepare", headers=HEADERS, json={}
- )
+ response = await prepare(app.client, app.machine_id)
assert response.status_code == 409
- assert "machine_capability" not in response.text
- saved = await machine_of(factory, request_id)
+ assert "enrollment_code" not in response.text
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.state == "error"
assert "workspace member" in saved.error
- assert saved.machine_capability_hash is None
+ assert saved.enrollment_code_revision is None
+ assert app.enrollment.minted == []
async def test_controller_observation_fixture_is_accepted(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
body = fixture("controller_observation.json")
await update_machine(
- factory, machine.id, state="provisioning", revision=body["revision"]
- )
- response = await client.post(
- f"/hosted-controller/machines/{machine.id}/observation",
- headers=HEADERS,
- json=body,
+ app.factory, app.machine_id, state="provisioning", revision=body["revision"]
)
- assert response.status_code == 200, response.text
- item = response.json()
+ item = await observe(app.client, app.machine_id, **body)
assert set(item) == MACHINE_ITEM_KEYS
assert item["state"] == "provisioning"
assert item["data_volume_id"] == body["data_volume_id"]
- saved = await machine_of(factory, request_id)
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.instance_id == body["instance_id"]
assert saved.instance_type == body["instance_type"]
assert saved.running_observed_at is not None
async def test_observation_rejects_unknown_fields_and_codes(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- path = f"/hosted-controller/machines/{machine.id}/observation"
+ app = controller_app
+ path = f"/hosted-controller/machines/{app.machine_id}/observation"
for body in (
{"state": "running", "revision": 1, "surprise": True},
{"state": "error", "revision": 1, "error_code": "worker_needs_attention"},
{"state": "sleeping", "revision": 1},
{"state": "running", "revision": 0},
):
- assert (await client.post(path, headers=HEADERS, json=body)).status_code == 422
- missing = await client.post(
+ response = await app.client.post(path, headers=HEADERS, json=body)
+ assert response.status_code == 422
+ missing = await app.client.post(
f"/hosted-controller/machines/{uuid4()}/observation",
headers=HEADERS,
json={"state": "running", "revision": 1},
@@ -803,114 +564,157 @@ async def test_observation_rejects_unknown_fields_and_codes(controller_app):
async def test_stale_observation_is_ignored_unless_it_carries_an_error(
controller_app,
):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
+ app = controller_app
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="stopping",
desired_state="stopped",
stop_reason="owner",
revision=2,
)
ignored = await observe(
- client, machine.id, state="running", revision=1, data_volume_id="vol-stale"
+ app.client,
+ app.machine_id,
+ state="running",
+ revision=1,
+ data_volume_id="vol-stale",
)
assert ignored["state"] == "stopping"
assert ignored["desired_state"] == "stopped"
assert ignored["data_volume_id"] is None
- ahead = await observe(client, machine.id, state="stopped", revision=3)
+ ahead = await observe(app.client, app.machine_id, state="stopped", revision=3)
assert ahead["state"] == "stopping"
recorded = await observe(
- client,
- machine.id,
+ app.client,
+ app.machine_id,
state="error",
revision=1,
error="The instance could not be stopped.",
error_code="machine_needs_attention",
)
assert recorded["state"] == "error"
- saved = await machine_of(factory, request_id)
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.error == "The instance could not be stopped."
assert saved.error_code == "machine_needs_attention"
+async def test_an_observation_ahead_of_core_is_ignored_even_with_an_error(
+ controller_app,
+):
+ app = controller_app
+ await update_machine(app.factory, app.machine_id, state="ready", revision=2)
+ item = await observe(
+ app.client,
+ app.machine_id,
+ state="error",
+ revision=3,
+ error="The instance failed its status checks.",
+ )
+ assert item["state"] == "ready"
+ assert (await machine_of(app.factory, app.machine_id)).error is None
+
+
@pytest.mark.parametrize("state", ["queued", "provisioning"])
async def test_stale_error_does_not_overwrite_a_newer_retry(controller_app, state):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_machine(factory, machine.id, state=state, revision=2)
+ app = controller_app
+ await update_machine(app.factory, app.machine_id, state=state, revision=2)
item = await observe(
- client,
- machine.id,
+ app.client,
+ app.machine_id,
state="error",
revision=1,
error="The instance failed its status checks.",
error_code="machine_needs_attention",
)
assert item["state"] == state
- saved = await machine_of(factory, request_id)
- assert saved.state == state
- assert saved.error is None
- assert saved.error_code is None
+ saved = await machine_of(app.factory, app.machine_id)
+ assert (saved.state, saved.error, saved.error_code) == (state, None, None)
async def test_stale_error_is_recorded_on_a_ready_machine(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_machine(factory, machine.id, state="ready", revision=2)
+ app = controller_app
+ await update_machine(app.factory, app.machine_id, state="ready", revision=2)
item = await observe(
- client,
- machine.id,
+ app.client,
+ app.machine_id,
state="error",
revision=1,
error="The instance failed its status checks.",
)
assert item["state"] == "error"
- assert (
- await machine_of(factory, request_id)
- ).error == "The instance failed its status checks."
+ saved = await machine_of(app.factory, app.machine_id)
+ assert saved.error == "The instance failed its status checks."
-async def test_running_observation_waits_for_a_heartbeat(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_machine(factory, machine.id, state="provisioning")
- first = await observe(client, machine.id, state="running", revision=1)
+async def test_running_observation_waits_for_a_status_report(controller_app):
+ app = controller_app
+ controller_id, _ = await enrolled(app, agents=0)
+ await update_machine(app.factory, app.machine_id, state="provisioning")
+ first = await observe(app.client, app.machine_id, state="running", revision=1)
assert first["state"] == "provisioning"
- observed = (await machine_of(factory, request_id)).running_observed_at
+ observed = (await machine_of(app.factory, app.machine_id)).running_observed_at
assert observed is not None
- again = await observe(client, machine.id, state="running", revision=1)
+ again = await observe(app.client, app.machine_id, state="running", revision=1)
assert again["state"] == "provisioning"
- assert (await machine_of(factory, request_id)).running_observed_at == observed
- await update_machine(factory, machine.id, state="ready")
- assert (await observe(client, machine.id, state="running", revision=1))[
- "state"
- ] == "ready"
- assert (await observe(client, machine.id, state="provisioning", revision=1))[
- "state"
- ] == "ready"
+ assert (
+ await machine_of(app.factory, app.machine_id)
+ ).running_observed_at == observed
+ await report_status(app, controller_id, sessions_running=0)
+ assert (await machine_of(app.factory, app.machine_id)).state == "ready"
+ for state in ("running", "provisioning"):
+ item = await observe(app.client, app.machine_id, state=state, revision=1)
+ assert item["state"] == "ready"
+
+
+async def test_a_status_report_does_not_ready_a_machine_being_stopped(
+ controller_app,
+):
+ app = controller_app
+ controller_id, _ = await enrolled(app, agents=0)
+ await update_machine(
+ app.factory,
+ app.machine_id,
+ state="provisioning",
+ desired_state="stopped",
+ stop_reason="owner",
+ running_observed_at=datetime.now(UTC),
+ )
+ await report_status(app, controller_id, sessions_running=0)
+ saved = await machine_of(app.factory, app.machine_id)
+ assert saved.state == "provisioning"
+ assert saved.heartbeat_at is not None
@pytest.mark.parametrize(
"state", ["stopping", "stopped", "deleting", "deleted", "retained"]
)
async def test_lifecycle_observations_are_copied(controller_app, state):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_machine(factory, machine.id, state="ready", error_code="disk_full")
- item = await observe(client, machine.id, state=state, revision=1)
+ app = controller_app
+ await update_machine(
+ app.factory, app.machine_id, state="ready", error_code="disk_full"
+ )
+ item = await observe(app.client, app.machine_id, state=state, revision=1)
assert item["state"] == state
- assert (await machine_of(factory, request_id)).error_code is None
+ assert (await machine_of(app.factory, app.machine_id)).error_code is None
+
+
+async def test_a_deleted_machine_leaves_the_listing_and_ignores_observations(
+ controller_app,
+):
+ app = controller_app
+ await observe(app.client, app.machine_id, state="deleted", revision=1)
+ assert await list_machines(app.client) == []
+ item = await observe(app.client, app.machine_id, state="running", revision=1)
+ assert item["state"] == "deleted"
async def test_error_observation_is_kept_until_retry(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_machine(factory, machine.id, state="provisioning")
+ app = controller_app
+ await update_machine(app.factory, app.machine_id, state="provisioning")
errored = await observe(
- client,
- machine.id,
+ app.client,
+ app.machine_id,
state="error",
revision=1,
error="The instance failed its status checks.",
@@ -918,106 +722,75 @@ async def test_error_observation_is_kept_until_retry(controller_app):
)
assert errored["state"] == "error"
for state in ("running", "provisioning", "stopped"):
- assert (await observe(client, machine.id, state=state, revision=1))[
- "state"
- ] == "error"
- saved = await machine_of(factory, request_id)
+ item = await observe(app.client, app.machine_id, state=state, revision=1)
+ assert item["state"] == "error"
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.error == "The instance failed its status checks."
assert saved.error_code == "machine_needs_attention"
- defaulted = await observe(client, machine.id, state="error", revision=1)
+ defaulted = await observe(app.client, app.machine_id, state="error", revision=1)
assert defaulted["state"] == "error"
- saved = await machine_of(factory, request_id)
+ saved = await machine_of(app.factory, app.machine_id)
assert "Retry" in saved.error
assert saved.error_code is None
[email protected]("state", ["retained", "deleting", "deleted"])
-async def test_retiring_an_errored_machine_replaces_its_error(controller_app, state):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_launch(factory, request_id, state="deleted", desired_state="deleted")
+async def _errored_retained(app: ControllerApp, retain_until: datetime) -> None:
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="error",
error="The instance failed its status checks.",
error_code="machine_needs_attention",
desired_state="retained",
- retain_until=datetime.now(UTC) + timedelta(days=7),
+ retain_until=retain_until,
revision=2,
)
- item = await observe(client, machine.id, state=state, revision=2)
+
+
[email protected]("state", ["retained", "deleting", "deleted"])
+async def test_retiring_an_errored_machine_replaces_its_error(controller_app, state):
+ app = controller_app
+ await _errored_retained(app, datetime.now(UTC) + timedelta(days=7))
+ item = await observe(app.client, app.machine_id, state=state, revision=2)
assert item["state"] == state
- saved = await machine_of(factory, request_id)
- assert saved.error is None
- assert saved.error_code is None
+ saved = await machine_of(app.factory, app.machine_id)
+ assert (saved.error, saved.error_code) == (None, None)
async def test_errored_machine_retained_after_its_last_agent_expires(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_launch(factory, request_id, state="deleted", desired_state="deleted")
- await update_machine(
- factory,
- machine.id,
- state="error",
- error="The instance failed its status checks.",
- error_code="machine_needs_attention",
- desired_state="retained",
- retain_until=datetime.now(UTC) - timedelta(seconds=1),
- revision=2,
+ app = controller_app
+ await _errored_retained(app, datetime.now(UTC) - timedelta(seconds=1))
+ await observe(app.client, app.machine_id, state="retained", revision=2)
+ [item] = await list_machines(app.client)
+ assert (item["state"], item["desired_state"], item["revision"]) == (
+ "retained",
+ "deleted",
+ 3,
)
- await observe(client, machine.id, state="retained", revision=2)
- [item] = await list_machines(client)
- assert item["state"] == "retained"
- assert item["desired_state"] == "deleted"
- assert item["revision"] == 3
async def test_retention_sweep_deletes_an_expired_errored_machine(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_launch(factory, request_id, state="deleted", desired_state="deleted")
- await update_machine(
- factory,
- machine.id,
- state="error",
- error="The instance failed its status checks.",
- error_code="machine_needs_attention",
- desired_state="retained",
- retain_until=datetime.now(UTC) - timedelta(seconds=1),
- revision=2,
- )
- [item] = await list_machines(client)
+ app = controller_app
+ await _errored_retained(app, datetime.now(UTC) - timedelta(seconds=1))
+ [item] = await list_machines(app.client)
assert (item["state"], item["desired_state"], item["revision"]) == (
"error",
"deleted",
3,
)
- item = await observe(client, machine.id, state="deleting", revision=3)
+ item = await observe(app.client, app.machine_id, state="deleting", revision=3)
assert item["state"] == "deleting"
- saved = await machine_of(factory, request_id)
+ saved = await machine_of(app.factory, app.machine_id)
assert (saved.error, saved.error_code) == (None, None)
async def test_retention_sweep_keeps_an_errored_machine_inside_its_window(
controller_app,
):
- client, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- await update_launch(factory, request_id, state="deleted", desired_state="deleted")
+ app = controller_app
until = datetime.now(UTC) + timedelta(days=3)
- await update_machine(
- factory,
- machine.id,
- state="error",
- error="The instance failed its status checks.",
- error_code="machine_needs_attention",
- desired_state="retained",
- retain_until=until,
- revision=2,
- )
- [item] = await list_machines(client)
+ await _errored_retained(app, until)
+ [item] = await list_machines(app.client)
assert (item["state"], item["desired_state"], item["revision"]) == (
"error",
"retained",
@@ -1026,180 +799,136 @@ async def test_retention_sweep_keeps_an_errored_machine_inside_its_window(
assert item["retain_until"] == until.isoformat()
-async def test_deleted_observation_with_live_launches_logs_an_invariant_failure(
- controller_app, caplog
-):
- client, request_id, agent_id, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- with caplog.at_level(logging.ERROR, logger="switch_core.gateway.hosted_controller"):
- item = await observe(client, machine.id, state="deleted", revision=1)
- assert item["state"] == "deleted"
- assert request_id in caplog.text
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch.state == "queued"
- assert await session.get(Agent, agent_id) is not None
- assert await list_machines(client) == []
+async def _idle_ready(
+ app: ControllerApp,
+ *,
+ minutes: int,
+ agents: int = 1,
+ sessions: int | None = 0,
+ reported: timedelta = timedelta(minutes=1),
+) -> str:
+ """A ready machine whose controller, with `agents` placed on it, last
+ reported `reported` ago that `sessions` run, and which went quiet 31
+ minutes ago. The controller id."""
+ app.config.hosted_idle_stop_minutes = minutes
+ controller_id, _ = await enrolled(app, agents=agents)
+ heartbeat: dict = {"disk": None, "memory": None}
+ if sessions is not None:
+ heartbeat["sessions_running"] = sessions
+ now = datetime.now(UTC)
+ await update_machine(
+ app.factory,
+ app.machine_id,
+ state="ready",
+ active_at=now - timedelta(minutes=31),
+ heartbeat=heartbeat,
+ heartbeat_at=now - reported,
+ running_observed_at=now - timedelta(hours=1),
+ )
+ return controller_id
-async def _idle_ready(
- controller_app, *, minutes: int, report: bool = True, busy: bool = False
-) -> AgentConnection:
- """A ready machine whose one launch and the machine itself went quiet 31 minutes ago."""
- _, request_id, agent_id, service, factory, _ = controller_app
- service.config.hosted_idle_stop_minutes = minutes
- machine = await machine_of(factory, request_id)
- long_ago = datetime.now(UTC) - timedelta(minutes=31)
- await update_machine(factory, machine.id, state="ready", active_at=long_ago)
- await update_launch(factory, request_id, state="ready", active_at=long_ago)
- conn = attach_worker(service, agent_id, request_id)
- if report:
- report_idle(service, conn, busy=busy)
- return conn
-
-
-async def test_idle_machine_sleeps_when_every_agent_and_the_machine_are_idle(
+async def test_idle_machine_sleeps_once_its_controller_reports_it_idle(
controller_app,
):
- client, request_id, *_ = controller_app
- await _idle_ready(controller_app, minutes=30)
- [item] = await list_machines(client)
- assert item["desired_state"] == "stopped"
- assert item["revision"] == 2
- saved = await machine_of(controller_app[4], request_id)
+ app = controller_app
+ await _idle_ready(app, minutes=30)
+ [item] = await list_machines(app.client)
+ assert (item["desired_state"], item["revision"]) == ("stopped", 2)
+ saved = await machine_of(app.factory, app.machine_id)
assert saved.stop_reason == "idle"
assert saved.running_observed_at is None
- async with controller_app[4]() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch.state == "ready"
- assert launch.desired_state == "running"
- assert launch.revision == 1
-
-
-async def test_auto_session_does_not_exempt_an_agent_from_idle_stop(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- await update_launch(factory, request_id, spec={**SPEC, "auto_session": False})
- await _idle_ready(controller_app, minutes=30)
- [item] = await list_machines(client)
- assert item["desired_state"] == "stopped"
+ assert saved.retain_until is None
async def test_recent_machine_activity_keeps_it_awake(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- await _idle_ready(controller_app, minutes=30)
- machine = await machine_of(factory, request_id)
+ app = controller_app
+ await _idle_ready(app, minutes=30)
await update_machine(
- factory, machine.id, active_at=datetime.now(UTC) - timedelta(minutes=5)
+ app.factory,
+ app.machine_id,
+ active_at=datetime.now(UTC) - timedelta(minutes=5),
)
- [item] = await list_machines(client)
- assert item["desired_state"] == "running"
+ [item] = await list_machines(app.client)
+ assert (item["desired_state"], item["revision"]) == ("running", 1)
-async def test_recent_agent_activity_keeps_it_awake(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- await _idle_ready(controller_app, minutes=30)
- await update_launch(
- factory, request_id, active_at=datetime.now(UTC) - timedelta(minutes=5)
- )
- [item] = await list_machines(client)
+async def test_running_sessions_keep_it_awake(controller_app):
+ app = controller_app
+ await _idle_ready(app, minutes=30, sessions=2)
+ [item] = await list_machines(app.client)
assert item["desired_state"] == "running"
-async def test_a_report_older_than_the_last_activity_is_not_evidence(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- await _idle_ready(controller_app, minutes=30)
- await update_launch(
- factory, request_id, active_at=datetime.now(UTC) - timedelta(seconds=1)
- )
- [item] = await list_machines(client)
- assert item["desired_state"] == "running"
-
-
-async def test_missing_idle_report_counts_as_busy(controller_app):
- client, request_id, _, _, factory, _ = controller_app
+async def test_a_report_of_running_sessions_renews_its_activity(controller_app):
+ app = controller_app
+ controller_id = await _idle_ready(app, minutes=30)
before = datetime.now(UTC)
- await _idle_ready(controller_app, minutes=30, report=False)
- [item] = await list_machines(client)
+ await report_status(app, controller_id, sessions_running=1)
+ await report_status(app, controller_id, sessions_running=0)
+ [item] = await list_machines(app.client)
assert item["desired_state"] == "running"
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch.active_at >= before
+ assert (await machine_of(app.factory, app.machine_id)).active_at >= before
-async def test_busy_agent_renews_its_activity_and_keeps_the_machine_awake(
- controller_app,
+ ("sessions", "reported"),
+ [(0, timedelta(minutes=10)), (None, timedelta(minutes=1))],
+ ids=["stopped-reporting", "no-session-count"],
+)
+async def test_a_machine_not_known_to_be_idle_is_kept_awake(
+ controller_app, sessions, reported
):
- client, request_id, _, _, factory, _ = controller_app
- before = datetime.now(UTC)
- await _idle_ready(controller_app, minutes=30, busy=True)
- [item] = await list_machines(client)
+ app = controller_app
+ await _idle_ready(app, minutes=30, sessions=sessions, reported=reported)
+ [item] = await list_machines(app.client)
assert item["desired_state"] == "running"
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch.active_at >= before
-async def _second_launch(controller_app, **values) -> str:
- _, request_id, _, _, factory, _ = controller_app
- machine = await machine_of(factory, request_id)
- second = str(uuid4())
- async with factory() as session:
- await seed_launch(
- session,
- machine=machine,
- request_id=second,
- name="cloud-second",
- state=values.pop("state", "ready"),
- desired_state=values.pop("desired_state", "running"),
- revision=1,
- agent_id=values.pop("agent_id", None),
- spec=SPEC,
- )
- await session.commit()
- await update_launch(factory, second, **values)
- return second
+async def test_idle_stop_is_off_when_the_setting_is_zero(controller_app):
+ app = controller_app
+ await _idle_ready(app, minutes=0)
+ [item] = await list_machines(app.client)
+ assert (item["desired_state"], item["revision"]) == ("running", 1)
-async def test_one_busy_agent_keeps_the_whole_machine_awake(controller_app):
- client, *_ = controller_app
- await _idle_ready(controller_app, minutes=30)
- await _second_launch(
- controller_app, active_at=datetime.now(UTC) - timedelta(minutes=31)
- )
- [item] = await list_machines(client)
+async def test_only_a_ready_running_machine_is_put_to_sleep(controller_app):
+ app = controller_app
+ await _idle_ready(app, minutes=30)
+ await update_machine(app.factory, app.machine_id, state="provisioning")
+ [item] = await list_machines(app.client)
assert item["desired_state"] == "running"
- "values",
- [
- {"state": "error", "error_code": "agent_crashed"},
- {"state": "stopped", "desired_state": "stopped"},
- ],
- ids=["crashed", "stopped"],
-)
-async def test_crashed_and_stopped_agents_do_not_keep_the_machine_awake(
- controller_app, values
-):
- client, *_ = controller_app
- await _idle_ready(controller_app, minutes=30)
- await _second_launch(controller_app, **values)
- [item] = await list_machines(client)
- assert item["desired_state"] == "stopped"
+async def test_idle_machine_whose_agents_were_removed_keeps_its_disk(controller_app):
+ app = controller_app
+ await _idle_ready(app, minutes=30, agents=0)
+ before = datetime.now(UTC)
+ [item] = await list_machines(app.client)
+ assert (item["desired_state"], item["revision"]) == ("retained", 2)
+ retain_until = datetime.fromisoformat(item["retain_until"])
+ assert before + timedelta(days=7) <= retain_until
+ assert retain_until <= datetime.now(UTC) + timedelta(days=7)
-async def test_machine_with_no_counted_agents_sleeps(controller_app):
- client, request_id, agent_id, service, factory, _ = controller_app
- await _idle_ready(controller_app, minutes=30, report=False)
- await update_launch(factory, request_id, state="stopped", desired_state="stopped")
- [item] = await list_machines(client)
- assert item["desired_state"] == "stopped"
+async def test_a_machine_asleep_with_agents_stays_asleep(controller_app):
+ app = controller_app
+ await enrolled(app, agents=1)
+ await update_machine(
+ app.factory,
+ app.machine_id,
+ state="stopped",
+ desired_state="stopped",
+ stop_reason="idle",
+ revision=2,
+ )
+ [item] = await list_machines(app.client)
+ assert (item["desired_state"], item["revision"]) == ("stopped", 2)
-async def _empty_machine(controller_app, **values) -> tuple[str, str]:
- """Another owner's machine that never hosted an agent, quiet for 31 minutes."""
- factory = controller_app[4]
- async with factory() as session:
+async def _empty_machine(app: ControllerApp, **values) -> tuple[str, str]:
+ """Another owner's machine whose controller never enrolled, quiet for 31 minutes."""
+ async with app.factory() as session:
owner = User(
name="empty", email="[email protected]", role="user", password_hash="x"
)
@@ -1208,26 +937,20 @@ async def _empty_machine(controller_app, **values) -> tuple[str, str]:
machine = await seed_machine(
session,
owner_id=owner.id,
- slot_id="slot-b",
state="ready",
desired_state="running",
stop_reason=None,
revision=1,
- generation=1,
)
machine.active_at = datetime.now(UTC) - timedelta(minutes=31)
+ machine.heartbeat = {"disk": None, "memory": None, "sessions_running": 0}
+ machine.heartbeat_at = datetime.now(UTC)
for key, value in values.items():
setattr(machine, key, value)
await session.commit()
return machine.id, owner.id
-async def _listed(client, machine_id: str) -> dict:
- return {item["machine_id"]: item for item in await list_machines(client)}[
- machine_id
- ]
-
-
@pytest.mark.parametrize(
("minutes", "values"),
[
@@ -1239,46 +962,46 @@ async def _listed(client, machine_id: str) -> dict:
async def test_machine_that_never_hosted_an_agent_is_released(
controller_app, minutes, values
):
- client, _, _, service, _, _ = controller_app
- service.config.hosted_idle_stop_minutes = minutes
- machine_id, _ = await _empty_machine(controller_app, **values)
+ app = controller_app
+ app.config.hosted_idle_stop_minutes = minutes
+ machine_id, _ = await _empty_machine(app, **values)
before = datetime.now(UTC)
- item = await _listed(client, machine_id)
+ item = await listed(app.client, machine_id)
assert (item["desired_state"], item["revision"]) == ("retained", 2)
assert before <= datetime.fromisoformat(item["retain_until"]) <= datetime.now(UTC)
- await observe(client, machine_id, state="retained", revision=2)
- assert (await _listed(client, machine_id))["desired_state"] == "deleted"
+ await observe(app.client, machine_id, state="retained", revision=2)
+ assert (await listed(app.client, machine_id))["desired_state"] == "deleted"
async def test_owner_stopped_empty_machine_is_left_alone(controller_app):
- client, *_ = controller_app
+ app = controller_app
machine_id, _ = await _empty_machine(
- controller_app, state="stopped", desired_state="stopped", stop_reason="owner"
+ app, state="stopped", desired_state="stopped", stop_reason="owner"
)
- item = await _listed(client, machine_id)
+ item = await listed(app.client, machine_id)
assert (item["desired_state"], item["revision"]) == ("stopped", 1)
async def test_errored_machine_that_never_hosted_an_agent_is_released(controller_app):
- client, _, _, service, _, _ = controller_app
- service.config.hosted_idle_stop_minutes = 30
+ app = controller_app
+ app.config.hosted_idle_stop_minutes = 30
machine_id, _ = await _empty_machine(
- controller_app,
+ app,
state="error",
error="The instance failed its status checks.",
error_code="machine_needs_attention",
updated_at=datetime.now(UTC) - timedelta(minutes=31),
)
before = datetime.now(UTC)
- item = await _listed(client, machine_id)
+ item = await listed(app.client, machine_id)
assert (item["state"], item["desired_state"], item["revision"]) == (
"error",
"retained",
2,
)
assert before <= datetime.fromisoformat(item["retain_until"]) <= datetime.now(UTC)
- await observe(client, machine_id, state="retained", revision=2)
- item = await _listed(client, machine_id)
+ await observe(app.client, machine_id, state="retained", revision=2)
+ item = await listed(app.client, machine_id)
assert (item["state"], item["desired_state"], item["revision"]) == (
"retained",
"deleted",
@@ -1286,21 +1009,18 @@ async def test_errored_machine_that_never_hosted_an_agent_is_released(controller
)
-async def test_errored_machine_that_hosted_an_agent_is_left_for_its_owner(
- controller_app,
-):
- client, request_id, _, service, factory, _ = controller_app
- service.config.hosted_idle_stop_minutes = 30
- machine = await machine_of(factory, request_id)
- await update_launch(factory, request_id, state="deleted", desired_state="deleted")
+async def test_errored_machine_that_enrolled_is_left_for_its_owner(controller_app):
+ app = controller_app
+ app.config.hosted_idle_stop_minutes = 30
+ await enrolled(app, agents=0)
await update_machine(
- factory,
- machine.id,
+ app.factory,
+ app.machine_id,
state="error",
error="The instance failed its status checks.",
updated_at=datetime.now(UTC) - timedelta(minutes=31),
)
- [item] = await list_machines(client)
+ [item] = await list_machines(app.client)
assert (item["state"], item["desired_state"], item["revision"]) == (
"error",
"running",
@@ -1309,15 +1029,15 @@ async def test_errored_machine_that_hosted_an_agent_is_left_for_its_owner(
async def test_recently_errored_empty_machine_is_left_alone(controller_app):
- client, _, _, service, _, _ = controller_app
- service.config.hosted_idle_stop_minutes = 30
+ app = controller_app
+ app.config.hosted_idle_stop_minutes = 30
machine_id, _ = await _empty_machine(
- controller_app,
+ app,
state="error",
error="The instance failed its status checks.",
updated_at=datetime.now(UTC) - timedelta(minutes=10),
)
- item = await _listed(client, machine_id)
+ item = await listed(app.client, machine_id)
assert (item["state"], item["desired_state"], item["revision"]) == (
"error",
"running",
@@ -1325,25 +1045,12 @@ async def test_recently_errored_empty_machine_is_left_alone(controller_app):
)
-async def test_idle_machine_whose_agents_were_removed_keeps_its_disk(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- await _idle_ready(controller_app, minutes=30, report=False)
- await update_launch(factory, request_id, state="deleted", desired_state="deleted")
- before = datetime.now(UTC)
- [item] = await list_machines(client)
- assert item["desired_state"] == "retained"
- retain_until = datetime.fromisoformat(item["retain_until"])
- assert before + timedelta(days=7) <= retain_until
- assert retain_until <= datetime.now(UTC) + timedelta(days=7)
-
-
async def _claim(factory, owner_id: str) -> HostedMachine:
async with factory() as session:
- await lock_launches(session)
+ await lock_claims(session)
machine = await HostedMachineStore().claim(
session,
owner_id=owner_id,
- slots=["slot-a", "slot-b"],
capacity=2,
now=datetime.now(UTC),
)
@@ -1352,679 +1059,19 @@ async def _claim(factory, owner_id: str) -> HostedMachine:
async def test_released_machine_is_revived_or_replaced_by_a_claim(controller_app):
- client, _, _, service, factory, _ = controller_app
- service.config.hosted_idle_stop_minutes = 30
- machine_id, owner_id = await _empty_machine(controller_app)
- assert (await _listed(client, machine_id))["desired_state"] == "retained"
- revived = await _claim(factory, owner_id)
+ app = controller_app
+ app.config.hosted_idle_stop_minutes = 30
+ machine_id, owner_id = await _empty_machine(app)
+ assert (await listed(app.client, machine_id))["desired_state"] == "retained"
+ revived = await _claim(app.factory, owner_id)
assert (revived.id, revived.desired_state, revived.retain_until) == (
machine_id,
"running",
None,
)
- await update_machine(factory, machine_id, state="deleted", desired_state="deleted")
- replaced = await _claim(factory, owner_id)
- assert replaced.id != machine_id
- assert (replaced.slot_id, replaced.desired_state) == ("slot-b", "running")
-
-
-async def test_idle_stop_is_off_when_the_setting_is_zero(controller_app):
- client, *_ = controller_app
- await _idle_ready(controller_app, minutes=0)
- [item] = await list_machines(client)
- assert item["desired_state"] == "running"
- assert item["revision"] == 1
-
-
-async def test_only_a_ready_running_machine_is_put_to_sleep(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- await _idle_ready(controller_app, minutes=30)
- machine = await machine_of(factory, request_id)
- await update_machine(factory, machine.id, state="provisioning")
- [item] = await list_machines(client)
- assert item["desired_state"] == "running"
-
-
-async def test_worker_renewal_is_bound_to_its_owner_and_selected_repository(
- controller_app,
-):
- client, request_id, agent_id, _, factory, _ = controller_app
- renewed = await client.post("/hosted/github-credential")
- assert renewed.status_code == 200, renewed.text
- assert renewed.json()["repository"] == "example/project"
- assert renewed.headers["cache-control"] == "no-store"
- await update_launch(factory, request_id, agent_id=str(uuid4()))
- denied = await client.post("/hosted/github-credential")
- assert denied.status_code == 403
- assert "SYNTHETIC" not in denied.text
-
-
-async def test_worker_claim_is_durable_and_stale_claim_is_not_replayed(controller_app):
- client, request_id, agent_id, service, factory, _ = controller_app
- conn = attach_worker(service, agent_id, request_id)
- operation_id = await _queued_operation(factory, request_id)
- claim_path = f"/hosted/operations/{operation_id}/claim"
- claimed_by = f"1:{conn.id}:{conn.stream_generation}"
- claim = await client.post(claim_path, json=fence(conn))
- assert claim.status_code == 200, claim.text
- assert claim.json()["id"] == operation_id
- assert claim.json()["state"] == "claimed"
- other_boot = attach_worker(
- service, agent_id, request_id, connection_id=conn.id, boot_id="boot-b"
- )
- second = await client.post(claim_path, json=fence(other_boot))
- assert second.status_code == 409
- conn = attach_worker(service, agent_id, request_id, connection_id=conn.id)
- async with factory() as session:
- row = await session.get(HostedOperation, (require_tenant_id(), operation_id))
- assert row.claimed_by == claimed_by
- assert row.claimed_boot_id == "boot-a"
- row.updated_at = datetime.now(UTC) - timedelta(minutes=6)
- await session.commit()
- async with factory() as session:
- await HostedLaunchStore().fail_stale_operations(session, request_id, 1)
- await session.commit()
- row = await session.get(HostedOperation, (require_tenant_id(), operation_id))
- assert row.state == "unknown"
- assert (
- await client.post(
- f"/hosted/operations/{operation_id}/result",
- json={**fence(conn), "state": "applied", "error": None},
- )
- ).status_code == 409
-
-
-async def _queued_operation(factory, request_id: str, revision: int = 1) -> str:
- operation_id = str(uuid4())
- async with factory() as session:
- session.add(
- HostedOperation(
- id=operation_id,
- launch_id=request_id,
- launch_revision=revision,
- session_id=str(uuid4()),
- action="start",
- )
- )
- await session.commit()
- return operation_id
-
-
-async def test_operation_claim_is_refused_to_a_non_worker(controller_app):
- client, request_id, agent_id, service, factory, _ = controller_app
- operation_id = await _queued_operation(factory, request_id)
- plain = service.connections.open(
- agent_id=agent_id,
- connection_id=str(uuid4()),
- scope="all",
- delivery_filter="all",
- spawn_capable=True,
- cursor=0,
- declaration=ClientDeclaration(speaks=7, accepts=1),
- expected_generation=None,
- )
- refused = await client.post(
- f"/hosted/operations/{operation_id}/claim", json=fence(plain)
- )
- assert refused.status_code == 403
- assert refused.json()["detail"]["code"] == "hosted_worker_only"
- worker = attach_worker(service, agent_id, request_id)
- stale = {**fence(worker), "generation": worker.stream_generation + 1}
- refused = await client.post(f"/hosted/operations/{operation_id}/claim", json=stale)
- assert refused.status_code == 409
- assert refused.json()["detail"]["code"] == "generation_changed"
-
-
-async def test_a_lost_claim_reply_is_offered_again_and_claimed_once(controller_app):
- client, request_id, agent_id, service, factory, _ = controller_app
- operation_id = await _queued_operation(factory, request_id)
- conn = attach_worker(service, agent_id, request_id)
- claim_path = f"/hosted/operations/{operation_id}/claim"
- assert (await client.post(claim_path, json=fence(conn))).status_code == 200
-
- idle = {
- "report_seq": 1,
- "relays_through": 0,
- "busy": False,
- "reasons": [],
- "sessions": {"total": 0, "live": 0, "parked": 0, "failed": 0},
- }
- report = await client.post(
- f"/agents/{agent_id}/connection/idle", json={**fence(conn), **idle}
- )
- assert report.status_code == 200, report.text
- assert report.json()["queued_operations"] == [operation_id]
-
- reattached = attach_worker(service, agent_id, request_id, connection_id=conn.id)
- again = await client.post(claim_path, json=fence(reattached))
- assert again.status_code == 200, again.text
- assert again.json()["state"] == "claimed"
- result = await client.post(
- f"/hosted/operations/{operation_id}/result",
- json={**fence(reattached), "state": "applied", "error": None},
- )
- assert result.status_code == 200
- report = await client.post(
- f"/agents/{agent_id}/connection/idle",
- json={**fence(reattached), **idle, "report_seq": 2},
- )
- assert report.json()["queued_operations"] == []
- assert (await client.post(claim_path, json=fence(reattached))).status_code == 409
-
-
-async def test_lost_result_is_reposted_from_a_later_generation(controller_app):
- client, request_id, agent_id, service, factory, _ = controller_app
- operation_id = await _queued_operation(factory, request_id)
- conn = attach_worker(service, agent_id, request_id)
- claimed = await client.post(
- f"/hosted/operations/{operation_id}/claim", json=fence(conn)
- )
- assert claimed.status_code == 200
- reattached = attach_worker(service, agent_id, request_id, connection_id=conn.id)
- result_path = f"/hosted/operations/{operation_id}/result"
- body = {**fence(reattached), "state": "applied", "error": None}
- first = await client.post(result_path, json=body)
- assert first.status_code == 200, first.text
- assert first.json()["state"] == "applied"
- again = await client.post(result_path, json=body)
- assert again.status_code == 200
- other_boot = attach_worker(
- service, agent_id, request_id, connection_id=conn.id, boot_id="boot-b"
- )
- refused = await client.post(
- result_path, json={**fence(other_boot), "state": "applied", "error": None}
- )
- assert refused.status_code == 409
-
-
-async def test_provider_refresh_and_revocation_are_owner_bound(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- response = await client.post("/hosted/provider-credential")
- assert response.status_code == 200
- assert response.json()["status"] == "connected"
- assert response.headers["cache-control"] == "no-store"
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- await ProviderConnectionStore().delete(session, launch.owner_id)
- await session.commit()
- assert (await client.post("/hosted/provider-credential")).json() == {
- "status": "revoked"
- }
- await update_launch(factory, request_id, desired_state="deleted")
- assert (await client.post("/hosted/provider-credential")).status_code == 403
- assert (await client.post("/hosted/github-credential")).status_code == 403
-
-
-async def test_provider_status_waits_then_rejects_replaced_revision(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- old = (await client.post("/hosted/provider-credential")).json()
- async with factory() as writer:
- launch = await writer.get(HostedLaunch, (require_tenant_id(), request_id))
- await ProviderConnectionStore().lock_user(writer, launch.owner_id)
- connection = await writer.get(
- ProviderConnection, (require_tenant_id(), launch.owner_id, "claude")
- )
- replacement = connection.verified_at + timedelta(seconds=1)
- connection.verified_at = replacement
- connection.verification_status = "configured"
- request = asyncio.create_task(
- client.post(
- "/hosted/provider-status",
- json={
- "authenticated": True,
- "revision": old["revision"],
- },
- )
- )
- try:
- await asyncio.sleep(0.05)
- assert not request.done()
- finally:
- await writer.commit()
- response = await asyncio.wait_for(request, timeout=5)
- assert response.status_code == 409
- async with factory() as db:
- saved = await db.get(
- ProviderConnection, (require_tenant_id(), launch.owner_id, "claude")
- )
- assert saved.verified_at == replacement
- assert saved.verification_status == "configured"
-
-
[email protected]("desired", ["stopped", "deleted"])
-async def test_provider_status_does_not_overwrite_stop_during_lock_wait(
- controller_app, desired
-):
- client, request_id, _, _, factory, _ = controller_app
- old = (await client.post("/hosted/provider-credential")).json()
- async with factory() as writer:
- launch = await writer.get(HostedLaunch, (require_tenant_id(), request_id))
- await ProviderConnectionStore().lock_user(writer, launch.owner_id)
- request = asyncio.create_task(
- client.post(
- "/hosted/provider-status",
- json={"authenticated": False, "revision": old["revision"]},
- )
- )
- try:
- await asyncio.sleep(0.05)
- assert not request.done()
- launch.desired_state = desired
- launch.state = "stopping"
- finally:
- await writer.commit()
- response = await asyncio.wait_for(request, timeout=5)
- assert response.status_code == 409
- async with factory() as db:
- saved = await db.get(HostedLaunch, (require_tenant_id(), request_id))
- assert saved.desired_state == desired
- assert saved.state == "stopping"
-
-
[email protected]("change", ["stop", "relink", "disconnect", "revoke_failure"])
-async def test_worker_token_is_revoked_when_authorization_changes_during_issue(
- controller_app, monkeypatch, change
-):
- client, request_id, _, _, factory, _ = controller_app
- revoke = AsyncMock(
- side_effect=RuntimeError("Synthetic revocation failure")
- if change == "revoke_failure"
- else None
- )
-
- async def issue(*args):
- async with factory() as session:
- await asyncio.wait_for(lock_launch(session, request_id), 1)
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- if change in ("stop", "revoke_failure"):
- launch.desired_state = "stopped"
- launch.revision += 1
- else:
- row = await session.get(
- ProviderConnection, (require_tenant_id(), launch.owner_id, "github")
- )
- if change == "disconnect":
- await session.delete(row)
- else:
- row.verified_at = datetime.now(UTC) + timedelta(seconds=1)
- await session.commit()
- return RepositoryCredential(
- "SYNTHETIC-REPOSITORY",
- datetime.now(UTC) + timedelta(hours=1),
- 456,
- "example/project",
- )
-
- monkeypatch.setattr(
- "switch_core.bridges.agent.api.hosted_routes.GitHubInstallationCredentials",
- lambda *_: SimpleNamespace(issue=issue, revoke=revoke),
- )
- response = await client.post("/hosted/github-credential")
- assert response.status_code == 409
- revoke.assert_awaited_once_with("SYNTHETIC-REPOSITORY")
- if change == "revoke_failure":
- async with factory() as session:
- queued = await session.scalar(
- select(GitHubIssuedToken).where(
- GitHubIssuedToken.revoke_requested.is_(True)
- )
- )
- assert queued is not None
- assert "SYNTHETIC-REPOSITORY" not in response.text
-
-
-async def test_local_registration_cannot_take_a_reserved_cloud_name(controller_app):
- _, request_id, agent_id, service, factory, _ = controller_app
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- owner_id = launch.owner_id
- with pytest.raises(AgentExistsError):
- await service.register_agent(
- name="cloud-helper",
- description="Local helper",
- connector_type="test",
- integration_profile=PROFILE,
- owner_id=owner_id,
- )
- async with factory() as session:
- agent = await session.scalar(select(Agent).where(Agent.name == "cloud-helper"))
- assert agent.id == agent_id
-
-
[email protected]("state", ["queued", "claimed"])
-async def test_operations_from_an_earlier_worker_are_not_claimed_or_completed(
- controller_app, state
-):
- client, request_id, agent_id, service, factory, _ = controller_app
- conn = attach_worker(service, agent_id, request_id, revision=2)
- operation_id = str(uuid4())
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- launch.revision = 2
- session.add(
- HostedOperation(
- id=operation_id,
- launch_id=request_id,
- launch_revision=1,
- session_id=str(uuid4()),
- action="start",
- state=state,
- )
- )
- await session.commit()
- claim = await client.post(
- f"/hosted/operations/{operation_id}/claim", json=fence(conn)
- )
- assert claim.status_code == 409
- response = await client.post(
- f"/hosted/operations/{operation_id}/result",
- json={**fence(conn), "state": "applied", "error": None},
- )
- assert response.status_code == 409
- async with factory() as session:
- await HostedLaunchStore().fail_stale_operations(session, request_id, 2)
- await session.commit()
- async with factory() as session:
- row = await session.get(HostedOperation, (require_tenant_id(), operation_id))
- assert row.state == ("unknown" if state == "claimed" else "failed")
- assert "worker changed" in row.error.lower()
-
-
-async def _issue_repository_token(client) -> None:
- response = await client.post("/hosted/github-credential")
- assert response.status_code == 200, response.text
-
-
[email protected]("cause", ["stop", "owner_loss", "disconnect", "expired"])
-async def test_repository_tokens_are_revoked_after_access_commit(
- controller_app, monkeypatch, cause
-):
- client, request_id, _, service, factory, _ = controller_app
- await _issue_repository_token(client)
- async with factory() as session:
- record = await session.scalar(select(GitHubIssuedToken))
- assert "SYNTHETIC" not in record.encrypted_token
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- owner = launch.owner_id
- if cause == "stop":
- launch.desired_state = "stopped"
- launch.revision += 1
- elif cause == "owner_loss":
- await session.delete(
- await session.get(TenantMember, (require_tenant_id(), owner))
- )
- elif cause == "disconnect":
- await session.delete(
- await session.get(
- ProviderConnection, (require_tenant_id(), owner, "github")
- )
- )
- else:
- record.expires_at = datetime.now(UTC) - timedelta(seconds=1)
- await session.commit()
-
- async def revoked(token):
- assert token == "SYNTHETIC-REPOSITORY"
- async with factory() as session:
- stored = await session.scalar(select(GitHubIssuedToken))
- assert stored.revoke_requested
-
- revoke = AsyncMock(side_effect=revoked)
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", revoke)
- async with factory() as session:
- assert await revoke_pending(session, service.config, ()) is False
- assert await revoke_pending(session, service.config, ()) is False
- assert await session.scalar(select(GitHubIssuedToken)) is None
- assert revoke.await_count == (0 if cause == "expired" else 1)
-
-
-async def test_machines_listing_drains_pending_revocations(controller_app, monkeypatch):
- client, request_id, _, _, factory, _ = controller_app
- await _issue_repository_token(client)
- await update_launch(factory, request_id, desired_state="stopped", revision=2)
- revoke = AsyncMock()
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", revoke)
- await list_machines(client)
- revoke.assert_awaited_once_with("SYNTHETIC-REPOSITORY")
- async with factory() as session:
- assert await session.scalar(select(GitHubIssuedToken)) is None
-
-
-async def test_failed_repository_revocation_remains_queued(controller_app, monkeypatch):
- client, request_id, _, service, factory, _ = controller_app
- await _issue_repository_token(client)
- async with factory() as session:
- await queue_revocation(session, (GitHubIssuedToken.launch_id == request_id,))
- await session.commit()
- revoke = AsyncMock(side_effect=RuntimeError("Synthetic failure"))
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", revoke)
- async with factory() as session:
- assert await revoke_pending(session, service.config, ()) is True
- record = await session.scalar(select(GitHubIssuedToken))
- assert record.revoke_requested
- await session.commit()
- revoke.side_effect = None
- assert await revoke_pending(session, service.config, ()) is False
-
-
-async def test_error_retry_issues_a_fresh_repository_token(controller_app, monkeypatch):
- client, request_id, _, service, factory, _ = controller_app
- await _issue_repository_token(client)
- await update_launch(factory, request_id, state="error")
- async with factory() as session:
- assert await revoke_pending(session, service.config, ()) is False
- assert await session.scalar(select(GitHubIssuedToken)) is None
- await update_launch(factory, request_id, state="queued", revision=2)
- issue = AsyncMock(
- return_value=RepositoryCredential(
- "SYNTHETIC-FRESH-REPOSITORY",
- datetime.now(UTC) + timedelta(hours=1),
- 456,
- "example/project",
- )
- )
- monkeypatch.setattr(
- "switch_core.bridges.agent.api.hosted_routes.GitHubInstallationCredentials",
- lambda *_: SimpleNamespace(issue=issue, revoke=AsyncMock()),
- )
- result = await client.post("/hosted/github-credential")
- assert result.status_code == 200, result.text
- assert result.json()["token"] == "SYNTHETIC-FRESH-REPOSITORY"
- async with factory() as session:
- record = await session.scalar(select(GitHubIssuedToken))
- assert not record.revoke_requested
- assert record.launch_revision == 2
-
-
-async def test_concurrent_revocation_drains_claim_once(controller_app, monkeypatch):
- client, request_id, _, service, factory, _ = controller_app
- await _issue_repository_token(client)
- async with factory() as session:
- await queue_revocation(session, ())
- await session.commit()
- entered, release = asyncio.Event(), asyncio.Event()
-
- async def revoke(token):
- entered.set()
- await release.wait()
-
- mocked = AsyncMock(side_effect=revoke)
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", mocked)
-
- async def drain():
- async with factory() as session:
- return await revoke_pending(session, service.config, ())
-
- first = asyncio.create_task(drain())
- try:
- await asyncio.wait_for(entered.wait(), 5)
- assert await drain() is True
- async with factory() as session:
- row = await session.scalar(select(GitHubIssuedToken))
- assert row.attempts == 1
- assert row.claim_until > datetime.now(UTC) + timedelta(seconds=60)
- mocked.assert_awaited_once()
- finally:
- release.set()
- assert await first is False
- async with factory() as session:
- assert await session.scalar(select(GitHubIssuedToken)) is None
-
-
-async def test_revocation_lock_timeout_preserves_the_committed_sweep(controller_app):
- client, request_id, _, _, factory, _ = controller_app
- await _issue_repository_token(client)
- machine = await machine_of(factory, request_id)
await update_machine(
- factory, machine.id, updated_at=datetime.now(UTC) - timedelta(minutes=11)
+ app.factory, machine_id, state="deleted", desired_state="deleted"
)
- async with factory() as locked:
- await locked.execute(
- text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
- {"key": f"github-revocation:{require_tenant_id()}"},
- )
- await queue_revocation(locked, ())
- listed = await list_machines(client)
- assert listed[0]["state"] == "error"
- await locked.rollback()
- assert (await machine_of(factory, request_id)).state == "error"
-
-
-async def test_failed_revocations_do_not_starve_newer_tokens(
- controller_app, monkeypatch
-):
- client, request_id, _, service, factory, _ = controller_app
- await _issue_repository_token(client)
- async with factory() as session:
- first = await session.scalar(select(GitHubIssuedToken))
- for number in range(8):
- session.add(
- GitHubIssuedToken(
- id=str(uuid4()),
- owner_id=first.owner_id,
- launch_id=request_id,
- launch_revision=1,
- encrypted_token=TEST_KEYRING.encrypt(f"SYNTHETIC-TOKEN-{number}"),
- expires_at=first.expires_at + timedelta(seconds=1),
- revoke_requested=True,
- attempts=0,
- )
- )
- first.revoke_requested = True
- await session.commit()
- seen = []
-
- async def fail(token):
- seen.append(token)
- raise RuntimeError("Synthetic outage")
-
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", fail)
- async with factory() as session:
- assert await revoke_pending(session, service.config, ()) is True
- assert len(set(seen)) == 8
- assert await revoke_pending(session, service.config, ()) is True
- assert len(set(seen)) == 9
-
-
-async def test_revocation_warning_is_scoped_to_action_owner(
- controller_app, monkeypatch
-):
- client, request_id, _, service, factory, _ = controller_app
- await _issue_repository_token(client)
- async with factory() as session:
- first = await session.scalar(select(GitHubIssuedToken))
- owner = first.owner_id
- other_user = User(
- name="other",
- role="user",
- password_hash="x",
- )
- session.add(other_user)
- await session.flush()
- other = other_user.id
- other_launch = HostedLaunch(
- id=str(uuid4()), owner_id=other, name="other-revocation-worker", spec={}
- )
- session.add(other_launch)
- await session.flush()
- session.add(
- GitHubIssuedToken(
- id=str(uuid4()),
- owner_id=other,
- launch_id=other_launch.id,
- launch_revision=1,
- encrypted_token=TEST_KEYRING.encrypt("SYNTHETIC-OTHER-TOKEN"),
- expires_at=first.expires_at,
- revoke_requested=True,
- attempts=0,
- )
- )
- await session.commit()
- revoke = AsyncMock(side_effect=RuntimeError("Synthetic outage"))
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", revoke)
- async with factory() as session:
- assert (
- await revoke_pending(
- session, service.config, (GitHubIssuedToken.owner_id == owner,)
- )
- is False
- )
- revoke.assert_not_awaited()
- assert (
- await revoke_pending(
- session, service.config, (GitHubIssuedToken.owner_id == other,)
- )
- is True
- )
-
-
-async def _interrupted_removal(controller_app, updated_at: datetime) -> None:
- """Leave the launch as a removal interrupted between its two commits."""
- _, request_id, agent_id, service, factory, _ = controller_app
- service.client_lifecycle.stop = AsyncMock()
- service.client_lifecycle.delete_record = AsyncMock(side_effect=ClientStore().delete)
- service.config.hosted_disk_retention_days = 7
- async with factory() as session:
- agent = await session.get(Agent, agent_id)
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- launch.desired_state = "deleted"
- launch.state = "deleting"
- launch.revision = 2
- launch.deletion_cleanup = {
- "client_id": agent.client_id,
- "key_id": agent.api_key_id,
- }
- launch.updated_at = updated_at
- await session.commit()
-
-
-async def test_sweep_finishes_an_interrupted_removal(controller_app):
- client, request_id, agent_id, _, factory, _ = controller_app
- async with factory() as session:
- key_id = (await session.get(Agent, agent_id)).api_key_id
- await _interrupted_removal(
- controller_app, datetime.now(UTC) - DELETING_RESUME_AFTER - timedelta(seconds=1)
- )
- [item] = await list_machines(client)
- assert item["desired_state"] == "retained"
- async with factory() as session:
- assert await session.get(Agent, agent_id) is None
- assert await session.get(ApiKey, key_id) is None
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert (launch.state, launch.name, launch.deletion_cleanup) == (
- "deleted",
- "removed:" + request_id,
- None,
- )
-
-
-async def test_sweep_leaves_a_recent_removal_alone(controller_app):
- client, request_id, agent_id, _, factory, _ = controller_app
- await _interrupted_removal(controller_app, datetime.now(UTC))
- [item] = await list_machines(client)
- assert item["desired_state"] == "running"
- async with factory() as session:
- assert await session.get(Agent, agent_id) is not None
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch.state == "deleting"
+ replaced = await _claim(app.factory, owner_id)
+ assert replaced.id != machine_id
+ assert replaced.desired_state == "running"
diff --git a/core/tests/switch_core/gateway/test_hosted_cutover.py b/core/tests/switch_core/gateway/test_hosted_cutover.py
deleted file mode 100644
index 9189b29f5..000000000
--- a/core/tests/switch_core/gateway/test_hosted_cutover.py
+++ /dev/null
@@ -1,570 +0,0 @@
-"""A retained worker volume's pre-cutover work, merged with Core's capture and decided once."""
-
-from __future__ import annotations
-
-import hashlib
-from datetime import UTC, datetime
-from typing import Any
-from uuid import uuid4
-
-import pytest
-from sqlalchemy import select
-from sqlalchemy.dialects.postgresql import insert
-
-from switch_core.bridges.agent.hosted_cutover import (
- CutoverConflict,
- CutoverManifest,
- RoomMessageRecord,
- apply_manifest,
- decide_room_message,
- merge_worker,
- owed_notices,
- post_owed_cutover_notices,
- record_blocked,
-)
-from switch_core.bridges.agent.hosted_mailbox import mailbox_upkeep
-from switch_core.db.models import (
- HostedCutoverItem,
- HostedCutoverVolume,
- HostedWakeMailbox,
- require_tenant_id,
-)
-from switch_core.db.stores.agent_store import AgentStore
-from tests.switch_core.gateway.test_hosted_mailbox import ( # noqa: F401
- attach,
- fail_first_send,
- mailbox_app,
- set_launch,
-)
-from tests.switch_core.gateway.test_hosted_workers import worker_app # noqa: F401
-
-
-def room_record(
- room_id: str,
- message_id: str,
- *,
- room_pending: bool,
- host: str | None,
- failure_notified: bool,
-) -> dict[str, Any]:
- return {
- "kind": "room_message",
- "session_id": "watcher-placeholder",
- "room_id": room_id,
- "message_id": message_id,
- "thread_id": None,
- "room_pending": room_pending,
- "failure_notified": failure_notified,
- "host": host,
- }
-
-
-async def core_item(app, kind: str, evidence: dict[str, Any], **columns: Any) -> None:
- """A row as the cutover-manifest revision captured it from the old session tables."""
- async with app.factory() as session:
- session.add(
- HostedCutoverItem(
- tenant_id=require_tenant_id(),
- id=str(uuid4()),
- agent_id=app.agent_id,
- launch_id=app.request_id,
- session_id="watcher-placeholder",
- kind=kind,
- evidence={"core": evidence},
- **columns,
- )
- )
- await session.commit()
-
-
-async def volume(app) -> None:
- """The row the cutover-manifest revision makes for the launch."""
- async with app.factory() as session:
- await session.execute(
- insert(HostedCutoverVolume)
- .values(tenant_id=require_tenant_id(), launch_id=app.request_id)
- .on_conflict_do_nothing()
- )
- await session.commit()
-
-
-async def record(
- app, items: list[dict[str, Any]], sha: str | None = None
-) -> CutoverManifest:
- """`hosted-cutover-upgrade record` with the manifest a preflight check answered."""
- manifest = CutoverManifest.model_validate(
- {
- "manifest_sha256": sha or hashlib.sha256(repr(items).encode()).hexdigest(),
- "items": items,
- }
- )
- await volume(app)
- async with app.factory() as session:
- await apply_manifest(
- session, agent_id=app.agent_id, launch_id=app.request_id, manifest=manifest
- )
- await session.commit()
- return manifest
-
-
-async def confirm(app, conn, manifest: CutoverManifest):
- """The worker's upload of the manifest its first boot wrote."""
- return await app.client.post(
- f"/agents/{app.agent_id}/connection/cutover-manifest",
- json={
- "connection_id": conn.id,
- "generation": conn.stream_generation,
- **manifest.model_dump(mode="json"),
- },
- )
-
-
-async def upload(app, conn, items: list[dict[str, Any]], sha: str | None = None):
- return await confirm(app, conn, await record(app, items, sha))
-
-
-async def items(app) -> list[HostedCutoverItem]:
- async with app.factory() as session:
- return list(
- await session.scalars(
- select(HostedCutoverItem)
- .where(HostedCutoverItem.tenant_id == require_tenant_id())
- .order_by(HostedCutoverItem.created_at, HostedCutoverItem.kind)
- )
- )
-
-
-async def mailbox(app) -> list[HostedWakeMailbox]:
- async with app.factory() as session:
- return list(
- await session.scalars(
- select(HostedWakeMailbox).where(
- HostedWakeMailbox.tenant_id == require_tenant_id()
- )
- )
- )
-
-
-async def ready(app):
- await set_launch(app, state="ready")
- await attach(app)
- (conn,) = app.service.connections.for_agent(app.agent_id)
- conn.worker_frames.drain()
- return conn
-
-
- ("core_status", "host", "disposition", "notices", "imported"),
- [
- ("accepted", "dispatched", "uncertain", 1, False),
- (None, "accepted", "import", 0, True),
- ("applied", None, "ran", 0, False),
- ("rejected", None, "unrecoverable", 1, False),
- ("unknown", None, "uncertain", 1, False),
- ],
-)
-async def test_cutover_merges_overlapping_records(
- mailbox_app, # noqa: F811
- core_status,
- host,
- disposition,
- notices,
- imported,
-):
- app = mailbox_app
- room = app.rooms[0]
- conn = await ready(app)
- if core_status is not None:
- await core_item(
- app,
- "room_message",
- {"command_id": "placeholder-command", "status": core_status, "code": None},
- room_id=room,
- message_id="$m1",
- )
- response = await upload(
- app,
- conn,
- [
- room_record(
- room, "$m1", room_pending=True, host=host, failure_notified=False
- )
- ],
- )
- assert response.status_code == 200, response.text
- (item,) = await items(app)
- assert (item.kind, item.room_id, item.message_id) == ("room_message", room, "$m1")
- assert item.disposition == disposition
- assert item.evidence["worker"]["host"] == host
- assert len(app.sent) == notices
- rows = await mailbox(app)
- if imported:
- (row,) = rows
- assert (row.room_id, row.message_id, row.origin) == (room, "$m1", "cutover")
- assert row.state == "offered"
- assert item.payload["payload"]["message_id"] == "$m1"
- (wake,) = [
- data for event, data in conn.worker_frames.drain() if event == "wake"
- ]
- assert wake["entries"][0]["origin"] == "cutover"
- else:
- assert rows == []
- assert item.payload is None
-
-
-async def test_cutover_is_applied_once_per_manifest(mailbox_app): # noqa: F811
- app = mailbox_app
- room = app.rooms[0]
- conn = await ready(app)
- manifest = [
- room_record(room, "$m1", room_pending=True, host=None, failure_notified=True),
- room_record(
- room, "$m2", room_pending=False, host="dispatched", failure_notified=False
- ),
- ]
- first = await upload(app, conn, manifest, sha="a" * 64)
- again = await upload(app, conn, manifest, sha="a" * 64)
- assert first.status_code == 200, first.text
- assert again.status_code == 200, again.text
- with pytest.raises(CutoverConflict, match="a" * 64):
- await record(app, manifest[:1], sha="b" * 64)
- other = await confirm(
- app,
- conn,
- CutoverManifest.model_validate(
- {"manifest_sha256": "b" * 64, "items": manifest[:1]}
- ),
- )
- assert other.status_code == 409
- assert other.json()["detail"]["code"] == "cutover_manifest_conflict"
-
- decided = {item.message_id: item.disposition for item in await items(app)}
- assert decided == {"$m1": "import", "$m2": "uncertain"}
- assert [row.message_id for row in await mailbox(app)] == ["$m1"]
- bodies = sorted(body for _, _, body in app.sent)
- assert len(bodies) == 2
- assert any("will process this message now" in body for body in bodies)
- assert any("may have been interrupted" in body for body in bodies)
- assert all(item.notice_posted_at is not None for item in await items(app))
- async with app.factory() as session:
- volume = await session.get(
- HostedCutoverVolume, (require_tenant_id(), app.request_id)
- )
- assert volume is not None
- assert (volume.preflight_state, volume.manifest_sha256) == (
- "complete",
- "a" * 64,
- )
-
-
-async def test_cutover_refuses_a_message_it_cannot_rebuild(mailbox_app): # noqa: F811
- app = mailbox_app
- room = app.rooms[0]
- conn = await ready(app)
- response = await upload(
- app,
- conn,
- [
- room_record(
- room, "$gone", room_pending=True, host=None, failure_notified=False
- )
- ],
- )
- assert response.status_code == 200, response.text
- (item,) = await items(app)
- assert item.disposition == "unrecoverable"
- assert await mailbox(app) == []
- ((_, _, body),) = app.sent
- assert "was not run" in body
-
-
-async def test_cutover_decides_items_that_are_not_room_messages(mailbox_app): # noqa: F811
- app = mailbox_app
- room = app.rooms[0]
- conn = await ready(app)
- await core_item(
- app,
- "console_command",
- {"command_id": "console-delivered", "status": "accepted"},
- )
- await core_item(
- app, "console_command", {"command_id": "console-lost", "status": "accepted"}
- )
- await core_item(
- app,
- "request_open",
- {"request_id": "request-1", "epoch": 1},
- room_id=room,
- thread_id=None,
- )
- await core_item(app, "session", {"epoch": 1})
- response = await upload(
- app,
- conn,
- [
- {
- "kind": "console_command",
- "session_id": "watcher-placeholder",
- "command_id": "console-delivered",
- "host": "accepted",
- },
- {
- "kind": "request_open",
- "session_id": "watcher-placeholder",
- "request_id": "request-1",
- "room_id": None,
- "thread_id": None,
- },
- {"kind": "reset_pending", "session_id": "watcher-placeholder"},
- ],
- )
- assert response.status_code == 200, response.text
- decided = {
- (
- item.kind,
- (item.evidence.get("core") or {}).get("command_id"),
- ): item.disposition
- for item in await items(app)
- }
- assert decided == {
- ("console_command", "console-delivered"): "settled_by_host",
- ("console_command", "console-lost"): "owner_notice",
- ("request_open", None): "interrupted",
- ("session", None): "preserved",
- ("reset_pending", None): "preserved",
- }
- ((notice_room, _, body),) = app.sent
- assert notice_room == room
- assert "approval" in body
- assert await mailbox(app) == []
-
-
-async def test_an_import_into_a_session_awaiting_a_reset_says_it_waits(mailbox_app): # noqa: F811
- app = mailbox_app
- room = app.rooms[0]
- conn = await ready(app)
- elsewhere = {
- **room_record(room, "$m3", room_pending=True, host=None, failure_notified=True),
- "session_id": "other-session",
- }
- response = await upload(
- app,
- conn,
- [
- {"kind": "reset_pending", "session_id": "watcher-placeholder"},
- room_record(
- room, "$m1", room_pending=True, host=None, failure_notified=True
- ),
- room_record(
- room, "$m2", room_pending=True, host=None, failure_notified=False
- ),
- elsewhere,
- ],
- )
- assert response.status_code == 200, response.text
-
- decided = {
- item.message_id: item.disposition
- for item in await items(app)
- if item.kind == "room_message"
- }
- assert decided == {"$m1": "import", "$m2": "import", "$m3": "import"}
- bodies = sorted(body for _, _, body in app.sent)
- assert len(bodies) == 3
- waiting = [body for body in bodies if "cannot continue" in body]
- assert len(waiting) == 2
- assert all("!reset @" in body for body in waiting)
- assert sum("will process this message now" in body for body in bodies) == 1
-
-
-async def test_cutover_refuses_a_manifest_recorded_for_no_volume(mailbox_app): # noqa: F811
- app = mailbox_app
- room = app.rooms[0]
- conn = await ready(app)
- manifest = CutoverManifest.model_validate(
- {
- "manifest_sha256": "c" * 64,
- "items": [
- room_record(
- room, "$m1", room_pending=True, host=None, failure_notified=False
- )
- ],
- }
- )
- missing = await confirm(app, conn, manifest)
- assert missing.status_code == 409
- assert missing.json()["detail"]["code"] == "cutover_manifest_unrecorded"
- await volume(app)
- pending = await confirm(app, conn, manifest)
- assert pending.status_code == 409
- assert "preflight pending" in pending.json()["detail"]["message"]
- assert await items(app) == []
- assert await mailbox(app) == []
-
-
-async def test_a_blocked_volume_completes_once_it_checks_clean(mailbox_app): # noqa: F811
- app = mailbox_app
- room = app.rooms[0]
- await volume(app)
- async with app.factory() as session:
- await record_blocked(session, app.request_id, "sessions at events.jsonl:3: bad")
- await session.commit()
- async with app.factory() as session:
- blocked = await session.get(
- HostedCutoverVolume, (require_tenant_id(), app.request_id)
- )
- assert blocked is not None
- assert (blocked.preflight_state, blocked.blocked_reason) == (
- "blocked",
- "sessions at events.jsonl:3: bad",
- )
- await record(
- app,
- [
- room_record(
- room, "$m1", room_pending=True, host=None, failure_notified=False
- )
- ],
- sha="d" * 64,
- )
- async with app.factory() as session:
- complete = await session.get(
- HostedCutoverVolume,
- (require_tenant_id(), app.request_id),
- populate_existing=True,
- )
- assert complete is not None
- assert (complete.preflight_state, complete.blocked_reason) == (
- "complete",
- None,
- )
- with pytest.raises(CutoverConflict):
- await record_blocked(session, app.request_id, "late")
-
-
-async def test_upkeep_posts_the_notices_a_recorded_volume_owes(mailbox_app): # noqa: F811
- app = mailbox_app
- room = app.rooms[0]
- await record(
- app,
- [
- room_record(
- room,
- "$m2",
- room_pending=False,
- host="dispatched",
- failure_notified=False,
- )
- ],
- )
- assert app.sent == []
- await mailbox_upkeep(app.service, datetime.now(UTC))
- ((notice_room, _, body),) = app.sent
- assert notice_room == room
- assert "may have been interrupted" in body
- (item,) = await items(app)
- assert item.notice_posted_at is not None
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert len(app.sent) == 1
-
-
-async def notice_dropped(app) -> list[str | None]:
- async with app.factory() as session:
- return list(
- await session.scalars(
- select(HostedCutoverItem.notice_dropped).where(
- HostedCutoverItem.tenant_id == require_tenant_id()
- )
- )
- )
-
-
-async def owe_one_notice(app) -> None:
- await record(
- app,
- [
- room_record(
- app.rooms[0],
- "$m2",
- room_pending=False,
- host="dispatched",
- failure_notified=False,
- )
- ],
- )
-
-
-async def test_upkeep_retries_a_cutover_notice_whose_send_failed(mailbox_app): # noqa: F811
- app = mailbox_app
- await owe_one_notice(app)
- fail_first_send(app)
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert app.sent == []
- (item,) = await items(app)
- assert item.notice_posted_at is None
- assert await notice_dropped(app) == [None]
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert len(app.sent) == 1
- (item,) = await items(app)
- assert item.notice_posted_at is not None
-
-
-async def test_a_deleted_agents_cutover_notice_is_dropped_not_posted(
- mailbox_app, # noqa: F811
- caplog,
-):
- app = mailbox_app
- await owe_one_notice(app)
- async with app.factory() as session:
- await AgentStore().delete(session, app.agent_id)
- await session.commit()
-
- assert await post_owed_cutover_notices(app.service) == 0
-
- assert app.sent == []
- (item,) = await items(app)
- assert item.notice_posted_at is None
- assert await notice_dropped(app) == ["agent_deleted"]
- async with app.factory() as session:
- assert await owed_notices(session, app.agent_id) == []
- assert any("was deleted" in record.message for record in caplog.records)
- caplog.clear()
- assert await post_owed_cutover_notices(app.service) == 0
- assert not any("was deleted" in record.message for record in caplog.records)
-
-
-def test_decide_room_message_prefers_the_strongest_evidence():
- records = [
- RoomMessageRecord(
- kind="room_message",
- session_id="a",
- room_id="!r:example.com",
- message_id="$m",
- thread_id=None,
- room_pending=True,
- failure_notified=True,
- host=None,
- ),
- RoomMessageRecord(
- kind="room_message",
- session_id="b",
- room_id="!r:example.com",
- message_id="$m",
- thread_id="$t",
- room_pending=False,
- failure_notified=False,
- host="dispatched",
- ),
- ]
- merged = merge_worker(records)
- assert merged is not None
- assert (merged.session_id, merged.host, merged.thread_id) == (
- "b",
- "dispatched",
- "$t",
- )
- assert merged.room_pending and merged.failure_notified
- assert decide_room_message("accepted", merged) == "uncertain"
- assert decide_room_message("applied", merged) == "ran"
- assert decide_room_message(None, merge_worker(records[:1])) == "import"
- assert decide_room_message("accepted", None) == "import"
diff --git a/core/tests/switch_core/gateway/test_hosted_launches.py b/core/tests/switch_core/gateway/test_hosted_launches.py
deleted file mode 100644
index 96a7e4f46..000000000
--- a/core/tests/switch_core/gateway/test_hosted_launches.py
+++ /dev/null
@@ -1,1037 +0,0 @@
-from datetime import UTC, datetime, timedelta
-from types import SimpleNamespace
-from unittest.mock import AsyncMock
-from uuid import uuid4
-
-import httpx
-import pytest
-from fastapi import FastAPI
-from sqlalchemy import func, select, text
-
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.bridges.agent.protocol.agent_core import AgentExistsError
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.db.models import (
- Agent,
- HostedLaunch,
- HostedMachine,
- HostedOperation,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.agent_session_store import AgentSessionStore
-from switch_core.db.stores.agent_store import AgentStore
-from switch_core.db.stores.client_store import ClientStore
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
-from switch_core.db.stores.hosted_machine_store import MACHINE_NEEDS_ATTENTION
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.db.stores.room_role_store import RoomRoleStore
-from switch_core.db.stores.room_store import RoomStore
-from switch_core.db.stores.user_store import UserStore
-from switch_core.gateway.agents import (
- update_addressing_policy,
- update_agent_display_name,
- update_agent_icon,
-)
-from switch_core.gateway.auth import get_current_user
-from switch_core.gateway.dependencies import get_config, get_protocol, get_session
-from switch_core.gateway.hosted_launches import router
-from switch_core.gateway.schemas import (
- UpdateAddressingPolicyRequest,
- UpdateAgentDisplayNameRequest,
- UpdateAgentIconRequest,
-)
-from switch_core.keys import Keyring
-from switch_core.providers.claude_verifier import ClaudeVerificationError
-from tests.switch_core.bridges.agent.protocol.registration_harness import (
- make_service,
- register,
-)
-from tests.switch_core.hosted_machine_helpers import seed_machine
-
-TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
-
-ICON = "https://cdn.example.com/9.x/bottts/png?seed=helper"
-
-SUMMARY_KEYS = {
- "request_id",
- "name",
- "provider",
- "state",
- "agent_id",
- "error",
- "error_code",
- "desired_state",
- "revision",
- "sleeping",
- "machine_id",
- "process_state",
- "process_restarts",
- "oom_kills",
-}
-
-
-async def launch_app(session_factory, monkeypatch):
- async with session_factory() as session:
- owner = User(
- id="cloud-owner",
- name="Owner",
- role="user",
- password_hash="unused",
- )
- session.add(owner)
- await session.flush()
- await ProviderConnectionStore().save(
- session,
- owner.id,
- "setup-token",
- TEST_KEYRING.encrypt("SYNTHETIC-CREDENTIAL"),
- datetime.now(UTC),
- )
- await session.commit()
- app = FastAPI()
- app.include_router(router)
- verifier = AsyncMock()
- app.state.claude_verifier = verifier
- app.state.github_connections = object()
- app.state.hosted_controller_settings = SimpleNamespace(
- tenant_id=require_tenant_id(),
- machine_slots=["slot-a", "slot-b"],
- )
- config = SimpleNamespace(
- hosted_launch_capacity=1,
- hosted_agents_per_owner=3,
- hosted_sessions_per_agent=2,
- hosted_disk_retention_days=7,
- keyring=TEST_KEYRING,
- )
- protocol = make_service(session_factory)
- protocol.connections = AgentConnectionRegistry()
- protocol.event_buffer = EventBuffer(sequence_base=1 << 32)
- protocol.client_lifecycle.stop = AsyncMock()
- protocol.client_lifecycle.delete_record = AsyncMock(
- side_effect=ClientStore().delete
- )
- identity = {"user": owner}
-
- async def sessions():
- async with session_factory() as session:
- yield session
-
- app.dependency_overrides[get_current_user] = lambda: identity["user"]
- app.dependency_overrides[get_session] = sessions
- app.dependency_overrides[get_config] = lambda: config
- app.dependency_overrides[get_protocol] = lambda: protocol
- access = AsyncMock(
- return_value={
- "installations": [
- {
- "id": 123,
- "repositories": [
- {
- "id": 456,
- "name": "Example/Project",
- "permissions": {"push": True},
- }
- ],
- }
- ]
- }
- )
- monkeypatch.setattr("switch_core.gateway.hosted_launches.connection_status", access)
- async with httpx.AsyncClient(
- transport=httpx.ASGITransport(app=app), base_url="https://switch.example.com"
- ) as client:
- yield SimpleNamespace(
- client=client,
- verifier=verifier,
- access=access,
- config=config,
- identity=identity,
- factory=session_factory,
- protocol=protocol,
- )
-
-
-def body(name: str = "helper"):
- return {
- "request_id": str(uuid4()),
- "name": name,
- "description": "Repository helper",
- "display_name": None,
- "icon_url": None,
- "instructions": "",
- "definition": f"---\nname: {name}\ndescription: Repository helper\n---\nRepository helper\n",
- "installation_id": 123,
- "repository_id": 456,
- "definition_attributes": {"model": "sonnet"},
- "auto_session": True,
- "auto_approve": False,
- "addressing_policy": None,
- }
-
-
-async def _launch(factory, request_id: str) -> HostedLaunch:
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch is not None
- return launch
-
-
-async def _machine(factory, machine_id: str) -> HostedMachine:
- async with factory() as session:
- machine = await session.get(HostedMachine, (require_tenant_id(), machine_id))
- assert machine is not None
- return machine
-
-
-async def _update(factory, model, key: str, **values) -> None:
- async with factory() as session:
- row = await session.get(model, (require_tenant_id(), key))
- assert row is not None
- for name, value in values.items():
- setattr(row, name, value)
- await session.commit()
-
-
-async def _lifecycle(app, request_id: str, action: str, revision: int):
- return await app.client.post(
- f"/hosted-launches/{request_id}/lifecycle",
- json={"action": action, "revision": revision},
- )
-
-
-async def test_request_is_durable_idempotent_and_never_returns_credentials(launch_app):
- app = launch_app
- request = body()
- result = await app.client.post("/hosted-launches", json=request)
- assert result.status_code == 202
- assert result.json()["state"] == "queued"
- assert "SYNTHETIC" not in result.text
- app.verifier.verify.assert_awaited_once_with("setup-token", "SYNTHETIC-CREDENTIAL")
- second = await app.client.post("/hosted-launches", json=request)
- assert second.json() == result.json()
- assert app.verifier.verify.await_count == 1
- assert app.access.await_count == 1
- async with app.factory() as session:
- assert await session.scalar(select(func.count()).select_from(HostedLaunch)) == 1
- assert await session.scalar(select(func.count()).select_from(Agent)) == 1
- app.identity["user"] = SimpleNamespace(id="someone-else")
- assert (
- await app.client.get("/hosted-launches/" + request["request_id"])
- ).status_code == 404
-
-
-async def test_create_claims_a_machine_and_registers_the_identity(launch_app):
- app = launch_app
- created = await app.client.post("/hosted-launches", json=body())
- assert created.status_code == 202, created.text
- summary = created.json()
- assert set(summary) == SUMMARY_KEYS
- assert summary["sleeping"] is False
- assert summary["process_restarts"] == 0 and summary["oom_kills"] == 0
- agent_id = summary["agent_id"]
- assert agent_id is not None
- launch = await _launch(app.factory, summary["request_id"])
- assert launch.repository == "Example/Project"
- assert launch.machine_id == summary["machine_id"]
- async with app.factory() as session:
- agent = await session.get(Agent, agent_id)
- assert agent is not None and agent.name == "helper"
- assert agent.metadata_["hosted_launch_id"] == launch.id
- assert (
- agent.metadata_["known_agent_options"]["repo_dir"]
- == f"/data/worktrees/{agent_id}/example/project"
- )
- machine = await _machine(app.factory, summary["machine_id"])
- assert (machine.slot_id, machine.state, machine.desired_state) == (
- "slot-a",
- "queued",
- "running",
- )
- versions = machine.agents_version
- assert versions >= 1
-
- other = await app.client.post("/hosted-launches", json=body("reviewer"))
- assert other.status_code == 202, other.text
- assert other.json()["machine_id"] == machine.id
- assert (await _machine(app.factory, machine.id)).agents_version > versions
-
-
-async def test_create_refuses_without_a_machine(session_factory, launch_app):
- app = launch_app
- async with app.factory() as session:
- session.add(
- User(
- id="other-owner",
- name="Other",
- role="user",
- password_hash="unused",
- )
- )
- await session.flush()
- await seed_machine(
- session,
- owner_id="other-owner",
- slot_id="slot-a",
- state="ready",
- desired_state="running",
- stop_reason=None,
- revision=1,
- generation=1,
- )
- await session.commit()
- full = await app.client.post("/hosted-launches", json=body())
- assert full.status_code == 409
- assert full.json()["detail"] == "no machine slot available"
-
- app.config.hosted_launch_capacity = 2
- created = await app.client.post("/hosted-launches", json=body())
- assert created.status_code == 202, created.text
- await _update(
- app.factory, HostedMachine, created.json()["machine_id"], state="error"
- )
- broken = await app.client.post("/hosted-launches", json=body("reviewer"))
- assert broken.status_code == 409
- assert broken.json()["detail"] == MACHINE_NEEDS_ATTENTION
-
-
-async def test_identity_failure_is_an_error_that_retry_registers(
- launch_app, monkeypatch
-):
- app = launch_app
- register_agent = app.protocol.register_agent
- monkeypatch.setattr(
- app.protocol,
- "register_agent",
- AsyncMock(side_effect=RuntimeError("synthetic registration failure")),
- raising=False,
- )
- created = await app.client.post("/hosted-launches", json=body())
- assert created.status_code == 202, created.text
- failed = created.json()
- assert (failed["state"], failed["error_code"], failed["agent_id"]) == (
- "error",
- "identity_failed",
- None,
- )
- assert failed["error"]
- async with app.factory() as session:
- assert await session.scalar(select(func.count()).select_from(Agent)) == 0
-
- monkeypatch.setattr(app.protocol, "register_agent", register_agent, raising=False)
- retried = await _lifecycle(app, failed["request_id"], "retry", failed["revision"])
- assert retried.status_code == 200, retried.text
- assert retried.json()["state"] == "queued"
- assert retried.json()["error_code"] is None
- assert retried.json()["agent_id"] is not None
- async with app.factory() as session:
- assert await session.get(Agent, retried.json()["agent_id"]) is not None
-
-
-async def test_identity_failure_deletes_the_partial_agent_under_the_launch_lock(
- launch_app, monkeypatch
-):
- app = launch_app
- register_agent = app.protocol.register_agent
- delete_agent = app.protocol.delete_agent
- partial: list[str] = []
- lock_free: list[bool] = []
-
- async def register_then_fail(**kwargs):
- await register_agent(**kwargs)
- partial.append(kwargs["reserved_agent_id"])
- raise RuntimeError("synthetic registration failure")
-
- async def probe_then_delete(*, agent_id):
- async with app.factory() as other:
- lock_free.append(
- await other.scalar(
- text("SELECT pg_try_advisory_xact_lock(hashtextextended(:key, 0))"),
- {
- "key": f"hosted-launch:{require_tenant_id()}:{request['request_id']}"
- },
- )
- )
- await other.rollback()
- await delete_agent(agent_id=agent_id)
-
- monkeypatch.setattr(
- app.protocol, "register_agent", register_then_fail, raising=False
- )
- monkeypatch.setattr(app.protocol, "delete_agent", probe_then_delete, raising=False)
- request = body()
- created = await app.client.post("/hosted-launches", json=request)
- assert created.status_code == 202, created.text
- assert lock_free == [False]
- failed = created.json()
- assert (failed["state"], failed["error_code"], failed["agent_id"]) == (
- "error",
- "identity_failed",
- None,
- )
- async with app.factory() as session:
- assert await session.get(Agent, partial[0]) is None
-
-
-async def test_a_taken_name_is_refused(launch_app, monkeypatch):
- app = launch_app
- await register(app.protocol, "helper", "cloud-owner")
- refused = await app.client.post("/hosted-launches", json=body())
- assert refused.status_code == 409
- assert refused.json()["detail"] == "An agent already uses this name."
-
- monkeypatch.setattr(
- app.protocol,
- "register_agent",
- AsyncMock(side_effect=AgentExistsError("taken meanwhile")),
- raising=False,
- )
- request = body("reviewer")
- raced = await app.client.post("/hosted-launches", json=request)
- assert raced.status_code == 422
- launch = await _launch(app.factory, request["request_id"])
- assert (launch.state, launch.error_code, launch.agent_id) == (
- "error",
- "identity_failed",
- None,
- )
-
-
-async def test_disabled_server_refuses_before_verifying_credentials(launch_app):
- app = launch_app
- app.config.hosted_launch_capacity = 0
- assert (await app.client.post("/hosted-launches", json=body())).status_code == 503
- app.verifier.verify.assert_not_called()
-
-
-async def test_expired_claude_or_missing_repository_never_queues(launch_app):
- app = launch_app
- app.verifier.verify.side_effect = ClaudeVerificationError(
- "Credential no longer works."
- )
- assert (await app.client.post("/hosted-launches", json=body())).status_code == 422
- app.access.assert_not_called()
- app.verifier.verify.side_effect = None
- app.access.return_value = {"installations": []}
- assert (await app.client.post("/hosted-launches", json=body())).status_code == 422
- async with app.factory() as session:
- assert await session.scalar(select(func.count()).select_from(HostedLaunch)) == 0
-
-
-async def test_changed_request_and_unexpected_credentials_are_rejected(launch_app):
- app = launch_app
- request = body()
- assert (await app.client.post("/hosted-launches", json=request)).status_code == 202
- assert (
- await app.client.post("/hosted-launches", json=request | {"name": "different"})
- ).status_code == 409
- assert (
- await app.client.post(
- "/hosted-launches", json=body() | {"credential": "SYNTHETIC"}
- )
- ).status_code == 422
-
-
[email protected]("change", [{"instructions": "x" * 32768}])
-async def test_rejects_unlaunchable_configuration_before_verifying_credentials(
- launch_app, change
-):
- app = launch_app
- response = await app.client.post("/hosted-launches", json=body() | change)
- assert response.status_code == 422
- app.verifier.verify.assert_not_awaited()
- app.access.assert_not_awaited()
-
-
-async def test_manual_cloud_agents_are_supported(launch_app):
- app = launch_app
- assert (
- await app.client.post("/hosted-launches", json=body() | {"auto_session": False})
- ).status_code == 202
-
-
-async def test_lifecycle_owner_and_revision_guards(launch_app):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- request_id = created["request_id"]
- owner = app.identity["user"]
- app.identity["user"] = SimpleNamespace(id="someone-else")
- assert (await _lifecycle(app, request_id, "stop", 1)).status_code == 404
- app.identity["user"] = owner
- stopped = await _lifecycle(app, request_id, "stop", 1)
- assert stopped.json()["desired_state"] == "stopped"
- assert stopped.json()["state"] == "stopping"
- assert stopped.json()["revision"] == 2
- assert (await _lifecycle(app, request_id, "start", 1)).status_code == 409
- assert (await _lifecycle(app, request_id, "restart", 2)).status_code == 409
- assert (await _lifecycle(app, request_id, "retry", 2)).status_code == 409
- started = await _lifecycle(app, request_id, "start", 2)
- assert (started.json()["desired_state"], started.json()["state"]) == (
- "running",
- "queued",
- )
- await _update(app.factory, HostedLaunch, request_id, state="ready")
- restarted = await _lifecycle(app, request_id, "restart", 3)
- assert restarted.status_code == 200, restarted.text
- assert (restarted.json()["state"], restarted.json()["revision"]) == ("queued", 4)
-
-
-async def test_start_wakes_a_stopped_machine(launch_app):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- machine_id = created["machine_id"]
- await _update(
- app.factory,
- HostedMachine,
- machine_id,
- desired_state="stopped",
- stop_reason="idle",
- state="stopped",
- revision=2,
- )
- url = f"/hosted-launches/{created['request_id']}"
- assert (await app.client.get(url)).json()["sleeping"] is True
- listed = (await app.client.get("/hosted-launches")).json()
- assert [launch["sleeping"] for launch in listed] == [True]
- stopped = await _lifecycle(app, created["request_id"], "stop", 1)
- assert (stopped.json()["state"], stopped.json()["sleeping"]) == ("stopped", True)
- await _update(app.factory, HostedMachine, machine_id, stop_reason="owner")
- started = await _lifecycle(app, created["request_id"], "start", 2)
- assert started.json()["desired_state"] == "running"
- assert started.json()["sleeping"] is False
- machine = await _machine(app.factory, machine_id)
- assert (machine.desired_state, machine.stop_reason, machine.revision) == (
- "running",
- None,
- 3,
- )
-
-
-async def test_retry_resets_the_process_counters(launch_app):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- request_id = created["request_id"]
- await _update(
- app.factory,
- HostedLaunch,
- request_id,
- state="error",
- error="The agent keeps crashing.",
- error_code="agent_crashed",
- process_restarts=5,
- process_oom_kills=2,
- )
- failed = (await app.client.get(f"/hosted-launches/{request_id}")).json()
- assert (failed["process_restarts"], failed["oom_kills"]) == (5, 2)
- retried = await _lifecycle(app, request_id, "retry", 1)
- assert retried.status_code == 200, retried.text
- summary = retried.json()
- assert (summary["state"], summary["error"], summary["error_code"]) == (
- "queued",
- None,
- None,
- )
- assert (summary["process_restarts"], summary["oom_kills"]) == (0, 0)
- assert summary["agent_id"] == created["agent_id"]
-
-
-async def test_remove_in_any_state_retains_the_machine_only_when_empty(launch_app):
- app = launch_app
- app.config.hosted_agents_per_owner = 3
- first = (await app.client.post("/hosted-launches", json=body())).json()
- second = (await app.client.post("/hosted-launches", json=body("reviewer"))).json()
- machine_id = first["machine_id"]
-
- removed = await _lifecycle(app, first["request_id"], "remove", 1)
- assert removed.status_code == 200, removed.text
- assert (removed.json()["state"], removed.json()["name"]) == (
- "deleted",
- "removed:" + first["request_id"],
- )
- assert (await _machine(app.factory, machine_id)).desired_state == "running"
- async with app.factory() as session:
- assert await session.get(Agent, first["agent_id"]) is None
- assert (await _lifecycle(app, first["request_id"], "start", 2)).status_code == 409
-
- await _update(
- app.factory,
- HostedLaunch,
- second["request_id"],
- state="error",
- error_code="agent_failed",
- )
- last = await _lifecycle(app, second["request_id"], "remove", 1)
- assert last.status_code == 200, last.text
- machine = await _machine(app.factory, machine_id)
- assert machine.desired_state == "retained"
- retention = machine.retain_until - datetime.now(UTC)
- assert timedelta(days=7) - timedelta(minutes=1) < retention <= timedelta(days=7)
- async with app.factory() as session:
- assert await session.scalar(select(func.count()).select_from(Agent)) == 0
- reclaimed = await app.client.post("/hosted-launches", json=body("helper"))
- assert reclaimed.status_code == 202, reclaimed.text
- assert reclaimed.json()["machine_id"] == machine_id
- assert (await _machine(app.factory, machine_id)).desired_state == "running"
-
-
-async def test_session_operation_status_is_owner_scoped(launch_app):
- app = launch_app
- request = body()
- await app.client.post("/hosted-launches", json=request)
- operation_id = str(uuid4())
- async with app.factory() as session:
- session.add(
- HostedOperation(
- id=operation_id,
- launch_id=request["request_id"],
- launch_revision=1,
- session_id=str(uuid4()),
- action="start",
- )
- )
- await session.commit()
- url = f"/hosted-launches/{request['request_id']}/sessions/{operation_id}"
- found = await app.client.get(url)
- assert found.status_code == 200
- assert found.json()["state"] == "queued"
- app.identity["user"] = SimpleNamespace(id="someone-else")
- assert (await app.client.get(url)).status_code == 404
-
-
-async def test_stop_on_a_sleeping_machine_prevents_mention_wake(launch_app):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- await _update(
- app.factory,
- HostedMachine,
- created["machine_id"],
- desired_state="stopped",
- stop_reason="idle",
- state="stopped",
- revision=2,
- )
- stopped = await _lifecycle(app, created["request_id"], "stop", 1)
- assert stopped.status_code == 200
- assert stopped.json()["state"] == "stopped"
- async with app.factory() as session:
- launch, machine = await HostedLaunchStore().note_addressed(
- session, created["request_id"]
- )
- await session.commit()
- assert launch.desired_state == "stopped"
- assert launch.revision == 2
- assert (machine.desired_state, machine.revision) == ("stopped", 2)
-
-
-async def test_read_only_repository_cannot_create_cloud_agent(launch_app):
- app = launch_app
- app.access.return_value["installations"][0]["repositories"][0]["permissions"] = {
- "pull": True
- }
- result = await app.client.post("/hosted-launches", json=body())
- assert result.status_code == 422
- assert "needs write access" in result.json()["detail"]
-
-
-async def _interrupted_registration(app, monkeypatch, request: dict) -> str:
- """A launch whose agent id was recorded but whose agent was never registered."""
- register_agent = app.protocol.register_agent
- monkeypatch.setattr(
- app.protocol,
- "register_agent",
- AsyncMock(side_effect=RuntimeError("synthetic registration failure")),
- raising=False,
- )
- created = await app.client.post("/hosted-launches", json=request)
- assert created.status_code == 202, created.text
- monkeypatch.setattr(app.protocol, "register_agent", register_agent, raising=False)
- reserved = str(uuid4())
- await _update(
- app.factory,
- HostedLaunch,
- request["request_id"],
- state="queued",
- error=None,
- error_code=None,
- agent_id=reserved,
- )
- return reserved
-
-
-async def test_create_retry_registers_an_interrupted_identity(launch_app, monkeypatch):
- app = launch_app
- request = body()
- reserved = await _interrupted_registration(app, monkeypatch, request)
- retried = await app.client.post("/hosted-launches", json=request)
- assert retried.status_code == 202, retried.text
- assert retried.json()["agent_id"] == reserved
- async with app.factory() as session:
- agent = await session.get(Agent, reserved)
- assert agent is not None
- assert agent.metadata_["hosted_launch_id"] == request["request_id"]
- again = await app.client.post("/hosted-launches", json=request)
- assert again.json()["agent_id"] == reserved
- async with app.factory() as session:
- assert await session.scalar(select(func.count()).select_from(Agent)) == 1
-
-
- ("state", "action"), [("error", "retry"), ("stopped", "start")]
-)
-async def test_lifecycle_registers_an_interrupted_identity(
- launch_app, monkeypatch, state, action
-):
- app = launch_app
- request = body()
- reserved = await _interrupted_registration(app, monkeypatch, request)
- await _update(
- app.factory,
- HostedLaunch,
- request["request_id"],
- state=state,
- desired_state="running" if state == "error" else "stopped",
- )
- launch = await _launch(app.factory, request["request_id"])
- result = await _lifecycle(app, request["request_id"], action, launch.revision)
- assert result.status_code == 200, result.text
- assert result.json()["agent_id"] == reserved
- async with app.factory() as session:
- assert await session.get(Agent, reserved) is not None
-
-
-async def _ready_launch(app) -> dict:
- created = (await app.client.post("/hosted-launches", json=body())).json()
- await _update(app.factory, HostedLaunch, created["request_id"], state="ready")
- await _update(
- app.factory, HostedMachine, created["machine_id"], state="ready", revision=2
- )
- return created
-
-
-async def _session_operation(app, request_id: str, operation_id: str | None = None):
- return await app.client.post(
- f"/hosted-launches/{request_id}/sessions",
- json={
- "id": operation_id or str(uuid4()),
- "session_id": str(uuid4()),
- "action": "start",
- },
- )
-
-
-async def _operation_count(factory) -> int:
- async with factory() as session:
- return await session.scalar(select(func.count()).select_from(HostedOperation))
-
-
- ("machine", "code", "message"),
- [
- (
- {"state": "error"},
- "machine_error",
- "The cloud machine needs attention. Retry it in Switch Console.",
- ),
- (
- {"state": "error", "error_code": "machine_needs_attention"},
- "machine_error",
- "The cloud machine needs attention. Contact your server administrator.",
- ),
- (
- {"state": "stopped", "desired_state": "stopped", "stop_reason": "owner"},
- "machine_stopped",
- "The owner stopped the cloud machine. Start it in Switch Console.",
- ),
- (
- {"state": "provisioning"},
- "worker_waking",
- "The cloud machine is starting. Try again in a moment.",
- ),
- ],
-)
-async def test_session_operation_refuses_a_machine_that_is_not_ready(
- launch_app, machine, code, message
-):
- app = launch_app
- created = await _ready_launch(app)
- await _update(app.factory, HostedMachine, created["machine_id"], **machine)
- refused = await _session_operation(app, created["request_id"])
- assert refused.status_code == 409
- assert refused.json() == {"detail": message, "code": code}
- assert await _operation_count(app.factory) == 0
- assert (await _machine(app.factory, created["machine_id"])).revision == 2
-
-
-async def test_session_operation_wakes_a_sleeping_machine(launch_app):
- app = launch_app
- created = await _ready_launch(app)
- await _update(
- app.factory,
- HostedMachine,
- created["machine_id"],
- state="stopped",
- desired_state="stopped",
- stop_reason="idle",
- )
- before = await _launch(app.factory, created["request_id"])
- waking = await _session_operation(app, created["request_id"])
- assert waking.status_code == 409
- assert waking.json()["code"] == "worker_waking"
- machine = await _machine(app.factory, created["machine_id"])
- assert (machine.desired_state, machine.stop_reason, machine.revision) == (
- "running",
- None,
- 3,
- )
- assert (
- await _launch(app.factory, created["request_id"])
- ).active_at > before.active_at
- assert await _operation_count(app.factory) == 0
-
-
-async def test_session_operation_is_queued_on_a_ready_machine_and_replayed(
- launch_app,
-):
- app = launch_app
- created = await _ready_launch(app)
- operation_id = str(uuid4())
- body_ = {"id": operation_id, "session_id": str(uuid4()), "action": "start"}
- url = f"/hosted-launches/{created['request_id']}/sessions"
- queued = await app.client.post(url, json=body_)
- assert queued.status_code == 202, queued.text
- await _update(
- app.factory,
- HostedMachine,
- created["machine_id"],
- state="stopped",
- desired_state="stopped",
- stop_reason="owner",
- )
- replayed = await app.client.post(url, json=body_)
- assert replayed.status_code == 202
- assert replayed.json()["id"] == operation_id
-
-
-async def _restartable(app, action: str, stop_reason: str) -> dict:
- """A launch that `action` accepts, on a machine stopped for `stop_reason`."""
- created = (await app.client.post("/hosted-launches", json=body())).json()
- launch_state = {"start": "stopped", "restart": "ready", "retry": "error"}[action]
- await _update(app.factory, HostedLaunch, created["request_id"], state=launch_state)
- await _update(
- app.factory,
- HostedMachine,
- created["machine_id"],
- state="stopped",
- desired_state="stopped",
- stop_reason=stop_reason,
- revision=2,
- )
- return created
-
-
[email protected]("action", ["restart", "retry"])
-async def test_restart_and_retry_do_not_start_an_owner_stopped_machine(
- launch_app, action
-):
- app = launch_app
- created = await _restartable(app, action, "owner")
- refused = await _lifecycle(app, created["request_id"], action, 1)
- assert refused.status_code == 409
- assert refused.json() == {
- "detail": "The owner stopped the cloud machine. Start it in Switch Console.",
- "code": "machine_stopped",
- }
- machine = await _machine(app.factory, created["machine_id"])
- assert (machine.desired_state, machine.stop_reason, machine.revision) == (
- "stopped",
- "owner",
- 2,
- )
- assert (await _launch(app.factory, created["request_id"])).revision == 1
-
-
[email protected]("action", ["start", "restart", "retry"])
- "error_code,message",
- [
- (
- None,
- "The cloud machine needs attention. Retry it in Switch Console.",
- ),
- (
- "machine_needs_attention",
- "The cloud machine needs attention. Contact your server administrator.",
- ),
- ],
-)
-async def test_restart_and_retry_refuse_an_errored_machine_even_if_owner_stopped(
- launch_app, action, error_code, message
-):
- app = launch_app
- created = await _restartable(app, action, "owner")
- machine_updates = {"state": "error", "revision": 3}
- if error_code is not None:
- machine_updates["error_code"] = error_code
- await _update(app.factory, HostedMachine, created["machine_id"], **machine_updates)
- refused = await _lifecycle(app, created["request_id"], action, 1)
- assert refused.status_code == 409
- assert refused.json() == {
- "detail": message,
- "code": "machine_error",
- }
- machine = await _machine(app.factory, created["machine_id"])
- assert machine.state == "error"
- assert (await _launch(app.factory, created["request_id"])).revision == 1
-
-
[email protected]("action", ["restart", "retry"])
-async def test_restart_and_retry_wake_an_idle_sleeping_machine(launch_app, action):
- app = launch_app
- created = await _restartable(app, action, "idle")
- result = await _lifecycle(app, created["request_id"], action, 1)
- assert result.status_code == 200, result.text
- assert result.json()["state"] == "queued"
- machine = await _machine(app.factory, created["machine_id"])
- assert (machine.desired_state, machine.stop_reason, machine.revision) == (
- "running",
- None,
- 3,
- )
-
-
-def _configuration(
- definition: str = "---\nname: helper\nmodel: opus\n---\nBe brief.\n",
-):
- return {
- "instructions": "Be brief.",
- "definition": definition,
- "definition_attributes": {"model": "opus"},
- }
-
-
-async def test_owner_updates_the_configuration_the_next_start_runs(launch_app):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- url = f"/hosted-launches/{created['request_id']}/configuration"
- assert (await app.client.get(url)).json() == {
- "description": "Repository helper",
- "instructions": "",
- "definition_attributes": {"model": "sonnet"},
- }
- updated = await app.client.put(url, json=_configuration())
- assert updated.status_code == 200, updated.text
- assert updated.json() == {
- "description": "Repository helper",
- "instructions": "Be brief.",
- "definition_attributes": {"model": "opus"},
- }
- assert (await app.client.get(url)).json() == updated.json()
- launch = await _launch(app.factory, created["request_id"])
- assert launch.revision == 1
- assert launch.spec["definition"] == _configuration()["definition"].strip()
- assert launch.spec["session_limit"] == 2
- assert launch.spec["name"] == "helper"
- await _lifecycle(app, created["request_id"], "stop", 1)
- started = await _lifecycle(app, created["request_id"], "start", 2)
- assert started.json()["revision"] == 3
- launch = await _launch(app.factory, created["request_id"])
- assert launch.spec["instructions"] == "Be brief."
- assert launch.spec["definition_attributes"] == {"model": "opus"}
-
-
-async def test_configuration_is_owner_scoped(launch_app):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- url = f"/hosted-launches/{created['request_id']}/configuration"
- app.identity["user"] = SimpleNamespace(id="someone-else")
- assert (await app.client.get(url)).status_code == 404
- assert (await app.client.put(url, json=_configuration())).status_code == 404
- launch = await _launch(app.factory, created["request_id"])
- assert launch.spec["instructions"] == ""
-
-
-async def test_configuration_of_a_removed_launch_is_refused(launch_app):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- await _lifecycle(app, created["request_id"], "remove", 1)
- response = await app.client.put(
- f"/hosted-launches/{created['request_id']}/configuration",
- json=_configuration(),
- )
- assert response.status_code == 409
-
-
- "change",
- [
- {"definition": ""},
- {"instructions": "x" * 32769},
- {"instructions": "x" * 20000, "definition": "y" * 20000},
- {"definition_attributes": "opus"},
- {"model": "opus"},
- ],
-)
-async def test_invalid_configuration_is_refused(launch_app, change):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- response = await app.client.put(
- f"/hosted-launches/{created['request_id']}/configuration",
- json=_configuration() | change,
- )
- assert response.status_code == 422
- launch = await _launch(app.factory, created["request_id"])
- assert launch.spec["definition_attributes"] == {"model": "sonnet"}
-
-
-async def test_identity_edits_keep_the_launch_spec_that_registers_it_again(
- launch_app,
-):
- app = launch_app
- created = (await app.client.post("/hosted-launches", json=body())).json()
- agent_id = created["agent_id"]
- owner = app.identity["user"]
- policy = {"rules": [{"users": [], "agents": [], "owner": True}]}
- async with app.factory() as session:
- await update_agent_display_name(
- agent_id,
- UpdateAgentDisplayNameRequest(display_name="Helper"),
- session,
- AgentStore(),
- owner,
- False,
- )
- await update_agent_icon(
- agent_id,
- UpdateAgentIconRequest(icon_url=ICON),
- session,
- AgentStore(),
- owner,
- False,
- )
- await update_addressing_policy(
- agent_id,
- UpdateAddressingPolicyRequest.model_validate({"policy": policy}),
- session,
- AgentStore(),
- RoomStore(),
- UserStore(),
- SimpleNamespace(
- agent_session_store=AgentSessionStore(),
- room_role_store=RoomRoleStore(),
- connections=AgentConnectionRegistry(),
- ),
- owner,
- False,
- )
- agent = await session.get(Agent, agent_id, populate_existing=True)
- assert agent is not None
- stored_policy = agent.addressing_policy
- launch = await _launch(app.factory, created["request_id"])
- assert launch.spec["display_name"] == "Helper"
- assert launch.spec["icon_url"] == ICON
- assert launch.spec["addressing_policy"] == stored_policy
- assert launch.spec["definition_attributes"] == {"model": "sonnet"}
diff --git a/core/tests/switch_core/gateway/test_hosted_machines.py b/core/tests/switch_core/gateway/test_hosted_machines.py
index 32fe0664c..0febe3098 100644
--- a/core/tests/switch_core/gateway/test_hosted_machines.py
+++ b/core/tests/switch_core/gateway/test_hosted_machines.py
@@ -1,65 +1,48 @@
-"""The owner's cloud machine: its summary, and stopping, starting and retrying it."""
+"""The owner's cloud machine: its summary, ensuring it, and stopping, starting
+and retrying it."""
from __future__ import annotations
import json
-from datetime import UTC, datetime, timedelta
+from datetime import UTC, datetime
from pathlib import Path
from types import SimpleNamespace
import pytest
-from switch_core.db.models import HostedWakeMailbox
+from switch_core.db.models import HostedMachine, require_tenant_id
+from switch_core.db.stores.hosted_machine_store import MACHINES_FULL
from switch_core.gateway.auth import get_current_user
-from switch_core.gateway.hosted_machines import router as machine_router
from tests.switch_core.gateway.test_hosted_controller import ( # noqa: F401
+ ControllerApp,
controller_app,
+ enrolled,
machine_of,
observe,
-)
-from tests.switch_core.gateway.test_hosted_mailbox import ( # noqa: F401
- address,
- addressed,
- attach,
- attached_conn,
- mailbox_app,
- rows,
- set_launch,
-)
-from tests.switch_core.gateway.test_hosted_supervisor import ( # noqa: F401
- heartbeat_body,
- supervisor,
-)
-from tests.switch_core.gateway.test_hosted_workers import ( # noqa: F401
- _launch,
- _machine,
- set_machine,
- worker_app,
+ report_status,
+ update_machine,
)
FIXTURES = Path(__file__).parent.parent / "fixtures" / "hosted_machines"
HEARTBEAT = {
- "disk": {
- "path": "/data",
- "total_bytes": 214748364800,
- "used_bytes": 10737418240,
- "available_bytes": 204010946560,
- },
+ "disk": {"total_bytes": 214748364800, "available_bytes": 204010946560},
"memory": {"total_bytes": 17179869184, "available_bytes": 12884901888},
+ "sessions_running": 0,
}
-async def _lifecycle(app, action: str, revision: int):
+async def _lifecycle(app: ControllerApp, action: str, revision: int):
return await app.client.post(
f"/hosted-machines/{app.machine_id}/lifecycle",
json={"action": action, "revision": revision},
)
-async def test_summary_has_the_contract_shape(mailbox_app): # noqa: F811
- app = mailbox_app
+async def test_summary_has_the_contract_shape(controller_app): # noqa: F811
+ app = controller_app
+ controller_id, agent_ids = await enrolled(app, agents=1)
expected = json.loads((FIXTURES / "machine_summary_sleeping.json").read_text())
- await set_machine(
+ await update_machine(
app.factory,
app.machine_id,
state="stopped",
@@ -76,17 +59,13 @@ async def test_summary_has_the_contract_shape(mailbox_app): # noqa: F811
assert set(summary) == set(expected)
assert set(summary["disk"]) == set(expected["disk"])
assert set(summary["memory"]) == set(expected["memory"])
- assert {
- key: value
- for key, value in summary.items()
- if key not in ("machine_id", "heartbeat_at", "agents")
- } == {
- key: value
- for key, value in expected.items()
- if key not in ("machine_id", "heartbeat_at", "agents")
+ varying = ("machine_id", "heartbeat_at", "agents", "controller_id")
+ assert {key: value for key, value in summary.items() if key not in varying} == {
+ key: value for key, value in expected.items() if key not in varying
}
assert summary["machine_id"] == app.machine_id
- assert summary["agents"] == [app.request_id]
+ assert summary["agents"] == agent_ids
+ assert summary["controller_id"] == controller_id
assert datetime.fromisoformat(summary["heartbeat_at"]) == datetime(
2026, 1, 1, tzinfo=UTC
)
@@ -94,22 +73,45 @@ async def test_summary_has_the_contract_shape(mailbox_app): # noqa: F811
assert single.json() == summary
-async def test_a_machine_without_a_heartbeat_reports_no_usage(mailbox_app): # noqa: F811
- app = mailbox_app
+async def test_a_machine_that_never_enrolled_reports_no_usage_or_agents(
+ controller_app, # noqa: F811
+):
+ app = controller_app
summary = (await app.client.get(f"/hosted-machines/{app.machine_id}")).json()
- assert (summary["disk"], summary["memory"], summary["heartbeat_at"]) == (
- None,
- None,
- None,
+ assert (
+ summary["disk"],
+ summary["memory"],
+ summary["heartbeat_at"],
+ summary["agents"],
+ summary["controller_id"],
+ summary["sleeping"],
+ ) == (None, None, None, [], None, False)
+
+
+async def test_the_summary_lists_the_agents_placed_on_its_controller(
+ controller_app, # noqa: F811
+):
+ app = controller_app
+ controller_id, agent_ids = await enrolled(app, agents=2)
+ await report_status(
+ app,
+ controller_id,
+ disk_free_bytes=10,
+ disk_total_bytes=100,
+ mem_free_bytes=5,
+ mem_total_bytes=50,
+ sessions_running=0,
)
- assert summary["sleeping"] is False
- await set_launch(app, state="deleted", desired_state="deleted")
summary = (await app.client.get(f"/hosted-machines/{app.machine_id}")).json()
- assert summary["agents"] == []
+ assert summary["agents"] == sorted(agent_ids)
+ assert summary["controller_id"] == controller_id
+ assert summary["disk"] == {"total_bytes": 100, "available_bytes": 10}
+ assert summary["memory"] == {"total_bytes": 50, "available_bytes": 5}
+ assert summary["heartbeat_at"] is not None
-async def test_other_owners_machines_read_as_missing(mailbox_app): # noqa: F811
- app = mailbox_app
+async def test_other_owners_machines_read_as_missing(controller_app): # noqa: F811
+ app = controller_app
fastapi_app = app.client._transport.app
fastapi_app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
id="someone-else"
@@ -119,31 +121,11 @@ async def test_other_owners_machines_read_as_missing(mailbox_app): # noqa: F811
await app.client.get(f"/hosted-machines/{app.machine_id}")
).status_code == 404
assert (await _lifecycle(app, "stop", 1)).status_code == 404
- assert (await _machine(app.factory, app.machine_id)).desired_state == "running"
+ assert (await machine_of(app.factory, app.machine_id)).desired_state == "running"
-async def test_stop_cancels_the_mailbox_and_start_resumes(mailbox_app): # noqa: F811
- app = mailbox_app
- first, second = app.rooms
- response, _ = await attach(app)
- conn = attached_conn(app)
- await address(app, addressed(first, "$m1"))
- async with app.factory() as session:
- session.add(
- HostedWakeMailbox(
- agent_id=app.agent_id,
- room_id=second,
- message_id="$n1",
- launch_id=app.request_id,
- thread_id="$thread-x",
- event={},
- addressed_at=datetime.now(UTC),
- updated_at=datetime.now(UTC),
- expires_at=datetime.now(UTC) + timedelta(hours=24),
- )
- )
- await session.commit()
-
+async def test_stop_then_start(controller_app): # noqa: F811
+ app = controller_app
stopped = await _lifecycle(app, "stop", 1)
assert stopped.status_code == 200, stopped.text
machine = stopped.json()["machine"]
@@ -153,20 +135,6 @@ async def test_stop_cancels_the_mailbox_and_start_resumes(mailbox_app): # noqa:
machine["sleeping"],
machine["revision"],
) == ("stopped", "owner", False, 2)
- assert await rows(app) == {
- (first, "$m1"): "cancel_requested",
- (second, "$n1"): "cancelled",
- }
- assert [(room, thread) for room, thread, _ in app.sent] == [(second, "$thread-x")]
- assert "stopped before I processed" in app.sent[0][2]
- cancels = [
- data for event, data in conn.worker_frames.drain() if event == "mailbox_cancel"
- ]
- assert [(e["room_id"], e["message_id"]) for e in cancels[0]["entries"]] == [
- (first, "$m1")
- ]
- launch = await _launch(app.factory, app.request_id)
- assert launch.desired_state == "running"
assert (await _lifecycle(app, "start", 1)).status_code == 409
started = await _lifecycle(app, "start", 2)
@@ -179,9 +147,9 @@ async def test_stop_cancels_the_mailbox_and_start_resumes(mailbox_app): # noqa:
)
-async def test_revision_leniency_only_while_idle_sleeping(mailbox_app): # noqa: F811
- app = mailbox_app
- await set_machine(
+async def test_revision_leniency_only_while_idle_sleeping(controller_app): # noqa: F811
+ app = controller_app
+ await update_machine(
app.factory,
app.machine_id,
desired_state="stopped",
@@ -198,10 +166,10 @@ async def test_revision_leniency_only_while_idle_sleeping(mailbox_app): # noqa:
assert (await _lifecycle(app, "stop", 5)).json() == {"detail": "revision mismatch"}
-async def test_retry_only_from_error(mailbox_app): # noqa: F811
- app = mailbox_app
+async def test_retry_only_from_error(controller_app): # noqa: F811
+ app = controller_app
assert (await _lifecycle(app, "retry", 1)).status_code == 409
- await set_machine(
+ await update_machine(
app.factory,
app.machine_id,
state="error",
@@ -219,9 +187,9 @@ async def test_retry_only_from_error(mailbox_app): # noqa: F811
assert machine["revision"] == 2
-async def test_retry_requeues_the_retire_of_a_machine_in_error(mailbox_app): # noqa: F811
- app = mailbox_app
- await set_machine(
+async def test_retry_requeues_the_retire_of_a_machine_in_error(controller_app): # noqa: F811
+ app = controller_app
+ await update_machine(
app.factory,
app.machine_id,
state="error",
@@ -252,41 +220,144 @@ async def test_retry_requeues_the_retire_of_a_machine_in_error(mailbox_app): #
],
)
@pytest.mark.parametrize("action", ["stop", "start", "retry"])
-async def test_a_retired_machine_refuses_lifecycle(mailbox_app, values, action): # noqa: F811
- app = mailbox_app
- await set_machine(app.factory, app.machine_id, **values)
+async def test_a_retired_machine_refuses_lifecycle(controller_app, values, action): # noqa: F811
+ app = controller_app
+ await update_machine(app.factory, app.machine_id, **values)
refused = await _lifecycle(app, action, 1)
assert refused.status_code == 409
-async def test_stop_then_start_before_the_stop_lands_waits_for_a_fresh_heartbeat(
- supervisor, # noqa: F811
+async def test_stop_then_start_before_the_stop_lands_waits_for_a_fresh_status_report(
+ controller_app, # noqa: F811
):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- client._transport.app.include_router(machine_router)
- heartbeat = f"/hosted/machines/{machine_id}/heartbeat"
- await observe(client, machine_id, state="running", revision=1)
- await client.post(heartbeat, headers=headers, json=heartbeat_body())
- assert (await machine_of(factory, request_id)).state == "ready"
-
- lifecycle = f"/hosted-machines/{machine_id}/lifecycle"
- stopped = await client.post(lifecycle, json={"action": "stop", "revision": 1})
+ app = controller_app
+ controller_id, _ = await enrolled(app, agents=1)
+ await observe(app.client, app.machine_id, state="running", revision=1)
+ await report_status(app, controller_id, sessions_running=0)
+ assert (await machine_of(app.factory, app.machine_id)).state == "ready"
+
+ stopped = await _lifecycle(app, "stop", 1)
assert stopped.status_code == 200, stopped.text
- started = await client.post(lifecycle, json={"action": "start", "revision": 2})
+ started = await _lifecycle(app, "start", 2)
assert started.status_code == 200, started.text
assert (
started.json()["machine"]["state"],
started.json()["machine"]["revision"],
- ) == (
- "provisioning",
- 3,
- )
+ ) == ("provisioning", 3)
- await observe(client, machine_id, state="running", revision=3)
- observed = await machine_of(factory, request_id)
+ await report_status(app, controller_id, sessions_running=0)
+ assert (await machine_of(app.factory, app.machine_id)).state == "provisioning"
+ await observe(app.client, app.machine_id, state="running", revision=3)
+ observed = await machine_of(app.factory, app.machine_id)
assert (observed.state, observed.running_observed_at is not None) == (
"provisioning",
True,
)
- await client.post(heartbeat, headers=headers, json=heartbeat_body())
- assert (await machine_of(factory, request_id)).state == "ready"
+ await report_status(app, controller_id, sessions_running=0)
+ assert (await machine_of(app.factory, app.machine_id)).state == "ready"
+
+
+async def _ensure(app: ControllerApp):
+ return await app.client.post("/hosted-machines/ensure")
+
+
+async def test_ensure_returns_the_owner_s_machine_and_claims_one_when_missing(
+ controller_app, # noqa: F811
+):
+ app = controller_app
+ reused = await _ensure(app)
+ assert reused.status_code == 200, reused.text
+ assert (reused.json()["machine_id"], reused.json()["state"]) == (
+ app.machine_id,
+ "queued",
+ )
+ await update_machine(
+ app.factory, app.machine_id, state="deleted", desired_state="deleted"
+ )
+ claimed = await _ensure(app)
+ assert claimed.status_code == 200, claimed.text
+ body = claimed.json()
+ assert body["machine_id"] != app.machine_id
+ assert (body["state"], body["desired_state"], body["agents"]) == (
+ "queued",
+ "running",
+ [],
+ )
+ async with app.factory() as session:
+ machine = await session.get(
+ HostedMachine, (require_tenant_id(), body["machine_id"])
+ )
+ assert machine is not None
+
+
+async def test_ensure_wakes_a_sleeping_machine_but_not_one_its_owner_stopped(
+ controller_app, # noqa: F811
+):
+ app = controller_app
+ await update_machine(
+ app.factory,
+ app.machine_id,
+ state="stopped",
+ desired_state="stopped",
+ stop_reason="idle",
+ revision=2,
+ )
+ woken = (await _ensure(app)).json()
+ assert (woken["desired_state"], woken["stop_reason"], woken["revision"]) == (
+ "running",
+ None,
+ 3,
+ )
+ await update_machine(
+ app.factory, app.machine_id, desired_state="stopped", stop_reason="owner"
+ )
+ left = (await _ensure(app)).json()
+ assert (left["desired_state"], left["stop_reason"], left["revision"]) == (
+ "stopped",
+ "owner",
+ 3,
+ )
+
+
+async def test_ensure_returns_a_machine_in_error_as_it_is(controller_app): # noqa: F811
+ app = controller_app
+ await update_machine(
+ app.factory, app.machine_id, state="error", error="The instance failed."
+ )
+ response = await _ensure(app)
+ assert response.status_code == 200, response.text
+ assert (response.json()["state"], response.json()["revision"]) == ("error", 1)
+
+
+async def test_ensure_refuses_when_every_machine_is_in_use(controller_app): # noqa: F811
+ app = controller_app
+ app.config.hosted_launch_capacity = 1
+ fastapi_app = app.client._transport.app
+ fastapi_app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
+ id="someone-else"
+ )
+ refused = await _ensure(app)
+ assert refused.status_code == 409
+ assert refused.json() == {"detail": MACHINES_FULL}
+
+
[email protected]("disabled", ["capacity", "settings", "tenant"])
+async def test_ensure_is_unavailable_when_machines_are_not_enabled(
+ controller_app, # noqa: F811
+ disabled,
+):
+ app = controller_app
+ fastapi_app = app.client._transport.app
+ if disabled == "capacity":
+ app.config.hosted_launch_capacity = 0
+ elif disabled == "settings":
+ fastapi_app.state.hosted_controller_settings = None
+ else:
+ fastapi_app.state.hosted_controller_settings = app.settings.model_copy(
+ update={"tenant_id": "another-tenant"}
+ )
+ response = await _ensure(app)
+ assert response.status_code == 503
+ assert response.json() == {
+ "detail": "Switch cloud machines are not enabled on this server."
+ }
diff --git a/core/tests/switch_core/gateway/test_hosted_mailbox.py b/core/tests/switch_core/gateway/test_hosted_mailbox.py
deleted file mode 100644
index 836eed41d..000000000
--- a/core/tests/switch_core/gateway/test_hosted_mailbox.py
+++ /dev/null
@@ -1,1146 +0,0 @@
-"""The wake mailbox end to end: addressed while asleep, woken, delivered once."""
-
-from __future__ import annotations
-
-from datetime import UTC, datetime, timedelta
-from types import SimpleNamespace
-from typing import Any
-from uuid import uuid4
-
-import pytest
-from sqlalchemy import insert, select
-
-from switch_core.bridges.agent.api.hosted_worker_routes import post_mailbox_notices
-from switch_core.bridges.agent.hosted_mailbox import mailbox_upkeep
-from switch_core.bridges.agent.protocol.agent_connections import (
- TAKEN_OVER,
- AgentConnectionRegistry,
-)
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.types import (
- AgentEvent,
- MessagePayload,
- TaskDelegatePayload,
-)
-from switch_core.clients.agent_consumer import (
- _HOSTED_ERROR_MESSAGE,
- _HOSTED_MACHINE_ERROR_MESSAGE,
- _HOSTED_MACHINE_STOPPED_MESSAGE,
- _HOSTED_REMOVED_MESSAGE,
- _HOSTED_STOPPED_MESSAGE,
- AUTO_REPLY_FLAG,
- AgentConsumer,
- _GateOutcome,
- _hosted_unavailable,
-)
-from switch_core.db.models import (
- Agent,
- ApiKey,
- Client,
- HostedLaunch,
- HostedWakeMailbox,
- Message,
- ProviderConnection,
- Room,
- require_tenant_id,
-)
-from switch_core.db.stores.agent_store import AgentStore
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
-from switch_core.db.stores.hosted_mailbox_store import MAILBOX_LIMIT, HostedMailboxStore
-from switch_core.events import TaskDelegate
-from switch_core.transport import InboundMedia, InboundMessage, RoomRef
-from tests.switch_core.gateway.test_hosted_workers import ( # noqa: F401
- _agent,
- _first_frames,
- _launch,
- _machine,
- _open,
- issue_capability,
- set_machine,
- worker_app,
-)
-
-
-async def mailbox_app(worker_app): # noqa: F811
- """The worker app plus a room with addressed messages and a recorded send."""
- client, request_id, agent_id, service, factory, prepared = worker_app
- async with factory() as session:
- rooms = []
- for index in range(2):
- room = Room(
- transport_room_id=f"!{uuid4().hex[:8]}:example.com",
- name=f"r{index}",
- description="",
- )
- session.add(room)
- await session.flush()
- rooms.append(room.id)
- names = ("$m1", "$m2", "$m3") if index == 0 else ("$n1", "$n2", "$n3")
- for seq, message_id in enumerate(names, start=1):
- session.add(
- Message(
- seq=seq,
- room_id=room.id,
- transport_event_id=message_id,
- sender_id="@someone:example.com",
- event_type="m.room.message",
- msgtype="m.text",
- body="hello",
- content={"body": "hello"},
- )
- )
- sender = await session.scalar(
- select(Client.transport_user_id)
- .join(Agent, Agent.client_id == Client.id)
- .where(Agent.id == agent_id)
- )
- await session.commit()
- sent: list[tuple[str, str | None, str]] = []
-
- async def send_message(agent, room, body, *, thread_id, extra_content):
- sent.append((room, thread_id, body))
- async with factory() as session:
- session.add(
- Message(
- seq=100 + len(sent),
- room_id=room,
- transport_event_id=f"$notice{len(sent)}",
- sender_id=sender,
- event_type="m.room.message",
- msgtype="m.notice",
- body=body,
- content={"body": body, **extra_content},
- )
- )
- await session.commit()
-
- service.send_message = send_message
- return SimpleNamespace(
- client=client,
- request_id=request_id,
- agent_id=agent_id,
- machine_id=prepared["machine_id"],
- service=service,
- factory=factory,
- rooms=rooms,
- sent=sent,
- )
-
-
-def addressed(room_id: str, message_id: str, thread_id: str | None = None):
- return AgentEvent(
- type="message",
- room_id=room_id,
- bridge_id=None,
- channel_type=None,
- payload=MessagePayload(
- addressed=True,
- sender="@someone:example.com",
- sender_name="Someone",
- message_id=message_id,
- body="hello",
- timestamp=1700000000000,
- thread_id=thread_id,
- ),
- )
-
-
-async def address(app, event: AgentEvent | None) -> Any:
- """`AgentConsumer._note_hosted_addressed`, as the agent's Matrix client runs it."""
- client = SimpleNamespace(
- _hosted_launch_store=HostedLaunchStore(),
- session_factory=app.factory,
- tenant_id=require_tenant_id(),
- _connections=app.service.connections,
- _event_buffer=app.service.event_buffer,
- )
- agent = await _agent(app.factory, app.agent_id)
- return await AgentConsumer._note_hosted_addressed(client, agent, event) # type: ignore[arg-type]
-
-
-async def set_launch(app, **values: Any) -> None:
- async with app.factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), app.request_id))
- assert launch is not None
- for key, value in values.items():
- setattr(launch, key, value)
- await session.commit()
-
-
-async def rows(app) -> dict[tuple[str, str], str]:
- async with app.factory() as session:
- found = await session.execute(
- select(
- HostedWakeMailbox.room_id,
- HostedWakeMailbox.message_id,
- HostedWakeMailbox.state,
- ).where(HostedWakeMailbox.tenant_id == require_tenant_id())
- )
- return {(room, message): state for room, message, state in found}
-
-
-async def capability(app) -> str:
- return await issue_capability(app.factory, app.request_id)
-
-
-def restart_core(app, boot: int) -> None:
- """A new Core process: nothing in memory survives, the database does."""
- app.service.connections = AgentConnectionRegistry()
- app.service.event_buffer = EventBuffer(sequence_base=boot << 32)
-
-
-async def attach(app, *, boot_id: str = "boot-a") -> tuple[Any, list[tuple[str, dict]]]:
- agent = await _agent(app.factory, app.agent_id)
- connection_id = str(uuid4())
- response = await _open(
- app.service,
- agent,
- capability=await capability(app),
- connection_id=connection_id,
- boot_id=boot_id,
- )
- conn = app.service.connections.get(connection_id)
- return response, conn.worker_frames.drain()
-
-
-async def ack(app, conn, *entries: tuple[str, str, str]):
- return await app.client.post(
- f"/agents/{app.agent_id}/connection/mailbox/ack",
- json={
- "connection_id": conn.id,
- "generation": conn.stream_generation,
- "entries": [
- {"room_id": room, "message_id": message, "outcome": outcome}
- for room, message, outcome in entries
- ],
- },
- )
-
-
-def wake_entries(frames: list[tuple[str, dict]]) -> list[tuple[str, str]]:
- return [
- (entry["room_id"], entry["message_id"])
- for event, data in frames
- if event == "wake"
- for entry in data["entries"]
- ]
-
-
-def attached_conn(app):
- conn = app.service.connections.attached_worker(app.agent_id)
- assert conn is not None
- return conn
-
-
-async def sleep_machine(app) -> None:
- """The machine idled out: stopped for idleness, its launch left running."""
- await set_machine(
- app.factory,
- app.machine_id,
- desired_state="stopped",
- stop_reason="idle",
- state="stopped",
- revision=2,
- )
- await set_launch(app, state="stopped")
-
-
[email protected]("restart", ["before_attach", "after_wake", "after_ack"])
-async def test_sleep_mention_wake_delivers_once(mailbox_app, restart):
- app = mailbox_app
- room = app.rooms[0]
- await sleep_machine(app)
- noted = await address(app, addressed(room, "$m1", "$thread"))
- assert noted.deliver is True and noted.refusal is None
- woken = await _machine(app.factory, app.machine_id)
- assert (woken.desired_state, woken.stop_reason, woken.revision) == (
- "running",
- None,
- 3,
- )
- launch = await _launch(app.factory, app.request_id)
- assert (launch.desired_state, launch.revision) == ("running", 1)
- assert await rows(app) == {(room, "$m1"): "pending"}
-
- if restart == "before_attach":
- restart_core(app, 2)
- _, frames = await attach(app)
- assert [event for event, _ in frames] == ["worker_attached", "wake"]
- (wake,) = [data for event, data in frames if event == "wake"]
- assert wake["entries"][0]["thread_id"] == "$thread"
- assert wake["entries"][0]["event"]["type"] == "message"
- assert wake["entries"][0]["event"]["payload"]["message_id"] == "$m1"
- assert await rows(app) == {(room, "$m1"): "offered"}
-
- if restart == "after_wake":
- restart_core(app, 3)
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert await rows(app) == {(room, "$m1"): "pending"}
- _, frames = await attach(app)
- assert wake_entries(frames) == [(room, "$m1")]
-
- conn = attached_conn(app)
- journaled = await ack(app, conn, (room, "$m1", "journaled"))
- assert journaled.status_code == 200, journaled.text
- assert journaled.json()["entries"] == [
- {"room_id": room, "message_id": "$m1", "state": "accepted"}
- ]
-
- if restart == "after_ack":
- restart_core(app, 4)
- await mailbox_upkeep(app.service, datetime.now(UTC))
- _, frames = await attach(app)
- assert wake_entries(frames) == []
- conn = attached_conn(app)
- admitted = await ack(
- app, conn, (room, "$m1", "admitted"), (room, "$m1", "journaled")
- )
- assert [e["state"] for e in admitted.json()["entries"]] == ["admitted", "admitted"]
- assert await rows(app) == {(room, "$m1"): "admitted"}
- assert app.sent == []
-
-
-async def test_core_crash_after_offer_reclaims(mailbox_app):
- app = mailbox_app
- room = app.rooms[0]
- _, frames = await attach(app)
- assert wake_entries(frames) == []
- noted = await address(app, addressed(room, "$m1"))
- assert noted.deliver is True
- assert await rows(app) == {(room, "$m1"): "offered"}
- async with app.factory() as session:
- row = await session.get(
- HostedWakeMailbox, (require_tenant_id(), app.agent_id, room, "$m1")
- )
- assert row is not None and row.offered_to is not None
- assert row.offered_to.startswith("1:")
-
- restart_core(app, 2)
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert await rows(app) == {(room, "$m1"): "pending"}
- _, frames = await attach(app)
- assert wake_entries(frames) == [(room, "$m1")]
-
-
-async def test_stream_killed_before_journal_is_offered_again(mailbox_app):
- app = mailbox_app
- room = app.rooms[0]
- _, _ = await attach(app)
- first = attached_conn(app)
- await address(app, addressed(room, "$m1"))
- app.service.connections.close(first.id, TAKEN_OVER)
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert await rows(app) == {(room, "$m1"): "pending"}
- _, frames = await attach(app, boot_id="boot-b")
- assert wake_entries(frames) == [(room, "$m1")]
- stale = await ack(app, first, (room, "$m1", "journaled"))
- assert stale.status_code == 409
-
-
-async def test_live_upkeep_reoffers_a_lapsed_lease(mailbox_app):
- app = mailbox_app
- room = app.rooms[0]
- await attach(app)
- conn = attached_conn(app)
- await address(app, addressed(room, "$m1"))
- async with app.factory() as session:
- row = await session.get(
- HostedWakeMailbox, (require_tenant_id(), app.agent_id, room, "$m1")
- )
- assert row is not None
- row.offered_until = datetime.now(UTC) - timedelta(seconds=1)
- await session.commit()
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert await rows(app) == {(room, "$m1"): "offered"}
- assert wake_entries(conn.worker_frames.drain()) == [(room, "$m1")]
-
-
-async def test_redelivered_event_is_not_delivered_twice(mailbox_app):
- app = mailbox_app
- room = app.rooms[0]
- assert (await address(app, addressed(room, "$m1"))).deliver is True
- assert (await address(app, addressed(room, "$m1"))).deliver is False
-
-
-async def test_task_delegate_is_written_under_its_hash(mailbox_app):
- app = mailbox_app
- room = app.rooms[0]
- event = AgentEvent(
- type="task_delegate",
- room_id=room,
- bridge_id=None,
- channel_type=None,
- payload=TaskDelegatePayload(
- task_id="task-1",
- requester_agent_id="agent-a",
- performer_agent_id=app.agent_id,
- summary="Do it",
- description="",
- ),
- )
- await address(app, event)
- ((key, state),) = (await rows(app)).items()
- assert key[1].startswith("task_delegate:") and state == "pending"
-
-
-async def test_commands_are_never_written(mailbox_app):
- app = mailbox_app
- await sleep_machine(app)
- noted = await address(app, None)
- assert noted.machine.desired_state == "running"
- assert noted.machine.revision == 3
- assert await rows(app) == {}
-
-
- ("state", "refusal"),
- [
- ({"desired_state": "stopped", "state": "stopped"}, _HOSTED_STOPPED_MESSAGE),
- ({"desired_state": "deleted", "state": "deleting"}, _HOSTED_REMOVED_MESSAGE),
- ({"state": "error"}, _HOSTED_ERROR_MESSAGE),
- ],
-)
-async def test_not_written_when_stopped_deleted_or_failed(mailbox_app, state, refusal):
- app = mailbox_app
- await set_launch(app, **state)
- noted = await address(app, addressed(app.rooms[0], "$m1"))
- assert (noted.refusal, noted.deliver) == (refusal, False)
- assert await rows(app) == {}
-
-
-async def test_wake_refused_while_provider_revoked(mailbox_app):
- app = mailbox_app
- await sleep_machine(app)
- async with app.factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), app.request_id))
- assert launch is not None
- connection = await session.get(
- ProviderConnection, (require_tenant_id(), launch.owner_id, "claude")
- )
- await session.delete(connection)
- await session.commit()
- noted = await address(app, addressed(app.rooms[0], "$m1"))
- assert noted.deliver is False
- assert "reconnect the provider" in noted.refusal
- machine = await _machine(app.factory, app.machine_id)
- assert (machine.desired_state, machine.stop_reason, machine.revision) == (
- "stopped",
- "idle",
- 2,
- )
- assert await rows(app) == {}
-
-
-async def test_full_mailbox_refuses_loudly(mailbox_app):
- app = mailbox_app
- now = datetime.now(UTC)
- async with app.factory() as session:
- await session.execute(
- insert(HostedWakeMailbox),
- [
- {
- "tenant_id": require_tenant_id(),
- "agent_id": app.agent_id,
- "room_id": app.rooms[1],
- "message_id": f"$filler{index}",
- "launch_id": app.request_id,
- "event": {},
- "addressed_at": now,
- "updated_at": now,
- "expires_at": now + timedelta(hours=24),
- }
- for index in range(MAILBOX_LIMIT)
- ],
- )
- await session.commit()
- noted = await address(app, addressed(app.rooms[0], "$m1"))
- assert noted.deliver is False
- assert f"{MAILBOX_LIMIT} messages waiting" in noted.refusal
- assert (app.rooms[0], "$m1") not in await rows(app)
-
-
-async def test_attach_leaves_what_does_not_fit_pending(mailbox_app, monkeypatch):
- app = mailbox_app
- room = app.rooms[0]
- for message_id in ("$m1", "$m2", "$m3"):
- await address(app, addressed(room, message_id))
- monkeypatch.setattr(
- "switch_core.bridges.agent.hosted_mailbox.WAKE_ENTRIES_PER_FRAME", 1
- )
- monkeypatch.setattr(
- "switch_core.bridges.agent.protocol.hosted_workers.FRAME_QUEUE_FRAMES", 2
- )
- _, frames = await attach(app)
- assert wake_entries(frames) == [(room, "$m1"), (room, "$m2")]
- assert await rows(app) == {
- (room, "$m1"): "offered",
- (room, "$m2"): "offered",
- (room, "$m3"): "pending",
- }
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert wake_entries(attached_conn(app).worker_frames.drain()) == [(room, "$m3")]
-
-
-async def test_idle_evidence_counts_the_mailbox(mailbox_app):
- app = mailbox_app
- room = app.rooms[0]
- await attach(app)
- await address(app, addressed(room, "$m1"))
-
- async def reasons() -> list[str]:
- async with app.factory() as session:
- launch = await session.get(
- HostedLaunch, (require_tenant_id(), app.request_id)
- )
- evidence = await HostedLaunchStore().idle_evidence(
- session, launch, app.service.connections
- )
- return evidence.reasons
-
- assert "mailbox_pending" in await reasons()
- conn = attached_conn(app)
- await ack(app, conn, (room, "$m1", "journaled"))
- assert "mailbox_pending" in await reasons()
- await ack(app, conn, (room, "$m1", "held"))
- assert "mailbox_pending" not in await reasons()
-
-
-async def test_ack_is_the_attached_workers_alone(mailbox_app):
- app = mailbox_app
- await attach(app)
- conn = attached_conn(app)
- wrong = await app.client.post(
- f"/agents/{app.agent_id}/connection/mailbox/ack",
- json={"connection_id": str(uuid4()), "generation": 0, "entries": []},
- )
- assert wrong.status_code == 409
- too_many = await app.client.post(
- f"/agents/{app.agent_id}/connection/mailbox/ack",
- json={
- "connection_id": conn.id,
- "generation": conn.stream_generation,
- "entries": [
- {"room_id": "r", "message_id": f"$m{i}", "outcome": "journaled"}
- for i in range(201)
- ],
- },
- )
- assert too_many.status_code == 422
- unknown = await ack(app, conn, (app.rooms[0], "$nothing", "journaled"))
- assert unknown.json()["entries"] == [
- {"room_id": app.rooms[0], "message_id": "$nothing", "state": None}
- ]
-
-
-async def stop(app) -> Any:
- launch = await _launch(app.factory, app.request_id)
- response = await app.client.post(
- f"/hosted-launches/{app.request_id}/lifecycle",
- json={"action": "stop", "revision": launch.revision},
- )
- assert response.status_code == 200, response.text
- return response
-
-
-async def test_stop_splits_the_mailbox_and_hard_stop_wins(mailbox_app):
- app = mailbox_app
- first, second = app.rooms
- # Never offered: addressed while the worker is away.
- await address(app, addressed(first, "$m1", "$thread-1"))
- await address(app, addressed(first, "$m2", "$thread-2"))
- await address(app, addressed(second, "$n1"))
- agent = await _agent(app.factory, app.agent_id)
- response = await _open(app.service, agent, capability=await capability(app))
- opened = await _first_frames(response, 3)
- assert [event for event, _ in opened] == [
- "connection_state",
- "worker_attached",
- "wake",
- ]
- assert len(wake_entries(opened)) == 3
- conn = attached_conn(app)
- await ack(app, conn, (first, "$m1", "journaled"))
- await ack(app, conn, (second, "$n1", "journaled"), (second, "$n1", "admitted"))
- # Offered to the worker, not yet journaled; plus one it never saw.
- await address(app, addressed(first, "$m3"))
- async with app.factory() as session:
- session.add(
- HostedWakeMailbox(
- agent_id=app.agent_id,
- room_id=second,
- message_id="$n2",
- launch_id=app.request_id,
- thread_id="$thread-x",
- event={},
- addressed_at=datetime.now(UTC),
- updated_at=datetime.now(UTC),
- expires_at=datetime.now(UTC) + timedelta(hours=24),
- )
- )
- await session.commit()
-
- await stop(app)
- closing = await _first_frames(response, 2)
- assert [event for event, _ in closing] == ["mailbox_cancel", "evicted"]
- assert sorted(
- (e["room_id"], e["message_id"]) for e in closing[0][1]["entries"]
- ) == sorted([(first, "$m1"), (first, "$m2"), (first, "$m3")])
- assert await rows(app) == {
- (first, "$m1"): "cancel_requested",
- (first, "$m2"): "cancel_requested",
- (first, "$m3"): "cancel_requested",
- (second, "$n1"): "admitted",
- (second, "$n2"): "cancelled",
- }
- assert [(room, thread) for room, thread, _ in app.sent] == [(second, "$thread-x")]
- assert "stopped before I processed" in app.sent[0][2]
-
- # Addressed after a hard Stop: never woken, never queued.
- noted = await address(app, addressed(first, "$m9"))
- assert noted.launch.desired_state == "stopped"
- assert (first, "$m9") not in await rows(app)
-
- launch = await _launch(app.factory, app.request_id)
- started = await app.client.post(
- f"/hosted-launches/{app.request_id}/lifecycle",
- json={"action": "start", "revision": launch.revision},
- )
- assert started.status_code == 200, started.text
- _, frames = await attach(app)
- (attached,) = [data for event, data in frames if event == "worker_attached"]
- assert sorted(
- (e["room_id"], e["message_id"], e["reason"]) for e in attached["cancelled"]
- ) == sorted(
- [
- (first, "$m1", "stopped"),
- (first, "$m2", "stopped"),
- (first, "$m3", "stopped"),
- ]
- )
- assert wake_entries(frames) == []
- conn = attached_conn(app)
- app.sent.clear()
- settled = await ack(
- app,
- conn,
- (first, "$m1", "admitted"),
- (first, "$m2", "cancelled"),
- (first, "$m3", "cancelled"),
- )
- assert [e["state"] for e in settled.json()["entries"]] == [
- "admitted",
- "cancelled",
- "cancelled",
- ]
- bodies = sorted(body for _, _, body in app.sent)
- assert len(bodies) == 2
- assert any("already started processing" in body for body in bodies)
- assert any("stopped before I processed" in body for body in bodies)
- _, frames = await attach(app, boot_id="boot-a")
- (attached,) = [data for event, data in frames if event == "worker_attached"]
- assert attached["cancelled"] == []
-
-
-async def test_stop_on_a_sleeping_machine_never_wakes_it(mailbox_app):
- app = mailbox_app
- await sleep_machine(app)
- async with app.factory() as session:
- session.add(
- HostedWakeMailbox(
- agent_id=app.agent_id,
- room_id=app.rooms[0],
- message_id="$m1",
- launch_id=app.request_id,
- event={},
- addressed_at=datetime.now(UTC),
- updated_at=datetime.now(UTC),
- expires_at=datetime.now(UTC) + timedelta(hours=24),
- )
- )
- await session.commit()
- response = await app.client.post(
- f"/hosted-launches/{app.request_id}/lifecycle",
- json={"action": "stop", "revision": 1},
- )
- assert response.status_code == 200, response.text
- assert response.json()["state"] == "stopped"
- assert response.json()["sleeping"] is True
- launch = await _launch(app.factory, app.request_id)
- assert (launch.desired_state, launch.revision) == ("stopped", 2)
- machine = await _machine(app.factory, app.machine_id)
- assert (machine.desired_state, machine.revision) == ("stopped", 2)
- assert await rows(app) == {(app.rooms[0], "$m1"): "cancelled"}
- assert len(app.sent) == 1
- await address(app, addressed(app.rooms[0], "$m2"))
- assert (await _machine(app.factory, app.machine_id)).desired_state == "stopped"
- assert (app.rooms[0], "$m2") not in await rows(app)
-
-
-async def test_remove_deletes_the_mailbox(mailbox_app):
- app = mailbox_app
- await address(app, addressed(app.rooms[0], "$m1"))
- await stop(app)
- await set_launch(app, state="stopped")
- launch = await _launch(app.factory, app.request_id)
- removed = await app.client.post(
- f"/hosted-launches/{app.request_id}/lifecycle",
- json={"action": "remove", "revision": launch.revision},
- )
- assert removed.status_code == 200, removed.text
- assert await rows(app) == {}
-
-
-async def test_remove_of_a_running_agent_is_synchronous(mailbox_app):
- app = mailbox_app
- first, second = app.rooms
- await address(app, addressed(first, "$m1", "$thread-1"))
- await address(app, addressed(first, "$m2"))
- await address(app, addressed(second, "$n1"))
- agent = await _agent(app.factory, app.agent_id)
- key_id = agent.api_key_id
- before = await _machine(app.factory, app.machine_id)
- removed = await app.client.post(
- f"/hosted-launches/{app.request_id}/lifecycle",
- json={"action": "remove", "revision": 1},
- )
- assert removed.status_code == 200, removed.text
- assert removed.json()["state"] == "deleted"
- assert removed.json()["desired_state"] == "deleted"
- assert removed.json()["name"] == "removed:" + app.request_id
- assert await rows(app) == {}
- assert sorted(room for room, _, _ in app.sent) == sorted([first, second])
- assert all("removed before I processed" in body for _, _, body in app.sent)
- async with app.factory() as session:
- assert await session.get(Agent, app.agent_id) is None
- assert await session.scalar(select(ApiKey).where(ApiKey.id == key_id)) is None
- machine = await _machine(app.factory, app.machine_id)
- assert machine.desired_state == "retained"
- assert machine.revision == before.revision + 1
- assert machine.agents_version > before.agents_version
- retention = machine.retain_until - datetime.now(UTC)
- assert timedelta(days=7) - timedelta(minutes=1) < retention <= timedelta(days=7)
- again = await app.client.post(
- f"/hosted-launches/{app.request_id}/lifecycle",
- json={"action": "remove", "revision": 2},
- )
- assert again.status_code == 409
-
-
-async def test_mention_to_an_owner_stopped_machine_is_refused_not_queued(
- mailbox_app,
-):
- app = mailbox_app
- await set_machine(
- app.factory,
- app.machine_id,
- desired_state="stopped",
- stop_reason="owner",
- state="stopped",
- revision=2,
- )
- noted = await address(app, addressed(app.rooms[0], "$m1"))
- assert await rows(app) == {}
- assert _hosted_unavailable(noted.launch, noted.machine) == (
- _HOSTED_MACHINE_STOPPED_MESSAGE
- )
- machine = await _machine(app.factory, app.machine_id)
- assert (machine.desired_state, machine.revision) == ("stopped", 2)
-
-
-async def test_mention_to_an_errored_owner_stopped_machine_reports_machine_error(
- mailbox_app,
-):
- app = mailbox_app
- await set_machine(
- app.factory,
- app.machine_id,
- desired_state="stopped",
- stop_reason="owner",
- state="error",
- revision=2,
- )
- noted = await address(app, addressed(app.rooms[0], "$m1"))
- assert await rows(app) == {}
- assert _hosted_unavailable(noted.launch, noted.machine) == (
- _HOSTED_MACHINE_ERROR_MESSAGE
- )
- machine = await _machine(app.factory, app.machine_id)
- assert machine.state == "error"
-
-
-async def test_mention_to_an_errored_running_machine_is_refused_not_queued(
- mailbox_app,
-):
- app = mailbox_app
- await set_machine(
- app.factory,
- app.machine_id,
- desired_state="running",
- state="error",
- revision=2,
- )
- noted = await address(app, addressed(app.rooms[0], "$m1"))
- assert await rows(app) == {}
- assert _hosted_unavailable(noted.launch, noted.machine) == (
- _HOSTED_MACHINE_ERROR_MESSAGE
- )
- machine = await _machine(app.factory, app.machine_id)
- assert (machine.state, machine.desired_state, machine.revision) == (
- "error",
- "running",
- 2,
- )
-
-
-async def agent_client(app) -> SimpleNamespace:
- """An `AgentConsumer` addressed by everything, with no live session, that
- records what it posts and what it hands the live buffer."""
- agent = await _agent(app.factory, app.agent_id)
- meta = SimpleNamespace(
- room_id=app.rooms[0], name="r0", bridge_id=None, channel_type=None
- )
- posted: list[str] = []
- enqueued: list[AgentEvent] = []
- reported: list[dict[str, Any]] = []
-
- async def send_message(_room_id: str, body: str, **_kwargs: Any) -> str:
- posted.append(body)
- return "$sent"
-
- async def resolve_room_meta(_matrix_room_id: str) -> Any:
- return meta
-
- async def yes(*_args: Any) -> bool:
- return True
-
- async def no(*_args: Any) -> bool:
- return False
-
- async def fresh_agent(_session: Any) -> Any:
- return agent
-
- async def gate_addressed(*_args: Any) -> _GateOutcome:
- return _GateOutcome(addressed=True, refusal=None)
-
- async def reply_when_unavailable_here(*_args: Any) -> str:
- return "I don't have a session connected to this room."
-
- client = SimpleNamespace(
- agent=agent,
- _hosted_launch_store=HostedLaunchStore(),
- session_factory=app.factory,
- tenant_id=require_tenant_id(),
- _connections=app.service.connections,
- _event_buffer=SimpleNamespace(
- boot=app.service.event_buffer.boot,
- enqueue=lambda _agent_id, _room_id, event: enqueued.append(event),
- ),
- _resolve_room_meta=resolve_room_meta,
- _addressed=yes,
- _addressed_without_lookup=lambda _event, _meta: True,
- _compute_addressed=yes,
- _fresh_agent=fresh_agent,
- _gate_addressed=gate_addressed,
- _is_available=no,
- _reply_when_unavailable_here=reply_when_unavailable_here,
- _triggered_by_auto_reply=AgentConsumer._triggered_by_auto_reply,
- _waking_notice_revisions={},
- _unreachable_notice_revisions={},
- actor=SimpleNamespace(send_message=send_message),
- _message_telemetry=SimpleNamespace(
- enabled=True,
- agent_addressed=lambda **kwargs: reported.append(kwargs),
- ),
- posted=posted,
- enqueued=enqueued,
- reported=reported,
- )
- for name in (
- "_note_hosted_addressed",
- "_emit_media",
- "_post_auto_reply",
- "_report_addressed",
- "_sender_handle",
- ):
- setattr(client, name, getattr(AgentConsumer, name).__get__(client))
- return client
-
-
-async def error_the_running_machine(app) -> None:
- await set_machine(
- app.factory,
- app.machine_id,
- desired_state="running",
- state="error",
- revision=2,
- )
-
-
-async def test_media_to_an_errored_running_machine_is_refused_not_queued(
- mailbox_app,
-):
- app = mailbox_app
- await error_the_running_machine(app)
- client = await agent_client(app)
- media = InboundMedia(
- room_id="!room:example.com",
- event_id="$m1",
- sender="@someone:example.com",
- timestamp=1700000000000,
- content={"msgtype": "m.file", "body": "notes.md", "sender_name": "someone"},
- body="notes.md",
- sender_name="someone",
- msgtype="m.file",
- uri="mxc://example.com/abc",
- mimetype="text/markdown",
- size=5,
- )
- await AgentConsumer.on_media(client, RoomRef(room_id="!room:example.com"), media) # type: ignore[arg-type]
- assert await rows(app) == {}
- assert client.posted == [f"@someone {_HOSTED_MACHINE_ERROR_MESSAGE}"]
- assert client.enqueued == []
- # Still a message the agent was asked, by someone it could not answer.
- [report] = client.reported
- assert report["agent_live"] is False
-
-
-async def test_task_delegate_to_an_errored_running_machine_is_refused(mailbox_app):
- app = mailbox_app
- await error_the_running_machine(app)
- client = await agent_client(app)
- delegate = TaskDelegate(
- task_id="task-1",
- requester_agent_id="agent-a",
- performer_agent_id=app.agent_id,
- summary="Do it",
- description="",
- )
- await AgentConsumer.on_task_delegate(
- client, # type: ignore[arg-type]
- RoomRef(room_id="!room:example.com"),
- delegate,
- )
- assert await rows(app) == {}
- assert client.posted == [_HOSTED_MACHINE_ERROR_MESSAGE]
- assert client.enqueued == []
-
-
-async def test_mention_to_an_errored_running_machine_is_answered_once(mailbox_app):
- app = mailbox_app
- await error_the_running_machine(app)
- client = await agent_client(app)
- message = InboundMessage(
- room_id="!room:example.com",
- event_id="$m1",
- sender="@someone:example.com",
- timestamp=1700000000000,
- content={"sender_name": "someone"},
- body="@agent hello",
- sender_name="someone",
- )
- await AgentConsumer.on_message(
- client, RoomRef(room_id="!room:example.com"), message
- ) # type: ignore[arg-type]
- assert await rows(app) == {}
- assert client.posted == [f"@someone {_HOSTED_MACHINE_ERROR_MESSAGE}"]
- assert client.enqueued == []
- [report] = client.reported
- assert report["agent_live"] is False
-
-
-async def test_a_mention_the_mailbox_takes_is_a_message_the_agent_received(
- mailbox_app,
-):
- """Held for the worker rather than refused: asked, while not yet live."""
- app = mailbox_app
- client = await agent_client(app)
- message = InboundMessage(
- room_id="!room:example.com",
- event_id="$m1",
- sender="@someone:example.com",
- timestamp=1700000000000,
- content={"sender_name": "someone"},
- body="@agent hello",
- sender_name="someone",
- )
- await AgentConsumer.on_message(
- client, RoomRef(room_id="!room:example.com"), message
- ) # type: ignore[arg-type]
- [report] = client.reported
- assert report["agent_live"] is False
-
-
- ("content", "posted"),
- [
- ({"sender_name": "other"}, [f"@other {_HOSTED_MACHINE_STOPPED_MESSAGE}"]),
- ({"sender_name": "other", AUTO_REPLY_FLAG: True}, []),
- ],
-)
-async def test_auto_reply_to_an_owner_stopped_machine_is_not_answered(
- mailbox_app, content, posted
-):
- app = mailbox_app
- await set_machine(
- app.factory,
- app.machine_id,
- desired_state="stopped",
- stop_reason="owner",
- state="stopped",
- revision=2,
- )
- client = await agent_client(app)
- message = InboundMessage(
- room_id="!room:example.com",
- event_id="$m1",
- sender="@other:example.com",
- timestamp=1700000000000,
- content=content,
- body=f"@agent {_HOSTED_MACHINE_STOPPED_MESSAGE}",
- sender_name="other",
- )
- await AgentConsumer.on_message(
- client, RoomRef(room_id="!room:example.com"), message
- ) # type: ignore[arg-type]
- assert await rows(app) == {}
- assert client.posted == posted
- assert client.enqueued == []
-
-
-def fail_first_send(app) -> None:
- """The homeserver refuses the next notice once, then takes them again."""
- send = app.service.send_message
- failures = [RuntimeError("homeserver unreachable")]
-
- async def flaky(*args: Any, **kwargs: Any) -> None:
- if failures:
- raise failures.pop()
- await send(*args, **kwargs)
-
- app.service.send_message = flaky
-
-
-async def owed(app) -> dict[tuple[str, str], str | None]:
- async with app.factory() as session:
- found = await session.execute(
- select(
- HostedWakeMailbox.room_id,
- HostedWakeMailbox.message_id,
- HostedWakeMailbox.notice_owed,
- ).where(HostedWakeMailbox.tenant_id == require_tenant_id())
- )
- return {(room, message): notice for room, message, notice in found}
-
-
-async def add_row(app, message_id: str, **values: Any) -> None:
- now = datetime.now(UTC)
- async with app.factory() as session:
- session.add(
- HostedWakeMailbox(
- agent_id=app.agent_id,
- room_id=app.rooms[0],
- message_id=message_id,
- launch_id=app.request_id,
- thread_id="$thread",
- event={},
- addressed_at=now,
- updated_at=now,
- expires_at=now + timedelta(hours=24),
- **values,
- )
- )
- await session.commit()
-
-
-async def assert_retried_once(app, key: tuple[str, str], body: str) -> None:
- assert app.sent == []
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert [(room, thread) for room, thread, _ in app.sent] == [(key[0], "$thread")]
- assert body in app.sent[0][2]
- assert (await owed(app))[key] is None
- await mailbox_upkeep(app.service, datetime.now(UTC))
- assert len(app.sent) == 1
-
-
-async def test_stop_notice_failed_send_is_retried_by_upkeep(mailbox_app):
- app = mailbox_app
- await add_row(app, "$m1")
- fail_first_send(app)
- await stop(app)
- key = (app.rooms[0], "$m1")
- assert await rows(app) == {key: "cancelled"}
- assert await owed(app) == {key: "stopped"}
- await assert_retried_once(app, key, "stopped before I processed")
-
-
- ("outcome", "reason", "body"),
- [
- ("cancelled", "stopped", "stopped before I processed"),
- ("admitted", "started_before_stop", "already started processing"),
- ],
-)
-async def test_ack_notice_failed_send_is_retried_by_upkeep(
- mailbox_app, outcome, reason, body
-):
- app = mailbox_app
- await add_row(
- app, "$m1", state="cancel_requested", cancel_reason="stopped", ever_offered=True
- )
- await attach(app)
- fail_first_send(app)
- key = (app.rooms[0], "$m1")
- acked = await ack(app, attached_conn(app), (*key, outcome))
- assert acked.status_code == 200, acked.text
- assert await rows(app) == {key: outcome}
- assert await owed(app) == {key: reason}
- # A repeated ack finds the row terminal and owes nothing new.
- await ack(app, attached_conn(app), (*key, outcome))
- await assert_retried_once(app, key, body)
-
-
-async def test_expiry_notice_failed_send_is_retried_by_upkeep(mailbox_app):
- app = mailbox_app
- await add_row(app, "$m1")
- await sleep_machine(app)
- async with app.factory() as session:
- row = await session.get(
- HostedWakeMailbox,
- (require_tenant_id(), app.agent_id, app.rooms[0], "$m1"),
- )
- assert row is not None
- row.expires_at = datetime.now(UTC) - timedelta(seconds=1)
- await session.commit()
- fail_first_send(app)
- await mailbox_upkeep(app.service, datetime.now(UTC))
- key = (app.rooms[0], "$m1")
- assert await rows(app) == {key: "expired"}
- assert await owed(app) == {key: "expired"}
- await assert_retried_once(app, key, "could not process this message in time")
-
-
-async def test_a_deleted_agents_notice_is_dropped_not_posted(mailbox_app, caplog):
- app = mailbox_app
- await add_row(app, "$m1", state="cancelled", notice_owed="stopped")
- async with app.factory() as session:
- await AgentStore().delete(session, app.agent_id)
- await session.commit()
- store = HostedMailboxStore()
- async with app.factory() as session:
- notices = await store.owed_notices(session, 10)
- assert len(notices) == 1
-
- await post_mailbox_notices(app.service, notices)
-
- key = (app.rooms[0], "$m1")
- assert app.sent == []
- assert await owed(app) == {key: "stopped"}
- async with app.factory() as session:
- row = await session.get(
- HostedWakeMailbox, (require_tenant_id(), app.agent_id, *key)
- )
- assert row is not None
- assert row.notice_dropped == "agent_deleted"
- assert await store.owed_notices(session, 10) == []
- assert any("was deleted" in record.message for record in caplog.records)
diff --git a/core/tests/switch_core/gateway/test_hosted_mailbox_gate.py b/core/tests/switch_core/gateway/test_hosted_mailbox_gate.py
deleted file mode 100644
index 6019ac2e2..000000000
--- a/core/tests/switch_core/gateway/test_hosted_mailbox_gate.py
+++ /dev/null
@@ -1,71 +0,0 @@
-"""The wake mailbox upkeep stays off on a server that does not run cloud agents."""
-
-from __future__ import annotations
-
-from datetime import UTC, datetime
-from types import SimpleNamespace
-from unittest.mock import AsyncMock
-
-import pytest
-
-from switch_core.bridges.agent import hosted_mailbox
-from switch_core.bridges.agent.hosted_mailbox import mailbox_upkeep
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from tests.switch_core.bridges.agent.protocol.registration_harness import make_service
-from tests.switch_core.gateway.test_hosted_workers import worker_app # noqa: F401
-
-
-def reclaim(monkeypatch) -> AsyncMock:
- spy = AsyncMock(return_value=0)
- monkeypatch.setattr(hosted_mailbox.HostedMailboxStore, "reclaim", spy)
- return spy
-
-
-def _unconfigured(config) -> None:
- config.hosted_controller_config_path = None
- config.hosted_launch_capacity = 0
-
-
-def _bare_service(session_factory):
- service = make_service(session_factory)
- service.connections = AgentConnectionRegistry()
- service.event_buffer = SimpleNamespace(boot=1)
- _unconfigured(service.config)
- return service
-
-
-async def test_unconfigured_server_with_no_hosted_work_skips_the_pass(
- session_factory, reclaim
-):
- await mailbox_upkeep(_bare_service(session_factory), datetime.now(UTC))
-
- reclaim.assert_not_awaited()
-
-
-async def test_unconfigured_server_still_tends_retained_hosted_work(
- worker_app, # noqa: F811
- reclaim,
-):
- _, _, _, service, _, _ = worker_app
- _unconfigured(service.config)
-
- await mailbox_upkeep(service, datetime.now(UTC))
-
- reclaim.assert_awaited_once()
-
-
- ("controller", "capacity"),
- [("/etc/switch/controller.json", 0), (None, 1)],
-)
-async def test_configured_server_runs_the_pass(
- session_factory, reclaim, controller, capacity
-):
- service = _bare_service(session_factory)
- service.config.hosted_controller_config_path = controller
- service.config.hosted_launch_capacity = capacity
-
- await mailbox_upkeep(service, datetime.now(UTC))
-
- reclaim.assert_awaited_once()
diff --git a/core/tests/switch_core/gateway/test_hosted_supervisor.py b/core/tests/switch_core/gateway/test_hosted_supervisor.py
deleted file mode 100644
index 05f1433ef..000000000
--- a/core/tests/switch_core/gateway/test_hosted_supervisor.py
+++ /dev/null
@@ -1,794 +0,0 @@
-import asyncio
-import hashlib
-import logging
-from datetime import UTC, datetime, timedelta
-from uuid import uuid4
-
-import pytest
-
-from switch_core.db.models import (
- Agent,
- ApiKey,
- HostedLaunch,
- ProviderConnection,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
-from switch_core.db.stores.hosted_machine_store import lock_machine
-from switch_core.keys import Keyring
-from tests.switch_core.gateway.test_hosted_controller import ( # noqa: F401
- HEADERS,
- SPEC,
- TOKEN,
- attach_worker,
- controller_app,
- fixture,
- machine_of,
- observe,
- register_hosted_agent,
- update_launch,
- update_machine,
-)
-from tests.switch_core.hosted_machine_helpers import seed_launch, seed_machine
-
-TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
-
-
-async def supervisor(controller_app): # noqa: F811
- """The controller app plus a prepared machine and the supervisor's headers."""
- client, request_id, agent_id, service, factory, settings = controller_app
- machine = await machine_of(factory, request_id)
- prepared = await client.post(
- f"/hosted-controller/machines/{machine.id}/prepare", headers=HEADERS, json={}
- )
- assert prepared.status_code == 200, prepared.text
- await update_launch(factory, request_id, spec={**SPEC, "provider": "claude"})
- headers = {
- "Authorization": "Bearer " + prepared.json()["machine_capability"],
- "X-Switch-Host-Boot-Id": "00000000-0000-4000-8000-00000000b001",
- "X-Switch-Host-Instance-Id": "i-0123456789abcdef0",
- }
- return client, request_id, agent_id, service, factory, machine.id, headers
-
-
-def heartbeat_body(**agent) -> dict:
- body = fixture("heartbeat_request.json")
- body["agents"] = [{**body["agents"][0], **agent}] if agent else []
- return body
-
-
-async def other_owner(factory) -> str:
- async with factory() as session:
- user = User(
- name="other",
- role="user",
- password_hash="x",
- )
- session.add(user)
- await session.commit()
- return user.id
-
-
-async def launch_row(factory, request_id: str) -> HostedLaunch:
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch is not None
- return launch
-
-
-async def test_supervisor_routes_require_the_machine_capability(supervisor):
- client, _, _, _, factory, machine_id, headers = supervisor
- async with factory() as session:
- other = await seed_machine(
- session,
- owner_id=await other_owner(factory),
- slot_id="slot-b",
- state="ready",
- desired_state="running",
- stop_reason=None,
- revision=1,
- generation=1,
- )
- await session.commit()
- for path in (
- f"/hosted/machines/{machine_id}/agents",
- f"/hosted/machines/{machine_id}/heartbeat",
- f"/hosted/machines/{other.id}/agents",
- f"/hosted/machines/{uuid4()}/agents",
- ):
- method = "POST" if path.endswith("heartbeat") else "GET"
- for authorization in (None, "Bearer wrong", "Bearer " + TOKEN):
- sent = {k: v for k, v in headers.items() if k != "Authorization"}
- if authorization is not None:
- sent["Authorization"] = authorization
- response = await client.request(
- method, path, headers=sent, json=heartbeat_body()
- )
- assert response.status_code == 401, path
- assert response.json() == {"detail": "invalid machine capability"}
- another = await client.get(f"/hosted/machines/{other.id}/agents", headers=headers)
- assert another.status_code == 401
-
-
-async def test_wrong_capability_is_refused_without_waiting_on_the_machine_lock(
- supervisor,
-):
- client, _, _, _, factory, machine_id, headers = supervisor
- sent = {**headers, "Authorization": "Bearer wrong"}
- async with factory() as holder:
- await lock_machine(holder, machine_id)
- try:
- response = await asyncio.wait_for(
- client.get(f"/hosted/machines/{machine_id}/agents", headers=sent),
- timeout=5,
- )
- finally:
- await holder.rollback()
- assert response.status_code == 401
- assert response.json() == {"detail": "invalid machine capability"}
-
-
- "missing", ["X-Switch-Host-Boot-Id", "X-Switch-Host-Instance-Id"]
-)
-async def test_supervisor_routes_require_the_host_headers(supervisor, missing):
- client, _, _, _, _, machine_id, headers = supervisor
- sent = {k: v for k, v in headers.items() if k != missing}
- response = await client.get(f"/hosted/machines/{machine_id}/agents", headers=sent)
- assert response.status_code == 400
-
-
- ("state", "desired"),
- [
- ("retained", "retained"),
- ("stopped", "retained"),
- ("deleting", "deleted"),
- ("ready", "deleted"),
- ],
-)
-async def test_retired_machine_gets_410(supervisor, state, desired):
- client, _, _, _, factory, machine_id, headers = supervisor
- await update_machine(factory, machine_id, state=state, desired_state=desired)
- for method, suffix in (("GET", "agents"), ("POST", "heartbeat")):
- response = await client.request(
- method,
- f"/hosted/machines/{machine_id}/{suffix}",
- headers=headers,
- json=heartbeat_body(),
- )
- assert response.status_code == 410
- assert response.json() == {"detail": "machine retired"}
-
-
-async def test_agents_lists_the_machine_agents_in_the_contract_shape(supervisor):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- response = await client.get(
- f"/hosted/machines/{machine_id}/agents", headers=headers
- )
- assert response.status_code == 200, response.text
- assert response.headers["cache-control"] == "no-store"
- body = response.json()
- expected = fixture("agents_response.json")
- assert set(body) == set(expected)
- assert body["machine_id"] == machine_id
- assert body["revision"] == 1
- assert body["desired_state"] == "running"
- [entry] = body["agents"]
- assert set(entry) == set(expected["agents"][0])
- assert set(entry["switch_credentials"]["env"]) == set(
- expected["agents"][0]["switch_credentials"]["env"]
- )
- assert entry["launch_id"] == request_id
- assert entry["agent_id"] == agent_id
- assert entry["name"] == "cloud-helper"
- assert entry["revision"] == 1
- assert entry["desired_state"] == "running"
- assert entry["provider"] == "claude"
- assert entry["provider_credential_kind"] == "setup-token"
- assert entry["repository"] == "example/project"
- assert entry["spec"]["instructions"] == SPEC["instructions"]
- env = entry["switch_credentials"]["env"]
- assert env["SWITCH_API_ENDPOINT"] == "https://switch.example.com/api/agent"
- assert env["SWITCH_AGENT_ID"] == agent_id
- async with factory() as session:
- agent = await session.get(Agent, agent_id)
- key = await session.get(ApiKey, agent.api_key_id)
- assert (
- key.key_hash == hashlib.sha256(env["SWITCH_API_TOKEN"].encode()).hexdigest()
- )
- assert [skill["slug"] for skill in entry["skills"]] == ["github"]
- assert entry["skills"][0]["files"]["SKILL.md"].startswith("---\nname: github\n")
- launch = await launch_row(factory, request_id)
- assert launch.state == "provisioning"
- assert launch.worker_capability_hash is not None
-
-
-async def test_agents_lists_the_configuration_edited_since_launch(supervisor):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- async with factory() as session:
- await HostedLaunchStore().merge_spec(
- session,
- request_id,
- {"instructions": "Be brief.", "definition_attributes": {"model": "opus"}},
- )
- await session.commit()
- response = await client.get(
- f"/hosted/machines/{machine_id}/agents", headers=headers
- )
- [entry] = response.json()["agents"]
- assert entry["spec"] == {
- **SPEC,
- "provider": "claude",
- "instructions": "Be brief.",
- "definition_attributes": {"model": "opus"},
- }
-
-
-async def test_worker_capability_is_stable_per_revision_and_rotates_on_a_new_one(
- supervisor,
-):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- path = f"/hosted/machines/{machine_id}/agents"
- first = (await client.get(path, headers=headers)).json()["agents"][0]
- again = (await client.get(path, headers=headers)).json()["agents"][0]
- assert first["worker_capability"] == again["worker_capability"]
- assert first["switch_credentials"] == again["switch_credentials"]
- await update_launch(factory, request_id, revision=2)
- rotated = (await client.get(path, headers=headers)).json()["agents"][0]
- assert rotated["revision"] == 2
- assert rotated["worker_capability"] != first["worker_capability"]
-
-
-async def test_agents_omits_deleted_and_unregistered_launches(supervisor):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- machine = await machine_of(factory, request_id)
- async with factory() as session:
- await seed_launch(
- session,
- machine=machine,
- request_id=str(uuid4()),
- name="cloud-unregistered",
- state="queued",
- desired_state="running",
- revision=1,
- agent_id=None,
- spec=SPEC,
- )
- await session.commit()
- path = f"/hosted/machines/{machine_id}/agents"
- listed = (await client.get(path, headers=headers)).json()["agents"]
- assert [entry["launch_id"] for entry in listed] == [request_id]
- await update_launch(factory, request_id, desired_state="deleted")
- assert (await client.get(path, headers=headers)).json()["agents"] == []
-
-
-UNAVAILABLE_KEYS = {
- "launch_id",
- "agent_id",
- "name",
- "revision",
- "desired_state",
- "unavailable",
-}
-KEY_MISSING_ERROR = (
- "Switch lost this cloud agent's credential, so its machine cannot run it. "
- "Remove the agent and create it again."
-)
-IDENTITY_MISSING_ERROR = (
- "Switch lost this cloud agent's identity, so its machine cannot run it. "
- "Retry it in Switch Console."
-)
-
-
-async def second_launch(
- factory, service, request_id: str, *, state: str, register: bool
-) -> tuple[str, str]:
- """Another launch on the fixture's machine, with its own agent if `register`."""
- machine = await machine_of(factory, request_id)
- launch_id = str(uuid4())
- agent_id = (
- await register_hosted_agent(
- service,
- owner=machine.owner_id,
- request_id=launch_id,
- name="cloud-second",
- spec=SPEC,
- )
- if register
- else str(uuid4())
- )
- async with factory() as session:
- await seed_launch(
- session,
- machine=machine,
- request_id=launch_id,
- name="cloud-second",
- state=state,
- desired_state="running",
- revision=1,
- agent_id=agent_id,
- spec={**SPEC, "provider": "claude"},
- )
- await session.commit()
- return launch_id, agent_id
-
-
-def split_entries(listed: list[dict], broken_id: str) -> tuple[dict, dict]:
- [healthy] = [entry for entry in listed if entry["launch_id"] != broken_id]
- [broken] = [entry for entry in listed if entry["launch_id"] == broken_id]
- return healthy, broken
-
-
-async def test_agents_lists_a_queued_launch_whose_agent_row_does_not_exist_yet_as_unavailable(
- supervisor, caplog
-):
- client, request_id, agent_id, service, factory, machine_id, headers = supervisor
- pending_id, pending_agent = await second_launch(
- factory, service, request_id, state="queued", register=False
- )
- with caplog.at_level(logging.ERROR):
- response = await client.get(
- f"/hosted/machines/{machine_id}/agents", headers=headers
- )
- assert response.status_code == 200, response.text
- healthy, broken = split_entries(response.json()["agents"], pending_id)
- assert set(healthy) == set(fixture("agents_response.json")["agents"][0])
- assert healthy["agent_id"] == agent_id
- assert broken == {
- "launch_id": pending_id,
- "agent_id": pending_agent,
- "name": "cloud-second",
- "revision": 1,
- "desired_state": "running",
- "unavailable": "agent_identity_missing",
- }
- assert "agent_identity_missing" in caplog.text
- launch = await launch_row(factory, pending_id)
- assert launch.state == "queued"
- assert launch.error_code is None
- assert launch.error is None
-
-
-async def test_agents_puts_a_stale_queued_launch_without_its_agent_row_in_error(
- supervisor,
-):
- client, request_id, _, service, factory, machine_id, headers = supervisor
- stale_id, stale_agent = await second_launch(
- factory, service, request_id, state="queued", register=False
- )
- await update_launch(
- factory, stale_id, updated_at=datetime.now(UTC) - timedelta(minutes=2)
- )
- response = await client.get(
- f"/hosted/machines/{machine_id}/agents", headers=headers
- )
- assert response.status_code == 200, response.text
- _, broken = split_entries(response.json()["agents"], stale_id)
- assert broken["agent_id"] == stale_agent
- assert broken["unavailable"] == "agent_identity_missing"
- launch = await launch_row(factory, stale_id)
- assert launch.state == "error"
- assert launch.error_code == "agent_identity_missing"
- assert launch.error == IDENTITY_MISSING_ERROR
-
-
-async def test_agents_lists_a_launch_whose_agent_row_is_gone_as_unavailable(
- supervisor,
-):
- client, request_id, _, service, factory, machine_id, headers = supervisor
- broken_id, broken_agent = await second_launch(
- factory, service, request_id, state="provisioning", register=False
- )
- path = f"/hosted/machines/{machine_id}/agents"
- response = await client.get(path, headers=headers)
- assert response.status_code == 200, response.text
- healthy, broken = split_entries(response.json()["agents"], broken_id)
- assert set(healthy) == set(fixture("agents_response.json")["agents"][0])
- assert "unavailable" not in healthy
- assert set(broken) == UNAVAILABLE_KEYS
- assert broken["agent_id"] == broken_agent
- assert broken["unavailable"] == "agent_identity_missing"
- launch = await launch_row(factory, broken_id)
- assert launch.state == "error"
- assert launch.error_code == "agent_identity_missing"
- assert launch.error == IDENTITY_MISSING_ERROR
- marked_at = launch.updated_at
- again = await client.get(path, headers=headers)
- assert again.status_code == 200, again.text
- assert (await launch_row(factory, broken_id)).updated_at == marked_at
-
-
-async def test_agents_lists_a_launch_without_its_key_as_unavailable(supervisor, caplog):
- client, request_id, agent_id, service, factory, machine_id, headers = supervisor
- broken_id, broken_agent = await second_launch(
- factory, service, request_id, state="queued", register=True
- )
- async with factory() as session:
- agent = await session.get(Agent, broken_agent)
- key = await session.get(ApiKey, agent.api_key_id)
- key.encrypted_key = ""
- await session.commit()
- path = f"/hosted/machines/{machine_id}/agents"
- with caplog.at_level(logging.ERROR):
- response = await client.get(path, headers=headers)
- assert response.status_code == 200, response.text
- healthy, broken = split_entries(response.json()["agents"], broken_id)
- assert set(healthy) == set(fixture("agents_response.json")["agents"][0])
- assert healthy["agent_id"] == agent_id
- assert broken == {
- "launch_id": broken_id,
- "agent_id": broken_agent,
- "name": "cloud-second",
- "revision": 1,
- "desired_state": "running",
- "unavailable": "agent_key_missing",
- }
- assert "agent_key_missing" in caplog.text
- launch = await launch_row(factory, broken_id)
- assert launch.state == "error"
- assert launch.error_code == "agent_key_missing"
- assert launch.error == KEY_MISSING_ERROR
- marked_at = launch.updated_at
- again = await client.get(path, headers=headers)
- assert again.status_code == 200, again.text
- assert (await launch_row(factory, broken_id)).updated_at == marked_at
- assert (await launch_row(factory, request_id)).state == "provisioning"
-
-
-async def test_agents_sends_no_skills_to_a_provider_without_a_skills_directory(
- supervisor, caplog
-):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- launch = await launch_row(factory, request_id)
- await update_launch(factory, request_id, spec={**SPEC, "provider": "cursor"})
- async with factory() as session:
- session.add(
- ProviderConnection(
- user_id=launch.owner_id,
- provider="cursor",
- kind="api-key",
- encrypted_credential=TEST_KEYRING.encrypt("SYNTHETIC-CURSOR"),
- verified_at=datetime.now(UTC),
- )
- )
- await session.commit()
- with caplog.at_level(logging.WARNING):
- response = await client.get(
- f"/hosted/machines/{machine_id}/agents", headers=headers
- )
- assert response.status_code == 200, response.text
- [entry] = response.json()["agents"]
- assert entry["skills"] == []
- assert entry["provider"] == "cursor"
- assert entry["provider_credential_kind"] == "api-key"
- assert "granted connection skills are not installed" in caplog.text
-
-
-async def test_agents_lists_a_launch_whose_provider_connection_is_gone(
- supervisor, caplog
-):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- launch = await launch_row(factory, request_id)
- async with factory() as session:
- await session.delete(
- await session.get(
- ProviderConnection, (require_tenant_id(), launch.owner_id, "claude")
- )
- )
- await session.commit()
- with caplog.at_level(logging.WARNING):
- response = await client.get(
- f"/hosted/machines/{machine_id}/agents", headers=headers
- )
- assert response.status_code == 200, response.text
- [entry] = response.json()["agents"]
- assert entry["launch_id"] == request_id
- assert "provider_credential_kind" in entry
- assert entry["provider_credential_kind"] is None
- assert "connection is gone" in caplog.text
-
-
-async def test_heartbeat_fixture_is_accepted_verbatim(supervisor):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- body = fixture("heartbeat_request.json")
- response = await client.post(
- f"/hosted/machines/{machine_id}/heartbeat", headers=headers, json=body
- )
- assert response.status_code == 200, response.text
- assert set(response.json()) == set(fixture("heartbeat_response.json"))
- assert response.json()["heartbeat_every_s"] == 15
- assert response.json()["machine_desired_state"] == "running"
- saved = await machine_of(factory, request_id)
- assert saved.heartbeat == body
- assert saved.heartbeat_at is not None
-
-
-async def test_heartbeat_records_process_state_and_a_crash(supervisor):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- response = await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(launch_id=request_id, agent_id=agent_id, revision=1),
- )
- assert response.status_code == 200, response.text
- launch = await launch_row(factory, request_id)
- assert launch.process_state == "crashed"
- assert launch.process_restarts == 5
- assert launch.process_oom_kills == 1
- assert launch.process_exit == {
- "code": None,
- "signal": 9,
- "result": "start-limit-hit",
- }
- assert launch.process_reported_at is not None
- assert launch.state == "error"
- assert launch.error_code == "agent_crashed"
- assert "5 times in 10 minutes" in launch.error
-
-
-async def test_heartbeat_marks_a_failed_agent(supervisor):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(
- launch_id=request_id, agent_id=agent_id, revision=1, process_state="failed"
- ),
- )
- launch = await launch_row(factory, request_id)
- assert launch.state == "error"
- assert launch.error_code == "agent_failed"
-
-
-async def test_heartbeat_confirms_a_requested_stop(supervisor):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- await update_launch(factory, request_id, state="stopping", desired_state="stopped")
- await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(
- launch_id=request_id,
- agent_id=agent_id,
- revision=1,
- process_state="stopped",
- exit=None,
- ),
- )
- assert (await launch_row(factory, request_id)).state == "stopped"
-
-
[email protected]("process_state", ["crashed", "failed"])
-async def test_crash_of_an_agent_meant_to_stop_stops_it(supervisor, process_state):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- await update_launch(
- factory,
- request_id,
- state="stopping",
- desired_state="stopped",
- error="An earlier failure.",
- error_code="agent_failed",
- )
- response = await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(
- launch_id=request_id,
- agent_id=agent_id,
- revision=1,
- process_state=process_state,
- ),
- )
- assert response.status_code == 200, response.text
- launch = await launch_row(factory, request_id)
- assert launch.state == "stopped"
- assert launch.error is None
- assert launch.error_code is None
-
-
-async def test_stopped_process_does_not_stop_an_agent_meant_to_run(supervisor):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(
- launch_id=request_id,
- agent_id=agent_id,
- revision=1,
- process_state="stopped",
- exit=None,
- ),
- )
- assert (await launch_row(factory, request_id)).state == "queued"
-
-
-async def test_running_process_is_ready_only_once_its_worker_listens(supervisor):
- client, request_id, agent_id, service, factory, machine_id, headers = supervisor
- await update_launch(factory, request_id, state="provisioning")
- body = heartbeat_body(
- launch_id=request_id,
- agent_id=agent_id,
- revision=1,
- process_state="running",
- exit=None,
- )
- path = f"/hosted/machines/{machine_id}/heartbeat"
- await client.post(path, headers=headers, json=body)
- assert (await launch_row(factory, request_id)).state == "provisioning"
- stale = attach_worker(service, agent_id, request_id, revision=2)
- await client.post(path, headers=headers, json=body)
- assert (await launch_row(factory, request_id)).state == "provisioning"
- attach_worker(service, agent_id, request_id, connection_id=stale.id)
- before = datetime.now(UTC)
- await client.post(path, headers=headers, json=body)
- launch = await launch_row(factory, request_id)
- assert launch.state == "ready"
- assert launch.active_at >= before
-
-
-async def post_running_report(supervisor, *, since: datetime, updated_at: datetime):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- await update_launch(
- factory, request_id, state="provisioning", updated_at=updated_at
- )
- response = await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(
- launch_id=request_id,
- agent_id=agent_id,
- revision=1,
- process_state="running",
- exit=None,
- since=since.isoformat(),
- ),
- )
- assert response.status_code == 200, response.text
- return await launch_row(factory, request_id)
-
-
-async def test_running_process_that_never_attaches_times_out(supervisor):
- long_ago = datetime.now(UTC) - timedelta(minutes=11)
- launch = await post_running_report(supervisor, since=long_ago, updated_at=long_ago)
- assert launch.state == "error"
- assert launch.error_code == "worker_attach_timeout"
- assert launch.error == (
- "The agent started but did not connect to Switch within 10 minutes. "
- "Retry it in Switch Console, or check its provider login."
- )
- assert launch.updated_at > long_ago
-
-
[email protected]("recent", ["since", "updated_at"])
-async def test_running_process_within_the_attach_deadline_stays_provisioning(
- supervisor, recent
-):
- long_ago = datetime.now(UTC) - timedelta(minutes=11)
- just_now = datetime.now(UTC) - timedelta(minutes=1)
- times = {"since": long_ago, "updated_at": long_ago, recent: just_now}
- launch = await post_running_report(supervisor, **times)
- assert launch.state == "provisioning"
- assert launch.error_code is None
-
-
-async def test_heartbeat_refuses_a_report_time_without_an_offset(supervisor):
- client, request_id, agent_id, _, _, machine_id, headers = supervisor
- response = await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(
- launch_id=request_id, agent_id=agent_id, since="2026-01-01T00:00:00"
- ),
- )
- assert response.status_code == 422
-
-
[email protected]("result", ["invalid-config", "ownership-invalid"])
-async def test_heartbeat_stores_a_worker_exit_result(supervisor, result):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- response = await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(
- launch_id=request_id,
- agent_id=agent_id,
- revision=1,
- process_state="failed",
- exit={"code": None, "signal": None, "result": result},
- ),
- )
- assert response.status_code == 200, response.text
- launch = await launch_row(factory, request_id)
- assert launch.process_exit is not None
- assert launch.process_exit["result"] == result
-
-
-async def test_stale_revision_report_writes_nothing(supervisor):
- client, request_id, agent_id, _, factory, machine_id, headers = supervisor
- reported_at = datetime.now(UTC) - timedelta(minutes=5)
- await update_launch(
- factory,
- request_id,
- revision=2,
- process_state="starting",
- process_restarts=0,
- process_oom_kills=0,
- process_exit=None,
- process_reported_at=reported_at,
- )
- response = await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(launch_id=request_id, agent_id=agent_id, revision=1),
- )
- assert response.status_code == 200, response.text
- launch = await launch_row(factory, request_id)
- assert (
- launch.process_state,
- launch.process_restarts,
- launch.process_oom_kills,
- launch.process_exit,
- launch.process_reported_at,
- ) == ("starting", 0, 0, None, reported_at)
- assert launch.state == "queued"
- assert launch.error_code is None
-
-
-async def test_report_for_another_agent_is_ignored(supervisor):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(launch_id=request_id, agent_id=str(uuid4()), revision=1),
- )
- launch = await launch_row(factory, request_id)
- assert launch.process_state is None
- assert launch.state == "queued"
-
-
-async def test_low_disk_is_flagged_and_cleared(supervisor):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- path = f"/hosted/machines/{machine_id}/heartbeat"
- low = heartbeat_body()
- low["disk"] = {**low["disk"], "available_bytes": 1024}
- await client.post(path, headers=headers, json=low)
- assert (await machine_of(factory, request_id)).error_code == "disk_full"
- await client.post(path, headers=headers, json=heartbeat_body())
- assert (await machine_of(factory, request_id)).error_code is None
- await update_machine(factory, machine_id, error_code="machine_needs_attention")
- await client.post(path, headers=headers, json=low)
- assert (await machine_of(factory, request_id)).error_code == (
- "machine_needs_attention"
- )
-
-
-async def test_machine_is_ready_after_a_running_observation_and_a_later_heartbeat(
- supervisor,
-):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- path = f"/hosted/machines/{machine_id}/heartbeat"
- await client.post(path, headers=headers, json=heartbeat_body())
- assert (await machine_of(factory, request_id)).state == "provisioning"
- await observe(client, machine_id, state="running", revision=1)
- await client.post(path, headers=headers, json=heartbeat_body())
- saved = await machine_of(factory, request_id)
- assert saved.state == "ready"
- assert saved.error is None
-
-
-async def test_heartbeat_before_the_running_observation_does_not_count(supervisor):
- client, request_id, _, _, factory, machine_id, headers = supervisor
- await update_machine(
- factory,
- machine_id,
- running_observed_at=datetime.now(UTC) + timedelta(minutes=1),
- )
- await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=headers,
- json=heartbeat_body(),
- )
- assert (await machine_of(factory, request_id)).state == "provisioning"
diff --git a/core/tests/switch_core/gateway/test_hosted_wire_fixtures.py b/core/tests/switch_core/gateway/test_hosted_wire_fixtures.py
deleted file mode 100644
index c2d73d495..000000000
--- a/core/tests/switch_core/gateway/test_hosted_wire_fixtures.py
+++ /dev/null
@@ -1,371 +0,0 @@
-"""The shared hosted-machine wire fixtures are what Core really sends and accepts.
-
-One launch is driven through the real routes: create, prepare, the controller's
-observation, the supervisor's agent list and heartbeat, the controller's
-machine list and the idle-stop sweep. Each response is compared with its
-fixture; each request fixture is posted as it stands.
-"""
-
-import json
-from datetime import UTC, datetime, timedelta
-from types import SimpleNamespace
-from typing import Any
-from unittest.mock import AsyncMock
-
-import httpx
-import pytest
-from fastapi import FastAPI
-
-from switch_core.bridges.agent.api.hosted_machine_routes import (
- router as hosted_machine_router,
-)
-from switch_core.bridges.agent.dependencies import get_config as get_worker_config
-from switch_core.bridges.agent.dependencies import get_protocol as get_worker_protocol
-from switch_core.bridges.agent.dependencies import (
- get_session_factory as get_worker_session_factory,
-)
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.db.models import (
- Agent,
- ApiKey,
- HostedLaunch,
- HostedMachine,
- TenantMember,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.gateway.auth import get_current_user
-from switch_core.gateway.dependencies import (
- get_config,
- get_protocol,
- get_session,
- get_session_factory,
-)
-from switch_core.gateway.hosted_controller import router as controller_router
-from switch_core.gateway.hosted_launches import router as launch_router
-from switch_core.gateway.hosted_machines import router as machine_router
-from switch_core.keys import Keyring
-from switch_core.providers.hosted import HostedControllerSettings
-from tests.switch_core.bridges.agent.protocol.registration_harness import (
- make_owner,
- make_service,
-)
-from tests.switch_core.hosted_wire_fixtures import (
- AGENT_ID,
- API_ENDPOINT,
- API_TOKEN,
- LAUNCH_ID,
- MACHINE_CAPABILITY,
- MACHINE_ID,
- TIMESTAMP,
- WORKER_CAPABILITY,
- assert_wire_fixture,
- read_fixture,
- request_fixture,
-)
-
-TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
-TOKEN = "SYNTHETIC-CONTROLLER-CREDENTIAL-FOR-TESTS"
-CONTROLLER = {"Authorization": "Bearer " + TOKEN}
-IDLE_STOP_MINUTES = 30
-
-LAUNCH_REQUEST = {
- "request_id": LAUNCH_ID,
- "name": "reviewer",
- "description": "Reviews pull requests",
- "display_name": None,
- "icon_url": None,
- "instructions": "Review each pull request you are asked about and post your findings in the room.",
- "provider": "claude",
- "definition": "---\nname: reviewer\ndescription: Reviews pull requests\n---\nYou review pull requests for correctness and clarity.\n",
- "installation_id": 123,
- "repository_id": 456,
- "definition_attributes": {"model": "sonnet"},
- "auto_session": True,
- "auto_approve": False,
- "addressing_policy": None,
-}
-
-RESPONSES: dict[str, dict[tuple[str, ...], Any]] = {
- "prepare_response.json": {("machine_capability",): MACHINE_CAPABILITY},
- "agents_response.json": {
- ("agents", "*", "worker_capability"): WORKER_CAPABILITY,
- ("agents", "*", "switch_credentials", "env", "SWITCH_API_TOKEN"): API_TOKEN,
- },
- "agents_response_unavailable.json": {},
- "heartbeat_response.json": {},
- "launch_summary.json": {},
- "machines_response.json": {},
- "machine_summary_sleeping.json": {("heartbeat_at",): TIMESTAMP},
-}
-
-
-async def wire(session_factory, monkeypatch, tmp_path):
- owner = await make_owner(session_factory)
- async with session_factory() as session:
- session.add(
- TenantMember(tenant_id=require_tenant_id(), user_id=owner, role="member")
- )
- await ProviderConnectionStore().save(
- session,
- owner,
- "setup-token",
- TEST_KEYRING.encrypt("SYNTHETIC-CLAUDE"),
- datetime.now(UTC),
- )
- await session.commit()
- settings = HostedControllerSettings(
- tenant_id=require_tenant_id(),
- token=TOKEN,
- machine_slots=["slot-a", "slot-b"],
- github_private_key_path="/tmp/synthetic-signing-key.pem",
- agent_api_endpoint=API_ENDPOINT,
- )
- settings_path = tmp_path / "controller.json"
- settings_path.write_text(
- json.dumps({**settings.model_dump(mode="json"), "token": TOKEN})
- )
- service = make_service(session_factory)
- service.connections = AgentConnectionRegistry()
- service.event_buffer = EventBuffer(sequence_base=1 << 32)
- service.config.hosted_launch_capacity = 4
- service.config.hosted_agents_per_owner = 3
- service.config.hosted_sessions_per_agent = 8
- service.config.hosted_disk_retention_days = 7
- service.config.hosted_idle_stop_minutes = IDLE_STOP_MINUTES
- service.config.hosted_controller_config_path = str(settings_path)
- monkeypatch.setattr(
- "switch_core.gateway.hosted_launches.connection_status",
- AsyncMock(
- return_value={
- "installations": [
- {
- "id": 123,
- "repositories": [
- {
- "id": 456,
- "name": "example-org/example-repo",
- "permissions": {"push": True},
- }
- ],
- }
- ]
- }
- ),
- )
- app = FastAPI()
- app.state.hosted_controller_settings = settings
- app.state.claude_verifier = AsyncMock()
- app.state.github_connections = object()
- app.include_router(controller_router)
- app.include_router(hosted_machine_router)
- app.include_router(launch_router)
- app.include_router(machine_router)
-
- async def sessions():
- async with session_factory() as session:
- yield session
-
- async def current_user():
- async with session_factory() as session:
- return await session.get(User, owner)
-
- app.dependency_overrides[get_current_user] = current_user
- app.dependency_overrides[get_session] = sessions
- app.dependency_overrides[get_session_factory] = lambda: session_factory
- app.dependency_overrides[get_config] = lambda: service.config
- app.dependency_overrides[get_protocol] = lambda: service
- app.dependency_overrides[get_worker_config] = lambda: service.config
- app.dependency_overrides[get_worker_session_factory] = lambda: session_factory
- app.dependency_overrides[get_worker_protocol] = lambda: service
- async with httpx.AsyncClient(
- transport=httpx.ASGITransport(app=app), base_url="https://switch.example.test"
- ) as client:
- yield SimpleNamespace(client=client, factory=session_factory)
-
-
-async def _row(factory, model, key: str):
- async with factory() as session:
- row = await session.get(model, (require_tenant_id(), key))
- assert row is not None
- return row
-
-
-def _ok(response: httpx.Response, status: int = 200) -> Any:
- assert response.status_code == status, response.text
- return response.json()
-
-
-async def wire_flow(wire) -> tuple[dict[str, Any], dict[str, str]]:
- """Drive one launch through the real routes, returning each response by fixture name."""
- client, factory = wire.client, wire.factory
- bodies: dict[str, Any] = {}
- created = _ok(await client.post("/hosted-launches", json=LAUNCH_REQUEST), 202)
- machine_id, agent_id = created["machine_id"], created["agent_id"]
- placeholders = {MACHINE_ID: machine_id, AGENT_ID: agent_id}
-
- bodies["prepare_response.json"] = prepared = _ok(
- await client.post(
- f"/hosted-controller/machines/{machine_id}/prepare",
- headers=CONTROLLER,
- json={},
- )
- )
-
- observed = _ok(
- await client.post(
- f"/hosted-controller/machines/{machine_id}/observation",
- headers=CONTROLLER,
- json=request_fixture("controller_observation.json", placeholders),
- )
- )
- assert observed["state"] == "provisioning"
- assert (await _row(factory, HostedMachine, machine_id)).running_observed_at
-
- heartbeat = request_fixture("heartbeat_request.json", placeholders)
- supervisor = {
- "Authorization": "Bearer " + prepared["machine_capability"],
- "X-Switch-Host-Boot-Id": heartbeat["boot_id"],
- "X-Switch-Host-Instance-Id": heartbeat["instance_id"],
- }
- bodies["agents_response.json"] = _ok(
- await client.get(f"/hosted/machines/{machine_id}/agents", headers=supervisor)
- )
-
- bodies["heartbeat_response.json"] = _ok(
- await client.post(
- f"/hosted/machines/{machine_id}/heartbeat",
- headers=supervisor,
- json=heartbeat,
- )
- )
- machine = await _row(factory, HostedMachine, machine_id)
- assert machine.state == "ready"
- assert machine.heartbeat == heartbeat
- [report] = heartbeat["agents"]
- launch = await _row(factory, HostedLaunch, LAUNCH_ID)
- assert (
- launch.process_state,
- launch.process_restarts,
- launch.process_oom_kills,
- launch.process_exit,
- ) == (
- report["process_state"],
- report["restarts"],
- report["oom_kills"],
- report["exit"],
- )
- assert (launch.state, launch.error_code) == ("error", "agent_crashed")
-
- bodies["launch_summary.json"] = _ok(
- await client.get(f"/hosted-launches/{LAUNCH_ID}")
- )
- bodies["machines_response.json"] = _ok(
- await client.get("/hosted-controller/machines", headers=CONTROLLER)
- )
-
- async with factory() as session:
- row = await session.get(HostedMachine, (require_tenant_id(), machine_id))
- row.active_at = datetime.now(UTC) - timedelta(minutes=IDLE_STOP_MINUTES + 1)
- await session.commit()
- [swept] = (
- _ok(await client.get("/hosted-controller/machines", headers=CONTROLLER))
- )["machines"]
- assert swept["desired_state"] == "stopped"
- _ok(
- await client.post(
- f"/hosted-controller/machines/{machine_id}/observation",
- headers=CONTROLLER,
- json={"state": "stopped", "revision": swept["revision"]},
- )
- )
- bodies["machine_summary_sleeping.json"] = _ok(
- await client.get(f"/hosted-machines/{machine_id}")
- )
-
- async with factory() as session:
- agent = await session.get(Agent, agent_id)
- key = await session.get(ApiKey, agent.api_key_id)
- key.encrypted_key = ""
- await session.commit()
- bodies["agents_response_unavailable.json"] = _ok(
- await client.get(f"/hosted/machines/{machine_id}/agents", headers=supervisor)
- )
- return bodies, placeholders
-
-
[email protected]("name", sorted(RESPONSES))
-async def test_response_fixture_is_real_core_output(wire, name):
- bodies, placeholders = await wire_flow(wire)
- assert_wire_fixture(
- name, bodies[name], placeholders=placeholders, volatile=RESPONSES[name]
- )
-
-
-def test_request_fixtures_carry_exactly_the_contract_fields():
- assert set(read_fixture("controller_observation.json")) == {
- "state",
- "revision",
- "error",
- "error_code",
- "data_volume_id",
- "instance_id",
- "instance_type",
- }
- heartbeat = read_fixture("heartbeat_request.json")
- assert set(heartbeat) == {
- "boot_id",
- "instance_id",
- "supervisor_version",
- "runtime_fingerprint",
- "disk",
- "memory",
- "agents",
- }
- assert set(heartbeat["disk"]) == {
- "path",
- "total_bytes",
- "used_bytes",
- "available_bytes",
- }
- assert set(heartbeat["memory"]) == {"total_bytes", "available_bytes"}
- [agent] = heartbeat["agents"]
- assert set(agent) == {
- "launch_id",
- "agent_id",
- "revision",
- "process_state",
- "restarts",
- "oom_kills",
- "exit",
- "since",
- }
- assert set(agent["exit"]) == {"code", "signal", "result"}
-
-
-def test_fixtures_name_the_same_machine_launch_and_agent():
- agents = read_fixture("agents_response.json")
- [listed] = agents["agents"]
- [reported] = read_fixture("heartbeat_request.json")["agents"]
- launch = read_fixture("launch_summary.json")
- assert (reported["launch_id"], reported["agent_id"], reported["revision"]) == (
- listed["launch_id"],
- listed["agent_id"],
- listed["revision"],
- )
- assert (launch["request_id"], launch["agent_id"]) == (LAUNCH_ID, AGENT_ID)
- assert (listed["launch_id"], listed["agent_id"]) == (LAUNCH_ID, AGENT_ID)
- assert {
- agents["machine_id"],
- launch["machine_id"],
- read_fixture("prepare_response.json")["machine_id"],
- read_fixture("machine_summary_sleeping.json")["machine_id"],
- *(
- item["machine_id"]
- for item in read_fixture("machines_response.json")["machines"]
- ),
- } == {MACHINE_ID}
- assert read_fixture("machine_summary_sleeping.json")["agents"] == [LAUNCH_ID]
diff --git a/core/tests/switch_core/gateway/test_hosted_workers.py b/core/tests/switch_core/gateway/test_hosted_workers.py
deleted file mode 100644
index 43708d6ca..000000000
--- a/core/tests/switch_core/gateway/test_hosted_workers.py
+++ /dev/null
@@ -1,1369 +0,0 @@
-"""A hosted agent's worker: attaching with its capability, and the up-calls only it may make."""
-
-from __future__ import annotations
-
-import asyncio
-import json
-import time
-from datetime import UTC, datetime, timedelta
-from types import SimpleNamespace
-from typing import Any
-from unittest.mock import AsyncMock
-from uuid import uuid4
-
-import httpx
-import pytest
-from fastapi import FastAPI, HTTPException, WebSocketDisconnect
-from sqlalchemy import event as orm_event
-from sqlalchemy import select
-
-from switch_core.bridges.agent.api import handlers
-from switch_core.bridges.agent.api.handlers import (
- connection_placements,
- connection_socket,
- poll_events,
-)
-from switch_core.bridges.agent.api.hosted_cutover_routes import (
- router as hosted_cutover_router,
-)
-from switch_core.bridges.agent.api.hosted_routes import router as hosted_routes_router
-from switch_core.bridges.agent.api.hosted_worker_routes import (
- router as hosted_worker_router,
-)
-from switch_core.bridges.agent.api.schemas import ConnectionPlacementsRequest
-from switch_core.bridges.agent.auth import get_agent_from_scope
-from switch_core.bridges.agent.dependencies import get_config as get_worker_config
-from switch_core.bridges.agent.dependencies import get_protocol as get_worker_protocol
-from switch_core.bridges.agent.dependencies import get_session as get_worker_session
-from switch_core.bridges.agent.protocol.agent_connections import (
- TAKEN_OVER,
- AgentConnectionRegistry,
-)
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.hosted_workers import ConsoleView
-from switch_core.db.models import (
- Agent,
- Client,
- HostedLaunch,
- HostedMachine,
- Message,
- ProviderConnection,
- Room,
- TenantMember,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.client_store import ClientStore
-from switch_core.db.stores.hosted_launch_store import HostedLaunchStore
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.db.stores.user_store import UserStore
-from switch_core.gateway import hosted_relay
-from switch_core.gateway.auth import (
- create_jwt,
- get_current_user,
- get_current_user_in_transaction,
-)
-from switch_core.gateway.dependencies import (
- get_config,
- get_protocol,
- get_session,
- get_session_factory,
- get_user_store,
-)
-from switch_core.gateway.hosted_launches import register_identity
-from switch_core.gateway.hosted_launches import router as launch_router
-from switch_core.gateway.hosted_machines import router as machine_router
-from switch_core.gateway.hosted_relay import router as relay_router
-from switch_core.keys import Keyring
-from switch_core.providers.github_installation import (
- GitHubInstallationCredentials,
- RepositoryCredential,
-)
-from switch_core.providers.hosted import HostedControllerSettings
-from tests.switch_core.bridges.agent.protocol.registration_harness import (
- make_owner,
- make_service,
-)
-from tests.switch_core.hosted_machine_helpers import seed_launch, seed_machine
-
-TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
-
-TOKEN = "SYNTHETIC-CONTROLLER-CREDENTIAL-FOR-TESTS"
-HEADERS = {"Authorization": "Bearer " + TOKEN}
-SPEC = {
- "description": "Cloud helper",
- "display_name": None,
- "icon_url": None,
- "instructions": "Help with the repository.",
- "auto_session": True,
- "auto_approve": False,
- "addressing_policy": None,
- "definition_attributes": {},
- "installation_id": 123,
- "repository_id": 456,
- "session_limit": 8,
-}
-
-
-async def worker_app(session_factory, monkeypatch, tmp_path):
- """A running machine with one launch whose identity Core registered.
-
- Yields `(client, request_id, agent_id, service, factory, prepared)`, where
- `prepared` holds the worker capability for the launch's revision and the
- machine id.
- """
- owner = await make_owner(session_factory)
- request_id = str(uuid4())
- async with session_factory() as session:
- session.add(
- TenantMember(tenant_id=require_tenant_id(), user_id=owner, role="member")
- )
- session.add(
- ProviderConnection(
- user_id=owner,
- provider="github",
- kind="oauth",
- encrypted_credential=TEST_KEYRING.encrypt(
- json.dumps(
- {
- "access_token": "SYNTHETIC-GITHUB",
- "expires_at": (
- datetime.now(UTC) + timedelta(hours=1)
- ).timestamp(),
- }
- ),
- ),
- verified_at=datetime.now(UTC),
- )
- )
- await ProviderConnectionStore().save(
- session,
- owner,
- "setup-token",
- TEST_KEYRING.encrypt("SYNTHETIC-CLAUDE"),
- datetime.now(UTC),
- )
- machine = await seed_machine(
- session,
- owner_id=owner,
- slot_id="slot-a",
- state="ready",
- desired_state="running",
- stop_reason=None,
- revision=1,
- generation=1,
- )
- machine_id = machine.id
- await seed_launch(
- session,
- machine=machine,
- request_id=request_id,
- name="cloud-helper",
- state="queued",
- desired_state="running",
- revision=1,
- agent_id=None,
- spec=SPEC,
- )
- await session.commit()
- service = make_service(session_factory)
- service.connections = AgentConnectionRegistry()
- service.event_buffer = EventBuffer(sequence_base=1 << 32)
- service.approval_outcomes = None
- service.config.hosted_sessions_per_agent = 8
- service.config.hosted_idle_stop_minutes = 0
- service.config.hosted_disk_retention_days = 7
- service.client_lifecycle.stop = AsyncMock()
- service.client_lifecycle.delete_record = AsyncMock(side_effect=ClientStore().delete)
- async with session_factory() as session:
- launch = await register_identity(session, service, request_id)
- agent_id = launch.agent_id
- assert agent_id is not None and launch.state == "queued"
- launch = await session.get(
- HostedLaunch, (require_tenant_id(), request_id), populate_existing=True
- )
- capability = HostedLaunchStore().issue_worker_capability(launch, TEST_KEYRING)
- await session.commit()
- settings = HostedControllerSettings(
- tenant_id=require_tenant_id(),
- token=TOKEN,
- machine_slots=["slot-a", "slot-b"],
- github_private_key_path="/tmp/synthetic-signing-key.pem",
- agent_api_endpoint="https://switch.example.com/api/agent",
- )
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", AsyncMock())
- app = FastAPI()
- app.state.hosted_controller_settings = settings
- app.state.github_connections = SimpleNamespace(client_id="synthetic-app")
- app.include_router(hosted_routes_router)
- app.include_router(hosted_worker_router, prefix="/agents")
- app.include_router(hosted_cutover_router, prefix="/agents")
- app.include_router(launch_router)
- app.include_router(relay_router)
- app.include_router(machine_router)
- settings_path = tmp_path / "controller.json"
- settings_path.write_text(
- json.dumps({**settings.model_dump(mode="json"), "token": TOKEN})
- )
- service.config.hosted_controller_config_path = str(settings_path)
- service.config.hosted_github_config_path = "/tmp/synthetic-github.json"
-
- async def worker_session():
- async with session_factory() as session:
- yield session
-
- async def current_user():
- async with session_factory() as session:
- return await session.get(User, owner)
-
- async def worker_agent():
- async with session_factory() as session:
- return await session.get(Agent, agent_id)
-
- app.dependency_overrides[get_current_user] = current_user
- app.dependency_overrides[get_current_user_in_transaction] = current_user
- app.dependency_overrides[get_session] = worker_session
- app.dependency_overrides[get_agent_from_scope] = worker_agent
- app.dependency_overrides[get_worker_config] = lambda: service.config
- app.dependency_overrides[get_worker_session] = worker_session
- app.dependency_overrides[get_session_factory] = lambda: session_factory
- app.dependency_overrides[get_config] = lambda: service.config
- app.dependency_overrides[get_protocol] = lambda: service
- app.dependency_overrides[get_worker_protocol] = lambda: service
- issue = AsyncMock(
- return_value=RepositoryCredential(
- "SYNTHETIC-REPOSITORY",
- datetime.now(UTC) + timedelta(hours=1),
- 456,
- "example/project",
- )
- )
- monkeypatch.setattr(
- "switch_core.bridges.agent.api.hosted_routes.GitHubConnections",
- lambda _: app.state.github_connections,
- )
- monkeypatch.setattr(
- "switch_core.bridges.agent.api.hosted_routes.GitHubInstallationCredentials",
- lambda *_: SimpleNamespace(issue=issue, revoke=AsyncMock()),
- )
- prepared = {
- "worker_capability": capability,
- "revision": 1,
- "machine_id": machine_id,
- }
- async with httpx.AsyncClient(
- transport=httpx.ASGITransport(app=app), base_url="https://switch.example.com"
- ) as client:
- yield client, request_id, agent_id, service, session_factory, prepared
-
-
-async def _agent(factory, agent_id: str) -> Agent:
- async with factory() as session:
- agent = await session.get(Agent, agent_id)
- assert agent is not None
- return agent
-
-
-async def _open(
- service,
- agent: Agent,
- *,
- capability: str | None,
- connection_id: str | None = None,
- boot_id: str = "boot-a",
- speaks: int = 7,
- state_version: int | None = 1,
- expected_generation: int | None = None,
-) -> Any:
- return await poll_events(
- agent.id,
- agent,
- service,
- service.config,
- accept="text/event-stream",
- connection_id=connection_id or str(uuid4()),
- scope="all",
- spawn_capable=True,
- protocol_version=speaks,
- expected_generation=expected_generation,
- worker_capability=capability,
- host_boot_id=boot_id,
- host_instance_id="instance-a",
- worker_state_version=state_version,
- )
-
-
-async def _refusal(coro) -> tuple[int, str]:
- with pytest.raises(HTTPException) as caught:
- await coro
- detail = caught.value.detail
- return caught.value.status_code, detail["code"] if isinstance(detail, dict) else ""
-
-
-async def _first_frames(response, count: int) -> list[tuple[str, dict]]:
- frames = []
- iterator = response.body_iterator
- while len(frames) < count:
- chunk = await asyncio.wait_for(anext(iterator), timeout=2)
- text = chunk.decode() if isinstance(chunk, bytes) else chunk
- if text.startswith(":"):
- continue
- fields = dict(
- line.split(": ", 1) for line in text.strip().splitlines() if ": " in line
- )
- frames.append((fields["event"], json.loads(fields["data"])))
- return frames
-
-
-async def _bump(factory, request_id: str) -> None:
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- launch.revision += 1
- await session.commit()
-
-
-async def issue_capability(factory, request_id: str) -> str:
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- capability = HostedLaunchStore().issue_worker_capability(launch, TEST_KEYRING)
- await session.commit()
- return capability
-
-
-async def set_machine(factory, machine_id: str, **values: Any) -> None:
- async with factory() as session:
- machine = await session.get(HostedMachine, (require_tenant_id(), machine_id))
- assert machine is not None
- for key, value in values.items():
- setattr(machine, key, value)
- await session.commit()
-
-
-async def _machine(factory, machine_id: str) -> HostedMachine:
- async with factory() as session:
- machine = await session.get(HostedMachine, (require_tenant_id(), machine_id))
- assert machine is not None
- return machine
-
-
-async def test_registered_identity_works_in_the_launch_worktree(worker_app):
- _, request_id, agent_id, _, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- options = agent.metadata_["known_agent_options"]
- assert options["repo_dir"] == f"/data/worktrees/{agent_id}/workspace"
- assert agent.metadata_["hosted_launch_id"] == request_id
- machine = await _machine(factory, prepared["machine_id"])
- assert machine.agents_version >= 1
-
-
-async def test_capability_is_idempotent_by_revision(worker_app):
- _, request_id, agent_id, service, factory, prepared = worker_app
- assert await issue_capability(factory, request_id) == prepared["worker_capability"]
- await _bump(factory, request_id)
- third = await issue_capability(factory, request_id)
- assert third != prepared["worker_capability"]
- agent = await _agent(factory, agent_id)
- assert await _refusal(
- _open(service, agent, capability=prepared["worker_capability"])
- ) == (403, "worker_capability_obsolete")
- await _open(service, agent, capability=third)
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert prepared["worker_capability"] not in (
- launch.worker_capability_encrypted or ""
- )
-
-
-async def test_worker_attach_requires_current_capability(worker_app):
- _, request_id, agent_id, service, factory, prepared = worker_app
- capability = prepared["worker_capability"]
- agent = await _agent(factory, agent_id)
- assert await _refusal(_open(service, agent, capability=None)) == (
- 403,
- "worker_capability_required",
- )
- assert await _refusal(_open(service, agent, capability="not-it")) == (
- 403,
- "worker_capability_obsolete",
- )
- assert await _refusal(_open(service, agent, capability=capability, speaks=6)) == (
- 426,
- "upgrade_required",
- )
- for state_version in (None, 0):
- assert await _refusal(
- _open(service, agent, capability=capability, state_version=state_version)
- ) == (426, "upgrade_required")
- assert service.connections.for_agent(agent_id) == []
-
- response = await _open(service, agent, capability=capability)
- (conn,) = service.connections.for_agent(agent_id)
- assert conn.worker is not None and conn.worker.launch_revision == 1
- frames = await _first_frames(response, 2)
- assert frames[0][0] == "connection_state"
- assert frames[1][0] == "worker_attached"
- assert frames[1][1]["launch_revision"] == 1
- assert frames[1][1]["cancelled"] == []
-
- await _bump(factory, request_id)
- service.connections.supersede(agent_id, 2)
- assert service.connections.get(conn.id) is None
- assert not service.connections.ring_worker(agent_id, "operation", {"id": "x"})
- assert await _refusal(
- _open(service, agent, capability=capability, connection_id=conn.id)
- ) == (403, "worker_capability_obsolete")
-
-
-async def test_local_console_cannot_take_over_hosted_stream(worker_app):
- _, _, agent_id, service, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- worker_id = str(uuid4())
- await _open(
- service,
- agent,
- capability=prepared["worker_capability"],
- connection_id=worker_id,
- )
- worker = service.connections.get(worker_id)
- generation = worker.stream_generation
-
- assert (await _refusal(_open(service, agent, capability=None)))[0] == 403
- assert (
- await _refusal(_open(service, agent, capability=None, connection_id=worker_id))
- )[0] == 403
- local_id = str(uuid4())
- placements = ConnectionPlacementsRequest(
- connection_id=local_id, generation=0, placements={}
- )
- assert await _refusal(
- connection_placements(agent_id, placements, agent, service)
- ) == (403, "hosted_worker_only")
- assert await _refusal(_poll(service, agent)) == (403, "hosted_worker_only")
- assert worker.stream_generation == generation
- assert service.connections.get(worker_id) is worker
-
-
-async def _poll(service, agent: Agent) -> Any:
- return await poll_events(
- agent.id, agent, service, service.config, timeout=0, accept="application/json"
- )
-
-
-async def test_second_worker_refused_while_first_alive(worker_app):
- _, _, agent_id, service, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- capability = prepared["worker_capability"]
- first = str(uuid4())
- await _open(service, agent, capability=capability, connection_id=first)
- assert await _refusal(
- _open(service, agent, capability=capability, boot_id="boot-b")
- ) == (
- 409,
- "worker_already_attached",
- )
- assert [conn.id for conn in service.connections.for_agent(agent_id)] == [first]
-
-
-async def test_same_boot_takeover_evicts_old(worker_app):
- _, _, agent_id, service, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- capability = prepared["worker_capability"]
- first = str(uuid4())
- await _open(service, agent, capability=capability, connection_id=first)
- old = service.connections.get(first)
- second = str(uuid4())
- await _open(service, agent, capability=capability, connection_id=second)
- assert service.connections.get(first) is None
- assert old.closure is not None and old.closure.code == TAKEN_OVER.code
- assert service.connections.attached_worker(agent_id).id == second
-
-
-async def test_same_connection_reattach_keeps_the_generation_fence(worker_app):
- _, _, agent_id, service, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- capability = prepared["worker_capability"]
- conn_id = str(uuid4())
- await _open(service, agent, capability=capability, connection_id=conn_id)
- held = service.connections.get(conn_id).stream_generation
- await _open(
- service,
- agent,
- capability=capability,
- connection_id=conn_id,
- expected_generation=held,
- )
- conn = service.connections.get(conn_id)
- assert conn.stream_generation != held and conn.worker is not None
- status, code = await _refusal(
- _open(
- service,
- agent,
- capability=capability,
- connection_id=conn_id,
- expected_generation=held,
- )
- )
- assert status == 409
-
-
-async def test_idle_report_answers_catch_up(worker_app):
- client, request_id, agent_id, service, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- conn_id = str(uuid4())
- await _open(
- service, agent, capability=prepared["worker_capability"], connection_id=conn_id
- )
- conn = service.connections.get(conn_id)
- body = {
- "connection_id": conn_id,
- "generation": conn.stream_generation,
- "report_seq": 1,
- "relays_through": 0,
- "busy": False,
- "reasons": [],
- "sessions": {"total": 0, "live": 0, "parked": 0, "failed": 0},
- }
- response = await client.post(f"/agents/{agent_id}/connection/idle", json=body)
- assert response.status_code == 200, response.text
- assert response.json()["queued_operations"] == []
- assert response.json()["credential_revision"] is not None
- assert service.connections.fresh_idle_report(agent_id, request_id, 1) is not None
- stale = await client.post(
- f"/agents/{agent_id}/connection/idle",
- json={**body, "generation": conn.stream_generation + 1},
- )
- assert stale.status_code == 409
-
-
-async def test_relay_reply_is_fenced(worker_app):
- client, request_id, agent_id, service, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- conn_id = str(uuid4())
- await _open(
- service, agent, capability=prepared["worker_capability"], connection_id=conn_id
- )
- conn = service.connections.get(conn_id)
- relay = service.connections.relays.register(
- tenant_id=require_tenant_id(),
- agent_id=agent_id,
- binding=conn.worker,
- relay_seq=None,
- core_boot=1,
- connection_id=conn_id,
- generation=conn.stream_generation,
- timeout_ms=5000,
- )
- fence = {"connection_id": conn_id, "generation": conn.stream_generation}
- path = f"/agents/{agent_id}/connection/relay/{relay.id}"
- unknown = await client.post(
- f"/agents/{agent_id}/connection/relay/{uuid4()}",
- json={**fence, "ok": True, "value": 1},
- )
- assert unknown.status_code == 404
- stale = await client.post(
- path, json={**fence, "generation": conn.stream_generation + 1, "ok": True}
- )
- assert stale.status_code == 409
- too_large = await client.post(
- path,
- content=json.dumps({**fence, "ok": True, "value": "x" * (2 * 1024 * 1024)}),
- headers={"content-type": "application/json"},
- )
- assert too_large.status_code == 413
- answered = await client.post(path, json={**fence, "ok": True, "value": {"a": 1}})
- assert answered.status_code == 200, answered.text
- assert (await relay.future) == {"ok": True, "value": {"a": 1}}
- again = await client.post(path, json={**fence, "ok": True, "value": 2})
- assert again.status_code == 409
-
-
-async def test_room_notice_requires_the_worker(worker_app):
- client, _, agent_id, service, factory, prepared = worker_app
- response = await client.post(
- f"/agents/{agent_id}/room-notices",
- json={
- "connection_id": str(uuid4()),
- "generation": 0,
- "room_id": "!room:example.com",
- "message_id": "$event",
- "thread_id": None,
- "reason": "startup",
- },
- )
- assert response.status_code == 409
- assert response.json()["detail"]["code"] == "generation_changed"
-
-
-async def _ready_worker(worker_app) -> Any:
- _, request_id, agent_id, service, factory, prepared = worker_app
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- launch.state = "ready"
- await session.commit()
- agent = await _agent(factory, agent_id)
- conn_id = str(uuid4())
- await _open(
- service, agent, capability=prepared["worker_capability"], connection_id=conn_id
- )
- conn = service.connections.get(conn_id)
- conn.worker_frames.drain()
- return conn
-
-
-async def _answer_relays(client, conn, value: Any, seen: list[dict]) -> None:
- while True:
- for event, data in conn.worker_frames.drain():
- if event != "relay":
- continue
- seen.append(data)
- await client.post(
- f"/agents/{conn.agent_id}/connection/relay/{data['id']}",
- json={
- "connection_id": conn.id,
- "generation": conn.stream_generation,
- "ok": True,
- "value": value,
- },
- )
- await asyncio.sleep(0.01)
-
-
-async def _launch(factory, request_id: str) -> HostedLaunch:
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch is not None
- return launch
-
-
-async def test_read_only_relay_does_not_renew_activity(worker_app):
- client, request_id, _, service, factory, _ = worker_app
- conn = await _ready_worker(worker_app)
- before = await _launch(factory, request_id)
- seen: list[dict] = []
- answering = asyncio.create_task(_answer_relays(client, conn, {"health": 1}, seen))
- try:
- response = await client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"health": True}, "timeout_ms": 5000},
- )
- finally:
- answering.cancel()
- assert response.status_code == 200, response.text
- assert response.json()["ok"] is True
- assert response.json()["value"] == {"health": 1}
- assert response.json()["worker"]["generation"] == conn.stream_generation
- assert seen[0]["relay_seq"] is None
- after = await _launch(factory, request_id)
- assert after.relay_seq == before.relay_seq
- assert after.active_at == before.active_at
-
-
-async def test_mutating_relay_takes_a_sequence(worker_app):
- client, request_id, _, service, factory, _ = worker_app
- conn = await _ready_worker(worker_app)
- seen: list[dict] = []
- answering = asyncio.create_task(_answer_relays(client, conn, {"accepted": 1}, seen))
- try:
- response = await client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"forget": str(uuid4())}, "timeout_ms": 5000},
- )
- finally:
- answering.cancel()
- assert response.status_code == 200, response.text
- assert seen[0]["relay_seq"] == 1
- assert seen[0]["deadline_ms"] > 0
- assert (await _launch(factory, request_id)).relay_seq == 1
-
-
-async def test_relay_queue_refusal_takes_no_sequence(worker_app):
- client, request_id, _, service, factory, _ = worker_app
- conn = await _ready_worker(worker_app)
- slots = [conn.worker_frames.reserve(1) for _ in range(64)]
- response = await client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"forget": str(uuid4())}, "timeout_ms": 5000},
- )
- assert response.status_code == 503
- assert response.json()["error"]["code"] == "worker_busy"
- assert (await _launch(factory, request_id)).relay_seq == 0
- for slot in slots:
- slot.release()
-
-
-async def test_gateway_cancel_after_commit_still_enqueues(worker_app):
- client, request_id, _, service, factory, _ = worker_app
- conn = await _ready_worker(worker_app)
- request = asyncio.create_task(
- client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"forget": str(uuid4())}, "timeout_ms": 5000},
- )
- )
- for _ in range(200):
- if (await _launch(factory, request_id)).relay_seq == 1:
- break
- await asyncio.sleep(0.01)
- request.cancel()
- with pytest.raises(asyncio.CancelledError):
- await request
- await asyncio.sleep(0.05)
- frames = [data for event, data in conn.worker_frames.drain() if event == "relay"]
- assert [frame["relay_seq"] for frame in frames] == [1]
-
-
-async def test_relay_refusals(worker_app):
- client, request_id, agent_id, service, factory, _ = worker_app
- path = f"/hosted-launches/{request_id}/relay"
- missing = await client.post(
- f"/hosted-launches/{uuid4()}/relay",
- json={"message": {"health": True}, "timeout_ms": 1000},
- )
- assert missing.status_code == 404
- starting = await client.post(
- path, json={"message": {"health": True}, "timeout_ms": 1000}
- )
- assert starting.status_code == 409
- assert starting.json()["error"]["code"] == "worker_waking"
- await _ready_worker(worker_app)
- for message in ({"ensure": {}}, {"room": {}}, {"approvals": {}}):
- refused = await client.post(path, json={"message": message, "timeout_ms": 1000})
- assert refused.status_code == 400
- assert refused.json()["error"]["code"] == "refused_message"
- too_large = await client.post(
- path,
- content=json.dumps({"message": {"health": "x" * (2 * 1024 * 1024)}}),
- headers={"content-type": "application/json"},
- )
- assert too_large.status_code == 413
-
-
-async def _relay_code(client, request_id: str, message: dict) -> tuple[int, dict]:
- response = await client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": message, "timeout_ms": 1000},
- )
- return response.status_code, response.json()
-
-
-READ_ONLY = {"health": True}
-
-
-async def test_relay_error_codes_in_order(worker_app):
- client, request_id, _, service, factory, prepared = worker_app
- machine_id = prepared["machine_id"]
- await _ready_worker(worker_app)
- mutating = {"forget": str(uuid4())}
-
- await set_machine(
- factory, machine_id, desired_state="stopped", stop_reason="owner", revision=2
- )
- await set_launch_values(
- factory, request_id, state="error", error_code="agent_crashed"
- )
- for message in (READ_ONLY, mutating):
- status, body = await _relay_code(client, request_id, message)
- assert (status, body["error"]["code"]) == (409, "machine_stopped")
- assert "wake_available" not in body
- await set_launch_values(factory, request_id, state="ready", error_code=None)
-
- await set_machine(factory, machine_id, stop_reason="idle", state="stopped")
- status, body = await _relay_code(client, request_id, READ_ONLY)
- assert (status, body["error"]["code"]) == (409, "worker_sleeping")
- assert body["wake_available"] is True
- assert body["worker"]["sleeping"] is True
- assert body["worker"]["request_id"] == request_id
- assert (await _machine(factory, machine_id)).desired_state == "stopped"
-
- status, body = await _relay_code(client, request_id, mutating)
- assert (status, body["error"]["code"]) == (409, "worker_waking")
- assert body["worker"]["sleeping"] is False
- woken = await _machine(factory, machine_id)
- assert (woken.desired_state, woken.stop_reason, woken.revision) == (
- "running",
- None,
- 3,
- )
- assert (await _launch(factory, request_id)).relay_seq == 0
- status, body = await _relay_code(client, request_id, READ_ONLY)
- assert body["error"]["code"] == "worker_waking"
-
- await set_machine(factory, machine_id, state="ready")
- await set_launch_values(
- factory, request_id, desired_state="stopped", state="stopped"
- )
- status, body = await _relay_code(client, request_id, mutating)
- assert (status, body["error"]["code"]) == (409, "agent_stopped")
-
- await set_launch_values(
- factory,
- request_id,
- desired_state="running",
- state="error",
- error_code="agent_crashed",
- )
- status, body = await _relay_code(client, request_id, READ_ONLY)
- assert (status, body["error"]["code"]) == (409, "agent_crashed")
-
- await set_launch_values(factory, request_id, state="ready", error_code=None)
- service.connections = AgentConnectionRegistry()
- status, body = await _relay_code(client, request_id, READ_ONLY)
- assert (status, body["error"]["code"]) == (409, "worker_not_attached")
- assert set(body["worker"]) >= {"machine_id", "process_state", "oom_kills"}
-
-
-async def test_relay_to_an_errored_machine_is_machine_error(worker_app):
- client, request_id, _, _, factory, prepared = worker_app
- machine_id = prepared["machine_id"]
- await set_machine(factory, machine_id, state="error")
- assert (await _launch(factory, request_id)).state == "queued"
- for message in (READ_ONLY, {"forget": str(uuid4())}):
- status, body = await _relay_code(client, request_id, message)
- assert (status, body["error"]["code"]) == (409, "machine_error")
- assert "wake_available" not in body
- assert (await _launch(factory, request_id)).relay_seq == 0
-
-
-async def test_relay_to_a_machine_needing_an_administrator_says_so(worker_app):
- client, request_id, _, _, factory, prepared = worker_app
- await set_machine(
- factory,
- prepared["machine_id"],
- state="error",
- error_code="machine_needs_attention",
- )
- status, body = await _relay_code(client, request_id, READ_ONLY)
- assert (status, body["error"]) == (
- 409,
- {
- "code": "machine_error",
- "message": "The cloud machine needs attention. Contact your server administrator.",
- },
- )
-
-
-async def test_relay_to_an_errored_owner_stopped_machine_is_machine_error(worker_app):
- client, request_id, _, _, factory, prepared = worker_app
- await _ready_worker(worker_app)
- await set_machine(
- factory,
- prepared["machine_id"],
- desired_state="stopped",
- stop_reason="owner",
- state="error",
- error_code="machine_needs_attention",
- revision=2,
- )
- for message in (READ_ONLY, {"forget": str(uuid4())}):
- status, body = await _relay_code(client, request_id, message)
- assert (status, body["error"]) == (
- 409,
- {
- "code": "machine_error",
- "message": "The cloud machine needs attention. Contact your server administrator.",
- },
- )
-
-
-async def test_relay_to_an_errored_sleeping_machine_does_not_wake_it(worker_app):
- client, request_id, _, _, factory, prepared = worker_app
- machine_id = prepared["machine_id"]
- await _ready_worker(worker_app)
- await set_machine(
- factory,
- machine_id,
- desired_state="stopped",
- stop_reason="idle",
- state="error",
- revision=2,
- )
- for message in (READ_ONLY, {"forget": str(uuid4())}):
- status, body = await _relay_code(client, request_id, message)
- assert (status, body["error"]["code"]) == (409, "machine_error")
- assert "wake_available" not in body
- machine = await _machine(factory, machine_id)
- assert (machine.desired_state, machine.stop_reason, machine.revision) == (
- "stopped",
- "idle",
- 2,
- )
-
-
-async def test_idle_sleep_wakes_only_for_a_running_agent(worker_app):
- client, request_id, _, _, factory, prepared = worker_app
- machine_id = prepared["machine_id"]
- await set_machine(
- factory,
- machine_id,
- desired_state="stopped",
- stop_reason="idle",
- state="stopped",
- revision=2,
- )
- await set_launch_values(
- factory, request_id, desired_state="stopped", state="stopped"
- )
- status, body = await _relay_code(client, request_id, {"forget": str(uuid4())})
- assert (status, body["error"]["code"]) == (409, "agent_stopped")
- assert (await _machine(factory, machine_id)).desired_state == "stopped"
-
-
-async def test_mutating_relay_renews_the_machine(worker_app):
- client, request_id, _, _, factory, prepared = worker_app
- conn = await _ready_worker(worker_app)
- before = await _machine(factory, prepared["machine_id"])
- seen: list[dict] = []
- answering = asyncio.create_task(_answer_relays(client, conn, {"accepted": 1}, seen))
- try:
- response = await client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"forget": str(uuid4())}, "timeout_ms": 5000},
- )
- finally:
- answering.cancel()
- assert response.status_code == 200, response.text
- after = await _machine(factory, prepared["machine_id"])
- assert after.active_at > before.active_at
-
-
-async def set_launch_values(factory, request_id: str, **values: Any) -> None:
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- assert launch is not None
- for key, value in values.items():
- setattr(launch, key, value)
- await session.commit()
-
-
-async def test_relay_timeout_cancels_on_the_worker(worker_app):
- client, request_id, _, service, factory, _ = worker_app
- conn = await _ready_worker(worker_app)
- response = await client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"health": True}, "timeout_ms": 50},
- )
- assert response.status_code == 504
- events = [event for event, _ in conn.worker_frames.drain()]
- assert events == ["relay", "relay_cancel"]
-
-
-async def test_auto_start_off_notice_is_posted_once(worker_app):
- client, _, agent_id, service, factory, _ = worker_app
- conn = await _ready_worker(worker_app)
- async with factory() as session:
- room = Room(
- transport_room_id=f"!{uuid4().hex[:8]}:example.com",
- name="r",
- description="",
- )
- session.add(room)
- await session.flush()
- room_id = room.id
- session.add(
- Message(
- seq=1,
- room_id=room_id,
- transport_event_id="$addressed",
- sender_id="@someone:example.com",
- event_type="m.room.message",
- msgtype="m.text",
- body="hello",
- content={"body": "hello"},
- )
- )
- sender = await session.scalar(
- select(Client.transport_user_id)
- .join(Agent, Agent.client_id == Client.id)
- .where(Agent.id == agent_id)
- )
- await session.commit()
- sent: list[tuple[str, str]] = []
-
- async def send_message(agent, room, body, *, thread_id, extra_content):
- sent.append((room, body))
- async with factory() as session:
- session.add(
- Message(
- seq=2 + len(sent),
- room_id=room,
- transport_event_id=f"$notice{len(sent)}",
- sender_id=sender,
- event_type="m.room.message",
- msgtype="m.notice",
- body=body,
- content={"body": body, **extra_content},
- )
- )
- await session.commit()
-
- service.send_message = send_message
- notice = {
- "connection_id": conn.id,
- "generation": conn.stream_generation,
- "room_id": room_id,
- "message_id": "$addressed",
- "thread_id": None,
- "reason": "auto_start_off",
- }
- first = await client.post(f"/agents/{agent_id}/room-notices", json=notice)
- second = await client.post(f"/agents/{agent_id}/room-notices", json=notice)
- assert first.status_code == 200, first.text
- assert first.json()["posted"] is True
- assert second.json()["posted"] is False
- assert len(sent) == 1
- assert "not set to start one automatically" in sent[0][1]
-
-
[email protected]("order", ["reply_then_teardown", "teardown_then_reply"])
[email protected]("stage", ["dispatching", "awaiting_reply"])
-async def test_gateway_cancel_with_teardown_and_immediate_reply(
- worker_app, caplog, order, stage
-):
- client, request_id, _, service, factory, _ = worker_app
- conn = await _ready_worker(worker_app)
- app = client._transport.app
- request_sessions: list[str] = []
-
- async def tracked_session():
- async with factory() as session:
- request_sessions.append("open")
- try:
- yield session
- finally:
- request_sessions.append("closed")
-
- app.dependency_overrides[get_session] = tracked_session
- loop = asyncio.get_running_loop()
- loop_errors: list[dict[str, Any]] = []
- loop.set_exception_handler(lambda _loop, context: loop_errors.append(context))
- try:
- request = asyncio.create_task(
- client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"forget": str(uuid4())}, "timeout_ms": 5000},
- )
- )
- frames: list[dict[str, Any]] = []
- deadline = loop.time() + 5
- while not frames and loop.time() < deadline:
- frames += [d for e, d in conn.worker_frames.drain() if e == "relay"]
- await asyncio.sleep(0.001)
- assert len(frames) == 1
- if stage == "awaiting_reply":
- for _ in range(20):
- await asyncio.sleep(0)
- reply = client.post(
- f"/agents/{conn.agent_id}/connection/relay/{frames[0]['id']}",
- json={
- "connection_id": conn.id,
- "generation": conn.stream_generation,
- "ok": True,
- "value": {"accepted": 1},
- },
- )
- if order == "reply_then_teardown":
- replying = asyncio.create_task(reply)
- request.cancel()
- else:
- request.cancel()
- replying = asyncio.create_task(reply)
- outcome, replied = await asyncio.gather(
- request, replying, return_exceptions=True
- )
- await asyncio.gather(*hosted_relay._dispatches)
- finally:
- loop.set_exception_handler(None)
- assert isinstance(outcome, asyncio.CancelledError)
- assert replied.status_code == 200, replied.text
- assert replied.json() == {"ok": True}
- assert frames[0]["relay_seq"] == 1
- assert conn.worker_frames.drain() == []
- assert (await _launch(factory, request_id)).relay_seq == 1
- assert request_sessions == ["open", "closed"]
- relays = service.connections.relays
- assert relays.get(frames[0]["id"]).future.done()
- assert not relays.mutating_pending(conn.agent_id, conn.worker.launch_id)
- assert loop_errors == []
- logged = caplog.text.lower()
- assert "session closed" not in logged
- assert "connection in use" not in logged
- assert "another operation is in progress" not in logged
-
-
-async def test_relay_stream_shows_a_full_worker_queue(worker_app):
- _, request_id, agent_id, service, factory, _ = worker_app
- conn = await _ready_worker(worker_app)
- session_id = str(uuid4())
- slots = [conn.worker_frames.reserve(1) for _ in range(64)]
- stream = hosted_relay.relay_events(
- service,
- require_tenant_id(),
- agent_id,
- request_id,
- ConsoleView(frozenset({session_id})),
- )
- try:
- assert (await anext(stream)).startswith(b"event: worker\n")
- refused = (await anext(stream)).decode()
- assert refused.startswith("event: error\n")
- assert json.loads(refused.split("data: ", 1)[1]) == {
- "sessionId": session_id,
- "code": "worker_busy",
- "message": "The worker's frame queue is full. Retry shortly.",
- }
- for slot in slots:
- slot.release()
- retried = asyncio.create_task(anext(stream))
- for _ in range(300):
- frames = [d for e, d in conn.worker_frames.drain() if e == "relay"]
- if frames:
- break
- await asyncio.sleep(0.01)
- assert [frame["message"] for frame in frames] == [{"subscribe": session_id}]
- retried.cancel()
- with pytest.raises(asyncio.CancelledError):
- await retried
- finally:
- await stream.aclose()
-
-
-WORKER_ATTACHED_FIELDS = {
- "launch_revision",
- "limits",
- "idle",
- "credential_revision",
- "queued_operations",
- "relay_fence",
- "cancelled",
-}
-RELAY_FIELDS = {"id", "deadline_ms", "relay_seq", "message"}
-
-
-async def test_worker_attaches_answers_a_relay_and_reports_idle_over_the_wire(
- worker_app,
-):
- client, request_id, agent_id, service, factory, prepared = worker_app
- async with factory() as session:
- launch = await session.get(HostedLaunch, (require_tenant_id(), request_id))
- launch.state = "ready"
- await session.commit()
- agent = await _agent(factory, agent_id)
- conn_id = str(uuid4())
- stream = await _open(
- service, agent, capability=prepared["worker_capability"], connection_id=conn_id
- )
- (state, _), (attached_event, attached) = await _first_frames(stream, 2)
- assert (state, attached_event) == ("connection_state", "worker_attached")
- assert set(attached) == WORKER_ATTACHED_FIELDS
- generation = service.connections.get(conn_id).stream_generation
-
- sent_at_ms = int(time.time() * 1000)
- relaying = asyncio.create_task(
- client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"health": True}, "timeout_ms": 5000},
- )
- )
- ((relay_event, relay),) = await _first_frames(stream, 1)
- assert relay_event == "relay"
- assert set(relay) == RELAY_FIELDS
- assert relay["relay_seq"] is None
- assert relay["message"] == {"health": True}
- assert sent_at_ms + 4000 <= relay["deadline_ms"] <= int(time.time() * 1000) + 5000
- fence = {"connection_id": conn_id, "generation": generation}
- reply = await client.post(
- f"/agents/{agent_id}/connection/relay/{relay['id']}",
- json={**fence, "ok": True, "value": {"health": 1}},
- )
- assert reply.status_code == 200, reply.text
- relayed = await relaying
- assert relayed.status_code == 200, relayed.text
- assert relayed.json()["value"] == {"health": 1}
-
- idle = await client.post(
- f"/agents/{agent_id}/connection/idle",
- json={
- **fence,
- "report_seq": 1,
- "relays_through": attached["relay_fence"],
- "busy": False,
- "reasons": [],
- "sessions": {"total": 0, "live": 0, "parked": 0, "failed": 0},
- },
- )
- assert idle.status_code == 200, idle.text
- assert set(idle.json()) >= {"queued_operations", "credential_revision"}
- assert idle.json()["credential_revision"] == attached["credential_revision"]
- assert service.connections.fresh_idle_report(agent_id, request_id, 1) is not None
-
-
-async def test_relay_with_real_auth_opens_two_transactions(worker_app):
- """Tenant resolution, then the user and the launch on one transaction."""
- client, request_id, _, service, factory, _ = worker_app
- await _ready_worker(worker_app)
- app = client._transport.app
- del app.dependency_overrides[get_current_user]
- del app.dependency_overrides[get_current_user_in_transaction]
- app.dependency_overrides[get_user_store] = UserStore
- service.config.gateway_tenant_choice_enabled = False
- owner = (await _launch(factory, request_id)).owner_id
- client.cookies.set(
- "switch_auth", create_jwt(owner, "owner@test", "user", TEST_KEYRING, None)
- )
- begins: list[object] = []
- engine = factory.kw["bind"].sync_engine
- listener = begins.append
- orm_event.listen(engine, "begin", listener)
- try:
- response = await client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"health": True}, "timeout_ms": 50},
- )
- finally:
- orm_event.remove(engine, "begin", listener)
- assert response.status_code == 504, response.text
- assert len(begins) == 2
- stranger = await client.post(
- f"/hosted-launches/{uuid4()}/relay",
- json={"message": {"health": True}, "timeout_ms": 50},
- )
- assert stranger.status_code == 404
- client.cookies.clear()
- assert (
- await client.post(
- f"/hosted-launches/{request_id}/relay",
- json={"message": {"health": True}, "timeout_ms": 50},
- )
- ).status_code == 401
-
-
-class _Socket:
- """The handler's side of a WebSocket, in this test's event loop.
-
- Records what is sent and how it closes, and hands the handler whatever the
- test queues as the client's messages; `None` is the client going away.
- """
-
- def __init__(self) -> None:
- self.sent: list[dict[str, Any]] = []
- self.close_code: int | None = None
- self._incoming: asyncio.Queue[dict[str, Any] | None] = asyncio.Queue()
- self._changed = asyncio.Event()
-
- async def accept(self) -> None:
- return None
-
- async def send_json(self, message: dict[str, Any]) -> None:
- self.sent.append(message)
- self._changed.set()
-
- async def receive_json(self) -> dict[str, Any]:
- message = await self._incoming.get()
- if message is None:
- raise WebSocketDisconnect(1000)
- return message
-
- async def close(self, code: int = 1000) -> None:
- if self.close_code is None:
- self.close_code = code
- self._changed.set()
-
- def leave(self) -> None:
- self._incoming.put_nowait(None)
-
- async def frames(self, count: int) -> list[tuple[str, dict[str, Any]]]:
- """The next `count` frames sent, pings skipped."""
-
- def received() -> list[tuple[str, dict[str, Any]]]:
- return [(m["event"], m["data"]) for m in self.sent if m["event"] != "ping"]
-
- async def wait() -> None:
- while len(received()) < count:
- self._changed.clear()
- await self._changed.wait()
-
- await asyncio.wait_for(wait(), timeout=2)
- return received()[:count]
-
-
-async def _open_socket(
- service,
- agent: Agent,
- *,
- capability: str | None,
- connection_id: str | None = None,
- speaks: int = 7,
- state_version: int | None = 1,
-) -> tuple[_Socket, asyncio.Task[None]]:
- socket = _Socket()
- task = asyncio.create_task(
- connection_socket(
- websocket=socket, # type: ignore[arg-type]
- agent_id=agent.id,
- agent=agent,
- protocol=service,
- config=service.config,
- connection_id=connection_id or str(uuid4()),
- scope="all",
- spawn_capable=True,
- protocol_version=speaks,
- worker_capability=capability,
- host_boot_id="boot-a",
- host_instance_id="instance-a",
- worker_state_version=state_version,
- )
- )
- return socket, task
-
-
- ("change", "refused"),
- [
- ({"capability": None}, (403, "worker_capability_required")),
- ({"capability": "not-it"}, (403, "worker_capability_obsolete")),
- ({"speaks": 6}, (426, "upgrade_required")),
- ({"state_version": None}, (426, "upgrade_required")),
- ],
-)
-async def test_the_socket_refuses_a_worker_open_as_the_stream_does(
- worker_app, change: dict[str, Any], refused: tuple[int, str]
-):
- _, _, agent_id, service, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- opening = {"capability": prepared["worker_capability"], **change}
- assert await _refusal(_open(service, agent, **opening)) == refused
-
- socket, task = await _open_socket(service, agent, **opening)
- await asyncio.wait_for(task, timeout=2)
-
- status, code = refused
- ((event, data),) = await socket.frames(1)
- assert event == "refused"
- assert data["status"] == status
- assert data["detail"]["code"] == code
- assert socket.close_code == 4000 + status
- assert service.connections.for_agent(agent_id) == []
-
-
-async def test_a_worker_over_the_socket_is_sent_its_frames(worker_app, monkeypatch):
- # Pings every 50 ms, so the socket notices its client leave promptly.
- monkeypatch.setattr(handlers, "HEARTBEAT_INTERVAL_SECONDS", 0.05)
- _, _, agent_id, service, factory, prepared = worker_app
- agent = await _agent(factory, agent_id)
- conn_id = str(uuid4())
- socket, task = await _open_socket(
- service, agent, capability=prepared["worker_capability"], connection_id=conn_id
- )
- try:
- (state, _), (attached, data) = await socket.frames(2)
- assert (state, attached) == ("connection_state", "worker_attached")
- assert set(data) == WORKER_ATTACHED_FIELDS
- conn = service.connections.get(conn_id)
- assert conn.worker is not None and conn.stream_transport == "websocket"
-
- assert service.connections.ring_worker(agent_id, "wake", {"entries": []})
- assert (await socket.frames(3))[2] == ("wake", {"entries": []})
- finally:
- socket.leave()
- await asyncio.wait_for(task, timeout=2)
diff --git a/core/tests/switch_core/gateway/test_provider_connections.py b/core/tests/switch_core/gateway/test_provider_connections.py
deleted file mode 100644
index 103fbc4ad..000000000
--- a/core/tests/switch_core/gateway/test_provider_connections.py
+++ /dev/null
@@ -1,450 +0,0 @@
-import json
-from datetime import UTC, datetime
-from types import SimpleNamespace
-from unittest.mock import AsyncMock, Mock
-
-import httpx
-import pytest
-from fastapi import FastAPI
-from sqlalchemy import select
-
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.db.models import (
- HostedLaunch,
- HostedMachine,
- ProviderConnection,
- Tenant,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.gateway.auth import get_current_user
-from switch_core.gateway.dependencies import get_config, get_protocol, get_session
-from switch_core.gateway.provider_connections import router
-from switch_core.keys import Keyring
-from switch_core.providers.claude_verifier import ClaudeVerificationError
-from switch_core.providers.credentials import validate_provider_credential
-from switch_core.tenant_context import tenant_scope
-from tests.switch_core.hosted_machine_helpers import seed_machine
-
-TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
-
-
-async def connection_app(session_factory):
- async with session_factory() as session:
- first = User(
- id="first",
- name="First",
- role="user",
- password_hash="unused",
- )
- second = User(
- id="second",
- name="Second",
- role="user",
- password_hash="unused",
- )
- session.add_all([first, second, Tenant(id="other", slug="other", name="Other")])
- await session.commit()
- identity = {"user": first}
- app = FastAPI()
- app.include_router(router, prefix="/provider-connections")
- verifier = AsyncMock()
- app.state.claude_verifier = verifier
-
- async def sessions():
- async with session_factory() as session:
- yield session
-
- app.dependency_overrides[get_session] = sessions
- app.dependency_overrides[get_protocol] = lambda: SimpleNamespace(
- connections=AgentConnectionRegistry()
- )
- app.dependency_overrides[get_current_user] = lambda: identity["user"]
- app.dependency_overrides[get_config] = lambda: SimpleNamespace(
- keyring=TEST_KEYRING,
- hosted_provider_verification_enabled=False,
- )
- async with httpx.AsyncClient(
- transport=httpx.ASGITransport(app=app), base_url="https://switch.example.com"
- ) as client:
- yield client, identity, verifier, session_factory, app
-
-
-async def test_save_is_encrypted_scoped_and_survives_new_request(connection_app):
- client, identity, verifier, factory, _ = connection_app
- credential = "sk-ant-api-SYNTHETIC-PLACEHOLDER"
- response = await client.put(
- "/provider-connections/claude",
- json={"kind": "api-key", "credential": credential},
- )
- assert response.status_code == 200
- assert credential not in response.text
- verifier.verify.assert_awaited_once_with("api-key", credential)
- async with factory() as session:
- row = await ProviderConnectionStore().get(session, "first")
- assert row.encrypted_credential != credential
- assert TEST_KEYRING.decrypt(row.encrypted_credential) == credential
- assert (await client.get("/provider-connections/claude")).json()[
- "status"
- ] == "connected"
- identity["user"] = SimpleNamespace(id="second")
- assert (await client.get("/provider-connections/claude")).json() == {
- "status": "not_connected"
- }
- await client.delete("/provider-connections/claude")
- identity["user"] = SimpleNamespace(id="first")
- with tenant_scope("other"):
- assert (await client.get("/provider-connections/claude")).json() == {
- "status": "not_connected"
- }
- await client.delete("/provider-connections/claude")
- assert (await client.get("/provider-connections/claude")).json()[
- "status"
- ] == "connected"
- assert (await client.delete("/provider-connections/claude")).status_code == 204
- assert (await client.get("/provider-connections/claude")).json()[
- "status"
- ] == "not_connected"
-
-
-async def test_failed_replacement_preserves_previous_connection(connection_app):
- client, _, verifier, _, _ = connection_app
- assert (
- await client.put(
- "/provider-connections/claude",
- json={
- "kind": "setup-token",
- "credential": "sk-ant-oat-SYNTHETIC-PLACEHOLDER",
- },
- )
- ).status_code == 200
- verifier.verify.side_effect = ClaudeVerificationError(
- "Claude could not complete the check."
- )
- result = await client.put(
- "/provider-connections/claude",
- json={"kind": "api-key", "credential": "sk-ant-api-SYNTHETIC-REPLACEMENT"},
- )
- assert result.status_code == 422
- assert "SYNTHETIC" not in result.text
- assert (await client.get("/provider-connections/claude")).json()[
- "kind"
- ] == "setup-token"
-
-
- "body",
- [
- {"kind": "wrong", "credential": "PRIVATE-PLACEHOLDER"},
- {"kind": "api-key", "credential": "sk-ant-oat-PRIVATE-PLACEHOLDER"},
- {"kind": "setup-token", "credential": ["PRIVATE-PLACEHOLDER"]},
- {"kind": "api-key", "credential": "sk-ant-api-PRIVATE PLACEHOLDER"},
- ],
-)
-async def test_bad_input_never_echoes_secret(connection_app, body):
- client, _, verifier, _, _ = connection_app
- result = await client.put("/provider-connections/claude", json=body)
- assert result.status_code == 400
- assert "PRIVATE" not in result.text
- verifier.verify.assert_not_awaited()
-
-
-async def test_disabled_verifier_and_oversize_body(connection_app):
- client, _, _, _, app = connection_app
- result = await client.put("/provider-connections/claude", content="x" * 21000)
- assert result.status_code == 413
- app.state.claude_verifier = None
- assert (await client.get("/provider-connections/claude")).status_code == 503
-
-
-async def test_provider_table_enforces_rls(rls_harness):
- async with rls_harness.owner() as session:
- session.add_all(
- [
- User(
- id="owner",
- name="Owner",
- role="user",
- password_hash="unused",
- ),
- Tenant(id="tenant-a", slug="tenant-a", name="A"),
- Tenant(id="tenant-b", slug="tenant-b", name="B"),
- ]
- )
- await session.commit()
- with tenant_scope("tenant-a"):
- async with rls_harness.restricted() as session:
- await ProviderConnectionStore().save(
- session, "owner", "api-key", "synthetic-ciphertext", datetime.now(UTC)
- )
- await session.commit()
- with tenant_scope("tenant-b"):
- async with rls_harness.restricted() as session:
- assert (
- await session.execute(select(ProviderConnection))
- ).scalars().all() == []
-
-
-async def test_simultaneous_connection_changes_fail_without_waiting(connection_app):
- client, _, verifier, factory, _ = connection_app
- async with factory() as session:
- await ProviderConnectionStore().lock_user(session, "first")
- result = await client.put(
- "/provider-connections/claude",
- json={"kind": "api-key", "credential": "sk-ant-api-SYNTHETIC-PLACEHOLDER"},
- )
- assert result.status_code == 409
- assert (await client.delete("/provider-connections/claude")).status_code == 409
- verifier.verify.assert_not_awaited()
-
-
- "provider,kind,credential",
- [
- ("codex", "api-key", "SYNTHETIC-PLACEHOLDER"),
- ("codex", "auth-json", '{"tokens":{"access_token":"SYNTHETIC-PLACEHOLDER"}}'),
- ("cursor", "api-key", "SYNTHETIC-PLACEHOLDER"),
- (
- "opencode",
- "auth-json",
- '{"opencode":{"type":"api","key":"SYNTHETIC-PLACEHOLDER"}}',
- ),
- ("antigravity", "auth-json", '{"access_token":"SYNTHETIC-PLACEHOLDER"}'),
- ],
-)
-async def test_other_provider_credentials_remain_unverified_until_worker_checks_them(
- connection_app, provider, kind, credential
-):
- client, identity, _, factory, _ = connection_app
- url = f"/provider-connections/{provider}"
- response = await client.put(url, json={"kind": kind, "credential": credential})
- assert response.status_code == 200
- assert response.json()["status"] == "configured"
- assert "SYNTHETIC" not in response.text
- async with factory() as session:
- row = await session.get(
- ProviderConnection, (require_tenant_id(), "first", provider)
- )
- assert row is not None
- assert row.encrypted_credential != credential
- assert TEST_KEYRING.decrypt(row.encrypted_credential) == credential
- identity["user"] = SimpleNamespace(id="second")
- assert (await client.get(url)).json() == {"status": "not_connected"}
- await client.delete(url)
- identity["user"] = SimpleNamespace(id="first")
- with tenant_scope("other"):
- assert (await client.get(url)).json() == {"status": "not_connected"}
- await client.delete(url)
- assert (await client.get(url)).json()["status"] == "configured"
- assert (await client.delete(url)).status_code == 204
- assert (await client.get(url)).json() == {"status": "not_connected"}
-
-
- "provider,kind,credential",
- [
- ("cursor", "auth-json", '{"PRIVATE":"PLACEHOLDER"}'),
- ("opencode", "api-key", "PRIVATE-PLACEHOLDER"),
- ("codex", "auth-json", "PRIVATE-INVALID-JSON"),
- ("codex", "api-key", "PRIVATE PLACEHOLDER"),
- ("antigravity", "auth-json", "[]"),
- ],
-)
-async def test_other_provider_invalid_credentials_are_not_echoed(
- connection_app, provider, kind, credential
-):
- client, _, _, _, _ = connection_app
- response = await client.put(
- f"/provider-connections/{provider}",
- json={"kind": kind, "credential": credential},
- )
- assert response.status_code == 400
- assert "PRIVATE" not in response.text
-
-
[email protected]("provider", ["claude", "codex"])
-async def test_disconnect_rings_owners_workers_to_revoke_without_superseding(
- connection_app, provider
-):
- client, _, _, factory, app = connection_app
- registry = Mock(spec=AgentConnectionRegistry)
- app.dependency_overrides[get_protocol] = lambda: SimpleNamespace(
- connections=registry
- )
- async with factory() as session:
- session.add_all(
- [
- HostedLaunch(
- id="matching",
- name="matching",
- owner_id="first",
- agent_id="matching-agent",
- spec={"provider": provider},
- state="ready",
- ),
- HostedLaunch(
- id="other-owner",
- name="other-owner",
- owner_id="second",
- agent_id="other-owner-agent",
- spec={"provider": provider},
- state="ready",
- ),
- HostedLaunch(
- id="other-provider",
- name="other-provider",
- owner_id="first",
- agent_id="other-provider-agent",
- spec={"provider": "cursor"},
- state="ready",
- ),
- ]
- )
- await session.commit()
- assert (await client.delete(f"/provider-connections/{provider}")).status_code == 204
- registry.ring_worker.assert_called_once_with(
- "matching-agent", "credential", {"revision": None}
- )
- registry.supersede.assert_not_called()
- async with factory() as session:
- for key in ["matching", "other-owner", "other-provider"]:
- launch = await session.get(HostedLaunch, (require_tenant_id(), key))
- assert launch.state == "ready"
- assert launch.revision == 1
- assert launch.error is None
-
-
-async def test_opencode_stores_only_its_own_login(connection_app):
- client, _, _, factory, _ = connection_app
- response = await client.put(
- "/provider-connections/opencode",
- json={
- "kind": "auth-json",
- "credential": '{"opencode":{"type":"api","key":"SYNTHETIC-PLACEHOLDER"},"other":{"key":"MUST-STAY-LOCAL"}}',
- },
- )
- assert response.status_code == 200
- async with factory() as session:
- row = await session.get(
- ProviderConnection, (require_tenant_id(), "first", "opencode")
- )
- assert (
- TEST_KEYRING.decrypt(row.encrypted_credential)
- == '{"opencode":{"type":"api","key":"SYNTHETIC-PLACEHOLDER"}}'
- )
-
-
-async def test_deeply_nested_auth_is_a_client_error(connection_app):
- client, _, _, _, _ = connection_app
- response = await client.put(
- "/provider-connections/opencode",
- json={
- "kind": "auth-json",
- "credential": "[" * 2000 + "0" + "]" * 2000,
- },
- )
- assert response.status_code == 400
-
-
-async def agents_version(factory, machine_id: str) -> int:
- async with factory() as session:
- machine = await session.get(HostedMachine, (require_tenant_id(), machine_id))
- assert machine is not None
- return machine.agents_version
-
-
- "provider,body",
- [
- ("claude", {"kind": "setup-token", "credential": "sk-ant-oat-SYNTHETIC"}),
- ("codex", {"kind": "api-key", "credential": "SYNTHETIC-PLACEHOLDER"}),
- ],
-)
-async def test_connection_changes_bump_the_owners_machine_agents(
- connection_app, provider, body
-):
- client, _, _, factory, _ = connection_app
- async with factory() as session:
- machine = await seed_machine(
- session,
- owner_id="first",
- slot_id="slot-a",
- state="ready",
- desired_state="running",
- stop_reason=None,
- revision=1,
- generation=1,
- )
- other = await seed_machine(
- session,
- owner_id="second",
- slot_id="slot-b",
- state="ready",
- desired_state="running",
- stop_reason=None,
- revision=1,
- generation=1,
- )
- await session.commit()
- machine_id, other_id = machine.id, other.id
- before = await agents_version(factory, machine_id)
- url = f"/provider-connections/{provider}"
- assert (await client.put(url, json=body)).status_code == 200
- assert await agents_version(factory, machine_id) == before + 1
- assert (await client.delete(url)).status_code == 204
- assert await agents_version(factory, machine_id) == before + 2
- assert await agents_version(factory, other_id) == before
-
-
- "provider,body",
- [
- ("claude", {"kind": "setup-token", "credential": "sk-ant-oat-SYNTHETIC"}),
- ("codex", {"kind": "api-key", "credential": "SYNTHETIC-PLACEHOLDER"}),
- ],
-)
-async def test_connection_changes_without_a_machine_succeed(
- connection_app, provider, body
-):
- client, _, _, factory, _ = connection_app
- url = f"/provider-connections/{provider}"
- assert (await client.put(url, json=body)).status_code == 200
- assert (await client.delete(url)).status_code == 204
- async with factory() as session:
- assert (
- await session.get(
- ProviderConnection, (require_tenant_id(), "first", provider)
- )
- ) is None
-
-
- "url", ["https://console.example.test", "https://console.example.test/"]
-)
-def test_opencode_account_url_is_preserved(url):
- credential = {
- "format": "switch-opencode-console-v1",
- "organization": "example",
- "account": {
- "id": "example",
- "email": "[email protected]",
- "url": url,
- "access_token": "SYNTHETIC",
- "refresh_token": "SYNTHETIC",
- "token_expiry": None,
- "time_created": 0,
- "time_updated": 0,
- },
- }
- saved = validate_provider_credential(
- "opencode", "auth-json", json.dumps(credential)
- )
- assert json.loads(saved)["account"]["url"] == url
- credential["account"]["url"] = "http://console.example.test"
- with pytest.raises(ValueError, match="Sign in to OpenCode"):
- validate_provider_credential("opencode", "auth-json", json.dumps(credential))
diff --git a/core/tests/switch_core/gateway/test_provider_verifications.py b/core/tests/switch_core/gateway/test_provider_verifications.py
deleted file mode 100644
index 9b39d2a91..000000000
--- a/core/tests/switch_core/gateway/test_provider_verifications.py
+++ /dev/null
@@ -1,412 +0,0 @@
-import asyncio
-from datetime import UTC, datetime, timedelta
-from types import SimpleNamespace
-
-import httpx
-import pytest
-from fastapi import FastAPI
-
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.db.models import (
- HostedMachine,
- ProviderConnection,
- ProviderVerification,
- TenantMember,
- User,
- require_tenant_id,
-)
-from switch_core.db.stores.provider_connection_store import ProviderConnectionStore
-from switch_core.gateway.auth import get_current_user
-from switch_core.gateway.dependencies import (
- get_config,
- get_protocol,
- get_session,
- get_session_factory,
-)
-from switch_core.gateway.provider_connections import router as connections
-from switch_core.gateway.provider_verifications import router as verifications
-from switch_core.keys import Keyring
-from switch_core.providers.hosted import HostedControllerSettings
-from tests.switch_core.bridges.agent.protocol.registration_harness import make_owner
-from tests.switch_core.hosted_machine_helpers import seed_machine
-
-TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
-
-CONTROLLER = "SYNTHETIC-CONTROLLER-VERIFICATION-TEST"
-HEADERS = {"Authorization": "Bearer " + CONTROLLER}
-KEY = "placeholder-encryption-key"
-
-
-async def verification_app(session_factory, tmp_path):
- owner = await make_owner(session_factory)
- async with session_factory() as session:
- user = await session.get(User, owner)
- session.add(
- TenantMember(tenant_id=require_tenant_id(), user_id=owner, role="member")
- )
- await session.commit()
- app = FastAPI()
- app.include_router(connections, prefix="/provider-connections")
- app.include_router(verifications)
- app.state.hosted_controller_settings = HostedControllerSettings(
- tenant_id=require_tenant_id(),
- token=CONTROLLER,
- machine_slots=["slot-a", "slot-b"],
- github_private_key_path=tmp_path / "unused.pem",
- agent_api_endpoint="https://switch.example.com",
- )
-
- async def sessions():
- async with session_factory() as session:
- yield session
-
- app.dependency_overrides[get_session] = sessions
- app.dependency_overrides[get_protocol] = lambda: SimpleNamespace(
- connections=AgentConnectionRegistry()
- )
- app.dependency_overrides[get_session_factory] = lambda: session_factory
- app.dependency_overrides[get_current_user] = lambda: user
- app.dependency_overrides[get_config] = lambda: SimpleNamespace(
- keyring=TEST_KEYRING, hosted_provider_verification_enabled=True
- )
- async with httpx.AsyncClient(
- transport=httpx.ASGITransport(app=app), base_url="https://switch.example.com"
- ) as client:
- yield client, session_factory, owner
-
-
-async def start(client, provider="codex"):
- response = await client.put(
- "/provider-connections/" + provider,
- json={"kind": "api-key", "credential": "placeholder-credential"},
- )
- assert response.status_code == 200, response.text
- assert response.json()["status"] == "verifying"
- job_id = response.json()["verification_id"]
- prepared = await client.post(
- "/provider-verifications/" + job_id + "/prepare", headers=HEADERS
- )
- assert prepared.status_code == 200
- return job_id, {"Authorization": "Bearer " + prepared.json()["token"]}
-
-
-async def test_verification_succeeds_before_cleanup_and_remains_queued_for_cleanup(
- verification_app,
-):
- client, factory, owner = verification_app
- job_id, worker = await start(client)
- base = "/provider-verifications/" + job_id
- assert (await client.get(base + "/credential")).status_code == 403
- credential = await client.get(base + "/credential", headers=worker)
- assert credential.json()["credential"] == "placeholder-credential"
- assert credential.headers["cache-control"] == "no-store"
- async with factory() as session:
- assert (
- await session.get(ProviderConnection, (require_tenant_id(), owner, "codex"))
- is None
- )
- assert (
- await client.post(
- base + "/observe",
- headers=HEADERS,
- json={"instance_id": "i-0123456789abcdef0", "terminated": False},
- )
- ).status_code == 200
- assert (
- await client.post(
- base + "/result",
- headers=worker,
- json={"succeeded": True, "credential": "placeholder-refreshed"},
- )
- ).status_code == 200
- assert (await client.get("/provider-connections/codex")).json()[
- "status"
- ] == "connected"
- async with factory() as session:
- job = await session.get(ProviderVerification, (require_tenant_id(), job_id))
- assert job.state == "finishing"
- assert job.encrypted_credential is None and job.encrypted_token is None
- connection = await session.get(
- ProviderConnection, (require_tenant_id(), owner, "codex")
- )
- assert (
- TEST_KEYRING.decrypt(connection.encrypted_credential)
- == "placeholder-refreshed"
- )
- verified_at = connection.verified_at
- job.deadline = datetime.now(UTC) - timedelta(seconds=1)
- await session.commit()
- pending = (await client.get("/provider-verifications", headers=HEADERS)).json()
- assert pending[0]["id"] == job_id and pending[0]["result"] is True
- assert (await client.get("/provider-connections/codex")).json()[
- "status"
- ] == "connected"
- receipt = {"instance_id": "i-0123456789abcdef0", "terminated": True}
- assert (await client.post(base + "/observe", headers=HEADERS, json=receipt)).json()[
- "state"
- ] == "succeeded"
- assert (await client.post(base + "/observe", headers=HEADERS, json=receipt)).json()[
- "state"
- ] == "succeeded"
- assert (await client.get("/provider-connections/codex")).json()[
- "status"
- ] == "connected"
- async with factory() as session:
- connection = await session.get(
- ProviderConnection, (require_tenant_id(), owner, "codex")
- )
- assert connection.verified_at == verified_at
- assert (
- TEST_KEYRING.decrypt(connection.encrypted_credential)
- == "placeholder-refreshed"
- )
- job = await session.get(ProviderVerification, (require_tenant_id(), job_id))
- assert job.encrypted_credential is None and job.encrypted_token is None
- assert (await client.get(base + "/credential", headers=worker)).status_code == 403
- assert (await client.delete("/provider-connections/codex")).status_code == 204
- assert (await client.get("/provider-verifications", headers=HEADERS)).json() == []
-
-
-async def test_timeout_denies_worker_and_never_verifies(verification_app):
- client, factory, _ = verification_app
- job_id, worker = await start(client)
- async with factory() as session:
- job = await session.get(ProviderVerification, (require_tenant_id(), job_id))
- job.deadline = datetime.now(UTC) - timedelta(seconds=1)
- await session.commit()
- assert (await client.get("/provider-connections/codex")).json()[
- "status"
- ] == "failed"
- assert (
- await client.post(
- "/provider-verifications/" + job_id + "/result",
- headers=worker,
- json={"succeeded": True},
- )
- ).status_code == 403
-
-
-async def test_deleted_connection_cannot_be_restored_by_late_result(verification_app):
- client, _, _ = verification_app
- job_id, worker = await start(client)
- base = "/provider-verifications/" + job_id
- await client.post(
- base + "/observe",
- headers=HEADERS,
- json={"instance_id": "i-0123456789abcdef0", "terminated": False},
- )
- assert (await client.delete("/provider-connections/codex")).status_code == 204
- assert (
- await client.post(base + "/result", headers=worker, json={"succeeded": True})
- ).status_code == 403
- assert (
- await client.post(
- base + "/observe",
- headers=HEADERS,
- json={"instance_id": "i-0123456789abcdef0", "terminated": True},
- )
- ).json()["state"] == "cancelled"
- assert (await client.get("/provider-connections/codex")).json()[
- "status"
- ] == "not_connected"
- assert (await client.get("/provider-verifications", headers=HEADERS)).json() == []
-
-
-async def test_checks_are_bounded_and_job_tokens_are_scoped(verification_app):
- client, _, _ = verification_app
- first, worker = await start(client)
- second, _ = await start(client, "cursor")
- duplicate, _ = await start(client)
- assert duplicate == first
- assert (
- await client.get(
- "/provider-verifications/" + second + "/credential", headers=worker
- )
- ).status_code == 403
- response = await client.put(
- "/provider-connections/opencode",
- json={
- "kind": "auth-json",
- "credential": '{"opencode":{"type":"api","key":"placeholder"}}',
- },
- )
- assert response.status_code == 429
- assert (await client.get("/provider-verifications")).status_code == 401
-
-
-async def test_failed_replacement_preserves_verified_connection(verification_app):
- client, factory, owner = verification_app
- job_id, worker = await start(client)
- base = "/provider-verifications/" + job_id
- await client.post(base + "/result", headers=worker, json={"succeeded": True})
- await client.post(
- base + "/observe",
- headers=HEADERS,
- json={"instance_id": "i-0123456789abcdef0", "terminated": True},
- )
- second, worker = await start(client)
- base = "/provider-verifications/" + second
- await client.post(base + "/result", headers=worker, json={"succeeded": False})
- await client.post(
- base + "/observe",
- headers=HEADERS,
- json={"instance_id": "i-0123456789abcdef1", "terminated": True},
- )
- assert (await client.get("/provider-connections/codex")).json()[
- "status"
- ] == "failed"
- async with factory() as session:
- saved = await session.get(
- ProviderConnection, (require_tenant_id(), owner, "codex")
- )
- assert saved.verification_status == "verified"
- assert (
- TEST_KEYRING.decrypt(saved.encrypted_credential) == "placeholder-credential"
- )
-
-
-async def agents_version(factory, machine_id: str) -> int:
- async with factory() as session:
- machine = await session.get(HostedMachine, (require_tenant_id(), machine_id))
- assert machine is not None
- return machine.agents_version
-
-
-async def test_only_a_successful_result_bumps_the_owners_machine_agents(
- verification_app,
-):
- client, factory, owner = verification_app
- async with factory() as session:
- machine = await seed_machine(
- session,
- owner_id=owner,
- slot_id="slot-a",
- state="ready",
- desired_state="running",
- stop_reason=None,
- revision=1,
- generation=1,
- )
- await session.commit()
- machine_id = machine.id
- job_id, worker = await start(client)
- before = await agents_version(factory, machine_id)
- succeeded = await client.post(
- "/provider-verifications/" + job_id + "/result",
- headers=worker,
- json={"succeeded": True},
- )
- assert succeeded.status_code == 200, succeeded.text
- assert await agents_version(factory, machine_id) == before + 1
- await client.post(
- "/provider-verifications/" + job_id + "/observe",
- headers=HEADERS,
- json={"instance_id": "i-0123456789abcdef0", "terminated": True},
- )
- second, worker = await start(client)
- before = await agents_version(factory, machine_id)
- failed = await client.post(
- "/provider-verifications/" + second + "/result",
- headers=worker,
- json={"succeeded": False},
- )
- assert failed.status_code == 200, failed.text
- assert await agents_version(factory, machine_id) == before
-
-
-async def test_changed_credential_cannot_silently_reuse_pending_check(verification_app):
- client, _, _ = verification_app
- await start(client)
- response = await client.put(
- "/provider-connections/codex",
- json={"kind": "api-key", "credential": "different-placeholder"},
- )
- assert response.status_code == 409
-
-
-async def test_removed_member_cannot_use_verification_token(verification_app):
- client, factory, owner = verification_app
- job_id, worker = await start(client)
- async with factory() as session:
- member = await session.get(TenantMember, (require_tenant_id(), owner))
- await session.delete(member)
- await session.commit()
- base = "/provider-verifications/" + job_id
- assert (await client.get(base + "/credential", headers=worker)).status_code == 403
- assert (
- await client.post(base + "/result", headers=worker, json={"succeeded": True})
- ).status_code == 403
-
-
-async def test_disconnect_after_success_keeps_cleanup_without_restoring_connection(
- verification_app,
-):
- client, _, _ = verification_app
- job_id, worker = await start(client)
- base = "/provider-verifications/" + job_id
- receipt = {"instance_id": "i-0123456789abcdef0", "terminated": False}
- await client.post(base + "/observe", headers=HEADERS, json=receipt)
- await client.post(base + "/result", headers=worker, json={"succeeded": True})
- assert (await client.get("/provider-connections/codex")).json()[
- "status"
- ] == "connected"
- assert (await client.delete("/provider-connections/codex")).status_code == 204
- pending = (await client.get("/provider-verifications", headers=HEADERS)).json()
- assert pending[0]["state"] == "cancelled"
- assert pending[0]["instance_id"] == receipt["instance_id"]
- receipt["terminated"] = True
- await client.post(base + "/observe", headers=HEADERS, json=receipt)
- assert (await client.get("/provider-connections/codex")).json()[
- "status"
- ] == "not_connected"
- assert (await client.get("/provider-verifications", headers=HEADERS)).json() == []
-
-
-async def test_repeated_result_cannot_replace_verified_credential(verification_app):
- client, factory, owner = verification_app
- job_id, worker = await start(client)
- base = "/provider-verifications/" + job_id
- await client.post(base + "/result", headers=worker, json={"succeeded": True})
- verified = (await client.get("/provider-connections/codex")).json()
- response = await client.post(
- base + "/result",
- headers=worker,
- json={"succeeded": True, "credential": "different-placeholder"},
- )
- assert response.status_code == 200
- assert (await client.get("/provider-connections/codex")).json() == verified
- assert (await client.get(base + "/credential", headers=worker)).status_code == 409
- async with factory() as session:
- saved = await session.get(
- ProviderConnection, (require_tenant_id(), owner, "codex")
- )
- assert (
- TEST_KEYRING.decrypt(saved.encrypted_credential) == "placeholder-credential"
- )
-
-
[email protected]("action", ["observe", "result"])
-async def test_worker_callback_waits_for_credential_write_lock(
- verification_app, action
-):
- client, factory, owner = verification_app
- job_id, worker = await start(client)
- async with factory() as writer:
- await ProviderConnectionStore().lock_user(writer, owner)
- request = asyncio.create_task(
- client.post(
- f"/provider-verifications/{job_id}/{action}",
- headers=HEADERS if action == "observe" else worker,
- json={"instance_id": "i-0123456789abcdef0", "terminated": False}
- if action == "observe"
- else {"succeeded": True},
- )
- )
- try:
- await asyncio.sleep(0.05)
- assert not request.done()
- finally:
- await writer.commit()
- response = await asyncio.wait_for(request, timeout=5)
- assert response.status_code == 200
diff --git a/core/tests/switch_core/gateway/test_session_requires_authentication.py b/core/tests/switch_core/gateway/test_session_requires_authentication.py
index 8babb988b..0bef2df02 100644
--- a/core/tests/switch_core/gateway/test_session_requires_authentication.py
+++ b/core/tests/switch_core/gateway/test_session_requires_authentication.py
@@ -39,7 +39,6 @@ from switch_core.gateway.auth import (
)
from switch_core.gateway.dependencies import get_session, get_system_session
from switch_core.gateway.hosted_controller import controller_session
-from switch_core.gateway.provider_verifications import worker_session
# The only routes that may open a session with no tenant bound. `get_session`
# is unavailable to them because there is no `get_current_user` to bind one:
@@ -215,30 +214,18 @@ def test_the_routes_that_never_bind_a_tenant_are_exactly_these() -> None:
assert {
route.key
for route in ROUTES
- if not route.binds_a_tenant
- and controller_session not in route.calls
- and worker_session not in route.calls
+ if not route.binds_a_tenant and controller_session not in route.calls
} == _ROUTES_THAT_NEVER_BIND_A_TENANT
def test_controller_credential_is_confined_to_the_controller_routes() -> None:
assert {route.key for route in ROUTES if controller_session in route.calls} == {
- ("GET", "/provider-verifications"),
- ("POST", "/provider-verifications/{job_id}/prepare"),
- ("POST", "/provider-verifications/{job_id}/observe"),
("GET", "/hosted-controller/machines"),
("POST", "/hosted-controller/machines/{machine_id}/prepare"),
("POST", "/hosted-controller/machines/{machine_id}/observation"),
}
-def test_verification_job_token_is_confined_to_its_worker_routes() -> None:
- assert {route.key for route in ROUTES if worker_session in route.calls} == {
- ("GET", "/provider-verifications/{job_id}/credential"),
- ("POST", "/provider-verifications/{job_id}/result"),
- }
-
-
def test_the_routes_with_no_tenant_bound_do_not_bind_a_tenant() -> None:
"""The two exemptions agree: anything allowed to open an unscoped session
is, necessarily, a route that resolves no tenant."""
diff --git a/core/tests/switch_core/gateway/test_signup.py b/core/tests/switch_core/gateway/test_signup.py
index 7e9489ff3..e0a5c35ad 100644
--- a/core/tests/switch_core/gateway/test_signup.py
+++ b/core/tests/switch_core/gateway/test_signup.py
@@ -17,6 +17,7 @@ from switch_core.db.stores.hosted_machine_store import (
MACHINE_BEING_REMOVED,
MACHINE_NEEDS_ADMIN,
MACHINE_NEEDS_ATTENTION,
+ MACHINES_FULL,
)
from switch_core.db.stores.user_store import UserStore
from switch_core.gateway.auth import decode_jwt, get_current_user, verify_password
@@ -29,7 +30,7 @@ from switch_core.gateway.dependencies import (
get_system_session,
get_user_store,
)
-from switch_core.gateway.hosted_launches import LAUNCH_DISABLED
+from switch_core.gateway.hosted_machines import MACHINES_DISABLED
from switch_core.gateway.hosted_machines import router as machine_router
from switch_core.keys import Keyring
from switch_core.tenant_context import tenant_scope
@@ -46,9 +47,7 @@ async def signup_app(session_factory):
app = FastAPI()
app.include_router(auth_router)
app.include_router(machine_router)
- app.state.hosted_controller_settings = SimpleNamespace(
- tenant_id=TENANT_ZERO_ID, machine_slots=["slot-a", "slot-b"]
- )
+ app.state.hosted_controller_settings = SimpleNamespace(tenant_id=TENANT_ZERO_ID)
config = SimpleNamespace(
gateway_signup_open=True,
gateway_signup_mode="default_tenant",
@@ -254,14 +253,14 @@ async def test_signup_succeeds_without_cloud_capacity(signup_app, caplog):
assert response.status_code == 201, response.text
assert response.json()["machine"] == {
"status": "unavailable",
- "reason": LAUNCH_DISABLED,
+ "reason": MACHINES_DISABLED,
}
assert await _machines(app) == []
assert "has no cloud machine warming" in caplog.text
await _user(app, "[email protected]")
-async def test_signup_succeeds_when_every_slot_is_taken(signup_app):
+async def test_signup_succeeds_when_every_machine_is_in_use(signup_app):
app = signup_app
app.config.hosted_launch_capacity = 1
assert (await _signup(app, email="[email protected]")).status_code == 201
@@ -269,7 +268,7 @@ async def test_signup_succeeds_when_every_slot_is_taken(signup_app):
assert response.status_code == 201, response.text
assert response.json()["machine"] == {
"status": "unavailable",
- "reason": "no machine slot available",
+ "reason": MACHINES_FULL,
}
assert len(await _machines(app)) == 1
@@ -431,7 +430,7 @@ async def test_ensure_refuses_when_cloud_machines_are_off(signup_app):
app.config.hosted_launch_capacity = 0
response = await _ensure(app, "[email protected]")
assert response.status_code == 503
- assert response.json()["detail"] == LAUNCH_DISABLED
+ assert response.json()["detail"] == MACHINES_DISABLED
async def _user_count(app) -> int:
diff --git a/core/tests/switch_core/gateway/test_tenant_api_routes.py b/core/tests/switch_core/gateway/test_tenant_api_routes.py
index 1b219096d..20094751a 100644
--- a/core/tests/switch_core/gateway/test_tenant_api_routes.py
+++ b/core/tests/switch_core/gateway/test_tenant_api_routes.py
@@ -35,8 +35,6 @@ from switch_core.db.models import (
Agent,
ApiKey,
Client,
- GitHubIssuedToken,
- HostedLaunch,
Invitation,
ProviderConnection,
Tenant,
@@ -62,7 +60,6 @@ from switch_core.gateway.invite_mail import (
)
from switch_core.gateway.tenants import router as tenants_router
from switch_core.keys import Keyring
-from switch_core.providers.github_installation import GitHubInstallationCredentials
_SECRET = "unit-test-jwt-key-unit-test-jwt-key-unit-test" # gitleaks:allow
_KEYRING = Keyring.parse("test:" + _SECRET, legacy_secret=None)
@@ -1453,7 +1450,7 @@ class TestMemberRoutes:
assert await session.get(TenantMember, (TENANT_A, owner_id)) is not None
async def test_removing_a_member_deletes_their_membership(
- self, session_factory: async_sessionmaker[AsyncSession], monkeypatch
+ self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
await _make_tenant(session_factory, TENANT_A)
owner_id = await _make_member(
@@ -1464,12 +1461,6 @@ class TestMemberRoutes:
)
token = _token(owner_id, "[email protected]", TENANT_A)
async with tenant_session(session_factory, TENANT_A) as session:
- launch = HostedLaunch(
- id=str(uuid.uuid4()),
- owner_id=target_id,
- name="pending-member-worker",
- spec={},
- )
session.add(
ProviderConnection(
user_id=target_id,
@@ -1481,30 +1472,8 @@ class TestMemberRoutes:
verified_at=datetime.now(UTC),
)
)
- session.add(launch)
- await session.flush()
- session.add(
- GitHubIssuedToken(
- id=str(uuid.uuid4()),
- owner_id=target_id,
- launch_id=launch.id,
- launch_revision=1,
- encrypted_token=_KEYRING.encrypt("SYNTHETIC-REPOSITORY"),
- expires_at=datetime.now(UTC) + timedelta(hours=1),
- revoke_requested=False,
- attempts=0,
- )
- )
await session.commit()
- async def revoke_after_removal(value):
- assert value == "SYNTHETIC-REPOSITORY"
- async with tenant_session(session_factory, TENANT_A) as session:
- assert await session.get(TenantMember, (TENANT_A, target_id)) is None
-
- revoke = AsyncMock(side_effect=revoke_after_removal)
- monkeypatch.setattr(GitHubInstallationCredentials, "revoke", revoke)
-
app = _app(session_factory)
github = SimpleNamespace(
flows={
@@ -1520,13 +1489,12 @@ class TestMemberRoutes:
assert response.status_code == 200, response.text
github.revoke.assert_awaited_once_with("SYNTHETIC-USER-TOKEN")
assert set(github.flows) == {"other"}
- revoke.assert_awaited_once()
async with session_factory() as session:
assert await session.get(TenantMember, (TENANT_A, target_id)) is None
assert (
await session.scalar(
- select(GitHubIssuedToken).where(
- GitHubIssuedToken.owner_id == target_id
+ select(ProviderConnection).where(
+ ProviderConnection.user_id == target_id
)
)
is None
diff --git a/core/tests/switch_core/hosted_machine_helpers.py b/core/tests/switch_core/hosted_machine_helpers.py
index fa13b3406..042dc8d12 100644
--- a/core/tests/switch_core/hosted_machine_helpers.py
+++ b/core/tests/switch_core/hosted_machine_helpers.py
@@ -7,30 +7,33 @@ from uuid import uuid4
from sqlalchemy.ext.asyncio import AsyncSession
-from switch_core.db.models import HostedLaunch, HostedMachine
+from switch_core.db.models import (
+ Agent,
+ AgentController,
+ AgentDefinition,
+ ApiKey,
+ Client,
+ HostedMachine,
+)
async def seed_machine(
session: AsyncSession,
*,
owner_id: str,
- slot_id: str,
state: str,
desired_state: str,
stop_reason: str | None,
revision: int,
- generation: int,
) -> HostedMachine:
now = datetime.now(UTC)
machine = HostedMachine(
id=str(uuid4()),
owner_id=owner_id,
- slot_id=slot_id,
state=state,
desired_state=desired_state,
stop_reason=stop_reason,
revision=revision,
- generation=generation,
active_at=now,
created_at=now,
updated_at=now,
@@ -40,33 +43,56 @@ async def seed_machine(
return machine
-async def seed_launch(
- session: AsyncSession,
- *,
- machine: HostedMachine,
- request_id: str,
- name: str,
- state: str,
- desired_state: str,
- revision: int,
- agent_id: str | None,
- spec: dict,
-) -> HostedLaunch:
- now = datetime.now(UTC)
- launch = HostedLaunch(
- id=request_id,
- owner_id=machine.owner_id,
- machine_id=machine.id,
+async def link_controller(
+ session: AsyncSession, machine: HostedMachine
+) -> AgentController:
+ """The cloud controller the machine enrolled as, linked to it."""
+ controller = AgentController(
+ owner_id=machine.owner_id, name="Switch cloud", kind="ec2"
+ )
+ session.add(controller)
+ await session.flush()
+ machine.controller_id = controller.id
+ await session.flush()
+ return controller
+
+
+async def place_managed_agent(
+ session: AsyncSession, *, owner_id: str, controller_id: str, name: str
+) -> AgentDefinition:
+ """A managed agent of `owner_id`, placed on `controller_id`."""
+ client = Client(type="agent", transport_user_id=f"@{name}:test", display_name=name)
+ session.add(client)
+ await session.flush()
+ api_key = ApiKey(
+ type="agent",
+ key_hash=f"hash-{uuid4()}",
+ encrypted_key=f"enc-{name}",
+ label=name,
+ user_id=owner_id,
+ )
+ session.add(api_key)
+ await session.flush()
+ agent = Agent(
name=name,
- spec=spec,
- state=state,
- desired_state=desired_state,
- revision=revision,
- agent_id=agent_id,
- active_at=now,
- created_at=now,
- updated_at=now,
+ description=f"{name} desc",
+ agent_type="session_passive",
+ connector_type="external",
+ integration_profile={"connection_model": "session_passive"},
+ client_id=client.id,
+ api_key_id=api_key.id,
+ owner_id=owner_id,
+ )
+ session.add(agent)
+ await session.flush()
+ definition = AgentDefinition(
+ agent_id=agent.id,
+ owner_id=owner_id,
+ controller_id=controller_id,
+ revision=1,
+ desired_state="running",
+ definition={"provider": "claude"},
)
- session.add(launch)
+ session.add(definition)
await session.flush()
- return launch
+ return definition
diff --git a/core/tests/switch_core/hosted_wire_fixtures.py b/core/tests/switch_core/hosted_wire_fixtures.py
deleted file mode 100644
index 3baf498bb..000000000
--- a/core/tests/switch_core/hosted_wire_fixtures.py
+++ /dev/null
@@ -1,105 +0,0 @@
-"""The hosted-machine wire fixtures, shared with the controller, the supervisor and Switch Console.
-
-Each response fixture is real Core output with its volatile values (ids Core
-mints, capabilities, API tokens, timestamps) swapped for fixed placeholders,
-and each request fixture is a body Core's route accepts. The Core tests fail
-when a route's output drifts from its fixture; run them with
-`SWITCH_UPDATE_FIXTURES=1` to rewrite the response fixtures from real output.
-"""
-
-from __future__ import annotations
-
-import json
-import os
-from pathlib import Path
-from typing import Any
-
-FIXTURES = Path(__file__).parent / "fixtures" / "hosted_machines"
-UPDATE_FIXTURES = os.environ.get("SWITCH_UPDATE_FIXTURES") == "1"
-
-MACHINE_ID = "3f1c2b4a-0000-4000-8000-000000000001"
-AGENT_ID = "3f1c2b4a-0000-4000-8000-0000000000a1"
-LAUNCH_ID = "3f1c2b4a-0000-4000-8000-0000000000c1"
-MACHINE_CAPABILITY = "mcap-test-00000000000000000000000000000000"
-WORKER_CAPABILITY = "wcap-test-00000000000000000000000000000000"
-API_TOKEN = "test-token-placeholder"
-API_ENDPOINT = "https://switch.example.test/agent-api"
-TIMESTAMP = "2026-01-01T00:00:00+00:00"
-
-JsonPath = tuple[str, ...]
-
-
-def read_fixture(name: str) -> Any:
- return json.loads((FIXTURES / name).read_text())
-
-
-def _substitute(value: Any, replacements: dict[str, str]) -> Any:
- if isinstance(value, dict):
- return {key: _substitute(item, replacements) for key, item in value.items()}
- if isinstance(value, list):
- return [_substitute(item, replacements) for item in value]
- if isinstance(value, str):
- return replacements.get(value, value)
- return value
-
-
-def request_fixture(name: str, placeholders: dict[str, str]) -> Any:
- """The request fixture with each placeholder id replaced by the real one."""
- return _substitute(read_fixture(name), placeholders)
-
-
-def _replace_volatile(value: Any, path: JsonPath, placeholder: Any, where: str) -> Any:
- if not path:
- assert type(value) is type(placeholder), (
- f"{where}: expected a {type(placeholder).__name__}, got {value!r}"
- )
- return placeholder
- head, rest = path[0], path[1:]
- if head == "*":
- assert isinstance(value, list), f"{where}: expected a list, got {value!r}"
- return [
- _replace_volatile(item, rest, placeholder, f"{where}[{index}]")
- for index, item in enumerate(value)
- ]
- assert isinstance(value, dict) and head in value, (
- f"{where}: expected an object with {head!r}, got {value!r}"
- )
- return {
- key: _replace_volatile(item, rest, placeholder, f"{where}.{key}")
- if key == head
- else item
- for key, item in value.items()
- }
-
-
-def assert_wire_fixture(
- name: str,
- actual: Any,
- *,
- placeholders: dict[str, str],
- volatile: dict[JsonPath, Any],
-) -> None:
- """Compare a real response body with its fixture, exactly.
-
- `placeholders` maps each placeholder id to the real value the test saw;
- every string equal to a real value is written as its placeholder. Each
- `volatile` path (`*` matches every list item) must hold a value of the
- placeholder's type and is written as the placeholder.
- """
- real = {value: placeholder for placeholder, value in placeholders.items()}
- assert len(real) == len(placeholders), "two placeholders share one real value"
- normalised = _substitute(actual, real)
- for path, placeholder in volatile.items():
- normalised = _replace_volatile(normalised, path, placeholder, name)
- if UPDATE_FIXTURES:
- (FIXTURES / name).write_text(json.dumps(normalised, indent=2) + "\n")
- return
- expected = read_fixture(name)
- drifted = (
- f"{name} no longer matches Core's output; rerun with SWITCH_UPDATE_FIXTURES=1 "
- "to rewrite it, and update its consumers."
- )
- assert normalised == expected, drifted
- assert json.dumps(normalised, sort_keys=True) == json.dumps(
- expected, sort_keys=True
- ), drifted
diff --git a/core/tests/switch_core/management/conftest.py b/core/tests/switch_core/management/conftest.py
index f15e256e4..c8a9e318e 100644
--- a/core/tests/switch_core/management/conftest.py
+++ b/core/tests/switch_core/management/conftest.py
@@ -7,6 +7,7 @@ import pytest
from switch_core.bridges.agent.operations.agent_management import (
disable_agent_management,
)
+from switch_core.gateway.cloud_controllers import set_cloud_enrollment
@pytest.fixture(autouse=True)
@@ -15,3 +16,4 @@ def _agent_management_operations_off_afterwards() -> Iterator[None]:
does in production; take them away again so no other test sees them."""
yield
disable_agent_management()
+ set_cloud_enrollment(None)
diff --git a/core/tests/switch_core/management/test_gateway_routes.py b/core/tests/switch_core/management/test_gateway_routes.py
index 26472e07c..1b8990715 100644
--- a/core/tests/switch_core/management/test_gateway_routes.py
+++ b/core/tests/switch_core/management/test_gateway_routes.py
@@ -133,7 +133,7 @@ class TestOwnerIsolation:
return found
routes = [r for r in gateway_router.routes if hasattr(r, "dependant")]
- assert len(routes) == 14
+ assert len(routes) == 17
for route in routes:
assert get_current_user in calls(route.dependant), route.path # type: ignore[attr-defined]
diff --git a/core/tests/switch_core/management/test_sealed_logins.py b/core/tests/switch_core/management/test_sealed_logins.py
new file mode 100644
index 000000000..04b5fd0f5
--- /dev/null
+++ b/core/tests/switch_core/management/test_sealed_logins.py
@@ -0,0 +1,303 @@
+"""Provider logins sealed to a controller's own key, end to end against Postgres.
+
+A controller registers its public key; its owner gives it a login sealed to
+that key, which the controller is told to take up through a `provider.login`
+operation and fetches as ciphertext; the owner can take it back. Switch
+checks the envelope and the key it names, and never sees inside.
+"""
+
+from __future__ import annotations
+
+import base64
+import os
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.management.sealed_logins import key_id
+from tests.switch_core.management.harness import (
+ EnrolledController,
+ Harness,
+ add_member,
+ build_harness,
+ cookies_for,
+ enroll_console,
+)
+
+
+def harness(session_factory: async_sessionmaker[AsyncSession]) -> Harness:
+ return build_harness(session_factory)
+
+
+def _b64(size: int) -> str:
+ return base64.b64encode(os.urandom(size)).decode()
+
+
+def _sealed(public_key: str, **overrides: str) -> dict[str, str]:
+ return {
+ "alg": "X25519-HKDF-SHA256-A256GCM",
+ "key_id": key_id(public_key),
+ "ephemeral_key": _b64(32),
+ "nonce": _b64(12),
+ "ciphertext": _b64(80),
+ **overrides,
+ }
+
+
+async def _register_key(client, controller: EnrolledController, key: str):
+ return await client.patch(
+ f"/v1/management/controllers/{controller.controller_id}",
+ json={"public_key": {"alg": "X25519", "key": key}},
+ headers=controller.headers,
+ )
+
+
+def _logins_path(controller: EnrolledController, provider: str | None = None) -> str:
+ path = f"/gateway/management/controllers/{controller.controller_id}/provider-logins"
+ return f"{path}/{provider}" if provider else path
+
+
+class TestPublicKey:
+ async def test_a_controller_registers_its_key_once(self, harness: Harness) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ key = _b64(32)
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ first = await _register_key(client, controller, key)
+ same = await _register_key(client, controller, key)
+ other = await _register_key(client, controller, _b64(32))
+ listed = await client.get(
+ "/gateway/management/controllers", cookies=cookies_for(owner)
+ )
+ assert first.status_code == 200, first.text
+ assert first.json()["public_key"] == {
+ "alg": "X25519",
+ "key": key,
+ "key_id": key_id(key),
+ }
+ assert same.status_code == 200, same.text
+ assert other.status_code == 409, other.text
+ assert listed.json()[0]["public_key"]["key"] == key
+
+ @pytest.mark.parametrize(
+ "public_key",
+ [
+ {"alg": "RSA", "key": base64.b64encode(b"k" * 32).decode()},
+ {"alg": "X25519", "key": base64.b64encode(b"short").decode()},
+ {"alg": "X25519", "key": "not base64!"},
+ ],
+ )
+ async def test_anything_but_an_x25519_key_is_refused(
+ self, harness: Harness, public_key: dict[str, str]
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ response = await client.patch(
+ f"/v1/management/controllers/{controller.controller_id}",
+ json={"public_key": public_key},
+ headers=controller.headers,
+ )
+ assert response.status_code == 422, response.text
+
+
+class TestSealedLogins:
+ async def test_the_owner_gives_a_login_the_controller_fetches(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ key = _b64(32)
+ sealed = _sealed(key)
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ await _register_key(client, controller, key)
+ given = await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": sealed},
+ cookies=cookies_for(owner),
+ )
+ fetched = await client.get(
+ f"/v1/management/controllers/{controller.controller_id}/provider-credentials/claude",
+ headers=controller.headers,
+ )
+ pending = await client.get(
+ f"/v1/management/controllers/{controller.controller_id}/operations",
+ params={"state": "pending"},
+ headers=controller.headers,
+ )
+ again = await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": _sealed(key)},
+ cookies=cookies_for(owner),
+ )
+ listed = await client.get(
+ _logins_path(controller), cookies=cookies_for(owner)
+ )
+ assert given.status_code == 200, given.text
+ body = given.json()
+ assert body["login"]["provider"] == "claude"
+ assert body["login"]["revision"] == 1
+ assert "ciphertext" not in body["login"]
+ assert body["operation"]["kind"] == "provider.login"
+ assert body["operation"]["params"] == {
+ "provider": "claude",
+ "method": "sealed",
+ "revision": 1,
+ }
+ assert fetched.status_code == 200, fetched.text
+ assert fetched.headers["cache-control"] == "no-store"
+ assert fetched.json() == {"provider": "claude", "revision": 1, "sealed": sealed}
+ assert [operation["kind"] for operation in pending.json()["operations"]] == [
+ "provider.login"
+ ]
+ assert again.json()["login"]["revision"] == 2
+ assert [(login["provider"], login["revision"]) for login in listed.json()] == [
+ ("claude", 2)
+ ]
+
+ async def test_a_missing_login_is_provider_login_missing(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ response = await client.get(
+ f"/v1/management/controllers/{controller.controller_id}/provider-credentials/codex",
+ headers=controller.headers,
+ )
+ assert response.status_code == 404, response.text
+ assert response.json()["error"]["code"] == "provider_login_missing"
+
+ async def test_a_login_sealed_to_another_key_is_refused(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ keyless = await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": _sealed(_b64(32))},
+ cookies=cookies_for(owner),
+ )
+ await _register_key(client, controller, _b64(32))
+ stale = await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": _sealed(_b64(32))},
+ cookies=cookies_for(owner),
+ )
+ assert keyless.status_code == 409, keyless.text
+ assert stale.status_code == 409, stale.text
+ assert "no longer has" in stale.json()["error"]["message"]
+
+ async def test_a_malformed_envelope_or_provider_is_refused(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ key = _b64(32)
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ await _register_key(client, controller, key)
+ short_nonce = await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": _sealed(key, nonce=_b64(8))},
+ cookies=cookies_for(owner),
+ )
+ plaintext = await client.put(
+ _logins_path(controller, "claude"),
+ json={"kind": "api-key", "credential": "sk-ant-api-placeholder"},
+ cookies=cookies_for(owner),
+ )
+ unknown = await client.put(
+ _logins_path(controller, "notepad"),
+ json={"sealed": _sealed(key)},
+ cookies=cookies_for(owner),
+ )
+ assert short_nonce.status_code == 422, short_nonce.text
+ assert plaintext.status_code == 422, plaintext.text
+ assert unknown.status_code == 422, unknown.text
+
+ async def test_only_the_owner_sees_or_gives_a_machine_logins(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ other = await add_member(harness.session_factory, "bob")
+ key = _b64(32)
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ await _register_key(client, controller, key)
+ given = await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": _sealed(key)},
+ cookies=cookies_for(other),
+ )
+ listed = await client.get(
+ _logins_path(controller), cookies=cookies_for(other)
+ )
+ assert given.status_code == 404, given.text
+ assert listed.status_code == 404, listed.text
+
+ async def test_withdrawing_a_login_has_the_controller_recheck(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ key = _b64(32)
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ await _register_key(client, controller, key)
+ await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": _sealed(key)},
+ cookies=cookies_for(owner),
+ )
+ withdrawn = await client.delete(
+ _logins_path(controller, "claude"), cookies=cookies_for(owner)
+ )
+ twice = await client.delete(
+ _logins_path(controller, "claude"), cookies=cookies_for(owner)
+ )
+ fetched = await client.get(
+ f"/v1/management/controllers/{controller.controller_id}/provider-credentials/claude",
+ headers=controller.headers,
+ )
+ pending = await client.get(
+ f"/v1/management/controllers/{controller.controller_id}/operations",
+ params={"state": "pending"},
+ headers=controller.headers,
+ )
+ assert withdrawn.status_code == 200, withdrawn.text
+ assert twice.status_code == 404, twice.text
+ assert fetched.status_code == 404, fetched.text
+ assert [operation["kind"] for operation in pending.json()["operations"]] == [
+ "provider.login",
+ "provider.recheck",
+ ]
+
+ async def test_revoking_the_machine_drops_its_logins(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ key = _b64(32)
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ await _register_key(client, controller, key)
+ await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": _sealed(key)},
+ cookies=cookies_for(owner),
+ )
+ revoked = await client.delete(
+ f"/gateway/management/controllers/{controller.controller_id}",
+ cookies=cookies_for(owner),
+ )
+ listed = await client.get(
+ _logins_path(controller), cookies=cookies_for(owner)
+ )
+ again = await client.put(
+ _logins_path(controller, "claude"),
+ json={"sealed": _sealed(key)},
+ cookies=cookies_for(owner),
+ )
+ assert revoked.status_code == 200, revoked.text
+ assert listed.json() == []
+ assert again.status_code == 409, again.text
diff --git a/core/tests/switch_core/management/test_stream.py b/core/tests/switch_core/management/test_stream.py
index 2f5adb9cf..8dd54f166 100644
--- a/core/tests/switch_core/management/test_stream.py
+++ b/core/tests/switch_core/management/test_stream.py
@@ -356,9 +356,7 @@ class TestLiveChanges:
surface="slack",
requester_name="Ada",
)
- assert harness.protocol.connections.relay_session_command(
- agent_id, command, worker_only=False
- )
+ assert harness.protocol.connections.relay_session_command(agent_id, command)
(frame,) = await take(stream, 1)
await stream.aclose()
diff --git a/core/tests/switch_core/providers/test_claude_verifier.py b/core/tests/switch_core/providers/test_claude_verifier.py
deleted file mode 100644
index f3ace72be..000000000
--- a/core/tests/switch_core/providers/test_claude_verifier.py
+++ /dev/null
@@ -1,75 +0,0 @@
-import asyncio
-import json
-import os
-import sys
-
-import pytest
-
-from switch_core.providers.claude_verifier import (
- ClaudeVerificationError,
- ClaudeVerifier,
-)
-
-
-def executable(tmp_path, body):
- path = tmp_path / "claude"
- path.write_text(f"#!{sys.executable}\n" + body)
- path.chmod(0o700)
- return str(path)
-
-
- "kind,variable",
- [("api-key", "ANTHROPIC_API_KEY"), ("setup-token", "CLAUDE_CODE_OAUTH_TOKEN")],
-)
-async def test_isolated_environment_and_disabled_tools(
- tmp_path, monkeypatch, kind, variable
-):
- monkeypatch.setenv("ANTHROPIC_API_KEY", "ambient-secret")
- monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", "ambient-secret")
- script = """import os, sys, json
-assert os.environ[VARIABLE] == 'SYNTHETIC'
-assert 'ambient-secret' not in os.environ.values()
-assert sys.argv[sys.argv.index('--tools') + 1] == ''
-assert '--no-session-persistence' in sys.argv
-assert os.path.realpath(os.getcwd()) == os.path.realpath(os.environ['HOME'])
-assert 'SYNTHETIC' not in sys.argv
-print(json.dumps({'type': 'result', 'subtype': 'success', 'is_error': False}))
-""".replace("VARIABLE", repr(variable))
- await ClaudeVerifier(executable(tmp_path, script)).verify(kind, "SYNTHETIC")
-
-
- "result", [{}, {"type": "result", "subtype": "error", "is_error": True}]
-)
-async def test_non_success_is_not_verified(tmp_path, result):
- verifier = ClaudeVerifier(executable(tmp_path, f"print({json.dumps(result)!r})"))
- with pytest.raises(ClaudeVerificationError):
- await verifier.verify("api-key", "SYNTHETIC")
-
-
-async def test_raw_provider_output_is_not_exposed(tmp_path):
- verifier = ClaudeVerifier(executable(tmp_path, "print('PRIVATE-PROVIDER-OUTPUT')"))
- with pytest.raises(ClaudeVerificationError) as error:
- await verifier.verify("api-key", "SYNTHETIC")
- assert "PRIVATE" not in str(error.value)
-
-
-async def test_cancellation_kills_process_and_cleans_home(tmp_path):
- marker = tmp_path / "process.json"
- script = f"import os, json, time\nopen({str(marker)!r}, 'w').write(json.dumps([os.getpid(), os.environ['HOME']]))\ntime.sleep(60)"
- task = asyncio.create_task(
- ClaudeVerifier(executable(tmp_path, script)).verify("api-key", "SYNTHETIC")
- )
- for _ in range(100):
- if marker.exists():
- break
- await asyncio.sleep(0.02)
- assert marker.exists()
- pid, home = json.loads(marker.read_text())
- task.cancel()
- with pytest.raises(asyncio.CancelledError):
- await task
- with pytest.raises(ProcessLookupError):
- os.kill(pid, 0)
- assert not os.path.exists(home)
diff --git a/core/tests/switch_core/providers/test_github_installation.py b/core/tests/switch_core/providers/test_github_installation.py
deleted file mode 100644
index 2ab278db0..000000000
--- a/core/tests/switch_core/providers/test_github_installation.py
+++ /dev/null
@@ -1,226 +0,0 @@
-import json
-import time
-from datetime import UTC, datetime
-from unittest.mock import AsyncMock
-
-import httpx
-import jwt
-import pytest
-from cryptography.hazmat.primitives import serialization
-from cryptography.hazmat.primitives.asymmetric import rsa
-
-from switch_core.providers.github import GitHubError, GitHubUnavailableError
-from switch_core.providers.github_installation import GitHubInstallationCredentials
-
-
-def signing(tmp_path, monkeypatch):
- monkeypatch.setattr(
- httpx.AsyncClient, "delete", AsyncMock(return_value=httpx.Response(204))
- )
- key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
- path = tmp_path / "app.pem"
- path.write_bytes(
- key.private_bytes(
- serialization.Encoding.PEM,
- serialization.PrivateFormat.PKCS8,
- serialization.NoEncryption(),
- )
- )
- path.chmod(0o600)
- return GitHubInstallationCredentials("example-client", str(path)), key
-
-
-def response_body():
- return {
- "token": "SYNTHETIC-INSTALLATION-TOKEN",
- "expires_at": datetime.fromtimestamp(time.time() + 3600, UTC).isoformat(),
- "permissions": {
- "contents": "write",
- "pull_requests": "write",
- "metadata": "read",
- },
- "repositories": [{"id": 789}],
- }
-
-
-def user_access():
- return AsyncMock(
- repositories=AsyncMock(
- return_value=[
- {
- "id": 456,
- "repositories": [
- {
- "id": 789,
- "name": "example/project",
- "permissions": {"push": True},
- }
- ],
- }
- ]
- )
- )
-
-
-async def test_signs_request_and_limits_worker_to_users_selected_repository(
- signing, monkeypatch
-):
- issuer, key = signing
-
- async def post(_client, url, **kwargs):
- assert url == "https://api.github.com/app/installations/456/access_tokens"
- assertion = kwargs["headers"]["Authorization"].removeprefix("Bearer ")
- claims = jwt.decode(
- assertion, key.public_key(), algorithms=["RS256"], issuer="example-client"
- )
- assert claims["exp"] - claims["iat"] == 600
- assert kwargs["json"] == {
- "repository_ids": [789],
- "permissions": {"contents": "write", "pull_requests": "write"},
- }
- return httpx.Response(201, json=response_body())
-
- monkeypatch.setattr(httpx.AsyncClient, "post", post)
- github = user_access()
- credential = await issuer.issue(github, "SYNTHETIC-USER-TOKEN", 456, 789)
- github.repositories.assert_awaited_once_with("SYNTHETIC-USER-TOKEN")
- assert credential.repository_name == "example/project"
- assert credential.token not in repr(credential)
-
-
- "installation,repository", [(456, 999), (999, 789), (0, 789), (True, 789)]
-)
-async def test_refuses_unowned_or_invalid_selection_before_minting(
- signing, monkeypatch, installation, repository
-):
- post = AsyncMock()
- monkeypatch.setattr(httpx.AsyncClient, "post", post)
- with pytest.raises(GitHubError):
- await signing[0].issue(
- user_access(), "SYNTHETIC-USER-TOKEN", installation, repository
- )
- post.assert_not_called()
-
-
- "change",
- [
- {"repositories": [{"id": 789}, {"id": 999}]},
- {
- "permissions": {
- "contents": "write",
- "pull_requests": "write",
- "issues": "write",
- }
- },
- {"expires_at": "2000-01-01T00:00:00Z"},
- {"token": "bad\ntoken"},
- {"expires_at": "not-a-date"},
- ],
-)
-async def test_rejects_overbroad_or_invalid_credential(signing, monkeypatch, change):
- body = response_body() | change
- monkeypatch.setattr(
- httpx.AsyncClient,
- "post",
- AsyncMock(return_value=httpx.Response(201, json=body)),
- )
- with pytest.raises(GitHubError) as raised:
- await signing[0].issue(user_access(), "SYNTHETIC-USER-TOKEN", 456, 789)
- assert "SYNTHETIC" not in str(raised.value)
-
-
-async def test_remote_errors_do_not_expose_tokens_or_response_bodies(
- signing, monkeypatch
-):
- monkeypatch.setattr(
- httpx.AsyncClient,
- "post",
- AsyncMock(
- return_value=httpx.Response(
- 403, text=json.dumps({"secret": "SYNTHETIC-SECRET"})
- )
- ),
- )
- with pytest.raises(GitHubError) as raised:
- await signing[0].issue(user_access(), "SYNTHETIC-USER-TOKEN", 456, 789)
- assert "SYNTHETIC" not in str(raised.value)
-
-
-def test_invalid_key_fails_without_key_material(tmp_path):
- path = tmp_path / "invalid.pem"
- path.write_text("SYNTHETIC-INVALID-KEY")
- with pytest.raises(ValueError, match="could not be loaded"):
- GitHubInstallationCredentials("example-client", str(path))
-
-
- "permissions", [{"pull": True}, {"triage": True}, {}, {"push": "true"}]
-)
-async def test_read_only_repository_cannot_mint_write_token(
- signing, monkeypatch, permissions
-):
- github = user_access()
- github.repositories.return_value[0]["repositories"][0]["permissions"] = permissions
- post = AsyncMock()
- monkeypatch.setattr(httpx.AsyncClient, "post", post)
- with pytest.raises(GitHubError, match="needs write access"):
- await signing[0].issue(github, "SYNTHETIC", 456, 789)
- post.assert_not_called()
-
-
[email protected]("status", [204, 401, 404, 422])
-async def test_repository_revocation_is_idempotent(monkeypatch, status):
- request = AsyncMock(return_value=httpx.Response(status))
- monkeypatch.setattr(httpx.AsyncClient, "delete", request)
- await GitHubInstallationCredentials.revoke("SYNTHETIC-REPOSITORY")
- assert request.call_args.args == ("https://api.github.com/installation/token",)
-
-
[email protected]("status", [429, 500])
-async def test_transient_token_issue_is_retryable(signing, monkeypatch, status):
- from_error = httpx.Response(status)
- monkeypatch.setattr(httpx.AsyncClient, "post", AsyncMock(return_value=from_error))
- with pytest.raises(GitHubUnavailableError):
- await signing[0].issue(user_access(), "SYNTHETIC-USER", 456, 789)
-
-
-async def test_rejected_minted_scope_is_revoked(signing, monkeypatch):
- body = response_body()
- body["permissions"]["contents"] = "read"
- monkeypatch.setattr(
- httpx.AsyncClient,
- "post",
- AsyncMock(return_value=httpx.Response(201, json=body)),
- )
- with pytest.raises(GitHubError):
- await signing[0].issue(user_access(), "SYNTHETIC-USER", 456, 789)
- httpx.AsyncClient.delete.assert_awaited_once()
- assert (
- httpx.AsyncClient.delete.call_args.kwargs["headers"]["Authorization"]
- == "Bearer SYNTHETIC-INSTALLATION-TOKEN"
- )
-
-
- "headers,body,retryable",
- [
- ({}, {"message": "Organization access refused"}, False),
- ({"x-ratelimit-remaining": "0"}, {}, True),
- ({"Retry-After": "30"}, {}, True),
- ({}, {"message": "You have exceeded a secondary rate limit"}, True),
- ],
-)
-async def test_only_rate_limited_403_is_retryable(
- signing, monkeypatch, headers, body, retryable
-):
- monkeypatch.setattr(
- httpx.AsyncClient,
- "post",
- AsyncMock(return_value=httpx.Response(403, headers=headers, json=body)),
- )
- with pytest.raises(GitHubError) as raised:
- await signing[0].issue(user_access(), "SYNTHETIC-USER", 456, 789)
- assert isinstance(raised.value, GitHubUnavailableError) == retryable
diff --git a/core/tests/switch_core/session_activity/test_command_relay.py b/core/tests/switch_core/session_activity/test_command_relay.py
index 8f95f2ce6..c87790e9f 100644
--- a/core/tests/switch_core/session_activity/test_command_relay.py
+++ b/core/tests/switch_core/session_activity/test_command_relay.py
@@ -8,12 +8,10 @@ from fastapi import FastAPI
from switch_core.bridges.agent.protocol.agent_connections import (
PROTOCOL_VERSION,
- SESSION_COMMAND_PROTOCOL_REVISION,
AgentConnectionRegistry,
ClientDeclaration,
)
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.hosted_workers import WorkerBinding
from switch_core.bridges.agent.protocol.stream import event_frames
from switch_core.db.models import User
from switch_core.gateway.agent_sessions import router
@@ -82,7 +80,7 @@ async def test_a_relayed_session_command_reaches_the_watcher_stream() -> None:
await anext(stream) # connection_state
waiting = asyncio.create_task(anext(stream))
await asyncio.sleep(0.05)
- assert registry.relay_session_command(AGENT, FRAME, worker_only=False) is True
+ assert registry.relay_session_command(AGENT, FRAME) is True
frame = await asyncio.wait_for(waiting, 5)
assert frame.event == "session_command"
assert frame.data == FRAME
@@ -101,7 +99,7 @@ async def test_a_command_nobody_can_take_is_not_relayed(speaks, scope) -> None:
registry = AgentConnectionRegistry()
conn, stream = _watcher(registry, speaks=speaks, scope=scope)
conn.stream_attached = True
- assert registry.relay_session_command(AGENT, FRAME, worker_only=False) is False
+ assert registry.relay_session_command(AGENT, FRAME) is False
assert conn.session_commands == []
await stream.aclose()
@@ -115,22 +113,3 @@ async def test_switch_does_not_answer_where_an_agents_sessions_are(
registry.place_session(AGENT, "session-1", "room-1", f"conn-all-{PROTOCOL_VERSION}")
async with _client(session_factory, registry, owner) as client:
assert (await client.get("/agent-sessions/room-health")).status_code == 404
-
-
-async def test_a_worker_only_command_skips_every_connection_but_the_worker() -> None:
- registry = AgentConnectionRegistry()
- plain, plain_stream = _watcher(registry, speaks=SESSION_COMMAND_PROTOCOL_REVISION)
- plain.stream_attached = True
- assert registry.relay_session_command(AGENT, FRAME, worker_only=True) is False
- worker, worker_stream = _watcher(
- registry, speaks=SESSION_COMMAND_PROTOCOL_REVISION, scope="single"
- )
- worker.stream_attached = True
- registry.bind_worker(
- worker, WorkerBinding("launch-1", 1, "boot-a", "instance-a"), {}
- )
- assert registry.relay_session_command(AGENT, FRAME, worker_only=True) is True
- assert worker.session_commands == [FRAME]
- assert plain.session_commands == []
- await plain_stream.aclose()
- await worker_stream.aclose()
diff --git a/core/tests/switch_core/test_migration_hosted_merge.py b/core/tests/switch_core/test_migration_hosted_merge.py
deleted file mode 100644
index 04d112692..000000000
--- a/core/tests/switch_core/test_migration_hosted_merge.py
+++ /dev/null
@@ -1,1188 +0,0 @@
-"""Both histories reach the merge revision `33e037ee949f` with their data.
-
-A pilot database reaches it through the cutover manifest `a3c9e5f71d28`, which
-has to run before main's `b9e4d2a71c05` drops the server-side session tables:
-the manifest is what keeps their pending work.
-
-The manifest only captures; the drop also waits for every retained volume's
-preflight check to be recorded (`hosted-cutover-upgrade record`), which the
-wrapper, `migrations/env.py` before the drop and the merge revision after it
-all enforce. `_upgrade_to` builds its own migration context and so exercises
-only the merge revision; the `test_real_upgrade_*` cases go through `env.py`
-the way `alembic upgrade` and Core's boot do.
-
-The hosted-agent chain (`ab921ef034cd` .. `95fc38e451b6`) was applied on a
-pilot before main grew its own head (`c4e9a1f7b203`), so a database can arrive
-at the merge from either side. The pilot side still has to run main's
-migrations, including the one that drops the server-side SDK session tables;
-main's side has to run the hosted chain. Each path starts from an empty
-database, seeds the rows its side owns, upgrades to heads and checks that the
-rows, the tenant-isolation policies and the runtime grants are all in place.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import json
-import uuid
-from collections.abc import AsyncIterator, Awaitable, Callable
-from pathlib import Path
-from types import SimpleNamespace
-from typing import Any, cast
-
-import pytest
-from alembic import command as alembic_command
-from alembic.config import Config
-from alembic.runtime.environment import EnvironmentContext
-from alembic.script import ScriptDirectory
-from sqlalchemy import Connection, text
-from sqlalchemy.engine import make_url
-from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine, create_async_engine
-
-import switch_core.db.models # noqa: F401 — registers every table on Base.metadata
-from switch_core.bridges.agent.hosted_cutover import CutoverManifest
-from switch_core.config import SwitchConfig
-from switch_core.db.hosted_cutover_gate import DROP_REVISION
-from switch_core.db.rls_ddl import POLICY_NAME, REQUIRE_TENANT_FUNCTION_NAME
-from switch_core.db.runtime_role import _require_every_policy, grant_runtime_role
-from switch_core.hosted_cutover_upgrade import (
- BlockedCheck,
- CutoverRefused,
- ManifestCheck,
- PreflightBlocked,
- cutover_problems,
- queue_all_imports,
- record,
- running_launches,
- upgrade,
-)
-from switch_core.main import _migrate_and_grant
-
-_CORE = Path(__file__).resolve().parents[2]
-
-_MERGE_REVISION = "33e037ee949f"
-_PILOT_HEAD = "95fc38e451b6"
-_MAIN_HEAD = "c4e9a1f7b203"
-# Main's head when it was last merged in: a database already there takes the
-# whole hosted chain on its next upgrade.
-_LATER_MAIN_HEAD = "a9e1c3f75b20"
-_MANIFEST_REVISION = "a3c9e5f71d28"
-# The last revision a database with cloud agents that are not removed can
-# reach: `b4e1d7a2c9f0` refuses one until they are.
-_BEFORE_MACHINES = "5c1e9b7d3f02"
-
-_HOSTED_TABLES = (
- "provider_connections",
- "provider_verifications",
- "hosted_launches",
- "hosted_operations",
- "github_issued_tokens",
- "hosted_wake_mailbox",
- "hosted_cutover_volumes",
- "hosted_cutover_items",
-)
-
-_EMPTY_MANIFEST = ManifestCheck(
- manifest=CutoverManifest(
- manifest_sha256="4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945",
- items=[],
- )
-)
-
-_PREDICATE = (
- f"(tenant_id = ( SELECT {REQUIRE_TENANT_FUNCTION_NAME}() "
- f"AS {REQUIRE_TENANT_FUNCTION_NAME}))"
-)
-
-
-def _script_directory() -> tuple[Config, ScriptDirectory]:
- config = Config(str(_CORE / "alembic.ini"))
- config.set_main_option("script_location", str(_CORE / "switch_core" / "migrations"))
- return config, ScriptDirectory.from_config(config)
-
-
-def _upgrade_to(revision: str) -> Any:
- def upgrade(connection: Connection) -> None:
- config, script = _script_directory()
-
- def do_upgrade(current_revision: str, context: Any) -> Any:
- return script._upgrade_revs(revision, current_revision)
-
- with EnvironmentContext(config, script, fn=do_upgrade) as environment:
- environment.configure(connection=connection)
- with environment.begin_transaction():
- environment.run_migrations()
-
- return upgrade
-
-
-async def _database(postgres_url: str, name: str) -> AsyncIterator[str]:
- admin = create_async_engine(postgres_url, isolation_level="AUTOCOMMIT")
- async with admin.connect() as connection:
- await connection.execute(text(f'DROP DATABASE IF EXISTS "{name}"'))
- await connection.execute(text(f'CREATE DATABASE "{name}"'))
- await admin.dispose()
-
- base, _, _ = postgres_url.rpartition("/")
- try:
- yield f"{base}/{name}"
- finally:
- admin = create_async_engine(postgres_url, isolation_level="AUTOCOMMIT")
- async with admin.connect() as connection:
- await connection.execute(text(f'DROP DATABASE IF EXISTS "{name}"'))
- await admin.dispose()
-
-
-async def pilot_url(postgres_url: str) -> AsyncIterator[str]:
- async for url in _database(postgres_url, "migration_hosted_merge_pilot"):
- yield url
-
-
-async def main_url(postgres_url: str) -> AsyncIterator[str]:
- async for url in _database(postgres_url, "migration_hosted_merge_main"):
- yield url
-
-
-async def _seed_tenant_and_user(connection: AsyncConnection) -> None:
- await connection.execute(
- text("INSERT INTO tenants (id, name, slug) VALUES ('t1', 'Tenant', 't1')")
- )
- await connection.execute(
- text(
- "INSERT INTO users (id, name, email, role, password_hash) "
- "VALUES ('u1', 'User', '[email protected]', 'user', 'x')"
- )
- )
-
-
-def _command(room_id: str | None, message_id: str | None, surface: str) -> str:
- return json.dumps(
- {
- "contractVersion": 1,
- "commandId": "placeholder-command",
- "sessionId": "s1",
- "epoch": "e1",
- "origin": {
- "actorId": "placeholder-actor",
- "surface": surface,
- "roomId": room_id,
- "threadId": "th1" if room_id else None,
- "messageId": message_id,
- },
- "body": {"type": "message.send", "text": "placeholder", "attachments": []},
- }
- )
-
-
-def _status(command_id: str, status: str) -> str:
- return json.dumps(
- {
- "type": "command.status",
- "commandId": command_id,
- "status": status,
- "code": None,
- "message": None,
- }
- )
-
-
-async def _seed_sdk_rows(connection: AsyncConnection) -> None:
- """#538's server-side session state for the hosted agent `a1` and a local agent `a9`."""
- # The agents, rooms and bridges these rows point at are beside the point here.
- await connection.execute(text("SET LOCAL session_replication_role = replica"))
- for session_id, agent_id in (("s1", "a1"), ("s9", "a9")):
- await connection.execute(
- text(
- "INSERT INTO sdk_sessions "
- "(tenant_id, id, agent_id, host_id, epoch, lease_expires_at, snapshot, host_sequence) "
- "VALUES ('t1', :id, :agent, 'h1', 'e1', now(), '{}', 7)"
- ),
- {"id": session_id, "agent": agent_id},
- )
- for session_id, command_id, room_id, message_id, surface, status in (
- ("s1", "c1", "r1", "m1", "slack", "dispatched"),
- ("s1", "c2", "r1", "m2", "slack", "applied"),
- ("s1", "c3", None, None, "console", "accepted"),
- ("s1", "c4", None, None, "console", "applied"),
- ("s9", "c5", "r1", "m3", "slack", "accepted"),
- ):
- await connection.execute(
- text(
- "INSERT INTO sdk_session_commands "
- "(tenant_id, session_id, command_id, accepted_sequence, command, status) "
- "VALUES ('t1', :session, :command_id, 1, CAST(:command AS jsonb), CAST(:status AS jsonb))"
- ),
- {
- "session": session_id,
- "command_id": command_id,
- "command": _command(room_id, message_id, surface),
- "status": _status(command_id, status),
- },
- )
- for post_id, removed in (("p1", None), ("p2", "now()")):
- await connection.execute(
- text(
- "INSERT INTO session_request_posts "
- "(tenant_id, id, bridge_id, token, handle, external_channel_id, "
- "external_post_id, room_id, thread_id, session_id, epoch, request_id, "
- "revision, form, removed_at) "
- f"VALUES ('t1', :id, 'b1', :id || '-placeholder-token', :id, 'C0', :id, 'r1', "
- f"'th1', 's1', 'e1', :request, 1, '{{}}', {removed or 'NULL'})"
- ),
- {"id": post_id, "request": f"q-{post_id}"},
- )
-
-
-_EXPECTED_MANIFEST = (
- [("l1", "pending")],
- [
- (
- "console_command",
- "s1",
- None,
- None,
- None,
- {"core": {"command_id": "c3", "status": "accepted"}},
- ),
- (
- "operation",
- "s1",
- None,
- None,
- None,
- {"core": {"operation_id": "o1", "action": "start", "state": "queued"}},
- ),
- (
- "request_open",
- "s1",
- "r1",
- None,
- "th1",
- {"core": {"request_id": "q-p1", "epoch": "e1"}},
- ),
- (
- "room_message",
- "s1",
- "r1",
- "m1",
- "th1",
- {"core": {"command_id": "c1", "status": "dispatched", "code": None}},
- ),
- (
- "room_message",
- "s1",
- "r1",
- "m2",
- "th1",
- {"core": {"command_id": "c2", "status": "applied", "code": None}},
- ),
- ("session", "s1", None, None, None, "session"),
- ],
-)
-
-
-def _config(url: str) -> SwitchConfig:
- return cast(SwitchConfig, SimpleNamespace(owner_database_url=url, database_url=url))
-
-
-async def _seed_import(connection: AsyncConnection) -> None:
- """A room message Core accepted for `a1` that no worker saw, with an attachment
- whose blob #538 tied to the session, and a session blob nothing imports."""
- await connection.execute(text("SET LOCAL session_replication_role = replica"))
- await connection.execute(
- text(
- "INSERT INTO rooms (tenant_id, id, matrix_room_id, name, description) "
- "VALUES ('t1', 'r2', '!r2:example.invalid', 'Room', '')"
- )
- )
- await connection.execute(
- text(
- "INSERT INTO messages (tenant_id, id, seq, room_id, transport_event_id, "
- "sender_id, event_type, msgtype, body, content) "
- "VALUES ('t1', 'row-m4', 1, 'r2', 'm4', '@person:example.invalid', "
- "'m.room.message', 'm.file', 'report.txt', '{}')"
- )
- )
- await connection.execute(
- text(
- "INSERT INTO message_attachments "
- "(tenant_id, id, message_id, position, uri, filename, mimetype, size) "
- "VALUES ('t1', 'att-m4', 'row-m4', 0, 'mxc://example.invalid/kept', "
- "'report.txt', 'text/plain', 5)"
- )
- )
- for blob, uri in (("blob-kept", "kept"), ("blob-dropped", "dropped")):
- await connection.execute(
- text(
- "INSERT INTO media_blobs (tenant_id, id, uri, size, data, sdk_session_id) "
- "VALUES ('t1', :id, :uri, 5, 'bytes', 's1')"
- ),
- {"id": blob, "uri": f"mxc://example.invalid/{uri}"},
- )
- await connection.execute(
- text(
- "INSERT INTO sdk_session_commands "
- "(tenant_id, session_id, command_id, accepted_sequence, command, status) "
- "VALUES ('t1', 's1', 'c6', 2, CAST(:command AS jsonb), CAST(:status AS jsonb))"
- ),
- {"command": _command("r2", "m4", "slack"), "status": _status("c6", "accepted")},
- )
-
-
-async def _pilot_at_manifest(url: str, *, with_import: bool) -> None:
- """A pilot database with a stopped hosted launch, prepared at the manifest revision."""
- engine = create_async_engine(url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_PILOT_HEAD))
- async with engine.begin() as connection:
- await _seed_tenant_and_user(connection)
- await connection.execute(
- text(
- "INSERT INTO hosted_launches "
- "(tenant_id, id, owner_id, name, spec, agent_id, state, desired_state) "
- "VALUES ('t1', 'l1', 'u1', 'pilot-agent', '{}', 'a1', 'stopped', 'stopped')"
- )
- )
- await _seed_sdk_rows(connection)
- if with_import:
- await _seed_import(connection)
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_MANIFEST_REVISION))
- finally:
- await engine.dispose()
-
-
-async def _ungated_heads_refused(url: str, match: str) -> None:
- """A bare `alembic upgrade heads` rolls back, the drop with it."""
- engine = create_async_engine(url)
- try:
- with pytest.raises(RuntimeError, match=match):
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to("heads"))
- async with engine.begin() as connection:
- assert (
- await connection.scalar(text("SELECT version_num FROM alembic_version"))
- == _MANIFEST_REVISION
- )
- assert await connection.scalar(text("SELECT to_regclass('sdk_sessions')"))
- finally:
- await engine.dispose()
-
-
-async def _cutover_rows(connection: AsyncConnection) -> tuple[list[Any], list[Any]]:
- volumes = (
- await connection.execute(
- text("SELECT launch_id, preflight_state FROM hosted_cutover_volumes")
- )
- ).all()
- items = (
- await connection.execute(
- text(
- "SELECT kind, session_id, room_id, message_id, thread_id, evidence "
- "FROM hosted_cutover_items WHERE agent_id = 'a1' AND disposition IS NULL "
- "ORDER BY kind, message_id"
- )
- )
- ).all()
- assert all(
- row.kind != "session" or row.evidence["core"]["host_sequence"] == 7
- for row in items
- )
- return [tuple(row) for row in volumes], [
- tuple(row)[:5] + (("session",) if row.kind == "session" else (row.evidence,))
- for row in items
- ]
-
-
-async def _assert_merged_schema(connection: AsyncConnection) -> None:
- _, script = _script_directory()
- heads = script.get_heads()
- assert len(heads) == 1
- ancestry = {
- revision.revision for revision in script.walk_revisions("base", heads[0])
- }
- assert _MERGE_REVISION in ancestry
- versions = (
- (await connection.execute(text("SELECT version_num FROM alembic_version")))
- .scalars()
- .all()
- )
- assert versions == heads
-
- sdk_tables = (
- (
- await connection.execute(
- text(
- "SELECT tablename FROM pg_tables "
- "WHERE schemaname = 'public' AND tablename LIKE 'sdk\\_%'"
- )
- )
- )
- .scalars()
- .all()
- )
- assert sdk_tables == []
-
- sdk_foreign_keys = (
- await connection.execute(
- text(
- "SELECT conname FROM pg_constraint c "
- "JOIN pg_class target ON target.oid = c.confrelid "
- "WHERE c.contype = 'f' AND target.relname LIKE 'sdk\\_%'"
- )
- )
- ).all()
- assert sdk_foreign_keys == []
-
- policies = {
- row.tablename: (row.rowsecurity, row.qual, row.with_check)
- for row in await connection.execute(
- text(
- "SELECT t.tablename, t.rowsecurity, p.qual, p.with_check "
- "FROM pg_tables t LEFT JOIN pg_policies p "
- "ON p.schemaname = t.schemaname AND p.tablename = t.tablename "
- "AND p.policyname = :name "
- "WHERE t.schemaname = 'public' AND t.tablename = ANY(:tables)"
- ),
- {"name": POLICY_NAME, "tables": list(_HOSTED_TABLES)},
- )
- }
- assert policies == {
- table: (True, _PREDICATE, _PREDICATE) for table in _HOSTED_TABLES
- }
- await _require_every_policy(connection)
-
-
-async def _remove_launches_and_upgrade(engine: AsyncEngine) -> None:
- """Remove every cloud agent, as `b4e1d7a2c9f0` requires, then upgrade to heads."""
- async with engine.begin() as connection:
- await connection.execute(
- text(
- "UPDATE hosted_launches SET state = 'deleted', desired_state = 'deleted'"
- )
- )
- await connection.run_sync(_upgrade_to("heads"))
- async with engine.begin() as connection:
- await _assert_merged_schema(connection)
- await _assert_runtime_grants(connection)
-
-
-async def _assert_runtime_grants(connection: AsyncConnection) -> None:
- role = f"switch_merge_test_{uuid.uuid4().hex[:12]}"
- await connection.execute(
- text(
- f'CREATE ROLE "{role}" NOLOGIN '
- "NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS"
- )
- )
- try:
- await grant_runtime_role(connection, role)
- for table in _HOSTED_TABLES:
- for privilege in ("SELECT", "INSERT", "UPDATE", "DELETE"):
- assert await connection.scalar(
- text("SELECT has_table_privilege(:role, :table, :privilege)"),
- {"role": role, "table": table, "privilege": privilege},
- ), (table, privilege)
- assert not await connection.scalar(
- text(
- "SELECT pg_has_role(:role, t.tableowner, 'USAGE') FROM pg_tables t WHERE t.tablename = :table"
- ),
- {"role": role, "table": table},
- ), table
- assert await connection.scalar(
- text("SELECT has_sequence_privilege(:role, 'agent_event_boot', 'USAGE')"),
- {"role": role},
- )
- finally:
- await connection.execute(text(f'DROP OWNED BY "{role}"'))
- await connection.execute(text(f'DROP ROLE "{role}"'))
-
-
-async def test_pilot_database_keeps_hosted_rows_through_the_merge(
- pilot_url: str,
-) -> None:
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_PILOT_HEAD))
- assert await connection.scalar(text("SELECT to_regclass('sdk_sessions')"))
-
- async with engine.begin() as connection:
- await _seed_tenant_and_user(connection)
- await connection.execute(
- text(
- "INSERT INTO provider_connections "
- "(tenant_id, user_id, provider, kind, encrypted_credential, verified_at) "
- "VALUES ('t1', 'u1', 'claude', 'api-key', 'sealed-credential', now())"
- )
- )
- await connection.execute(
- text(
- "INSERT INTO provider_verifications "
- "(tenant_id, id, user_id, provider, kind, token_hash, state, "
- "created_at, deadline) "
- "VALUES ('t1', 'v1', 'u1', 'claude', 'api-key', 'hash', 'queued', "
- "now(), now())"
- )
- )
- await connection.execute(
- text(
- "INSERT INTO hosted_launches "
- "(tenant_id, id, owner_id, name, spec, agent_id, state, desired_state) "
- "VALUES ('t1', 'l1', 'u1', 'pilot-agent', "
- "'{\"auto_session\": true}', 'a1', 'stopped', 'stopped')"
- )
- )
- await connection.execute(
- text(
- "INSERT INTO hosted_operations "
- "(tenant_id, id, launch_id, launch_revision, session_id, action) "
- "VALUES ('t1', 'o1', 'l1', 1, 's1', 'start')"
- )
- )
- await connection.execute(
- text(
- "INSERT INTO github_issued_tokens "
- "(tenant_id, id, owner_id, launch_id, launch_revision, "
- "encrypted_token, expires_at, revoke_requested, attempts) "
- "VALUES ('t1', 'g1', 'u1', 'l1', 1, 'sealed-token', now(), false, 0)"
- )
- )
- await _seed_sdk_rows(connection)
-
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_MANIFEST_REVISION))
-
- async with engine.begin() as connection:
- assert await connection.scalar(text("SELECT to_regclass('sdk_sessions')"))
- captured = await _cutover_rows(connection)
-
- await record(_config(pilot_url), "l1", _EMPTY_MANIFEST)
- assert await cutover_problems(_config(pilot_url)) == []
-
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_BEFORE_MACHINES))
-
- async with engine.begin() as connection:
- connections = (
- await connection.execute(
- text("SELECT user_id, provider, kind FROM provider_connections")
- )
- ).all()
- verifications = (
- await connection.execute(
- text("SELECT id, state FROM provider_verifications")
- )
- ).all()
- launches = (
- await connection.execute(
- text("SELECT id, name, agent_id, state, spec FROM hosted_launches")
- )
- ).all()
- operations = (
- await connection.execute(
- text(
- "SELECT id, launch_id, launch_revision, action "
- "FROM hosted_operations"
- )
- )
- ).all()
- tokens = (
- await connection.execute(
- text(
- "SELECT id, launch_id, encrypted_token FROM github_issued_tokens"
- )
- )
- ).all()
- kept = await _cutover_rows(connection)
- decided = {
- (row.kind, row.message_id): row.disposition
- for row in await connection.execute(
- text(
- "SELECT kind, message_id, disposition FROM hosted_cutover_items "
- "WHERE agent_id = 'a1'"
- )
- )
- }
- await _remove_launches_and_upgrade(engine)
- finally:
- await engine.dispose()
-
- assert [tuple(row) for row in connections] == [("u1", "claude", "api-key")]
- assert [tuple(row) for row in verifications] == [("v1", "queued")]
- assert [tuple(row) for row in launches] == [
- ("l1", "pilot-agent", "a1", "stopped", {"auto_session": True})
- ]
- assert [tuple(row) for row in operations] == [("o1", "l1", 1, "start")]
- assert [tuple(row) for row in tokens] == [("g1", "l1", "sealed-token")]
- assert captured == _EXPECTED_MANIFEST
- assert kept == ([("l1", "complete")], [])
- assert decided == {
- ("console_command", None): "owner_notice",
- ("operation", None): "preserved",
- ("request_open", None): "interrupted",
- ("room_message", "m1"): "uncertain",
- ("room_message", "m2"): "ran",
- ("session", None): "preserved",
- }
-
-
-async def test_pilot_upgrade_refuses_to_drop_sessions_before_the_manifest(
- pilot_url: str,
-) -> None:
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_PILOT_HEAD))
- async with engine.begin() as connection:
- await _seed_tenant_and_user(connection)
- await connection.execute(
- text(
- "INSERT INTO hosted_launches "
- "(tenant_id, id, owner_id, name, spec, agent_id, state) "
- "VALUES ('t1', 'l1', 'u1', 'pilot-agent', '{}', 'a1', 'stopped')"
- )
- )
- await _seed_sdk_rows(connection)
-
- with pytest.raises(RuntimeError, match="just hosted-cutover-upgrade"):
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to("heads"))
-
- async with engine.begin() as connection:
- version = await connection.scalar(
- text("SELECT version_num FROM alembic_version")
- )
- commands = await connection.scalar(
- text("SELECT count(*) FROM sdk_session_commands")
- )
- finally:
- await engine.dispose()
-
- assert version == _PILOT_HEAD
- assert commands == 5
-
- config = cast(
- SwitchConfig,
- SimpleNamespace(owner_database_url=pilot_url, database_url=pilot_url),
- )
- assert await running_launches(config) == ["l1"]
- with pytest.raises(CutoverRefused, match="l1"):
- await upgrade(config)
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.execute(
- text("UPDATE hosted_launches SET desired_state = 'stopped'")
- )
- finally:
- await engine.dispose()
- assert await running_launches(config) == []
-
-
-async def test_main_database_keeps_session_activity_through_the_merge(
- main_url: str,
-) -> None:
- engine = create_async_engine(main_url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_MAIN_HEAD))
- assert (
- await connection.scalar(text("SELECT to_regclass('hosted_launches')"))
- is None
- )
-
- async with engine.begin() as connection:
- await _seed_tenant_and_user(connection)
- # The agent these rows belong to is beside the point here.
- await connection.execute(
- text("SET LOCAL session_replication_role = replica")
- )
- await connection.execute(
- text(
- """
- INSERT INTO approval_requests
- (tenant_id, agent_id, session_id, request_id, turn_id,
- kind, title, options, questions, state)
- VALUES ('t1', 'a1', 's1', 'r1', 'turn-1', 'approval',
- 'Write file?',
- '[{"id": "0", "label": "Allow", "decision": "accept"}]',
- '[]', 'open')
- """
- )
- )
- await connection.execute(
- text(
- """
- INSERT INTO session_activity_items
- (tenant_id, agent_id, session_id, turn_id, item_id, kind,
- revision, status, title, text, occurred_at)
- VALUES ('t1', 'a1', 's1', 'turn-1', 'item-1', 'tool-activity', 1,
- 'completed', 'Read file', 'README.md', now())
- """
- )
- )
-
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to("heads"))
-
- async with engine.begin() as connection:
- await _assert_merged_schema(connection)
- await _assert_runtime_grants(connection)
- requests = (
- await connection.execute(
- text("SELECT request_id, title, state FROM approval_requests")
- )
- ).all()
- items = (
- await connection.execute(
- text("SELECT item_id, status, title FROM session_activity_items")
- )
- ).all()
- cutover = await _cutover_rows(connection)
- finally:
- await engine.dispose()
-
- assert [tuple(row) for row in requests] == [("r1", "Write file?", "open")]
- assert [tuple(row) for row in items] == [("item-1", "completed", "Read file")]
- assert cutover == ([], [])
-
-
-async def test_later_main_database_takes_the_hosted_chain(main_url: str) -> None:
- engine = create_async_engine(main_url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_LATER_MAIN_HEAD))
- assert (
- await connection.scalar(text("SELECT to_regclass('hosted_launches')"))
- is None
- )
-
- async with engine.begin() as connection:
- await _seed_tenant_and_user(connection)
- await connection.execute(
- text(
- "INSERT INTO tenant_join_domains (tenant_id, domain, created_by) "
- "VALUES ('t1', 'example.invalid', 'u1')"
- )
- )
-
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to("heads"))
-
- async with engine.begin() as connection:
- await _assert_merged_schema(connection)
- await _assert_runtime_grants(connection)
- domains = (
- await connection.execute(
- text("SELECT tenant_id, domain FROM tenant_join_domains")
- )
- ).all()
- cutover = await _cutover_rows(connection)
- finally:
- await engine.dispose()
-
- assert [tuple(row) for row in domains] == [("t1", "example.invalid")]
- assert cutover == ([], [])
-
-
-async def test_pilot_upgrade_refuses_the_drop_until_every_volume_is_recorded(
- pilot_url: str,
-) -> None:
- config = _config(pilot_url)
- await _pilot_at_manifest(pilot_url, with_import=True)
-
- problems = await cutover_problems(config)
- assert any("launch l1 has no recorded preflight check" in p for p in problems)
- assert any("launch l1 has cutover items no manifest decided" in p for p in problems)
- with pytest.raises(CutoverRefused, match="l1"):
- await upgrade(config)
- await _ungated_heads_refused(
- pilot_url, "the volume of launch l1 has no recorded preflight check"
- )
-
- await record(
- config,
- "l1",
- BlockedCheck(
- blocked=PreflightBlocked(
- step="sessions",
- file="/data/state/sessions/placeholder/events.jsonl",
- line=3,
- error="the line is not JSON",
- )
- ),
- )
- (blocked,) = [p for p in await cutover_problems(config) if "blocked" in p]
- assert (
- "launch l1: sessions at /data/state/sessions/placeholder/events.jsonl:3"
- in blocked
- )
- assert "the line is not JSON" in blocked
- with pytest.raises(CutoverRefused, match="events.jsonl:3"):
- await upgrade(config)
- await _ungated_heads_refused(
- pilot_url, "the preflight check blocked on the volume of launch l1"
- )
-
- await record(config, "l1", _EMPTY_MANIFEST)
- assert await cutover_problems(config) == []
-
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.execute(
- text(
- "UPDATE media_blobs SET sdk_session_id = 's1' WHERE id = 'blob-kept'"
- )
- )
- assert any(
- "would be dropped with its session" in p
- for p in await cutover_problems(config)
- )
- await _ungated_heads_refused(
- pilot_url,
- "import for launch l1 mxc://example.invalid/kept is gone",
- )
- await record(config, "l1", _EMPTY_MANIFEST)
- assert await cutover_problems(config) == []
-
- async with engine.begin() as connection:
- await connection.execute(
- text(
- "UPDATE hosted_cutover_items SET payload = NULL WHERE message_id = 'm4'"
- )
- )
- assert any(
- "the import l1 r2 m4 has no event" in p
- for p in await cutover_problems(config)
- )
- await _ungated_heads_refused(pilot_url, "the import l1 r2 m4 has no event")
- async with engine.begin() as connection:
- await connection.execute(
- text(
- "UPDATE hosted_cutover_items SET payload = CAST(:event AS jsonb) "
- "WHERE message_id = 'm4'"
- ),
- {
- "event": json.dumps(
- {"type": "message", "payload": {}, "missed": None}
- )
- },
- )
- finally:
- await engine.dispose()
-
-
-async def test_pilot_upgrade_refuses_a_capture_the_old_core_outran(
- pilot_url: str,
-) -> None:
- config = _config(pilot_url)
- await _pilot_at_manifest(pilot_url, with_import=False)
- await record(config, "l1", _EMPTY_MANIFEST)
- assert await cutover_problems(config) == []
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.execute(
- text(
- "UPDATE sdk_session_commands SET status = CAST(:status AS jsonb) "
- "WHERE command_id = 'c1'"
- ),
- {"status": _status("c1", "applied")},
- )
- finally:
- await engine.dispose()
- (stale,) = await cutover_problems(config)
- assert stale.startswith("c1 changed in the old session tables after `prepare`")
-
-
-async def test_pilot_upgrade_refuses_a_launch_restarted_or_added_after_prepare(
- pilot_url: str,
-) -> None:
- config = _config(pilot_url)
- await _pilot_at_manifest(pilot_url, with_import=False)
- await record(config, "l1", _EMPTY_MANIFEST)
- assert await cutover_problems(config) == []
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.execute(
- text("UPDATE hosted_launches SET desired_state = 'running'")
- )
- assert await cutover_problems(config) == ["launch l1 is not stopped"]
- await _ungated_heads_refused(pilot_url, "launch l1 is not stopped")
- async with engine.begin() as connection:
- await connection.execute(
- text("UPDATE hosted_launches SET desired_state = 'stopped'")
- )
- await connection.execute(
- text("SET LOCAL session_replication_role = replica")
- )
- await connection.execute(
- text(
- "INSERT INTO hosted_launches "
- "(tenant_id, id, owner_id, name, spec, agent_id, state, desired_state) "
- "VALUES ('t1', 'l2', 'u1', 'late-agent', '{}', 'a2', 'stopped', 'stopped')"
- )
- )
- finally:
- await engine.dispose()
- (missing,) = await cutover_problems(config)
- assert missing.startswith("launch l2 has no cutover volume")
- await _ungated_heads_refused(pilot_url, "launch l2 has no cutover volume")
-
-
-async def test_pilot_upgrade_keeps_what_the_recorded_volumes_import(
- pilot_url: str,
-) -> None:
- config = _config(pilot_url)
- await _pilot_at_manifest(pilot_url, with_import=True)
- await record(config, "l1", _EMPTY_MANIFEST)
- assert await cutover_problems(config) == []
-
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_BEFORE_MACHINES))
- assert await cutover_problems(config) == []
- await queue_all_imports(config)
- await queue_all_imports(config)
- async with engine.begin() as connection:
- blobs = (
- await connection.scalars(text("SELECT id FROM media_blobs ORDER BY id"))
- ).all()
- mailbox = (
- await connection.execute(
- text(
- "SELECT agent_id, room_id, message_id, launch_id, state, origin, "
- "event->'payload'->'attachments'->0->>'mxc' AS mxc "
- "FROM hosted_wake_mailbox"
- )
- )
- ).all()
- queued = await connection.scalar(
- text("SELECT imports_queued_at IS NOT NULL FROM hosted_cutover_volumes")
- )
- await _remove_launches_and_upgrade(engine)
- finally:
- await engine.dispose()
-
- assert blobs == ["blob-kept"]
- assert [tuple(row) for row in mailbox] == [
- ("a1", "r2", "m4", "l1", "pending", "cutover", "mxc://example.invalid/kept")
- ]
- assert queued
-
-
-def _migration_config(url: str, monkeypatch: pytest.MonkeyPatch) -> SwitchConfig:
- """The environment `migrations/env.py` reads, pointed at `url`."""
- parts = make_url(url)
- environment = {
- "DB_HOST": parts.host,
- "DB_PORT": str(parts.port),
- "DB_USER": parts.username,
- "DB_PASSWORD": parts.password,
- "DB_NAME": parts.database,
- "MATRIX_SERVER_NAME": "example.invalid",
- "AGENT_REGISTRATION_TOKEN": "placeholder-registration-token",
- "SECRET_KEYS": "test:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
- "GATEWAY_ADMIN_EMAIL": "[email protected]",
- "GATEWAY_ADMIN_PASSWORD": "placeholder-password",
- }
- for name, value in environment.items():
- assert value is not None
- monkeypatch.setenv(name, value)
- monkeypatch.delenv("DB_OWNER_USER", raising=False)
- monkeypatch.delenv("DB_OWNER_PASSWORD", raising=False)
- return SwitchConfig() # type: ignore[call-arg]
-
-
-async def _alembic_upgrade_heads(config: SwitchConfig) -> None:
- await asyncio.to_thread(
- alembic_command.upgrade, Config(str(_CORE / "alembic.ini")), "heads"
- )
-
-
-_MIGRATE_PATHS = pytest.mark.parametrize(
- "migrate",
- [_alembic_upgrade_heads, _migrate_and_grant],
- ids=["alembic-upgrade-heads", "core-boot"],
-)
-
-
-async def _head(url: str) -> tuple[list[str], bool]:
- engine = create_async_engine(url)
- try:
- async with engine.connect() as connection:
- versions = list(
- await connection.scalars(
- text("SELECT version_num FROM alembic_version")
- )
- )
- sessions = await connection.scalar(
- text("SELECT to_regclass('sdk_sessions') IS NOT NULL")
- )
- finally:
- await engine.dispose()
- return versions, bool(sessions)
-
-
-@_MIGRATE_PATHS
- ("with_import", "statements", "match"),
- [
- pytest.param(
- False,
- [
- "UPDATE sdk_session_commands SET status = CAST('"
- + _status("c1", "applied")
- + "' AS jsonb) WHERE command_id = 'c1'"
- ],
- "c1 changed in the old session tables after `prepare`",
- id="stale-capture",
- ),
- pytest.param(
- False,
- [
- "SET LOCAL session_replication_role = replica",
- "INSERT INTO hosted_launches "
- "(tenant_id, id, owner_id, name, spec, agent_id, state, desired_state) "
- "VALUES ('t1', 'l2', 'u1', 'late-agent', '{}', 'a2', 'stopped', 'stopped')",
- ],
- "launch l2 has no cutover volume",
- id="missing-volume",
- ),
- pytest.param(
- True,
- ["UPDATE hosted_cutover_items SET payload = NULL WHERE message_id = 'm4'"],
- "the import l1 r2 m4 has no event",
- id="missing-import-payload",
- ),
- pytest.param(
- False,
- ["UPDATE hosted_launches SET desired_state = 'running'"],
- "launch l1 is not stopped",
- id="launch-running",
- ),
- ],
-)
-async def test_real_upgrade_refuses_an_incomplete_cutover_before_the_drop(
- pilot_url: str,
- monkeypatch: pytest.MonkeyPatch,
- migrate: Callable[[SwitchConfig], Awaitable[None]],
- with_import: bool,
- statements: list[str],
- match: str,
-) -> None:
- await _pilot_at_manifest(pilot_url, with_import=with_import)
- await record(_config(pilot_url), "l1", _EMPTY_MANIFEST)
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- for statement in statements:
- await connection.execute(text(statement))
- finally:
- await engine.dispose()
- config = _migration_config(pilot_url, monkeypatch)
-
- with pytest.raises(RuntimeError, match=match):
- await migrate(config)
-
- assert await _head(pilot_url) == ([_MANIFEST_REVISION], True)
-
-
-async def _alembic_upgrade_to_drop(config: SwitchConfig) -> None:
- await asyncio.to_thread(
- alembic_command.upgrade, Config(str(_CORE / "alembic.ini")), DROP_REVISION
- )
-
-
- "migrate",
- [_alembic_upgrade_to_drop, _alembic_upgrade_heads, _migrate_and_grant],
- ids=["alembic-upgrade-drop", "alembic-upgrade-heads", "core-boot"],
-)
-async def test_real_upgrade_refuses_a_cutover_never_prepared(
- pilot_url: str,
- monkeypatch: pytest.MonkeyPatch,
- migrate: Callable[[SwitchConfig], Awaitable[None]],
-) -> None:
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(_PILOT_HEAD))
- await _seed_tenant_and_user(connection)
- await connection.execute(
- text(
- "INSERT INTO hosted_launches "
- "(tenant_id, id, owner_id, name, spec, agent_id, state, desired_state) "
- "VALUES ('t1', 'l1', 'u1', 'pilot-agent', '{}', 'a1', 'stopped', 'stopped')"
- )
- )
- await _seed_sdk_rows(connection)
- finally:
- await engine.dispose()
-
- with pytest.raises(
- RuntimeError, match=r"launch l1 .*run `just hosted-cutover-upgrade prepare`"
- ):
- await migrate(_migration_config(pilot_url, monkeypatch))
-
- assert await _head(pilot_url) == ([_PILOT_HEAD], True)
-
-
-@_MIGRATE_PATHS
-async def test_real_upgrade_completes_a_prepared_cutover(
- pilot_url: str,
- monkeypatch: pytest.MonkeyPatch,
- migrate: Callable[[SwitchConfig], Awaitable[None]],
-) -> None:
- await _pilot_at_manifest(pilot_url, with_import=True)
- await record(_config(pilot_url), "l1", _EMPTY_MANIFEST)
-
- with pytest.raises(RuntimeError, match="hosted_machines: 1 cloud agent"):
- await migrate(_migration_config(pilot_url, monkeypatch))
- assert await _head(pilot_url) == ([_MANIFEST_REVISION], True)
-
- engine = create_async_engine(pilot_url)
- try:
- async with engine.begin() as connection:
- await connection.execute(
- text(
- "UPDATE hosted_launches SET state = 'deleted', desired_state = 'deleted'"
- )
- )
- finally:
- await engine.dispose()
- await migrate(_migration_config(pilot_url, monkeypatch))
-
- _, script = _script_directory()
- assert await _head(pilot_url) == ([script.get_current_head()], False)
-
-
-@_MIGRATE_PATHS
- "start", [None, "545f80e11f13"], ids=["fresh", "main-before-drop"]
-)
-async def test_real_upgrade_leaves_databases_without_hosted_launches_alone(
- main_url: str,
- monkeypatch: pytest.MonkeyPatch,
- migrate: Callable[[SwitchConfig], Awaitable[None]],
- start: str | None,
-) -> None:
- if start is not None:
- engine = create_async_engine(main_url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to(start))
- finally:
- await engine.dispose()
-
- await migrate(_migration_config(main_url, monkeypatch))
-
- _, script = _script_directory()
- assert await _head(main_url) == ([script.get_current_head()], False)
diff --git a/core/tests/switch_core/test_migration_parity.py b/core/tests/switch_core/test_migration_parity.py
index efb1e6940..4967547fc 100644
--- a/core/tests/switch_core/test_migration_parity.py
+++ b/core/tests/switch_core/test_migration_parity.py
@@ -117,35 +117,3 @@ async def test_migrations_match_the_models(migrated_url: str) -> None:
"the migrations and the models disagree; autogenerate would emit:\n"
+ "\n".join(f" {entry}" for entry in drift)
)
-
-
-def _downgrade_github_cleanup(connection: Connection) -> None:
- config = Config(str(_CORE / "alembic.ini"))
- script = _script_directory(config)
-
- def do_downgrade(revision: str, context: Any) -> Any:
- return script._downgrade_revs("62cf05b128e3", revision)
-
- with EnvironmentContext(config, script, fn=do_downgrade) as environment:
- environment.configure(connection=connection, target_metadata=Base.metadata)
- with environment.begin_transaction():
- environment.run_migrations()
-
-
-async def test_github_cleanup_migrations_round_trip(migrated_url: str) -> None:
- engine = create_async_engine(migrated_url)
- try:
- async with engine.begin() as connection:
- await connection.run_sync(_upgrade_to_head)
- await connection.run_sync(_downgrade_github_cleanup)
- assert (
- await connection.scalar(
- text("SELECT to_regclass('github_issued_tokens')")
- )
- is None
- )
- await connection.run_sync(_upgrade_to_head)
- diff = await connection.run_sync(_diff)
- assert not [entry for entry in diff if _is_real_drift(entry)]
- finally:
- await engine.dispose()
diff --git a/core/tests/switch_core/test_migration_remove_hosted_worker.py b/core/tests/switch_core/test_migration_remove_hosted_worker.py
new file mode 100644
index 000000000..d695a2f0b
--- /dev/null
+++ b/core/tests/switch_core/test_migration_remove_hosted_worker.py
@@ -0,0 +1,175 @@
+"""`f5d2b8e6c3a1` removes the hosted worker.
+
+It refuses to run while a cloud machine that runs the hosted worker is not
+deleted, then drops the worker's tables and columns and the agent providers'
+credentials held for it, keeping GitHub connections.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import pytest
+from alembic.config import Config
+from alembic.runtime.environment import EnvironmentContext
+from alembic.script import ScriptDirectory
+from sqlalchemy import Connection, text
+from sqlalchemy.ext.asyncio import create_async_engine
+
+_CORE = Path(__file__).resolve().parents[2]
+_DB = "migration_remove_hosted_worker"
+
+_PRE_MIGRATION_REVISION = "e7a2c4b9d013"
+_MIGRATION_UNDER_TEST = "f5d2b8e6c3a1"
+
+_WORKER_TABLES = (
+ "hosted_launches",
+ "hosted_operations",
+ "hosted_wake_mailbox",
+ "hosted_cutover_items",
+ "hosted_cutover_volumes",
+ "github_issued_tokens",
+ "provider_verifications",
+)
+
+
+def _script_directory(config: Config) -> ScriptDirectory:
+ config.set_main_option("script_location", str(_CORE / "switch_core" / "migrations"))
+ return ScriptDirectory.from_config(config)
+
+
+def _upgrade(revision: str) -> Any:
+ def migrate(connection: Connection) -> None:
+ config = Config(str(_CORE / "alembic.ini"))
+ script = _script_directory(config)
+
+ def revisions(current_revision: str, context: Any) -> Any:
+ return script._upgrade_revs(revision, current_revision)
+
+ with EnvironmentContext(config, script, fn=revisions) as environment:
+ environment.configure(connection=connection)
+ with environment.begin_transaction():
+ environment.run_migrations()
+
+ return migrate
+
+
+async def database_url(postgres_url: str) -> Any:
+ admin = create_async_engine(postgres_url, isolation_level="AUTOCOMMIT")
+ async with admin.connect() as connection:
+ await connection.execute(text(f'DROP DATABASE IF EXISTS "{_DB}"'))
+ await connection.execute(text(f'CREATE DATABASE "{_DB}"'))
+ await admin.dispose()
+
+ base, _, _ = postgres_url.rpartition("/")
+ try:
+ yield f"{base}/{_DB}"
+ finally:
+ admin = create_async_engine(postgres_url, isolation_level="AUTOCOMMIT")
+ async with admin.connect() as connection:
+ await connection.execute(text(f'DROP DATABASE IF EXISTS "{_DB}"'))
+ await admin.dispose()
+
+
+async def test_refuses_a_live_worker_machine_then_removes_the_worker(
+ database_url: str,
+) -> None:
+ engine = create_async_engine(database_url)
+ try:
+ async with engine.begin() as connection:
+ await connection.run_sync(_upgrade(_PRE_MIGRATION_REVISION))
+
+ async with engine.begin() as connection:
+ # Foreign keys to tenants and users are beside the point here.
+ await connection.execute(
+ text("SET LOCAL session_replication_role = replica")
+ )
+ await connection.execute(
+ text(
+ """
+ INSERT INTO hosted_machines
+ (tenant_id, id, owner_id, slot_id, generation, state,
+ desired_state, runtime, active_at)
+ VALUES ('t', 'worker', 'u', 'slot-a', 1, 'ready', 'running',
+ 'worker', now()),
+ ('t', 'controller', 'v', 'slot-b', 1, 'ready',
+ 'running', 'controller', now())
+ """
+ )
+ )
+ await connection.execute(
+ text(
+ """
+ INSERT INTO provider_connections
+ (tenant_id, user_id, provider, kind, encrypted_credential,
+ verified_at)
+ VALUES ('t', 'u', 'claude', 'api-key', 'sealed', now()),
+ ('t', 'u', 'github', 'oauth', 'sealed', now())
+ """
+ )
+ )
+
+ with pytest.raises(RuntimeError) as raised:
+ async with engine.begin() as connection:
+ await connection.run_sync(_upgrade(_MIGRATION_UNDER_TEST))
+ assert str(raised.value) == (
+ "remove the hosted worker: 0 cloud agent(s) of the hosted worker are "
+ "not removed and 1 cloud machine(s) that run it are not deleted. "
+ "Remove those cloud agents in Switch Console, wait until their "
+ "machines are deleted, then upgrade again."
+ )
+
+ async with engine.begin() as connection:
+ await connection.execute(
+ text(
+ "UPDATE hosted_machines SET state = 'deleted', "
+ "desired_state = 'deleted' WHERE id = 'worker'"
+ )
+ )
+ await connection.run_sync(_upgrade(_MIGRATION_UNDER_TEST))
+
+ async with engine.connect() as connection:
+ tables = {
+ table: await connection.scalar(
+ text("SELECT to_regclass(:table)"), {"table": table}
+ )
+ for table in _WORKER_TABLES
+ }
+ columns = {
+ row[0]
+ for row in await connection.execute(
+ text(
+ "SELECT column_name FROM information_schema.columns "
+ "WHERE table_name = 'hosted_machines'"
+ )
+ )
+ }
+ machines = (
+ await connection.execute(
+ text("SELECT id, state FROM hosted_machines ORDER BY id")
+ )
+ ).all()
+ providers = (
+ await connection.execute(
+ text("SELECT provider FROM provider_connections")
+ )
+ ).all()
+ finally:
+ await engine.dispose()
+
+ assert tables == dict.fromkeys(_WORKER_TABLES)
+ assert (
+ not {
+ "runtime",
+ "machine_capability_hash",
+ "machine_capability_encrypted",
+ "machine_capability_revision",
+ "agents_version",
+ }
+ & columns
+ )
+ assert {"controller_id", "enrollment_code_encrypted"} <= columns
+ assert machines == [("controller", "ready"), ("worker", "deleted")]
+ assert providers == [("github",)]
diff --git a/deploy/hosted/Dockerfile.connections b/deploy/hosted/Dockerfile.connections
deleted file mode 100644
index a454d65f3..000000000
--- a/deploy/hosted/Dockerfile.connections
+++ /dev/null
@@ -1,16 +0,0 @@
-ARG SWITCH_CORE_IMAGE
-FROM ${SWITCH_CORE_IMAGE} AS claude
-RUN python - <<'PY'
-import hashlib
-import urllib.request
-url = "https://downloads.claude.ai/claude-code-releases/2.1.267/linux-x64/claude"
-with urllib.request.urlopen(url, timeout=120) as response:
- data = response.read()
-assert hashlib.sha256(data).hexdigest() == "0399c793ff571d5946ef923d80b4f330d05ac4b6842a6b0775468f5d389403c0"
-with open("/claude", "wb") as output:
- output.write(data)
-PY
-RUN chmod 755 /claude
-FROM ${SWITCH_CORE_IMAGE}
-COPY --from=claude /claude /usr/local/bin/claude
-ENV HOSTED_CLAUDE_VERIFIER_PATH=/usr/local/bin/claude
diff --git a/deploy/hosted/README.md b/deploy/hosted/README.md
index 3f7bb1e1d..b64fabae9 100644
--- a/deploy/hosted/README.md
+++ b/deploy/hosted/README.md
@@ -1,463 +1,183 @@
-# Hosted EC2 workers
-
-This directory implements the bounded cloud-worker pilot: a controller service,
-a trusted VM launcher, a Kubernetes chart and generic Terraform. Core keeps one
-machine record per user. The controller syncs these machines from the
-authenticated gateway and keeps its own durable AWS database. Operator commands
-remain available for lifecycle management.
-
-Start from [the architecture proposal](../../docs/planning/hosted-execution-backend-proposal.md).
-One ordinary EC2 VM (a machine) runs all cloud agents of one user, with a
-retained encrypted EBS data disk. Each machine uses one machine slot from the
-operator-configured pool. The watcher starts a separate session for each addressed
-room on that VM.
-Creating an agent does not create a room. The controller runs on existing EKS;
-workers never join that cluster.
-
-## Components
-
-- `controller/`: Python CLI/service using boto3, SQLite durable desired/observed
- state and an exclusive reconciliation lock. Create/start/stop/delete requests
- are local operator actions, not an unauthenticated web API.
-- `worker/`: root-owned AMI launcher. It validates the exact attached EBS volume,
- retrieves one scoped Secrets Manager document, prepares tmpfs credential files
- and starts the runtime as an unprivileged account. Retained boot identity prevents
- old process IDs from being treated as ownership evidence after a reboot.
-- `terraform/`: a separate worker VPC, private worker subnet, NAT egress, no inbound
- worker access, restricted worker roles, and an IRSA role for the controller.
-- `chart/`: a digest-pinned controller image, one replica with Recreate rollout,
- private durable database volume and no exposed application port.
-
-Stop/start preserves the disk and saved sessions. Automatic replacement requires
-confirmed termination of the old VM, a detached disk, and matching assignment
-identity. The new worker accepts only that exact predecessor. Recovery attempts
-are bounded. Cross-AZ migration and arbitrary disk adoption remain disabled.
-See [worker image upgrades](worker/README.md) for the explicit operator workflow.
-
-## GitHub App credential preparation
-
-The backend helper `switch_core.providers.github_installation` can issue a
-repository-scoped installation token after checking that the initiating user's
-GitHub account still has access to that repository in the selected installation.
-It signs with an operator-supplied RSA key and requests only contents and pull
-request write access. Credentials must stay on the backend and in the worker's
-private credential transport, never in Console responses or persisted launch specs.
-
-Console uses the shared New Agent form, a saved provider connection and a selected
-GitHub repository. A user's first cloud agent reserves a machine slot from the
-operator-configured pool. The controller creates the data volume, writes the
-assignment secret with that volume ID, and then starts the VM. Later agents of the
-same user run on that machine. An agent is ready only after its watcher connects. Agents can start sessions automatically when addressed or use
-manual sessions. Both paths use the existing session form and conversation view.
-
-Cloud sessions appear in the agent sidebar and under their connected rooms.
-The conversation supports messages, permission requests, interruption, stop,
-resume and restart. The machine card provides start, stop and retry for the machine. Each
-agent provides its own start, stop, restart, retry and removal actions.
-See [machine lifecycle](#machine-lifecycle) for what removal does to the machine
-and its disk. Uncertain operations are reported explicitly and are not
-automatically repeated.
-
-Each user has their own VM, encrypted disk and scoped machine credentials. All
-agents of that user share them. The server launch capacity and the controller's
-machine slot pool impose separate global limits. See
-[machine lifecycle](#machine-lifecycle) for the settings.
-
-Managed workers request a fresh installation token before checkout and each Git
-or GitHub CLI command. The agent-authenticated renewal route checks the saved
-assignment, workspace membership and current GitHub repository access. It never
-accepts a caller-selected repository. Personal tokens remain an operator option.
-
-### Enable Console launches
-
-Mount a private backend JSON file through `HOSTED_CONTROLLER_CONFIG_PATH` with
-`tenant_id`, a dedicated `token` of at least 32 characters, `machine_slots`,
-`github_private_key_path` and the HTTPS `agent_api_endpoint`. `machine_slots` is
-a list of 1–100 unique slot IDs. The IDs must match the keys of the Terraform
-`machine_slots` variable. Core rejects a file with the old `agent_ids` key at
-startup. Set `HOSTED_LAUNCH_CAPACITY` no higher than the number of machine slots.
-Zero disables creation. The backend chart exposes
-`switchCore.hostedControllerSecret` (files `controller.json` and the referenced
-key), `switchCore.hostedLaunchCapacity`, `switchCore.hostedIdleStopMinutes` and
+# Switch cloud machines on EC2
+
+A Switch cloud machine is an EC2 VM, one per user, that runs the standard
+`switch-agent-controller`: the program a laptop or an SSH host runs agents
+with. The user's managed agents placed on it run there, each as a Linux user
+of its own. Each machine has a retained, encrypted EBS data disk.
+
+This directory holds what runs the machines:
+
+- `controller/`: the hosted controller, a Python service on the existing EKS
+ cluster that creates, starts, stops and deletes the VMs and their disks. It
+ keeps desired and observed state in SQLite, behind an exclusive
+ reconciliation lock. Operator commands queue desired state for it; it exposes
+ no web API.
+- `machine/`: the machine image and its boot. See [its README](machine/README.md)
+ for how a machine enrolls its controller and starts.
+- `terraform/`: a separate VPC, a private subnet with NAT egress and no inbound
+ access, the machines' one instance role, which holds no permissions, and an
+ IRSA role for the hosted controller.
+- `chart/`: the hosted controller's chart: a digest-pinned image, one replica
+ with a Recreate rollout, a private retained volume for its database, and no
+ exposed port.
+
+Machines never join the cluster. They reach Switch over its public,
+authenticated HTTPS API, as any agents controller does.
+
+## How a user gets a machine
+
+Picking "Switch cloud" in Switch Console claims the user's machine
+(`POST /gateway/hosted-machines/ensure`) and creates a managed agent on its
+controller once the machine is ready. Self sign-up claims one too, so it warms
+while the user signs in. The hosted controller polls Core for the machines that
+should exist, creates the data volume, and launches the VM with a bundle that
+carries a one-time enrollment code as its user data. The machine enrolls its
+controller with that code, and its status reports are its heartbeat from then
+on. An instance's user data is set when it launches: when the machine must
+enroll again, the hosted controller terminates the stopped instance and
+launches a new one with the new code, on the same data volume.
+
+Provider logins are given to the machine on demand from Switch Console, sealed
+to its controller's own key: Core relays only ciphertext.
+
+## Enable cloud machines
+
+Core needs agent management (`AGENT_MANAGEMENT_ENABLED`, with
+`CONTROLLER_TOKEN_SECRET`): it refuses to start with `HOSTED_LAUNCH_CAPACITY`
+above 0 without it.
+
+Mount a private JSON file through `HOSTED_CONTROLLER_CONFIG_PATH` with
+`tenant_id`, a dedicated `token` of at least 32 characters and the HTTPS
+`agent_api_endpoint` the machines enroll against. Core refuses unknown keys.
+`HOSTED_LAUNCH_CAPACITY` is how many machines may exist at once, 0–100; 0
+disables cloud machines. Keep it no higher than the hosted controller's
+`max_machines`. The backend chart exposes
+`switchCore.hostedControllerSecret` (file `controller.json`),
+`switchCore.hostedLaunchCapacity`, `switchCore.hostedIdleStopMinutes` and
`switchCore.hostedDiskRetentionDays`.
-Mount a controller secret with `gateway.json` containing `origin`, the matching
-`token` and `instance_type`. The controller uses `instance_type` for every new
-machine. It must be one of `allowed_instance_types` in both the Terraform and the
-controller configuration. The recommended value, and the Terraform default, is
-`c7i.2xlarge`. Set the controller chart's `gatewaySecretName` to that secret. The
-token authorizes only the hosted controller routes for its configured tenant. It
-is not a user or agent API key. The machine slot IDs must match in the Terraform,
-controller and Core configurations. Keep all values and private keys outside this
-public repository.
-
-## Prepare a deployable environment
-
-1. Select a dedicated test account/environment and read-only inventory its EKS
- version, OIDC issuer, StorageClass, IAM boundaries, quotas and regional capacity.
- Do not copy internal resource identifiers or secret values into this public tree.
-2. Build a pinned Linux x86 AMI using the worker installer and its documented
- prerequisites. The runtime artifact must include this branch's reviewed
- bootstrap and ownership identity support; an older released runtime is insufficient.
- After installing the console workspace dependencies, build self-contained files:
- `node deploy/hosted/build-runtime.mjs /path/to/runtime-output`. Copy the three
- `.mjs` files and SHA256 manifest to the worker installer's documented location;
- pin and verify those hashes as part of the AMI build.
- Validate the AMI has the configured root-device name, exactly one EBS root mapping,
- and an approved source-snapshot encryption key. This slice creates disks with
- `alias/aws/ebs`; customer-managed EBS keys need additional reviewed permissions.
- No provider credential is baked in.
-3. Create one assignment secret per machine slot outside Terraform. Keep the
- secret ARN stable. Do not put a value in it. The controller writes the bundle
- itself:
- `{"version":2,"machineId","assignment":{…},"machineCapability","apiEndpoint"}`.
- The bundle holds the machine and slot identity, the data volume ID, an opaque
- machine credential and the Switch API endpoint, and nothing else. The worker
- refuses to start until the bundle matches its attached data volume.
-4. Configure Terraform in the **private deployment overlay**, selecting the worker
- AMI/AZ, CIDRs, allowed instance types, controller namespace/service account and
- the per-slot secret/KMS references in `machine_slots`. Review its plan before
- applying. The module does not create secret values, and each machine can read
- only its own slot's secret.
-5. Build the controller image and record its immutable digest. Put Terraform's
- worker subnet/security-group/AZ/role outputs into the controller configuration;
- root/data disk sizes must match the Terraform policy bounds. Deploy its chart
- with the controller IRSA role and a CSI-backed persistent StorageClass.
-6. Test authorization with EC2 DryRun where available and exercise the bounded
- lifecycle experiment before onboarding real users. Terraform validation and
- boto3 stub tests are not evidence of successful AWS authorization.
-
-The module deliberately provisions a new NAT gateway and public IPv4 allocation.
-Their charges continue when agents are stopped; they must be included in the
-reviewed plan and removed with the test network when no longer needed. This is
-not a reuse of a developer VM's network or IAM role.
-
-Workers require a publicly reachable, authenticated HTTPS Switch API, GitHub HTTPS,
-model-provider APIs and package registries. They cannot reach a Kubernetes-only
-Service or private control-plane database. There is no peering, SSH ingress,
-Git-over-SSH egress, host Docker socket or blanket sudo for repository scripts.
-The dedicated network blocks RFC1918 outbound traffic, but VPC NACLs do not block
-IMDS or the AWS DNS resolver. Treat access to the machine's instance role as
-possible for code of every agent on that machine, and grant it no infrastructure
-authority.
-DNS/private-address access and any organization-specific address ranges still
-require live isolation tests. Workloads requiring other ports need a reviewed
-supported-environment change.
+Mount a secret with `gateway.json` for the hosted controller, holding `origin`,
+the matching `token` and `instance_type`, and set the chart's
+`gatewaySecretName` to it. `instance_type` is used for every new machine and
+must be one of `allowed_instance_types` in both Terraform and the controller
+configuration; `c7i.2xlarge` is recommended. The token authorizes only the
+hosted controller routes for its tenant. It is not a user or agent API key.
+
+Only the configured tenant may claim machines. Keep every value and key out of
+this public repository.
+
+## Prepare an environment
+
+1. Select a dedicated account and inventory its EKS version, OIDC issuer,
+ StorageClass, IAM boundaries, quotas and regional capacity.
+2. Bake the machine image as [machine/README.md](machine/README.md) describes.
+ It must have the configured root device name and exactly one EBS root
+ mapping. Disks are created with `alias/aws/ebs`.
+3. Configure Terraform in the private deployment overlay: the image, AZ,
+ CIDRs, allowed instance types, and the controller's namespace and service
+ account. Review the plan before applying.
+4. Build the hosted controller image and record its digest. Put Terraform's
+ subnet, security group, AZ and `machine_instance_profile_arn` outputs into
+ the controller configuration, and deploy the chart with its IRSA role and a
+ CSI-backed StorageClass.
+
+The network has its own NAT gateway and public IPv4 address, which cost money
+while machines are stopped too. Machines need the Switch API, the model
+providers' APIs and the package registries. There is no peering, no SSH
+ingress and no Docker socket. VPC NACLs block RFC1918 egress but not IMDS or the
+AWS resolver: agents cannot reach instance metadata (their units deny it), but
+grant the instance role no authority anyway. The boot reads the machine's
+bundle from instance metadata as root. Anyone in the AWS account who can read
+an instance's user data can read its enrollment code, which is spent at the
+first boot and expires after 30 minutes.
## Controller configuration
-The chart accepts a non-secret `controllerConfig` map with:
+The chart takes a non-secret `controllerConfig` map:
- `installation_id`, `region`, `availability_zone`, `subnet_id`, `security_group_ids`
- `image_id`, `root_device_name`, `allowed_instance_types`
-- `max_machines`: 1–100, and no more than the number of machine slots. It caps
- both the machines that are not deleted and the active EC2 instances.
+- `max_machines`: 1–100. It caps both the machines not deleted and the
+ running instances.
- `root_volume_gib`, `data_volume_gib`, `poll_interval_seconds`
-- `machine_slots`: slot ID to `{instance_profile_arn, assignment_secret_arn}`.
- Copy it from the Terraform output `machine_slots`.
-
-The instance type of a new machine comes from `instance_type` in the controller's
-`gateway.json` (see [Enable Console launches](#enable-console-launches)). It must
-be one of `allowed_instance_types`. The recommended value is `c7i.2xlarge`.
-
-The chart fixes `state_db_path` and `lock_path` on the same retained PVC. A standalone
-operator installation must supply absolute paths for both. Keep the database and
-assignment configuration private and backed up. Do not run two installations
-against separate databases with the same cloud installation ID. Do not manually
-scale the Deployment or bypass its reconciliation lock. Startup/readiness/liveness
-probes check a local progress timestamp, not cloud or provider readiness; handled
-AWS failures remain visible in machine status without causing restart loops.
-
-Before a standalone controller upgrade, stop the old reconciler and back up its
-SQLite database. The Helm chart uses Recreate so the old pod stops before the new
-one starts. The observation-schema upgrade preserves assignment and cloud resource
-identities, but old observations must be refreshed before they can authorize deletion.
-Do not run an older controller binary against the upgraded database. To upgrade
-from a per-agent controller, follow
-[Moving to one machine per user](#moving-to-one-machine-per-user).
-
-The controller is trusted to provision all configured machine slots. Its IAM role
-can pass the configured worker roles; IAM is not a substitute for the controller's
-slot-to-role binding checks. Audit secret resource policies and KMS key policies
-as well as the supplied identity policies; an external broad resource policy can
-invalidate the intended cross-machine denial.
+- `instance_profile_arn`: the machines' instance profile, the Terraform
+ `machine_instance_profile_arn` output. An instance that runs with another
+ profile needs attention, so do not change it while machines exist.
+
+The chart fixes `state_db_path` and `lock_path` on its retained volume; a
+standalone installation must give absolute paths for both. Back up the database.
+Never run two installations with the same installation ID, scale the
+Deployment, or bypass the lock. The probes check a local progress timestamp, not
+cloud or provider readiness.
+
+The hosted controller may change only the data retention of the instances it
+tagged as its own, never their user data or security groups.
## Operator commands
-Install the controller with its locked dependencies, then supply an absolute path
-to the private controller JSON. In Kubernetes, run state commands inside the
-controller pod using the same config and database; they queue desired state for
-the resident reconciler. Each command takes a machine slot ID.
+Run them in the controller pod, with the same configuration and database. They
+queue desired state for the running reconciler.
```sh
-switch-hosted-controller --config /etc/switch-hosted/controller.json create example-slot --instance-type c7i.2xlarge
-switch-hosted-controller --config /etc/switch-hosted/controller.json status example-slot
-switch-hosted-controller --config /etc/switch-hosted/controller.json stop example-slot
-switch-hosted-controller --config /etc/switch-hosted/controller.json start example-slot
-switch-hosted-controller --config /etc/switch-hosted/controller.json delete example-slot --confirm-slot-id example-slot --retain-volume
switch-hosted-controller --config /etc/switch-hosted/controller.json list
+switch-hosted-controller --config /etc/switch-hosted/controller.json status <machine-id>
+switch-hosted-controller --config /etc/switch-hosted/controller.json stop <machine-id>
+switch-hosted-controller --config /etc/switch-hosted/controller.json start <machine-id>
+switch-hosted-controller --config /etc/switch-hosted/controller.json delete <machine-id> --confirm-machine-id <machine-id> --retain-volume
+switch-hosted-controller --config /etc/switch-hosted/controller.json upgrade <machine-id> --confirm-instance-id i-0123456789abcdef0
```
-For standalone reconciliation:
+`upgrade` moves a stopped machine whose instance is terminated onto the
+configured image, keeping its disk. A queued command is not confirmation that
+AWS has done it. Deletion needs `--confirm-machine-id` and exactly one of
+`--retain-volume` or `--delete-volume`. Deleting a machine does not remove
+snapshots, roles, the NAT gateway or the controller's database.
-```sh
-switch-hosted-controller --config /etc/switch-hosted/controller.json reconcile-once
-switch-hosted-controller --config /etc/switch-hosted/controller.json serve
-```
+## Machine lifecycle
-A queued command is not confirmation that AWS has completed it. Desired-state
-changes invalidate prior observations; stopped evidence must belong to the current
-operation before deletion is accepted. A delayed response from an older operation
-cannot certify a newer one. After deletion is accepted, a late-started instance is
-stopped and terminated before disk cleanup proceeds. `Running` describes
-the infrastructure, not provider authentication/readiness. Inspect worker service
-status and Switch session state before calling the coding agent ready. Stop/start
-retains disk contents and native state; it does not promise to resume an interrupted
-provider action or automatically replay a room message.
-
-Deletion requires `--confirm-slot-id` with the same slot ID and exactly one of
-`--retain-volume` or `--delete-volume`; consult `delete --help` and the
-controller's documented stop precondition.
-Retaining a disk retains its charges and sensitive state. Deleting compute/disk
-through this controller does not remove Secrets Manager documents, snapshots, IAM
-roles, NAT gateways or the controller's own database. Their lifecycle remains the
-private deployment workflow's responsibility.
-
-See [the implementation verification record](VERIFICATION.md) for local results and
-remaining acceptance gates.
-
-## Moving to one machine per user
-
-Earlier versions ran one EC2 VM per cloud agent. This version runs one VM per
-user. There is no in-place migration: you remove the old cloud agents, upgrade,
-and create the agents again. Do these steps in order.
-
-1. Before you upgrade, record the settings of each cloud agent: name, provider,
- instructions, repository, and the auto-session and approval settings. You need
- them to create the agent again in step 9.
-2. In Switch Console on the old version, stop each cloud agent, then remove it.
- Run `list` on the old controller. Wait until every row shows desired state
- `deleted` and observed state `deleted`.
-3. Removal kept each old data disk. These disks use the one-agent layout, and the
- new worker refuses them. Snapshot each disk that holds data you need, then
- delete the disks. You do not have to wait before you create a new cloud
- agent: machine EC2 idempotency tokens use a different prefix, so they cannot
- match the tokens of the old resources.
-4. Stop the controller. Back up its SQLite database.
-5. In the Terraform overlay, rename the variable `assignments` to `machine_slots`.
- Keep the keys, so no role, instance profile or secret is replaced. The output
- `worker_assignments` is now `machine_slots`. Run a plan and review it: only
- policy conditions, IAM tags and defaults change. The `data_volume_gib` default
- is now 200. It must equal the controller's `data_volume_gib`. If the overlay
- does not set it, set it to the controller's current value. Apply.
- The controller binds its database to `data_volume_gib`,
- `allowed_instance_types` and the other disk and network settings, and refuses
- to start when one of them changes. To change them, for example to use 200 GiB
- disks or to allow `c7i.2xlarge`, start the new controller on a new, empty
- database. This is safe here because every old row is deleted.
-6. Update the configuration files:
- - `controller.json`: replace `worker_assignments` with `machine_slots` (copy
- it from the Terraform output `machine_slots`), and `max_agents` with
- `max_machines`.
- - Core controller settings file: replace `agent_ids` with `machine_slots`,
- the list of slot IDs.
- - `gateway.json`: set `instance_type`, for example `c7i.2xlarge`.
- - Set `HOSTED_DISK_RETENTION_DAYS` and `HOSTED_IDLE_STOP_MINUTES` if you do
- not want the defaults.
-7. Build and roll out the new worker AMI. Set `image_id` in `controller.json` and
- `worker_image_id` in the Terraform overlay to the new AMI, and apply Terraform.
-8. Upgrade Core. Its database migration refuses to run while any cloud agent is
- not removed, and names this section. Then upgrade the controller. It refuses
- to start while its database holds per-agent rows that are not deleted, and
- also names this section. The controller keeps the old `agents` table; it does
- not drop it. The controller creates its `machines` table on first start. A
- database from a pre-release build of this controller is not supported; in
- that case, start the controller on a new database.
-9. Create each agent again in Switch Console with the settings from step 1. The
- first agent of a user creates that user's machine. Later agents of the same
- user share it.
-
-## Verification and rollout boundary
-
-Local checks cover state transitions/retries, cloud request shapes, ownership,
-credential transport, retained boot state and infrastructure rendering. All live
-AWS/AMI, block-device, IAM/KMS, network, provider-authentication and end-to-end
-GitHub gates remain mandatory before deployment acceptance. A generated chart or
-an EC2 instance in `running` state alone proves none of those gates.
-
-The controller PVC is annotated `keep` so Helm uninstall retains it. That is not a
-backup and does not override the underlying StorageClass/PV reclaim policy. Back
-up the assignment database and retain resource identities before uninstalling; a
-fresh empty database must not be used to guess ownership of existing workers.
-
-### GitHub authentication slice
-
-The optional worker secret fields described in [the worker contract](worker/README.md#optional-github-credential-delivery)
-provide a personal GitHub.com token to Git HTTPS and GitHub CLI without storing
-it in a workspace/config or passing it as a command argument. Bootstrap checks
-the token before starting the provider. Repository permission checks and actual
-clone/build/push/PR operations remain the coding task's responsibility; managed onboarding uses the renewable installation-token flow described above.
-
-## User-owned Claude connections
-
-The gateway exposes authenticated `GET`, `PUT`, and `DELETE` routes at
-`/gateway/provider-connections/claude`. The PUT body has `kind` (`api-key` or
-`setup-token`) and `credential`. GET returns connection status, kind and the last
-successful verification time, never the credential. Operations are scoped to the
-signed-in user's tenant and user ID; administrator status does not grant access
-to another user's connection. Concurrent changes to the same connection return
-409 so deletion and verification cannot race.
-
-The verifier is opt-in. Build the ordinary switch-core image from this checkout,
-then build the Linux amd64 hosted variant:
-
-```sh
-docker build --platform linux/amd64 -f deploy/hosted/Dockerfile.connections \
- --build-arg SWITCH_CORE_IMAGE=your-built-core-image \
- -t switch-core-with-claude .
-```
+When the last managed agent leaves a user's machine, its instance is
+terminated and its disk kept for `HOSTED_DISK_RETENTION_DAYS`. A new agent in
+that time starts the machine again on the kept disk; after it, the disk is
+deleted and the machine with it.
-Pin the base image by digest for deployment. The variant includes a checksum-pinned
-Claude Code executable and sets `HOSTED_CLAUDE_VERIFIER_PATH`. A non-container
-installation can set that variable to an absolute executable path. An invalid
-configured path fails startup; an unset path leaves connections unavailable.
-Run the normal database migration before rolling out the backend.
-
-Each check uses a temporary private home and a minimal environment with only the
-chosen credential. It runs one fixed Haiku request with tools, MCP servers, skills
-and session persistence disabled. It accepts only a successful Claude result.
-Checks time out after 25 seconds, process groups are killed on exit/cancellation,
-and temporary files are removed. There are at most two checks per gateway process.
-The verification process runs as the service user and accepts no user prompts,
-commands, repository paths or tool configuration; it is not a worker sandbox.
-
-Only verified credentials are written, using the existing server encryption key
-and the tenant-scoped `provider_connections` table. Failed replacement leaves the
-previous connection intact. Back up and rotate the server encryption key with the
-same care as other encrypted credentials. Removal deletes the database record;
-revocation at Anthropic and database-backup retention are separate concerns.
-
-Worker assignment bundles contain no provider credential. Managed runtimes fetch
-current credentials from the authenticated worker
-API. Revocation denies further credential and control requests and stops the
-affected workers. Reconnect the provider, then use Retry to start them again.
-
-Codex, Cursor, OpenCode and Antigravity use the same owner-scoped connection API
-under their provider IDs. Enable `HOSTED_PROVIDER_VERIFICATION_ENABLED` (Helm:
-`switchCore.hostedProviderVerificationEnabled`) after deploying the verification
-API, controller IAM policy, and a worker image with `--verify-credential` support.
-This requires a hosted controller. With the setting off, credentials keep the
-existing configured-until-worker-check behavior.
-
-With verification enabled, saving a credential queues a durable connection check.
-Console polls its status and shows **Checking connection**. A temporary worker
-uses the native provider adapter to send one fixed model request in an empty
-workspace. It has no repository, agent assignment, instance profile, or retained
-data volume. Its encrypted root volume is deleted on termination. The controller
-allows at most two checks at a time. Each worker schedules its own shutdown after
-eight minutes; the controller also terminates checks past their ten-minute deadline.
-Checks and cleanup continue when Console closes.
-
-The connection becomes verified only after the model replies and the controller
-observes instance termination. Refreshed subscription credentials are saved with
-the result. Failed checks preserve an existing verified credential and show a retry
-action. Job credentials and bootstrap tokens are cleared when the job finishes.
-Test each provider with its intended account before deployment acceptance.
-
-### GitHub App connections
-
-To enable browser authorization, set `HOSTED_GITHUB_CONFIG_PATH` to a private JSON
-file containing `client_id`, `client_secret`, `slug`, and the public HTTPS Switch
-`origin`. In the backend Helm chart, `switchCore.githubConnectionsSecret` mounts
-an existing Secret's `github.json` key. Never put the client secret in image layers
-or Console build configuration.
-
-Register `<origin>/gateway/provider-connections/github/callback` as the GitHub App
-callback. Enable expiring user tokens. Contents and pull requests need read/write
-permissions for the planned coding workflow; metadata read access is mandatory.
-The Console requests user authorization, confirms the account, and offers GitHub
-App installation to select repositories. Organization approval may be required.
-
-Stored credentials are encrypted and scoped to the current user and tenant.
-The backend refreshes expiring user tokens and asks GitHub for current repository
-access. Disconnect deletes local connection storage, not the installation on
-GitHub. Authorization attempts expire after ten minutes and on backend restart.
-Workers receive repository-scoped installation tokens. GitHub uninstall and suspend
-webhooks are not implemented; access is checked again when tokens are issued.
-
-### Machine lifecycle
-
-Removing a cloud agent removes it at once, in any state. Removal revokes its
-Switch API key, removes its agent and room memberships, and frees the agent name.
-Server-side sessions are removed with the agent.
-
-When the last agent on a user's machine is removed, the instance is terminated.
-The data disk is kept for `HOSTED_DISK_RETENTION_DAYS` days. If the same user
-creates a new agent within that time, the machine starts again on the kept disk.
-After that time, the disk is deleted and the slot returns to the pool. A reused
-slot gets a new generation. Do not attach a kept disk to another user's machine.
-A kept disk keeps its charges and sensitive state.
-
-A machine stops when its agents are idle for `HOSTED_IDLE_STOP_MINUTES`. It starts
-again when an agent is started or addressed, or when the user creates an agent.
+An idle machine is put to sleep after `HOSTED_IDLE_STOP_MINUTES`: its
+controller reports no running session and nothing kept it active. A message
+addressed to one of its agents wakes it, and the room is told so. A machine its
+owner stopped stays stopped.
| Setting | Default | Meaning |
|---|---|---|
-| `HOSTED_AGENTS_PER_OWNER` | 3 | Cloud agents per user. They share the user's machine. |
-| `HOSTED_SESSIONS_PER_AGENT` | 8 | Sessions per agent. |
-| `HOSTED_IDLE_STOP_MINUTES` | 30 | Idle minutes before a machine stops. 0 disables. Maximum 1440. |
-| `HOSTED_DISK_RETENTION_DAYS` | 7 | Days a data disk is kept after its last agent is removed. 1–90. |
-| `HOSTED_LAUNCH_CAPACITY` | 0 | Maximum live machines. No more than the number of machine slots. 0 disables creation. |
-
-In the backend Helm chart, `switchCore.hostedIdleStopMinutes`,
-`switchCore.hostedDiskRetentionDays` and `switchCore.hostedLaunchCapacity` set
-these values when `switchCore.hostedControllerSecret` is set.
-
-A machine that does not start, or does not connect to Switch, within 10 minutes
-goes to error. An agent goes to error when its watcher does not connect within 10
-minutes after it starts, or when it does not stop within 10 minutes. Use Retry in
-Switch Console.
-
-### GitHub sign-in availability
-
-Run the Switch API (`switchCore`) with one replica. The shipped Helm chart pins
-`switchCore.replicaCount` to `1` and rejects other values. GitHub sign-in flows
-live in that process alongside the live agent state. A restart interrupts an
-unconfirmed sign-in; start it again from Switch Console. Multiple API replicas
-require shared flow storage before they can be supported.
-
-### GitHub credential revocation
-
-Repository tokens are encrypted and bound to the workspace, owner, worker, and
-worker revision. Stop, removal, disconnect, relink, and owner removal queue their
-revocation after the access change commits. A bounded batch retries failed
-revocations on each controller poll; expired records are deleted. Disconnect and
-relink revoke only the old OAuth token, not the user's entire GitHub App grant.
-A failed revoke shows a warning and does not undo the access change.
-
-Tokens minted before token tracking was deployed, or minted just before a lost
-response or request cancellation, can remain valid until expiry (up to one hour for repository
-tokens). Stored tokens use the same encryption key as other provider credentials;
-separate encryption keys and key rotation remain a follow-up.
-
-Keep query strings out of access logs on every proxy in front of the GitHub
-callback. Its URL carries a single-use OAuth code in the query on
-`/gateway/provider-connections/github/callback`. Exclude this path if you later
-enable ALB access logs or WAF request logging.
-The API strips query strings from access logs, and the shipped nginx gateway
-disables access logging for that path. An external ingress, load balancer, WAF,
-or CDN needs the same protection before GitHub sign-in is enabled.
-
-Worker cards show general provider and GitHub recovery guidance for setup failures.
-They do not yet distinguish each GitHub access failure, such as missing write
-permission or a missing App installation.
-
-The controller uses an attached managed policy for assignment secret access.
-The deploy identity needs IAM policy create, version, attach, detach, and delete
-permissions for that policy. The pool is limited by AWS policy size: 6,144
-characters for assignment access and 10,240 for inline controller permissions.
-Terraform checks these limits during planning. The supported number of machine
-slots depends on ARN lengths; use a smaller pool if a size check fails.
-
-The loopback callback URL can remain in browser history. Its authorization code
-is single-use, consumed during sign-in, and bound to the flow's PKCE verifier.
+| `HOSTED_LAUNCH_CAPACITY` | 0 | Machines at once, at most 100. 0 disables. |
+| `HOSTED_IDLE_STOP_MINUTES` | 30 | Idle minutes before a machine sleeps. 0 disables. At most 1440. |
+| `HOSTED_DISK_RETENTION_DAYS` | 7 | Days a disk is kept after its last agent leaves. 1–90. |
+
+A machine that does not start, or whose controller does not report, within 10
+minutes goes to error. Use Retry in Switch Console.
+
+## Moving off machine slots
+
+Machines used to borrow one of a fixed pool of slots, each an IAM role and an
+assignment secret the machine read its bundle from at boot. A machine placed on
+a slot cannot move off it: its image reads the slot's secret, which is gone.
+Before upgrading an installation that has cloud machines:
+
+1. Remove every cloud agent in Switch Console and wait until Core shows no
+ cloud machine, or remove the machines in Core directly.
+2. Terminate their instances and delete their data disks.
+3. Delete the hosted controller's state database, or let it start: it forgets
+ a slot-era database whose machines are all deleted, and refuses one that
+ still has a live machine.
+4. Apply Terraform, which removes the slots' roles, and delete the slots'
+ assignment secrets and their KMS key yourself.
+5. Bake an image from this `machine/` and set it as `image_id`.
+
+## GitHub connections
+
+GitHub connections are not part of cloud machines. To let users link their
+GitHub account, set `HOSTED_GITHUB_CONFIG_PATH` to a private JSON file with
+`client_id`, `client_secret`, `slug` and the public HTTPS Switch `origin`
+(chart: `switchCore.githubConnectionsSecret`, key `github.json`), and register
+`<origin>/gateway/provider-connections/github/callback` as the GitHub App
+callback with expiring user tokens. Sign-in flows live in the Switch API
+process, so run it with one replica. Keep query strings out of every access
+log in front of the callback: its URL carries a single-use OAuth code.
diff --git a/deploy/hosted/VERIFICATION.md b/deploy/hosted/VERIFICATION.md
deleted file mode 100644
index 3a1c99e66..000000000
--- a/deploy/hosted/VERIFICATION.md
+++ /dev/null
@@ -1,94 +0,0 @@
-# Hosted integration verification
-
-## Console and worker integration
-
-The current implementation adds cloud sessions to the existing Console, manual
-session control, worker lifecycle actions, provider credential refresh, quotas
-and automatic replacement with a retained disk.
-
-Verification includes 361 backend tests, 367 provider tests, 36 controller tests,
-25 worker tests, and 55 focused Console tests. Backend and desktop type checks
-pass. All five provider executables pass startup probes in a clean Linux image.
-
-Live Claude checks passed for manual creation, initial prompts, approval and
-denial, interruption, stop/resume, session restart and worker restart. A worker
-image upgrade retained conversation history and native context. Automatic VM
-replacement retained the same disk and a command marker with exactly one write.
-The interrupted command was not repeated. Two workers ran independently, and a
-cross-worker operation update was rejected. Credential revocation denied worker
-access and stopped compute; reconnect plus explicit retry restored the sessions.
-Worker removal terminated compute and retained its encrypted disk.
-
-The live checks found and corrected termination-state validation, saved-path
-compatibility and an expensive controller health probe. The deployed lightweight
-probe completed in under one second under the configured CPU limit.
-
-Authenticated model requests for Codex, Cursor, OpenCode and Antigravity remain
-an acceptance gate. Clean-image startup checks do not replace those account tests.
-Quota concurrency and tenant isolation have database-backed coverage; live scale
-verification used two workers. This is not a large-scale load test.
-
-## Historical infrastructure checkpoint
-
-Branch: `codex/hosted-ec2-workers`, based on committed SDK revision `522da979`.
-The reviewed headless runtime foundation is included. Uncommitted SDK workspace
-changes are not part of this branch.
-
-## Implemented boundary
-
-Operator CLI and durable controller; EC2 create/start/stop/delete; encrypted,
-retained EBS; per-assignment roles and secret references; root launcher and
-unprivileged runtime; retained machine/boot identity; generic Terraform and Helm;
-standalone runtime artifact build and AMI installer contract.
-
-This is not full hosted onboarding. Switch service authorization/assignment APIs,
-Console creation, GitHub credential/setup flow, agent readiness reporting,
-additional provider enablement and safe cross-instance recovery remain follow-up
-work. One ordinary VM runs one configured Claude session. A stopped instance can
-restart on its original disk; a different instance cannot adopt that disk through
-this implementation.
-
-## Local verification
-
-- Controller: locked dependencies, 24 unit/cloud-request tests passed, Ruff, Docker build,
- nonroot identity, CLI and health-probe smoke checks.
-- Worker: 16 Python unit tests passed with mocked OS/cloud effects; installer shell syntax passed.
- Tests cover credential validation/cleanup, disk inspection, boot identity and
- interrupted ownership quarantine. They do not exercise real mounts or systemd.
-- TypeScript: 34 focused tests passed across hosted bootstrap, identity,
- ownership, supervisor and watcher; typecheck, package build, lint and format passed.
-- Standalone bootstrap/daemon bundles built; both entrypoints loaded successfully
- and returned their expected missing-argument errors.
-- Terraform formatting/validation and two mocked tests passed. Mocked `apply`
- executes no real cloud operations. Helm lint and template rendering passed.
-- Broader recovery tests: 10 failed because the sandbox denies `ps` process
- inspection (`EPERM`), consistent with the previously observed environment limit.
- Focused desktop tests: 13 passed, one failed on the same restriction.
- Three SSH integration cases skipped because test credentials were unavailable.
-
-The initial Sol review was followed by a fresh Astra medium adversarial review,
-which found two P1 lifecycle defects: terminal responses were validated as live
-instances, and stale stopped observations could admit deletion during startup.
-Both were fixed and covered by 10 additional lifecycle regression cases. Astra's
-follow-up review found no remaining actionable defects and independently passed
-all 24 controller tests. Orchestration checks also reproduced both fixes and
-verified legacy database migration. No live cloud acceptance is implied.
-
-The fix binds observations to the desired revision and operation, invalidates them
-on desired-state changes, and discards stale success/error writes. Accepted deletion
-stops late-started compute before terminating it. Terminal identity and ownership
-checks no longer require live network attachments, and confirmed termination remains
-usable after EC2 stops returning the instance. Disk cleanup still requires terminal
-evidence and honors the explicit retain/delete choice.
-
-## Required before pilot acceptance
-
-Build and inspect the real Linux AMI; verify controller IRSA and worker IAM/KMS
-permissions; prove network and cross-agent isolation; run provider authentication
-with test credentials; exercise write/stop/restart persistence and process cleanup;
-verify GitHub work; establish worker/provider readiness; test backups and the
-separate fenced replacement/recovery design. Review the actual deployment plan,
-including ongoing NAT, IPv4, storage and secret charges.
-
-No live AWS resources were created, no cluster deployment was made, and no
-provider login or GitHub coding task was exercised by this checkpoint.
diff --git a/deploy/hosted/build-runtime.mjs b/deploy/hosted/build-runtime.mjs
deleted file mode 100644
index e907f9820..000000000
--- a/deploy/hosted/build-runtime.mjs
+++ /dev/null
@@ -1,59 +0,0 @@
-#!/usr/bin/env node
-import { createHash } from 'node:crypto';
-import { mkdir, readFile, writeFile } from 'node:fs/promises';
-import { createRequire } from 'node:module';
-import { resolve } from 'node:path';
-import { fileURLToPath } from 'node:url';
-
-if (process.argv.length !== 3) {
- throw new Error('Usage: node deploy/hosted/build-runtime.mjs <output-directory>');
-}
-const repository = fileURLToPath(new URL('../../', import.meta.url));
-const require = createRequire(new URL('../../console/package.json', import.meta.url));
-const { build } = require('esbuild');
-const output = resolve(process.argv[2]);
-await mkdir(output, { recursive: true });
-const names = ['hosted-bootstrap', 'shared-host-daemon'];
-await build({
- absWorkingDir: repository,
- entryPoints: {
- 'hosted-bootstrap': 'console/packages/agent-providers/src/host/hosted-bootstrap-cli.ts',
- 'shared-host-daemon': 'console/packages/agent-providers/src/host/shared-daemon.ts',
- },
- outdir: output,
- outExtension: { '.js': '.mjs' },
- bundle: true,
- platform: 'node',
- format: 'esm',
- target: 'node24',
- tsconfig: 'console/packages/agent-providers/tsconfig.json',
- banner: {
- js: "import { createRequire as createHostedRequire } from 'node:module'; const require = createHostedRequire(import.meta.url);",
- },
- plugins: [
- {
- name: 'headless-runtime-only',
- setup(context) {
- context.onResolve(
- { filter: /^(electron|better-sqlite3|drizzle-orm|@main\/db)(\/|$)/ },
- (args) => ({
- errors: [{ text: `Hosted runtime cannot import desktop dependency: ${args.path}` }],
- })
- );
- },
- },
- ],
- logLevel: 'info',
-});
-
-const files = {};
-for (const name of names) {
- const file = `${name}.mjs`;
- files[file] = createHash('sha256')
- .update(await readFile(resolve(output, file)))
- .digest('hex');
-}
-await writeFile(
- resolve(output, 'manifest.json'),
- JSON.stringify({ version: 1, nodeMajor: 24, files }, null, 2) + '\n'
-);
diff --git a/deploy/hosted/chart/Chart.yaml b/deploy/hosted/chart/Chart.yaml
index e47c790b7..1e3c7ea62 100644
--- a/deploy/hosted/chart/Chart.yaml
+++ b/deploy/hosted/chart/Chart.yaml
@@ -1,5 +1,5 @@
apiVersion: v2
name: switch-hosted-controller
-description: Single-controller EC2 hosted worker lifecycle service
-version: 0.1.1
+description: Single-controller lifecycle service for Switch cloud machines on EC2
+version: 0.1.3
type: application
diff --git a/deploy/hosted/chart/values.schema.json b/deploy/hosted/chart/values.schema.json
index e9b071120..757077fff 100644
--- a/deploy/hosted/chart/values.schema.json
+++ b/deploy/hosted/chart/values.schema.json
@@ -57,7 +57,7 @@
"max_machines",
"root_volume_gib",
"data_volume_gib",
- "machine_slots",
+ "instance_profile_arn",
"poll_interval_seconds"
]
},
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/cli.py b/deploy/hosted/controller/src/switch_hosted_controller/cli.py
index c31b41838..c196caa42 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/cli.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/cli.py
@@ -4,7 +4,6 @@ import argparse
import json
import logging
import os
-import re
import signal
import sys
import threading
@@ -17,14 +16,13 @@ from typing import Any
import boto3
from .cloud import Ec2Cloud
-from .config import ConfigError, ControllerConfig, validate_slot_id
+from .config import ConfigError, ControllerConfig
from .gateway import Gateway, GatewayConfig
from .health import check_health
from .lock import ControllerAlreadyRunning, ControllerLock
from .model import DesiredState, Machine
from .reconciler import Reconciler
from .store import MachineStore, StoreError
-from .verification import VerificationWorkers
def parser() -> argparse.ArgumentParser:
@@ -33,27 +31,25 @@ def parser() -> argparse.ArgumentParser:
result.add_argument("--gateway-config", type=Path)
subparsers = result.add_subparsers(dest="command", required=True)
- create = subparsers.add_parser("create", help="create a machine in one configured slot")
- create.add_argument("slot_id")
+ create = subparsers.add_parser("create", help="create a machine")
create.add_argument("--instance-type", required=True)
for command in ("start", "stop", "status"):
child = subparsers.add_parser(command)
- child.add_argument("slot_id")
+ child.add_argument("machine_id")
delete = subparsers.add_parser("delete")
- delete.add_argument("slot_id")
- delete.add_argument("--confirm-slot-id", required=True)
+ delete.add_argument("machine_id")
+ delete.add_argument("--confirm-machine-id", required=True)
cleanup = delete.add_mutually_exclusive_group(required=True)
cleanup.add_argument("--retain-volume", action="store_true")
cleanup.add_argument("--delete-volume", action="store_true")
upgrade = subparsers.add_parser(
- "upgrade", help="use the configured image after the old worker is stopped and terminated"
+ "upgrade", help="use the configured image after the old instance is stopped and terminated"
)
- upgrade.add_argument("slot_id")
+ upgrade.add_argument("machine_id")
upgrade.add_argument("--confirm-instance-id", required=True)
- upgrade.add_argument("--previous-runtime-fingerprint", required=True)
subparsers.add_parser("list")
health = subparsers.add_parser("health")
@@ -82,19 +78,13 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
store = MachineStore(config.state_db_path, config.fingerprint())
try:
if args.command == "create":
- slot_id = validate_slot_id(args.slot_id)
if args.instance_type not in config.allowed_instance_types:
raise ConfigError("instance type is not in allowed_instance_types")
- slot = config.slot(slot_id)
machine = store.insert(
machine_id=str(uuid.uuid4()),
- slot_id=slot_id,
- generation=store.next_generation(slot_id),
core_revision=0,
instance_type=args.instance_type,
image_id=config.image_id,
- assignment_secret_arn=slot.assignment_secret_arn,
- instance_profile_arn=slot.instance_profile_arn,
max_machines=config.max_machines,
)
_print_machine(machine)
@@ -104,8 +94,11 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
json.dumps([_machine_output(machine) for machine in store.list()], sort_keys=True)
)
return 0
- slot_id = validate_slot_id(args.slot_id)
- machine = store.latest(slot_id)
+ try:
+ machine_id = str(uuid.UUID(args.machine_id))
+ except ValueError:
+ raise ConfigError("machine_id must be a machine's UUID") from None
+ machine = store.get(machine_id)
if args.command == "status":
_print_machine(machine)
return 0
@@ -116,8 +109,8 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
_print_machine(store.set_desired(machine.machine_id, DesiredState.STOPPED, None))
return 0
if args.command == "delete":
- if args.confirm_slot_id != slot_id:
- raise StoreError("--confirm-slot-id must exactly match slot_id")
+ if args.confirm_machine_id != machine.machine_id:
+ raise StoreError("--confirm-machine-id must exactly match machine_id")
desired = DesiredState.DELETED if args.delete_volume else DesiredState.RETAINED
_print_machine(store.set_desired(machine.machine_id, desired, None))
return 0
@@ -130,10 +123,6 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
raise StoreError(
"stop the machine and confirm its recorded instance before upgrading"
)
- if not re.fullmatch(r"sha256:[0-9a-f]{64}", args.previous_runtime_fingerprint):
- raise ConfigError(
- "previous runtime fingerprint must be the sha256:<64 hex> runtimeFingerprint from the trusted disk marker"
- )
cloud = Ec2Cloud(boto3.client("ec2", region_name=config.region), config)
instance = cloud.get_instance(machine)
volume = cloud.get_volume(machine)
@@ -149,9 +138,7 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
)
cloud.validate_image(replace(machine, image_id=config.image_id))
claim = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)
- _print_machine(
- store.upgrade_terminated(claim, config.image_id, args.previous_runtime_fingerprint)
- )
+ _print_machine(store.upgrade_terminated(claim, config.image_id))
return 0
raise AssertionError(f"unhandled command {args.command}")
finally:
@@ -165,16 +152,8 @@ def _reconcile_command(config: ControllerConfig, command: str, gateway_path: Pat
ec2 = boto3.client("ec2", region_name=config.region)
reconciler = Reconciler(store, Ec2Cloud(ec2, config))
gateway = (
- Gateway(
- GatewayConfig.load(gateway_path),
- config,
- store,
- boto3.client("secretsmanager", region_name=config.region),
- )
- if gateway_path
- else None
+ Gateway(GatewayConfig.load(gateway_path), config, store) if gateway_path else None
)
- verification = VerificationWorkers(ec2, config, gateway) if gateway else None
_touch_health()
if command == "reconcile-once":
try:
@@ -190,13 +169,6 @@ def _reconcile_command(config: ControllerConfig, command: str, gateway_path: Pat
signal.signal(signal.SIGTERM, request_stop)
signal.signal(signal.SIGINT, request_stop)
while not stop.is_set():
- if verification:
- try:
- verification.reconcile()
- except Exception as error:
- logging.error(
- "Provider verification reconciliation failed: %s", type(error).__name__
- )
listed = None
if gateway:
try:
@@ -245,8 +217,6 @@ def _print_machine(machine: Machine) -> None:
def _machine_output(machine: Machine) -> dict[str, Any]:
return {
"machine_id": machine.machine_id,
- "slot_id": machine.slot_id,
- "generation": machine.generation,
"instance_type": machine.instance_type,
"desired_state": machine.desired_state.value,
"desired_revision": machine.desired_revision,
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/cloud.py b/deploy/hosted/controller/src/switch_hosted_controller/cloud.py
index 4dd304909..ffecf31db 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/cloud.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/cloud.py
@@ -1,8 +1,6 @@
from __future__ import annotations
-import base64
import hashlib
-import json
from typing import Any
from botocore.exceptions import ClientError
@@ -53,9 +51,7 @@ class Ec2Cloud:
def discover_volume(self, machine: Machine) -> dict[str, Any] | None:
volumes = self._describe_volumes(Filters=self._resource_filters(machine, "data"))
if len(volumes) > 1:
- raise CloudResourceError(
- "multiple controller-owned data volumes exist for machine slot"
- )
+ raise CloudResourceError("multiple controller-owned data volumes exist for the machine")
if not volumes:
return None
self._validate_volume(volumes[0], machine)
@@ -93,7 +89,7 @@ class Ec2Cloud:
instances = self._describe_instances(Filters=self._resource_filters(machine, "worker"))
active = [instance for instance in instances if instance["State"]["Name"] != "terminated"]
if len(active) > 1:
- raise CloudResourceError("multiple controller-owned instances exist for machine slot")
+ raise CloudResourceError("multiple controller-owned instances exist for the machine")
if not active:
return None
self._validate_instance(active[0], machine)
@@ -147,13 +143,15 @@ class Ec2Cloud:
def run_instance(self, machine: Machine) -> str:
if machine.data_volume_id is None:
raise CloudResourceError("cannot launch without a recorded data volume")
+ if machine.bundle is None:
+ raise CloudResourceError("cannot launch without a prepared bundle")
response = self._ec2.run_instances(
ImageId=machine.image_id,
InstanceType=machine.instance_type,
MinCount=1,
MaxCount=1,
ClientToken=self._launch_token(machine),
- IamInstanceProfile={"Arn": machine.instance_profile_arn},
+ IamInstanceProfile={"Arn": self._config.instance_profile_arn},
Placement={"AvailabilityZone": self._config.availability_zone},
NetworkInterfaces=[
{
@@ -189,7 +187,7 @@ class Ec2Cloud:
{"ResourceType": "volume", "Tags": self._tags(machine, "root")},
{"ResourceType": "network-interface", "Tags": self._tags(machine, "network")},
],
- UserData=self._user_data(machine),
+ UserData=machine.bundle,
)
instances = response.get("Instances", [])
if len(instances) != 1:
@@ -269,7 +267,7 @@ class Ec2Cloud:
if instance.get("State", {}).get("Name") in {"shutting-down", "terminated"}:
return
profile = instance.get("IamInstanceProfile") or {}
- if profile.get("Arn") != machine.instance_profile_arn:
+ if profile.get("Arn") != self._config.instance_profile_arn:
raise CloudResourceError("instance profile differs from immutable spec")
if instance.get("SubnetId") != self._config.subnet_id:
raise CloudResourceError("instance is in the wrong subnet")
@@ -313,8 +311,6 @@ class Ec2Cloud:
def _tags(self, machine: Machine, purpose: str) -> list[dict[str, str]]:
return [
{"Key": "switch:installation-id", "Value": self._config.installation_id},
- {"Key": "switch:slot-id", "Value": machine.slot_id},
- {"Key": "switch:generation", "Value": str(machine.generation)},
{"Key": "switch:machine-id", "Value": machine.machine_id},
{"Key": "switch:purpose", "Value": purpose},
{"Key": "switch:managed-by", "Value": MANAGED_BY},
@@ -323,8 +319,7 @@ class Ec2Cloud:
def _resource_filters(self, machine: Machine, purpose: str) -> list[dict[str, Any]]:
return [
{"Name": "tag:switch:installation-id", "Values": [self._config.installation_id]},
- {"Name": "tag:switch:slot-id", "Values": [machine.slot_id]},
- {"Name": "tag:switch:generation", "Values": [str(machine.generation)]},
+ {"Name": "tag:switch:machine-id", "Values": [machine.machine_id]},
{"Name": "tag:switch:purpose", "Values": [purpose]},
{"Name": "tag:switch:managed-by", "Values": [MANAGED_BY]},
]
@@ -333,40 +328,9 @@ class Ec2Cloud:
return self._token(machine, f"instance-{machine.instance_seq}")
def _token(self, machine: Machine, resource: str) -> str:
- material = (
- f"{self._config.installation_id}:{machine.slot_id}:{machine.generation}:{resource}"
- )
+ material = f"{self._config.installation_id}:{machine.machine_id}:{resource}"
return f"switch-m-{hashlib.sha256(material.encode()).hexdigest()[:48]}"
- def _user_data(self, machine: Machine) -> str:
- metadata: dict[str, Any] = {
- "version": 2,
- "installationId": self._config.installation_id,
- "slotId": machine.slot_id,
- "generation": machine.generation,
- "assignmentSecretId": machine.assignment_secret_arn,
- "dataVolumeId": machine.data_volume_id,
- "dataDevice": DATA_DEVICE,
- "mountPath": "/data",
- }
- if machine.previous_instance_id:
- metadata["previousInstanceId"] = machine.previous_instance_id
- if machine.previous_runtime_fingerprint:
- metadata["previousRuntimeFingerprint"] = machine.previous_runtime_fingerprint
- encoded = base64.b64encode(json.dumps(metadata, separators=(",", ":")).encode()).decode()
- return "\n".join(
- [
- "#cloud-config",
- "write_files:",
- " - path: /etc/switch-hosted/assignment.json",
- " owner: root:root",
- " permissions: '0600'",
- " encoding: b64",
- f" content: {encoded}",
- "",
- ]
- )
-
def _describe_instances(self, **kwargs: Any) -> list[dict[str, Any]]:
instances: list[dict[str, Any]] = []
request = dict(kwargs)
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/config.py b/deploy/hosted/controller/src/switch_hosted_controller/config.py
index 77129c5f2..1a3a6fcb0 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/config.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/config.py
@@ -8,7 +8,6 @@ from pathlib import Path
from typing import Any
_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
-_SLOT_ID = re.compile(r"^[a-z0-9][a-z0-9-]{2,39}$")
_INSTANCE_TYPE = re.compile(r"^[a-z0-9][a-z0-9.]{1,30}$")
_ARN = re.compile(r"^arn:(aws|aws-us-gov|aws-cn):[a-z0-9-]+:[a-z0-9-]*:[0-9]{12}:.+$")
@@ -17,12 +16,6 @@ class ConfigError(ValueError):
pass
-@dataclass(frozen=True)
-class MachineSlot:
- instance_profile_arn: str
- assignment_secret_arn: str
-
-
@dataclass(frozen=True)
class ControllerConfig:
installation_id: str
@@ -36,7 +29,7 @@ class ControllerConfig:
max_machines: int
root_volume_gib: int
data_volume_gib: int
- machine_slots: dict[str, MachineSlot]
+ instance_profile_arn: str
state_db_path: Path
lock_path: Path
poll_interval_seconds: float
@@ -65,7 +58,7 @@ class ControllerConfig:
"max_machines",
"root_volume_gib",
"data_volume_gib",
- "machine_slots",
+ "instance_profile_arn",
"state_db_path",
"lock_path",
"poll_interval_seconds",
@@ -117,38 +110,9 @@ class ControllerConfig:
if not 0.2 <= float(poll_interval) <= 300:
raise ConfigError("poll_interval_seconds must be between 0.2 and 300")
- slots_raw = raw["machine_slots"]
- if not isinstance(slots_raw, dict) or not slots_raw:
- raise ConfigError("machine_slots must be a non-empty object")
- slots: dict[str, MachineSlot] = {}
- for slot_id, slot_raw in slots_raw.items():
- _validated_value(slot_id, "machine_slots key", _SLOT_ID)
- if not isinstance(slot_raw, dict) or set(slot_raw) != {
- "instance_profile_arn",
- "assignment_secret_arn",
- }:
- raise ConfigError(
- f"machine slot {slot_id!r} must contain only instance_profile_arn and assignment_secret_arn"
- )
- profile = _validated_value(
- slot_raw["instance_profile_arn"], "instance_profile_arn", _ARN
- )
- secret = _validated_value(
- slot_raw["assignment_secret_arn"], "assignment_secret_arn", _ARN
- )
- if ":iam::" not in profile or ":instance-profile/" not in profile:
- raise ConfigError(f"machine slot {slot_id!r} has an invalid instance profile ARN")
- if ":secretsmanager:" not in secret or ":secret:" not in secret:
- raise ConfigError(f"machine slot {slot_id!r} has an invalid secret ARN")
- slots[slot_id] = MachineSlot(profile, secret)
- if len(slots) < max_machines:
- raise ConfigError("max_machines exceeds configured machine slots")
- profiles = [slot.instance_profile_arn for slot in slots.values()]
- secrets = [slot.assignment_secret_arn for slot in slots.values()]
- if len(set(profiles)) != len(profiles):
- raise ConfigError("machine slot instance profiles must be unique")
- if len(set(secrets)) != len(secrets):
- raise ConfigError("machine slot secrets must be unique")
+ instance_profile_arn = _validated_string(raw, "instance_profile_arn", _ARN)
+ if ":iam::" not in instance_profile_arn or ":instance-profile/" not in instance_profile_arn:
+ raise ConfigError("instance_profile_arn is not an IAM instance profile ARN")
state_db_path = _absolute_path(raw, "state_db_path")
lock_path = _absolute_path(raw, "lock_path")
@@ -167,7 +131,7 @@ class ControllerConfig:
max_machines=max_machines,
root_volume_gib=root_volume_gib,
data_volume_gib=data_volume_gib,
- machine_slots=slots,
+ instance_profile_arn=instance_profile_arn,
state_db_path=state_db_path,
lock_path=lock_path,
poll_interval_seconds=float(poll_interval),
@@ -188,16 +152,6 @@ class ControllerConfig:
payload = json.dumps(immutable, separators=(",", ":"), sort_keys=True)
return hashlib.sha256(payload.encode()).hexdigest()
- def slot(self, slot_id: str) -> MachineSlot:
- try:
- return self.machine_slots[slot_id]
- except KeyError as exc:
- raise ConfigError(f"slot {slot_id!r} is not in machine_slots") from exc
-
-
-def validate_slot_id(slot_id: str) -> str:
- return _validated_value(slot_id, "slot_id", _SLOT_ID)
-
def _validated_string(raw: dict[str, Any], key: str, pattern: re.Pattern[str]) -> str:
return _validated_value(raw[key], key, pattern)
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/gateway.py b/deploy/hosted/controller/src/switch_hosted_controller/gateway.py
index c366ba3f1..85fe7db25 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/gateway.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/gateway.py
@@ -13,15 +13,13 @@ from urllib.parse import urlsplit
from urllib.request import HTTPRedirectHandler, Request, build_opener
from uuid import NAMESPACE_URL, UUID, uuid5
-from botocore.exceptions import ClientError
-
-from .config import ConfigError, ControllerConfig, validate_slot_id
+from .config import ConfigError, ControllerConfig
from .model import DesiredState, Machine, ObservedState
-from .store import MachineStore, SlotInUseError
+from .store import MachineStore
logger = logging.getLogger(__name__)
-CAPABILITY_RE = re.compile(r"^[\x21-\x7e]{16,4096}$")
+ENROLLMENT_CODE_RE = re.compile(r"^swce_[A-Za-z0-9_-]{16,128}$")
CORE_DESIRED_STATES = {"running", "stopped", "retained", "deleted"}
SETUP_FAILED_MESSAGE = (
"Cloud machine setup failed. Retry; if it still fails, contact your administrator."
@@ -62,8 +60,6 @@ class GatewayError(RuntimeError):
@dataclass(frozen=True)
class CoreMachine:
machine_id: str
- slot_id: str
- generation: int
state: str
desired_state: str
revision: int
@@ -74,10 +70,7 @@ class CoreMachine:
def parse(cls, raw: Any) -> CoreMachine:
if not isinstance(raw, dict):
raise ConfigError("Cloud gateway returned an invalid machine.")
- generation = raw.get("generation")
revision = raw.get("revision")
- if not isinstance(generation, int) or isinstance(generation, bool) or generation < 1:
- raise ConfigError("Cloud gateway returned an invalid machine generation.")
if not isinstance(revision, int) or isinstance(revision, bool) or revision < 1:
raise ConfigError("Cloud gateway returned an invalid machine revision.")
if raw.get("desired_state") not in CORE_DESIRED_STATES:
@@ -91,8 +84,6 @@ class CoreMachine:
raise ConfigError("Cloud gateway returned an invalid data volume id.")
return cls(
machine_id=str(UUID(raw["machine_id"])),
- slot_id=validate_slot_id(raw["slot_id"]),
- generation=generation,
state=raw["state"],
desired_state=raw["desired_state"],
revision=revision,
@@ -140,24 +131,13 @@ class Gateway:
settings: GatewayConfig,
config: ControllerConfig,
store: MachineStore,
- secrets_client: Any,
):
if settings.instance_type not in config.allowed_instance_types:
raise ConfigError("Cloud gateway instance type is not allowed.")
self.settings = settings
self.config = config
self.store = store
- self.secrets = secrets_client
self.prepare_failures: dict[str, float] = {}
- for machine in store.list():
- if machine.observed_state is ObservedState.DELETED:
- continue
- slot = config.slot(machine.slot_id)
- if (
- slot.assignment_secret_arn != machine.assignment_secret_arn
- or slot.instance_profile_arn != machine.instance_profile_arn
- ):
- raise ConfigError("A machine slot in use cannot change its secret or IAM identity.")
def request(
self, path: str, body: dict | None = None, *, prefix: str = "/gateway/hosted-controller"
@@ -252,36 +232,20 @@ class Gateway:
)
def _row(self, core: CoreMachine) -> Machine | None:
- machine = self.store.find(core.slot_id, core.generation)
- if machine is None or machine.machine_id != core.machine_id:
- return None
- return machine
+ return self.store.find(core.machine_id)
def sync_machine(self, core: CoreMachine) -> None:
- slot = self.config.slot(core.slot_id)
- machine = self.store.find(core.slot_id, core.generation)
- if machine is not None and machine.machine_id != core.machine_id:
- raise ConfigError(
- "Cloud gateway returned a different machine for a stored slot generation."
- )
+ machine = self.store.find(core.machine_id)
if machine is None:
if core.state == "error" and core.desired_state in {"running", "stopped"}:
return
- try:
- machine = self.store.insert(
- machine_id=core.machine_id,
- slot_id=core.slot_id,
- generation=core.generation,
- core_revision=core.revision,
- instance_type=self.settings.instance_type,
- image_id=self.config.image_id,
- assignment_secret_arn=slot.assignment_secret_arn,
- instance_profile_arn=slot.instance_profile_arn,
- max_machines=self.config.max_machines,
- )
- except SlotInUseError as error:
- logger.warning("Waiting for machine %s: %s", core.machine_id, error)
- return
+ machine = self.store.insert(
+ machine_id=core.machine_id,
+ core_revision=core.revision,
+ instance_type=self.settings.instance_type,
+ image_id=self.config.image_id,
+ max_machines=self.config.max_machines,
+ )
machine = self.store.record_core_revision(machine.machine_id, core.revision)
if machine.core_revision > core.revision:
logger.warning(
@@ -338,17 +302,8 @@ class Gateway:
)
if machine.data_volume_id is None or machine.bundle_token == token:
return
- secret_id = machine.assignment_secret_arn
- if self.promote_bundle(secret_id, token):
- self.store.record_bundle(machine.machine_id, token)
- return
prepared = self.request(f"/machines/{core.machine_id}/prepare", {})
- if (
- not isinstance(prepared, dict)
- or str(UUID(prepared["machine_id"])) != core.machine_id
- or prepared["slot_id"] != core.slot_id
- or prepared["generation"] != core.generation
- ):
+ if not isinstance(prepared, dict) or str(UUID(prepared["machine_id"])) != core.machine_id:
raise ConfigError("Cloud gateway prepared a different machine.")
if prepared["revision"] != machine.core_revision:
logger.warning(
@@ -361,38 +316,9 @@ class Gateway:
if prepared["bundle_revision"] != machine.core_revision:
raise ConfigError("Cloud gateway prepared a bundle for a different revision.")
bundle = self.bundle(prepared, machine)
- try:
- self.secrets.put_secret_value(
- SecretId=secret_id,
- ClientRequestToken=token,
- SecretString=json.dumps(bundle, separators=(",", ":")),
- )
- except ClientError as error:
- if error.response.get("Error", {}).get("Code") != "ResourceExistsException":
- raise
- # Secrets Manager answers a repeated ClientRequestToken with this error
- # only when the stored version's content differs from the request.
- raise ConfigError(
- "The assignment secret already holds a different bundle for this revision."
- ) from error
- self.store.record_bundle(machine.machine_id, token)
-
- def promote_bundle(self, secret_id: str, token: str) -> bool:
- """Make the bundle version `token` AWSCURRENT if it exists; False if it does not."""
- versions = self.secrets.describe_secret(SecretId=secret_id).get("VersionIdsToStages", {})
- if token not in versions:
- return False
- if "AWSCURRENT" in versions[token]:
- return True
- holder = {
- "RemoveFromVersionId": version
- for version, stages in versions.items()
- if "AWSCURRENT" in stages
- }
- self.secrets.update_secret_version_stage(
- SecretId=secret_id, VersionStage="AWSCURRENT", MoveToVersionId=token, **holder
+ self.store.record_bundle(
+ machine.machine_id, token, json.dumps(bundle, separators=(",", ":"))
)
- return True
def report_observations(self, listed: list[dict]) -> None:
for item in listed:
@@ -457,24 +383,32 @@ class Gateway:
}
def bundle(self, prepared: dict, machine: Machine) -> dict:
- capability = prepared.get("machine_capability")
- if not isinstance(capability, str) or not CAPABILITY_RE.fullmatch(capability):
- raise ConfigError("Cloud gateway returned no valid machine capability.")
+ """The bundle a machine boots its agents controller from, its instance's
+ user data: the controller it enrolled as, or the one-time code it
+ enrolls with until it has one. Never a long-lived credential: the
+ machine keeps the one it enrolls with on its own data volume."""
endpoint = prepared.get("api_endpoint")
url = urlsplit(endpoint) if isinstance(endpoint, str) else None
if url is None or url.scheme != "https" or not url.hostname:
raise ConfigError("Cloud gateway returned no valid API endpoint.")
+ controller = prepared.get("controller")
+ if not isinstance(controller, dict):
+ raise ConfigError("Cloud gateway returned no controller for the machine.")
+ controller_id = controller.get("id")
+ code = controller.get("enrollment_code")
+ if controller_id is not None:
+ if code is not None or not isinstance(controller_id, str):
+ raise ConfigError("Cloud gateway returned an invalid controller.")
+ controller_id = str(UUID(controller_id))
+ elif not isinstance(code, str) or not ENROLLMENT_CODE_RE.fullmatch(code):
+ raise ConfigError("Cloud gateway returned no valid enrollment code.")
return {
- "version": 2,
+ "version": 4,
+ "installationId": self.config.installation_id,
"machineId": machine.machine_id,
- "assignment": {
- "installationId": self.config.installation_id,
- "slotId": machine.slot_id,
- "generation": machine.generation,
- "dataVolumeId": machine.data_volume_id,
- },
- "machineCapability": capability,
+ "dataVolumeId": machine.data_volume_id,
"apiEndpoint": endpoint,
+ "controller": {"id": controller_id, "enrollmentCode": code},
}
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/model.py b/deploy/hosted/controller/src/switch_hosted_controller/model.py
index 60db3204b..4e11d5c92 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/model.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/model.py
@@ -27,8 +27,6 @@ class ObservedState(StrEnum):
@dataclass(frozen=True)
class Machine:
machine_id: str
- slot_id: str
- generation: int
desired_state: DesiredState
desired_revision: int
operation_id: str
@@ -36,11 +34,8 @@ class Machine:
retain_until: datetime | None
instance_type: str
image_id: str
- assignment_secret_arn: str
- instance_profile_arn: str
instance_id: str | None
previous_instance_id: str | None
- previous_runtime_fingerprint: str | None
instance_seq: int
recovery_count: int
data_volume_id: str | None
@@ -60,3 +55,5 @@ class Machine:
required_bundle_revision: int | None
required_bundle_token: str | None
bundle_token: str | None
+ bundle: str | None
+ instance_bundle: str | None
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py b/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py
index f6f398c96..5bf9a701d 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+import json
import logging
import re
from datetime import UTC, datetime, timedelta
@@ -78,7 +79,7 @@ class Reconciler:
instance = self._cloud.get_instance(machine)
if machine.instance_id is None:
if instance is not None:
- return self._store.record_instance(machine.machine_id, instance["InstanceId"])
+ return self._store.record_instance(machine.machine_id, instance["InstanceId"], None)
if volume["State"] != "available" or (
not machine.instance_launch_issued and not _bundle_ready(machine)
):
@@ -103,7 +104,7 @@ class Reconciler:
if rejected(exc):
self._store.clear_unlaunched_instance(machine)
raise
- return self._store.record_instance(machine.machine_id, instance_id)
+ return self._store.record_instance(machine.machine_id, instance_id, machine.bundle)
if instance is None:
return self._attention(
claim,
@@ -127,7 +128,18 @@ class Reconciler:
return self._store.replace_terminated(terminated, unexpected=unexpected)
if state == "stopped":
if _bundle_ready(machine) and self._unchanged(claim, DesiredState.RUNNING):
- self._cloud.start_instance(machine)
+ if machine.bundle is not None and needs_new_user_data(
+ machine.instance_bundle, machine.bundle
+ ):
+ # An instance's user data is set when it launches, so it
+ # makes way for one launched with the new bundle; the data
+ # volume is kept.
+ machine = self._store.mark_instance_terminate_issued(claim)
+ if not self._same_claim(claim, machine):
+ return machine
+ self._cloud.terminate_instance(machine)
+ else:
+ self._cloud.start_instance(machine)
return self._store.set_observed(claim, ObservedState.PROVISIONING, None)
if state in {"pending", "stopping", "shutting-down"}:
return self._store.set_observed(claim, ObservedState.PROVISIONING, None)
@@ -164,7 +176,7 @@ class Reconciler:
instance = self._cloud.get_instance(machine)
if machine.instance_id is None:
if instance is not None:
- return self._store.record_instance(machine.machine_id, instance["InstanceId"])
+ return self._store.record_instance(machine.machine_id, instance["InstanceId"], None)
if machine.instance_launch_intent and not machine.instance_launch_issued:
self._store.cancel_queued_instance_launch(claim)
return self._store.set_observed(claim, ObservedState.STOPPED, None)
@@ -271,7 +283,7 @@ class Reconciler:
return self._store.set_observed(claim, busy, None)
self._store.clear_unlaunched_instance(machine)
return None
- return self._store.record_instance(machine.machine_id, instance["InstanceId"])
+ return self._store.record_instance(machine.machine_id, instance["InstanceId"], None)
if machine.instance_id is None:
return None
instance = self._cloud.get_instance(machine)
@@ -342,6 +354,26 @@ def _bundle_ready(machine: Machine) -> bool:
)
+def needs_new_user_data(current: str | None, wanted: str) -> bool:
+ """Whether an instance booting from the bundle `current` must make way for
+ one launched with `wanted`.
+
+ Not when `wanted` only names the controller the machine enrolled as with
+ the code in `current`: the boot keeps an enrollment made with that code.
+ """
+ if current is None:
+ return True
+ have, want = json.loads(current), json.loads(wanted)
+ have_controller, want_controller = have.pop("controller"), want.pop("controller")
+ if have != want:
+ return True
+ if want_controller["enrollmentCode"] is not None:
+ return want_controller["enrollmentCode"] != have_controller["enrollmentCode"]
+ return have_controller["enrollmentCode"] is None and (
+ have_controller["id"] != want_controller["id"]
+ )
+
+
def _volume_mapping(instance: dict[str, Any], volume_id: str) -> dict[str, Any] | None:
for mapping in instance.get("BlockDeviceMappings", []):
ebs = mapping.get("Ebs") or {}
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/store.py b/deploy/hosted/controller/src/switch_hosted_controller/store.py
index a08911e76..fc711a314 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/store.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/store.py
@@ -10,6 +10,10 @@ from .model import DesiredState, Machine, ObservedState
LEGACY_ROWS_MESSAGE = (
"legacy per-agent rows present; see 'Moving to one machine per user' in deploy/hosted/README.md"
)
+SLOT_ROWS_MESSAGE = (
+ "the state database holds machines placed on slots that are not deleted; "
+ "see 'Moving off machine slots' in deploy/hosted/README.md"
+)
RECOVERY_LIMIT = 3
@@ -23,10 +27,6 @@ class CapacityError(StoreError):
pass
-class SlotInUseError(StoreError):
- pass
-
-
class MachineNotFoundError(StoreError):
pass
@@ -40,12 +40,11 @@ class MachineStore:
self._connection.execute("PRAGMA synchronous=FULL")
self._connection.execute("PRAGMA foreign_keys=ON")
self._refuse_legacy_rows()
+ self._drop_slot_rows()
self._connection.executescript(
"""
CREATE TABLE IF NOT EXISTS machines (
- machine_id TEXT NOT NULL UNIQUE,
- slot_id TEXT NOT NULL,
- generation INTEGER NOT NULL CHECK (generation >= 1),
+ machine_id TEXT PRIMARY KEY,
desired_state TEXT NOT NULL
CHECK (desired_state IN ('running', 'stopped', 'retained', 'deleted')),
observed_state TEXT NOT NULL,
@@ -54,15 +53,12 @@ class MachineStore:
operation_id TEXT NOT NULL,
instance_type TEXT NOT NULL,
image_id TEXT NOT NULL,
- assignment_secret_arn TEXT NOT NULL,
- instance_profile_arn TEXT NOT NULL,
instance_seq INTEGER NOT NULL DEFAULT 0,
recovery_count INTEGER NOT NULL DEFAULT 0,
data_volume_id TEXT,
volume_az TEXT,
instance_id TEXT,
previous_instance_id TEXT,
- previous_runtime_fingerprint TEXT,
retain_until TEXT,
observed_revision INTEGER NOT NULL DEFAULT 0,
observed_operation_id TEXT,
@@ -77,19 +73,19 @@ class MachineStore:
required_bundle_revision INTEGER,
required_bundle_token TEXT,
bundle_token TEXT,
+ bundle TEXT,
+ instance_bundle TEXT,
error TEXT,
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
- updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
- PRIMARY KEY (slot_id, generation)
+ updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
);
- CREATE UNIQUE INDEX IF NOT EXISTS machines_one_live_row_per_slot
- ON machines (slot_id) WHERE observed_state <> 'deleted';
CREATE TABLE IF NOT EXISTS controller_metadata (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
"""
)
+ self._keep_instance_bundles()
self._bind_fingerprint(controller_fingerprint)
def close(self) -> None:
@@ -108,6 +104,47 @@ class MachineStore:
self._connection.close()
raise StoreError(LEGACY_ROWS_MESSAGE)
+ def _drop_slot_rows(self) -> None:
+ """Forget the machines of a database from before machine slots were removed.
+
+ Only ones observed deleted are forgotten: a machine still on a slot
+ boots from that slot's secret, which no longer exists.
+ """
+ columns = {row["name"] for row in self._connection.execute("PRAGMA table_info(machines)")}
+ if "slot_id" not in columns:
+ return
+ live = self._connection.execute(
+ "SELECT COUNT(*) FROM machines WHERE observed_state <> 'deleted'"
+ ).fetchone()[0]
+ if live:
+ self._connection.close()
+ raise StoreError(SLOT_ROWS_MESSAGE)
+ self._connection.executescript(
+ """
+ BEGIN IMMEDIATE;
+ DROP INDEX IF EXISTS machines_one_live_row_per_slot;
+ DROP TABLE machines;
+ COMMIT;
+ """
+ )
+
+ def _keep_instance_bundles(self) -> None:
+ """Move a database that kept only the token of an instance's bundle to
+ keeping the bundle itself; one no longer current is not known."""
+ columns = {row["name"] for row in self._connection.execute("PRAGMA table_info(machines)")}
+ if "instance_bundle_token" not in columns:
+ return
+ self._connection.executescript(
+ """
+ BEGIN IMMEDIATE;
+ ALTER TABLE machines ADD COLUMN instance_bundle TEXT;
+ UPDATE machines SET instance_bundle = bundle
+ WHERE instance_bundle_token IS NOT NULL AND instance_bundle_token = bundle_token;
+ ALTER TABLE machines DROP COLUMN instance_bundle_token;
+ COMMIT;
+ """
+ )
+
def _bind_fingerprint(self, fingerprint: str) -> None:
self._connection.execute("BEGIN IMMEDIATE")
try:
@@ -134,37 +171,18 @@ class MachineStore:
self,
*,
machine_id: str,
- slot_id: str,
- generation: int,
core_revision: int,
instance_type: str,
image_id: str,
- assignment_secret_arn: str,
- instance_profile_arn: str,
max_machines: int,
) -> Machine:
- """Insert a machine for a slot whose earlier generations are all observed deleted."""
+ """Insert a machine, within the configured capacity."""
self._connection.execute("BEGIN IMMEDIATE")
try:
if self._connection.execute(
"SELECT 1 FROM machines WHERE machine_id = ?", (machine_id,)
).fetchone():
raise StoreError(f"machine {machine_id!r} already exists")
- latest = self._connection.execute(
- "SELECT MAX(generation) FROM machines WHERE slot_id = ?", (slot_id,)
- ).fetchone()[0]
- if latest is not None and generation <= latest:
- raise StoreError(
- f"slot {slot_id!r} generation {generation} is not newer than stored generation {latest}"
- )
- live = self._connection.execute(
- "SELECT generation FROM machines WHERE slot_id = ? AND observed_state <> ?",
- (slot_id, ObservedState.DELETED.value),
- ).fetchone()
- if live is not None:
- raise SlotInUseError(
- f"slot {slot_id!r} still has generation {live['generation']} that is not deleted"
- )
count = self._connection.execute(
"SELECT COUNT(*) FROM machines WHERE desired_state != ?",
(DesiredState.DELETED.value,),
@@ -175,24 +193,19 @@ class MachineStore:
self._connection.execute(
"""
INSERT INTO machines (
- machine_id, slot_id, generation, desired_state, observed_state,
- core_revision, desired_revision, operation_id, instance_type, image_id,
- assignment_secret_arn, instance_profile_arn, observed_revision,
- observed_operation_id
- ) VALUES (?, ?, ?, ?, ?, ?, 1, ?, ?, ?, ?, ?, 1, ?)
+ machine_id, desired_state, observed_state, core_revision,
+ desired_revision, operation_id, instance_type, image_id,
+ observed_revision, observed_operation_id
+ ) VALUES (?, ?, ?, ?, 1, ?, ?, ?, 1, ?)
""",
(
machine_id,
- slot_id,
- generation,
DesiredState.RUNNING.value,
ObservedState.PENDING.value,
core_revision,
operation_id,
instance_type,
image_id,
- assignment_secret_arn,
- instance_profile_arn,
operation_id,
),
)
@@ -203,12 +216,6 @@ class MachineStore:
self._connection.execute("ROLLBACK")
raise
- def next_generation(self, slot_id: str) -> int:
- latest = self._connection.execute(
- "SELECT MAX(generation) FROM machines WHERE slot_id = ?", (slot_id,)
- ).fetchone()[0]
- return 1 if latest is None else latest + 1
-
def record_core_revision(self, machine_id: str, revision: int) -> Machine:
"""Record Core's revision; an older one than already recorded leaves the machine unchanged."""
self._connection.execute(
@@ -289,9 +296,7 @@ class MachineStore:
def mark_volume_create_intent(self, claim: Machine) -> Machine:
return self._mark_intent(claim, "volume_create_intent")
- def upgrade_terminated(
- self, claim: Machine, image_id: str, previous_runtime_fingerprint: str
- ) -> Machine:
+ def upgrade_terminated(self, claim: Machine, image_id: str) -> Machine:
if (
claim.desired_state is not DesiredState.STOPPED
or not claim.instance_id
@@ -304,7 +309,8 @@ class MachineStore:
raise StoreError("the machine already uses this image")
cursor = self._connection.execute(
"""UPDATE machines SET previous_instance_id = instance_id, instance_id = NULL,
- image_id = ?, previous_runtime_fingerprint = ?, instance_seq = instance_seq + 1,
+ instance_bundle = NULL,
+ image_id = ?, instance_seq = instance_seq + 1,
desired_revision = desired_revision + 1, operation_id = ?,
instance_launch_intent = 0, instance_launch_issued = 0, instance_launch_issued_at = NULL,
instance_terminate_issued = 0,
@@ -313,7 +319,6 @@ class MachineStore:
AND desired_state = 'stopped' AND instance_id = ? AND instance_terminal_observed = 1""",
(
image_id,
- previous_runtime_fingerprint,
str(uuid.uuid4()),
claim.machine_id,
claim.desired_revision,
@@ -337,7 +342,8 @@ class MachineStore:
require_recovery_allowed(claim)
self._connection.execute(
"""UPDATE machines SET previous_instance_id = instance_id, instance_id = NULL,
- previous_runtime_fingerprint = NULL, instance_seq = instance_seq + 1,
+ instance_bundle = NULL,
+ instance_seq = instance_seq + 1,
recovery_count = recovery_count + ?, instance_launch_intent = 0,
instance_launch_issued = 0, instance_launch_issued_at = NULL,
instance_terminate_issued = 0,
@@ -361,7 +367,8 @@ class MachineStore:
raise StoreError("release requires a confirmed terminated instance")
self._connection.execute(
"""UPDATE machines SET previous_instance_id = instance_id, instance_id = NULL,
- previous_runtime_fingerprint = NULL, instance_seq = instance_seq + 1,
+ instance_bundle = NULL,
+ instance_seq = instance_seq + 1,
instance_launch_intent = 0, instance_launch_issued = 0, instance_launch_issued_at = NULL,
instance_terminate_issued = 0,
instance_terminal_observed = 0, updated_at = CURRENT_TIMESTAMP
@@ -434,8 +441,10 @@ class MachineStore:
)
return self.get(machine_id)
- def record_instance(self, machine_id: str, instance_id: str) -> Machine:
- self._set_once(machine_id, "instance_id", instance_id)
+ def record_instance(self, machine_id: str, instance_id: str, bundle: str | None) -> Machine:
+ """Record the machine's instance, launched with `bundle` as its user data,
+ or None when that is not known."""
+ self._set_once(machine_id, "instance_id", instance_id, extra=("instance_bundle", bundle))
return self.get(machine_id)
def mark_instance_terminal_observed(self, machine_id: str, instance_id: str) -> Machine:
@@ -489,39 +498,32 @@ class MachineStore:
)
return self.get(machine_id)
- def record_bundle(self, machine_id: str, token: str) -> Machine:
+ def record_bundle(self, machine_id: str, token: str, bundle: str) -> Machine:
+ """Keep the bundle of the required revision, which the machine's next
+ instance launch or start boots from."""
self._connection.execute(
"""
- UPDATE machines SET bundle_token = ?, updated_at = CURRENT_TIMESTAMP
+ UPDATE machines SET bundle_token = ?, bundle = ?, updated_at = CURRENT_TIMESTAMP
WHERE machine_id = ? AND required_bundle_token = ?
""",
- (token, machine_id, token),
+ (token, bundle, machine_id, token),
)
return self.get(machine_id)
def get(self, machine_id: str) -> Machine:
return self._get_row(machine_id)
- def find(self, slot_id: str, generation: int) -> Machine | None:
+ def find(self, machine_id: str) -> Machine | None:
row = self._connection.execute(
- "SELECT * FROM machines WHERE slot_id = ? AND generation = ?", (slot_id, generation)
+ "SELECT * FROM machines WHERE machine_id = ?", (machine_id,)
).fetchone()
return None if row is None else _machine(row)
- def latest(self, slot_id: str) -> Machine:
- row = self._connection.execute(
- "SELECT * FROM machines WHERE slot_id = ? ORDER BY generation DESC LIMIT 1",
- (slot_id,),
- ).fetchone()
- if row is None:
- raise MachineNotFoundError(f"no machine for slot {slot_id!r}")
- return _machine(row)
-
def list(self) -> list[Machine]:
return [
_machine(row)
for row in self._connection.execute(
- "SELECT * FROM machines ORDER BY slot_id, generation"
+ "SELECT * FROM machines ORDER BY created_at, machine_id"
)
]
@@ -555,7 +557,11 @@ class MachineStore:
return self.get(claim.machine_id)
def _set_once(
- self, machine_id: str, column: str, value: str, extra: tuple[str, str] | None = None
+ self,
+ machine_id: str,
+ column: str,
+ value: str,
+ extra: tuple[str, str | None] | None = None,
) -> None:
if column not in {"instance_id", "data_volume_id"}:
raise ValueError("invalid set-once column")
@@ -566,10 +572,10 @@ class MachineStore:
if existing is not None and existing != value:
raise StoreError(f"refusing to replace recorded {column}")
assignments = [f"{column} = ?", "updated_at = CURRENT_TIMESTAMP"]
- values: list[str] = [value]
+ values: list[str | None] = [value]
if extra is not None:
extra_column, extra_value = extra
- if extra_column != "volume_az":
+ if extra_column not in {"volume_az", "instance_bundle"}:
raise ValueError("invalid extra column")
assignments.insert(1, f"{extra_column} = ?")
values.append(extra_value)
@@ -608,8 +614,6 @@ def _timestamp(value: datetime | None) -> str | None:
def _machine(row: sqlite3.Row) -> Machine:
return Machine(
machine_id=row["machine_id"],
- slot_id=row["slot_id"],
- generation=row["generation"],
desired_state=DesiredState(row["desired_state"]),
desired_revision=row["desired_revision"],
operation_id=row["operation_id"],
@@ -617,11 +621,8 @@ def _machine(row: sqlite3.Row) -> Machine:
retain_until=(datetime.fromisoformat(row["retain_until"]) if row["retain_until"] else None),
instance_type=row["instance_type"],
image_id=row["image_id"],
- assignment_secret_arn=row["assignment_secret_arn"],
- instance_profile_arn=row["instance_profile_arn"],
instance_id=row["instance_id"],
previous_instance_id=row["previous_instance_id"],
- previous_runtime_fingerprint=row["previous_runtime_fingerprint"],
instance_seq=row["instance_seq"],
recovery_count=row["recovery_count"],
data_volume_id=row["data_volume_id"],
@@ -645,4 +646,6 @@ def _machine(row: sqlite3.Row) -> Machine:
required_bundle_revision=row["required_bundle_revision"],
required_bundle_token=row["required_bundle_token"],
bundle_token=row["bundle_token"],
+ bundle=row["bundle"],
+ instance_bundle=row["instance_bundle"],
)
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/verification.py b/deploy/hosted/controller/src/switch_hosted_controller/verification.py
deleted file mode 100644
index 5ee059a93..000000000
--- a/deploy/hosted/controller/src/switch_hosted_controller/verification.py
+++ /dev/null
@@ -1,194 +0,0 @@
-from __future__ import annotations
-
-import json
-import logging
-import time
-from typing import Any
-from uuid import UUID
-
-from .config import ControllerConfig
-from .gateway import Gateway
-
-MANAGED_BY = "switch-provider-verification"
-
-
-def user_data(origin: str, job_id: str, token: str) -> str:
- # The bootstrap token can retrieve only this short-lived check's credential.
- config = json.dumps(
- {"url": origin + "/gateway/provider-verifications/" + str(UUID(job_id)), "token": token}
- )
- return (
- """#!/bin/bash
-set -eu
-systemctl stop switch-hosted-worker.service || true
-shutdown -P +8 >/dev/null 2>&1
-trap 'shutdown -P now' EXIT
-python3 - <<'PYTHON'
-import json, os, pwd, subprocess, urllib.request
-from pathlib import Path
-"""
- + "config = "
- + repr(json.loads(config))
- + """
-def request(suffix, data=None):
- req = urllib.request.Request(config['url'] + suffix,
- data=json.dumps(data).encode() if data is not None else None,
- headers={'Authorization': 'Bearer ' + config['token'], 'Content-Type': 'application/json'})
- with urllib.request.urlopen(req, timeout=20) as response:
- return json.load(response)
-result = {'succeeded': False}
-try:
- credential = request('/credential')
- runtime = json.loads(Path('/etc/switch-hosted/runtime.json').read_text())
- account = pwd.getpwnam(runtime['agentUser'])
- home = Path('/run/switch-verification')
- home.mkdir(mode=0o700)
- os.chown(home, account.pw_uid, account.pw_gid)
- provider = credential['provider']
- binary = runtime['providerBinaryPath'] if provider == 'claude' else runtime['providers'][provider]['path']
- process = subprocess.run(
- ['setpriv', '--reuid=' + str(account.pw_uid), '--regid=' + str(account.pw_gid), '--clear-groups',
- runtime['nodePath'], runtime['bootstrapPath'], '--verify-credential'],
- input=json.dumps({**credential, 'binaryPath': binary, 'home': str(home)}),
- text=True, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=120,
- env={'HOME': str(home), 'PATH': runtime['path']}, cwd=home)
- if process.returncode == 0:
- result = json.loads(process.stdout.splitlines()[-1])
-except Exception:
- result = {'succeeded': False}
-try:
- request('/result', result)
-finally:
- subprocess.run(['shutdown', '-P', 'now'], check=False)
-PYTHON
-"""
- )
-
-
-class VerificationWorkers:
- def __init__(self, ec2: Any, config: ControllerConfig, gateway: Gateway):
- self.ec2 = ec2
- self.config = config
- self.gateway = gateway
-
- def request(self, path: str, body: dict | None = None) -> Any:
- return self.gateway.request(path, body, prefix="/gateway/provider-verifications")
-
- def reconcile(self) -> None:
- filters = [
- {"Name": "tag:switch:managed-by", "Values": [MANAGED_BY]},
- {"Name": "tag:switch:installation-id", "Values": [self.config.installation_id]},
- ]
- instances = []
- page = self.ec2.describe_instances(Filters=filters)
- while True:
- instances.extend(
- item for group in page.get("Reservations", []) for item in group["Instances"]
- )
- if not page.get("NextToken"):
- break
- page = self.ec2.describe_instances(Filters=filters, NextToken=page["NextToken"])
- now = time.time()
- by_job: dict[str, dict] = {}
- active = 0
- for instance in instances:
- tags = {tag["Key"]: tag["Value"] for tag in instance.get("Tags", [])}
- job_id = str(UUID(tags["switch:verification-id"]))
- if instance["State"]["Name"] != "terminated":
- active += 1
- if now >= float(tags["switch:deadline"]):
- self.ec2.terminate_instances(InstanceIds=[instance["InstanceId"]])
- if job_id in by_job and by_job[job_id]["InstanceId"] != instance["InstanceId"]:
- raise RuntimeError("Multiple instances found for one connection check.")
- by_job[job_id] = instance
- for job in self.request(""):
- try:
- active = self.reconcile_job(job, by_job, now, active)
- except Exception as error:
- # A launch may have succeeded before its receipt failed.
- active += 1
- logging.error(
- "Connection check %s failed to reconcile (%s)", job["id"], type(error).__name__
- )
-
- def reconcile_job(self, job: dict, by_job: dict[str, dict], now: float, active: int) -> int:
- job_id = str(UUID(job["id"]))
- instance = by_job.get(job_id)
- if instance is None and job["instance_id"]:
- found = self.ec2.describe_instances(InstanceIds=[job["instance_id"]]).get(
- "Reservations", []
- )
- exact = [item for group in found for item in group["Instances"]]
- if len(exact) != 1:
- raise RuntimeError("Could not confirm verification instance cleanup.")
- instance = exact[0]
- if instance is not None:
- tags = {tag["Key"]: tag["Value"] for tag in instance.get("Tags", [])}
- if (
- tags.get("switch:managed-by") != MANAGED_BY
- or tags.get("switch:installation-id") != self.config.installation_id
- or tags.get("switch:verification-id") != job_id
- ):
- raise RuntimeError("Connection check instance ownership mismatch.")
- instance_id = instance["InstanceId"]
- if job["instance_id"] and job["instance_id"] != instance_id:
- raise RuntimeError("Connection check instance identity mismatch.")
- terminated = instance["State"]["Name"] == "terminated"
- if not terminated and (
- job["result"] is not None or job["state"] == "cancelled" or now >= job["deadline"]
- ):
- self.ec2.terminate_instances(InstanceIds=[instance_id])
- self.request(
- "/" + job_id + "/observe",
- {"instance_id": instance_id, "terminated": terminated},
- )
- elif job["instance_id"] or now >= job["deadline"] or job["state"] == "cancelled":
- self.request(
- "/" + job_id + "/observe",
- {"instance_id": job["instance_id"], "terminated": True},
- )
- elif active < 2:
- prepared = self.request("/" + job_id + "/prepare", {})
- tags = [
- {"Key": "switch:managed-by", "Value": MANAGED_BY},
- {"Key": "switch:installation-id", "Value": self.config.installation_id},
- {"Key": "switch:verification-id", "Value": job_id},
- {"Key": "switch:deadline", "Value": str(job["deadline"])},
- ]
- response = self.ec2.run_instances(
- ImageId=self.config.image_id,
- InstanceType=self.gateway.settings.instance_type,
- MinCount=1,
- MaxCount=1,
- ClientToken="switch-verify-" + job_id,
- SubnetId=self.config.subnet_id,
- SecurityGroupIds=list(self.config.security_group_ids),
- InstanceInitiatedShutdownBehavior="terminate",
- MetadataOptions={
- "HttpEndpoint": "enabled",
- "HttpTokens": "required",
- "HttpPutResponseHopLimit": 1,
- },
- BlockDeviceMappings=[
- {
- "DeviceName": self.config.root_device_name,
- "Ebs": {
- "Encrypted": True,
- "DeleteOnTermination": True,
- "VolumeType": "gp3",
- "VolumeSize": self.config.root_volume_gib,
- },
- }
- ],
- TagSpecifications=[
- {"ResourceType": kind, "Tags": tags}
- for kind in ("instance", "volume", "network-interface")
- ],
- UserData=user_data(self.gateway.settings.origin, job_id, prepared["token"]),
- )
- instance_id = response["Instances"][0]["InstanceId"]
- self.request(
- "/" + job_id + "/observe", {"instance_id": instance_id, "terminated": False}
- )
- active += 1
- return active
diff --git a/deploy/hosted/controller/tests/fixtures/controller.json b/deploy/hosted/controller/tests/fixtures/controller.json
index 6d79dc134..5dca21745 100644
--- a/deploy/hosted/controller/tests/fixtures/controller.json
+++ b/deploy/hosted/controller/tests/fixtures/controller.json
@@ -10,12 +10,7 @@
"max_machines": 1,
"root_volume_gib": 20,
"data_volume_gib": 40,
- "machine_slots": {
- "slot-a": {
- "instance_profile_arn": "arn:aws:iam::123456789012:instance-profile/test-worker",
- "assignment_secret_arn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:test-slot"
- }
- },
+ "instance_profile_arn": "arn:aws:iam::123456789012:instance-profile/test-worker",
"state_db_path": "/var/lib/switch-hosted/state.db",
"lock_path": "/var/lib/switch-hosted/controller.lock",
"poll_interval_seconds": 1
diff --git a/deploy/hosted/controller/tests/test_bundle_revisions.py b/deploy/hosted/controller/tests/test_bundle_revisions.py
index bc655edd8..47129dd82 100644
--- a/deploy/hosted/controller/tests/test_bundle_revisions.py
+++ b/deploy/hosted/controller/tests/test_bundle_revisions.py
@@ -2,14 +2,11 @@ from __future__ import annotations
import json
from datetime import UTC, datetime
-from unittest.mock import patch
from uuid import NAMESPACE_URL, uuid5
import pytest
-from botocore.exceptions import ClientError
from test_controller import MACHINE_ID, config
-from switch_hosted_controller.config import ConfigError
from switch_hosted_controller.gateway import CoreMachine, Gateway, GatewayConfig, GatewayError
from switch_hosted_controller.model import DesiredState, ObservedState
from switch_hosted_controller.reconciler import Reconciler
@@ -19,80 +16,35 @@ VOLUME_ID = "vol-0123456789abcdef0"
INSTANCE_ID = "i-0123456789abcdef0"
-class FakeSecrets:
- def __init__(self):
- self.versions: dict[str, dict] = {}
- self.puts: list[dict] = []
- self.promotions: list[dict] = []
- self.lose_put_response = False
-
- def describe_secret(self, SecretId):
- return {
- "VersionIdsToStages": {
- token: list(version["stages"]) for token, version in self.versions.items()
- }
- }
-
- def put_secret_value(self, SecretId, ClientRequestToken, SecretString):
- existing = self.versions.get(ClientRequestToken)
- if existing is not None:
- if existing["string"] != SecretString:
- raise ClientError({"Error": {"Code": "ResourceExistsException"}}, "PutSecretValue")
- return {"VersionId": ClientRequestToken}
- self.puts.append({"token": ClientRequestToken, "string": SecretString})
- for version in self.versions.values():
- version["stages"].discard("AWSCURRENT")
- self.versions[ClientRequestToken] = {"string": SecretString, "stages": {"AWSCURRENT"}}
- if self.lose_put_response:
- self.lose_put_response = False
- raise ConnectionError("Simulated lost PutSecretValue response")
- return {"VersionId": ClientRequestToken}
-
- def update_secret_version_stage(self, SecretId, VersionStage, MoveToVersionId, **kwargs):
- self.promotions.append({"MoveToVersionId": MoveToVersionId, **kwargs})
- for version in self.versions.values():
- version["stages"].discard(VersionStage)
- self.versions[MoveToVersionId]["stages"].add(VersionStage)
-
- def get_secret_value(self, SecretId, VersionStage):
- for token, version in self.versions.items():
- if VersionStage in version["stages"]:
- return {"SecretString": version["string"], "VersionId": token}
- raise ClientError({"Error": {"Code": "ResourceNotFoundException"}}, "GetSecretValue")
-
- def current(self) -> tuple[str, dict]:
- response = self.get_secret_value(SecretId="assignment", VersionStage="AWSCURRENT")
- return response["VersionId"], json.loads(response["SecretString"])
+def code(revision: int) -> str:
+ return f"swce_SyntheticCodeRevision{revision:04d}"
class FakeCore:
- """Core's controller routes: one machine, a revision, one capability per revision."""
+ """Core's controller routes: one machine, a revision, one enrollment code per revision."""
def __init__(self, machine_id: str):
self.machine_id = machine_id
- self.slot_id = "slot-1"
- self.generation = 1
self.state = "provisioning"
self.revision = 1
self.desired_state = "running"
self.retain_until: str | None = None
- self.capability_revision: int | None = None
- self.capability = ""
+ self.code_revision: int | None = None
+ self.code = ""
self.prepared_revisions: list[int] = []
self.observations: list[dict] = []
self.prepare_failures = 0
+ self.controller_id: str | None = None
def machine(self, revision: int | None = None) -> dict:
return {
"machine_id": self.machine_id,
- "slot_id": self.slot_id,
- "generation": self.generation,
"state": self.state,
"desired_state": self.desired_state,
"revision": self.revision if revision is None else revision,
"data_volume_id": None,
"retain_until": self.retain_until,
- "bundle_revision": self.capability_revision,
+ "bundle_revision": self.code_revision,
"owner_hint": "ignored by the controller",
}
@@ -107,18 +59,20 @@ class FakeCore:
if self.prepare_failures:
self.prepare_failures -= 1
raise GatewayError(503)
- if self.capability_revision != self.revision:
- self.capability_revision = self.revision
- self.capability = f"SYNTHETIC-CAPABILITY-REVISION-{self.revision:04d}"
self.prepared_revisions.append(self.revision)
+ if self.controller_id is not None:
+ controller = {"id": self.controller_id, "enrollment_code": None}
+ else:
+ if self.code_revision != self.revision:
+ self.code_revision = self.revision
+ self.code = code(self.revision)
+ controller = {"id": None, "enrollment_code": self.code}
return {
"machine_id": self.machine_id,
- "slot_id": self.slot_id,
- "generation": self.generation,
"revision": self.revision,
- "bundle_revision": self.capability_revision,
- "machine_capability": self.capability,
+ "bundle_revision": self.revision,
"api_endpoint": "https://switch.example.test/agent-api",
+ "controller": controller,
"extra": "ignored",
}
@@ -138,8 +92,9 @@ class FakeCloud:
self.store = store
self.volume: dict | None = None
self.instance: dict | None = None
+ self.user_data: str | None = None
self.calls: list[str] = []
- self.bundles_at_boot: list[str | None] = []
+ self.codes_at_boot: list[str] = []
def get_volume(self, machine):
return self.volume
@@ -155,6 +110,12 @@ class FakeCloud:
return VOLUME_ID
def get_instance(self, machine):
+ if (
+ machine.instance_id is None
+ and self.instance is not None
+ and self.instance["State"]["Name"] == "terminated"
+ ):
+ return None
return self.instance
def validate_image(self, machine):
@@ -163,21 +124,23 @@ class FakeCloud:
def validate_capacity(self):
pass
- def _boot(self, machine) -> None:
- current = self.store.get(machine.machine_id)
- assert current.bundle_token == current.required_bundle_token
- self.bundles_at_boot.append(current.bundle_token)
+ def _boot(self) -> None:
assert self.instance is not None and self.volume is not None
+ assert self.user_data is not None
+ self.codes_at_boot.append(json.loads(self.user_data)["controller"]["enrollmentCode"])
+ self.calls.append("boot")
self.instance["State"] = {"Name": "running"}
self.volume["State"] = "in-use"
self.volume["Attachments"] = [
- {"InstanceId": INSTANCE_ID, "Device": "/dev/sdf", "State": "attached"}
+ {"InstanceId": self.instance["InstanceId"], "Device": "/dev/sdf", "State": "attached"}
]
def run_instance(self, machine):
self.calls.append("run_instance")
+ self.user_data = machine.bundle
+ launched = self.calls.count("run_instance")
self.instance = {
- "InstanceId": INSTANCE_ID,
+ "InstanceId": INSTANCE_ID if launched == 1 else f"i-{launched:017x}",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sdf",
@@ -185,12 +148,21 @@ class FakeCloud:
}
],
}
- self._boot(machine)
- return INSTANCE_ID
+ self._boot()
+ return self.instance["InstanceId"]
+
+ def terminate_instance(self, machine):
+ self.calls.append("terminate_instance")
+ assert self.instance is not None and self.instance["State"]["Name"] == "stopped"
+ assert self.volume is not None
+ self.instance["State"] = {"Name": "terminated"}
+ self.volume["State"] = "available"
+ self.volume["Attachments"] = []
+ self.user_data = None
def start_instance(self, machine):
self.calls.append("start_instance")
- self._boot(machine)
+ self._boot()
def stop_instance(self, machine):
self.calls.append("stop_instance")
@@ -205,45 +177,47 @@ def token(machine_id: str, revision: int) -> str:
def harness(tmp_path):
cfg = config(tmp_path)
store = MachineStore(cfg.state_db_path, cfg.fingerprint())
- secrets = FakeSecrets()
core = FakeCore(MACHINE_ID)
gateway = Gateway(
GatewayConfig("https://switch.example.com", "SYNTHETIC-CONTROLLER", "m6i.large"),
cfg,
store,
- secrets,
)
gateway.request = core.request
- return store, secrets, core, gateway
+ return store, core, gateway
def sync(gateway: Gateway, core: FakeCore, revision: int | None = None) -> None:
gateway.sync_machine(CoreMachine.parse(core.machine(revision)))
+def stored_bundle(store: MachineStore) -> dict:
+ bundle = store.get(MACHINE_ID).bundle
+ assert bundle is not None
+ return json.loads(bundle)
+
+
def launched(tmp_path, *, instance_launch_issued: bool = False):
- store, secrets, core, gateway = harness(tmp_path)
+ store, core, gateway = harness(tmp_path)
sync(gateway, core)
store.record_volume(MACHINE_ID, VOLUME_ID, "us-east-1a")
sync(gateway, core)
if instance_launch_issued:
machine = store.mark_instance_launch_intent(store.get(MACHINE_ID))
machine = store.mark_instance_launch_issued(machine, datetime.now(UTC))
- store.record_instance(MACHINE_ID, INSTANCE_ID)
+ store.record_instance(MACHINE_ID, INSTANCE_ID, machine.bundle)
assert store.get(MACHINE_ID).instance_launch_issued is instance_launch_issued
- return store, secrets, core, gateway
+ return store, core, gateway
@pytest.mark.parametrize("instance_launch_issued", [False, True])
-def test_new_revision_prepares_once_and_writes_one_current_version(
- tmp_path, instance_launch_issued
-):
- store, secrets, core, gateway = launched(
- tmp_path, instance_launch_issued=instance_launch_issued
- )
+def test_new_revision_prepares_once_and_records_its_bundle(tmp_path, instance_launch_issued):
+ store, core, gateway = launched(tmp_path, instance_launch_issued=instance_launch_issued)
first = token(core.machine_id, 1)
+ first_bundle = store.get(MACHINE_ID).bundle
assert core.prepared_revisions == [1]
- assert [put["token"] for put in secrets.puts] == [first]
+ assert store.get(MACHINE_ID).bundle_token == first
+ assert stored_bundle(store)["controller"]["enrollmentCode"] == code(1)
core.revision = 2
sync(gateway, core)
@@ -251,93 +225,83 @@ def test_new_revision_prepares_once_and_writes_one_current_version(
second = token(core.machine_id, 2)
assert core.prepared_revisions == [1, 2]
- assert [put["token"] for put in secrets.puts] == [first, second]
- assert secrets.versions[second]["stages"] == {"AWSCURRENT"}
- assert secrets.versions[first]["stages"] == set()
- version_id, bundle = secrets.current()
- assert version_id == second
- assert bundle["machineCapability"] == "SYNTHETIC-CAPABILITY-REVISION-0002"
- assert bundle["assignment"]["dataVolumeId"] == VOLUME_ID
+ bundle = stored_bundle(store)
+ assert bundle == {
+ "version": 4,
+ "installationId": "test-installation",
+ "machineId": MACHINE_ID,
+ "dataVolumeId": VOLUME_ID,
+ "apiEndpoint": "https://switch.example.test/agent-api",
+ "controller": {"id": None, "enrollmentCode": code(2)},
+ }
machine = store.get(MACHINE_ID)
assert machine.required_bundle_token == machine.bundle_token == second
assert machine.required_bundle_revision == 2
+ assert machine.instance_bundle == (first_bundle if instance_launch_issued else None)
store.close()
def test_older_revision_is_a_no_op(tmp_path):
- store, secrets, core, gateway = launched(tmp_path)
+ store, core, gateway = launched(tmp_path)
core.revision = 2
sync(gateway, core)
sync(gateway, core, 1)
assert core.prepared_revisions == [1, 2]
- assert len(secrets.puts) == 2
- assert secrets.promotions == []
- assert secrets.current()[0] == token(core.machine_id, 2)
machine = store.get(MACHINE_ID)
assert machine.required_bundle_revision == 2
assert machine.bundle_token == token(core.machine_id, 2)
+ assert stored_bundle(store)["controller"]["enrollmentCode"] == code(2)
store.close()
def test_revision_moved_before_prepare_writes_nothing(tmp_path):
- store, secrets, core, gateway = launched(tmp_path)
+ store, core, gateway = launched(tmp_path)
core.revision = 3
sync(gateway, core, 2)
- assert len(secrets.puts) == 1
- assert secrets.current()[0] == token(core.machine_id, 1)
machine = store.get(MACHINE_ID)
assert machine.bundle_token == token(core.machine_id, 1)
assert machine.required_bundle_token == token(core.machine_id, 2)
+ assert stored_bundle(store)["controller"]["enrollmentCode"] == code(1)
sync(gateway, core)
assert core.prepared_revisions == [1, 3, 3]
- assert secrets.current()[0] == token(core.machine_id, 3)
assert store.get(MACHINE_ID).bundle_token == token(core.machine_id, 3)
+ assert stored_bundle(store)["controller"]["enrollmentCode"] == code(3)
store.close()
-def test_bundle_present_but_not_current_is_promoted(tmp_path):
- store, secrets, core, gateway = launched(tmp_path)
- second = token(core.machine_id, 2)
- secrets.versions[second] = {"string": "{}", "stages": set()}
+def test_failed_prepare_keeps_the_previous_bundle_until_a_retry_succeeds(tmp_path):
+ store, core, gateway = launched(tmp_path)
core.revision = 2
+ core.prepare_failures = 1
+ with pytest.raises(GatewayError):
+ sync(gateway, core)
+ machine = store.get(MACHINE_ID)
+ assert machine.required_bundle_token == token(core.machine_id, 2)
+ assert machine.bundle_token == token(core.machine_id, 1)
+ assert stored_bundle(store)["controller"]["enrollmentCode"] == code(1)
sync(gateway, core)
-
- assert core.prepared_revisions == [1]
- assert secrets.promotions == [
- {"MoveToVersionId": second, "RemoveFromVersionId": token(core.machine_id, 1)}
- ]
- assert secrets.current()[0] == second
- assert store.get(MACHINE_ID).bundle_token == second
+ assert core.prepared_revisions == [1, 2]
+ assert store.get(MACHINE_ID).bundle_token == token(core.machine_id, 2)
+ assert stored_bundle(store)["controller"]["enrollmentCode"] == code(2)
store.close()
-def test_lost_put_response_and_resource_exists_retry_keep_one_version(tmp_path):
- store, secrets, core, gateway = launched(tmp_path)
- core.revision = 2
- secrets.lose_put_response = True
- with pytest.raises(ConnectionError):
- sync(gateway, core)
- second = token(core.machine_id, 2)
- assert store.get(MACHINE_ID).bundle_token == token(core.machine_id, 1)
-
- original = secrets.describe_secret
- with patch.object(
- secrets, "describe_secret", side_effect=[{"VersionIdsToStages": {}}, original(SecretId="")]
- ):
- sync(gateway, core)
-
- assert [put["token"] for put in secrets.puts].count(second) == 1
- assert secrets.current()[1]["machineCapability"] == "SYNTHETIC-CAPABILITY-REVISION-0002"
- assert store.get(MACHINE_ID).bundle_token == second
+def test_a_bundle_for_a_superseded_revision_is_not_recorded(tmp_path):
+ store, core, gateway = launched(tmp_path)
+ store.require_bundle(MACHINE_ID, 2, token(core.machine_id, 2))
+ store.record_bundle(MACHINE_ID, token(core.machine_id, 1), '{"stale":true}')
+ machine = store.get(MACHINE_ID)
+ assert machine.bundle_token == token(core.machine_id, 1)
+ assert stored_bundle(store)["controller"]["enrollmentCode"] == code(1)
store.close()
def test_sleep_wake_refreshes_the_bundle_before_the_instance_starts(tmp_path):
- store, secrets, core, gateway = harness(tmp_path)
+ store, core, gateway = harness(tmp_path)
cloud = FakeCloud(store)
reconciler = Reconciler(store, cloud)
@@ -348,11 +312,11 @@ def test_sleep_wake_refreshes_the_bundle_before_the_instance_starts(tmp_path):
for _ in range(4):
poll()
- assert store.get(MACHINE_ID).observed_state is ObservedState.RUNNING
first = token(core.machine_id, 1)
- booted, bundle = secrets.current()
- assert booted == first
- assert bundle["machineCapability"] == "SYNTHETIC-CAPABILITY-REVISION-0001"
+ machine = store.get(MACHINE_ID)
+ assert machine.observed_state is ObservedState.RUNNING
+ assert machine.instance_bundle == machine.bundle
+ assert cloud.codes_at_boot == [code(1)]
assert core.observations[-1] == {
"state": "running",
"revision": 1,
@@ -375,44 +339,70 @@ def test_sleep_wake_refreshes_the_bundle_before_the_instance_starts(tmp_path):
core.prepare_failures = 1
poll()
assert store.get(MACHINE_ID).desired_state is DesiredState.RUNNING
+ assert "terminate_instance" not in cloud.calls
assert cloud.calls.count("start_instance") == 0
assert cloud.instance["State"]["Name"] == "stopped"
assert core.observations[-1]["state"] == "provisioning"
- assert secrets.current()[0] == first
+ assert store.get(MACHINE_ID).bundle_token == first
poll()
woken = token(core.machine_id, 3)
- version_id, bundle = secrets.current()
- assert version_id == woken
- assert bundle["machineCapability"] == "SYNTHETIC-CAPABILITY-REVISION-0003"
assert core.prepared_revisions == [1, 3]
- assert [put["token"] for put in secrets.puts] == [first, woken]
- assert cloud.calls.count("start_instance") == 1
- assert cloud.bundles_at_boot == [first, woken]
+ assert cloud.calls[-1] == "terminate_instance"
+ poll()
+ poll()
+ assert cloud.calls[-2:] == ["run_instance", "boot"]
+ assert cloud.codes_at_boot == [code(1), code(3)]
+ assert json.loads(cloud.user_data) == stored_bundle(store)
+ assert store.get(MACHINE_ID).bundle_token == woken
+ assert store.get(MACHINE_ID).instance_bundle == store.get(MACHINE_ID).bundle
poll()
machine = store.get(MACHINE_ID)
assert machine.observed_state is ObservedState.RUNNING
assert machine.instance_launch_issued
+ assert machine.data_volume_id == VOLUME_ID
+ assert machine.previous_instance_id == INSTANCE_ID
+ assert machine.recovery_count == 0
+ assert cloud.calls.count("terminate_instance") == 1
store.close()
-def test_existing_bundle_version_with_different_content_fails_loud(tmp_path):
- store, secrets, core, gateway = harness(tmp_path)
- sync(gateway, core)
- store.record_volume(MACHINE_ID, VOLUME_ID, "us-east-1a")
- first = token(core.machine_id, 1)
- secrets.versions["older"] = {"string": '{"stale":true}', "stages": {"AWSCURRENT"}}
- describe = secrets.describe_secret
+def test_an_enrolled_machine_wakes_on_the_user_data_it_enrolled_with(tmp_path):
+ store, core, gateway = harness(tmp_path)
+ cloud = FakeCloud(store)
+ reconciler = Reconciler(store, cloud)
- def describe_before_the_racing_write(SecretId):
- response = describe(SecretId=SecretId)
- secrets.versions[first] = {"string": '{"conflicting":true}', "stages": set()}
- return response
+ def poll() -> None:
+ gateway.sync_machines(gateway.machines())
+ reconciler.reconcile_all()
+ gateway.report_observations(gateway.machines())
- secrets.describe_secret = describe_before_the_racing_write
- with pytest.raises(ConfigError, match="different bundle"):
- sync(gateway, core)
- assert secrets.promotions == []
- assert secrets.current()[0] == "older"
- assert store.get(MACHINE_ID).bundle_token is None
- store.close()
+ def sleep_and_wake() -> None:
+ core.sleep()
+ poll()
+ poll()
+ assert cloud.instance is not None and cloud.instance["State"]["Name"] == "stopped"
+ core.wake()
+ poll()
+
+ for _ in range(4):
+ poll()
+ enrolled_with = store.get(MACHINE_ID).bundle
+ core.controller_id = "9a1c2b4a-0000-4000-8000-0000000000aa"
+
+ sleep_and_wake()
+ assert "terminate_instance" not in cloud.calls
+ assert cloud.calls[-2:] == ["start_instance", "boot"]
+ assert json.loads(store.get(MACHINE_ID).bundle)["controller"] == {
+ "id": core.controller_id,
+ "enrollmentCode": None,
+ }
+ assert store.get(MACHINE_ID).instance_bundle == cloud.user_data == enrolled_with
+
+ core.controller_id = None
+ sleep_and_wake()
+ poll()
+ poll()
+ assert cloud.calls[-3:] == ["terminate_instance", "run_instance", "boot"]
+ assert cloud.codes_at_boot == [code(1), code(1), code(5)]
+ assert store.get(MACHINE_ID).instance_bundle == cloud.user_data == store.get(MACHINE_ID).bundle
diff --git a/deploy/hosted/controller/tests/test_cli.py b/deploy/hosted/controller/tests/test_cli.py
index ca67a4031..d5327f256 100644
--- a/deploy/hosted/controller/tests/test_cli.py
+++ b/deploy/hosted/controller/tests/test_cli.py
@@ -1,12 +1,11 @@
-import base64
import json
+import uuid
from pathlib import Path
from unittest.mock import Mock, patch
import pytest
from switch_hosted_controller import cli
-from switch_hosted_controller.cloud import Ec2Cloud
from switch_hosted_controller.config import ControllerConfig
from switch_hosted_controller.model import DesiredState, ObservedState
from switch_hosted_controller.store import MachineStore
@@ -40,17 +39,15 @@ def observe(config_path: Path, machine_id: str, observed: ObservedState) -> None
store.close()
-def test_create_status_and_delete_are_keyed_by_slot(capsys, config_path):
- code, created = run(capsys, config_path, "create", "slot-a", "--instance-type", "m6i.large")
+def test_create_status_and_delete_are_keyed_by_machine_id(capsys, config_path):
+ code, created = run(capsys, config_path, "create", "--instance-type", "m6i.large")
assert code == 0
- assert created["slot_id"] == "slot-a"
- assert created["generation"] == 1
+ machine_id = created["machine_id"]
+ assert str(uuid.UUID(machine_id)) == machine_id
assert created["desired_state"] == "running"
assert created["retain_until"] is None
assert set(created) == {
"machine_id",
- "slot_id",
- "generation",
"instance_type",
"desired_state",
"desired_revision",
@@ -61,47 +58,66 @@ def test_create_status_and_delete_are_keyed_by_slot(capsys, config_path):
"retain_until",
"error",
}
- assert run(capsys, config_path, "status", "slot-a")[1] == created
+ assert run(capsys, config_path, "status", machine_id)[1] == created
+ code, error = run(capsys, config_path, "create", "--instance-type", "m6i.large")
+ assert code == 2
+ assert "capacity" in error["error"]
+
+ other_id = str(uuid.uuid4())
code, error = run(
- capsys, config_path, "delete", "slot-a", "--confirm-slot-id", "slot-b", "--retain-volume"
+ capsys,
+ config_path,
+ "delete",
+ machine_id,
+ "--confirm-machine-id",
+ other_id,
+ "--retain-volume",
)
assert code == 2
- assert "confirm-slot-id" in error["error"]
+ assert "confirm-machine-id" in error["error"]
code, retained = run(
- capsys, config_path, "delete", "slot-a", "--confirm-slot-id", "slot-a", "--retain-volume"
+ capsys,
+ config_path,
+ "delete",
+ machine_id,
+ "--confirm-machine-id",
+ machine_id,
+ "--retain-volume",
)
assert code == 0
assert retained["desired_state"] == "retained"
- observe(config_path, created["machine_id"], ObservedState.RETAINED)
+ observe(config_path, machine_id, ObservedState.RETAINED)
code, deleted = run(
- capsys, config_path, "delete", "slot-a", "--confirm-slot-id", "slot-a", "--delete-volume"
+ capsys,
+ config_path,
+ "delete",
+ machine_id,
+ "--confirm-machine-id",
+ machine_id,
+ "--delete-volume",
)
assert code == 0
assert deleted["desired_state"] == "deleted"
- code, error = run(capsys, config_path, "create", "slot-a", "--instance-type", "m6i.large")
- assert code == 2
- observe(config_path, created["machine_id"], ObservedState.DELETED)
- code, reused = run(capsys, config_path, "create", "slot-a", "--instance-type", "m6i.large")
+ code, replacement = run(capsys, config_path, "create", "--instance-type", "m6i.large")
assert code == 0
- assert reused["generation"] == 2
- assert reused["machine_id"] != created["machine_id"]
- assert run(capsys, config_path, "status", "slot-a")[1] == reused
+ assert replacement["machine_id"] != machine_id
+ assert run(capsys, config_path, "status", machine_id)[1]["desired_state"] == "deleted"
+ assert run(capsys, config_path, "status", replacement["machine_id"])[1] == replacement
code, listed = run(capsys, config_path, "list")
- assert [(item["slot_id"], item["generation"]) for item in listed] == [
- ("slot-a", 1),
- ("slot-a", 2),
- ]
+ assert sorted(item["machine_id"] for item in listed) == sorted(
+ [machine_id, replacement["machine_id"]]
+ )
@pytest.mark.parametrize(
"argv",
[
- ("create", "slot-a", "--instance-type", "m6i.xlarge"),
- ("create", "slot-z", "--instance-type", "m6i.large"),
- ("status", "slot-a"),
+ ("create", "--instance-type", "m6i.xlarge"),
+ ("status", "3f1c2b4a-0000-4000-8000-000000000001"),
+ ("status", "not-a-uuid"),
],
)
def test_invalid_requests_exit_2(capsys, config_path, argv):
@@ -110,50 +126,30 @@ def test_invalid_requests_exit_2(capsys, config_path, argv):
assert error["error"]
-FINGERPRINT = "sha256:" + "0123456789abcdef" * 4
-
-
def stopped_with_instance(config_path: Path, machine_id: str) -> None:
config = ControllerConfig.load(config_path)
store = MachineStore(config.state_db_path, config.fingerprint())
try:
store.record_volume(machine_id, "vol-0123456789abcdef0", config.availability_zone)
- store.record_instance(machine_id, "i-0123456789abcdef0")
+ store.record_instance(machine_id, "i-0123456789abcdef0", "bundle-1")
store.set_desired(machine_id, DesiredState.STOPPED, None)
finally:
store.close()
-def upgrade(capsys, config_path: Path, fingerprint: str):
+def upgrade(capsys, config_path: Path, machine_id: str):
return run(
capsys,
config_path,
"upgrade",
- "slot-a",
+ machine_id,
"--confirm-instance-id",
"i-0123456789abcdef0",
- "--previous-runtime-fingerprint",
- fingerprint,
)
- "fingerprint", ["0123456789abcdef" * 4, "sha256:0000", "sha256:" + "A" * 64]
-)
-def test_upgrade_refuses_a_fingerprint_not_in_the_disk_marker_format(
- capsys, config_path, fingerprint
-):
- _, created = run(capsys, config_path, "create", "slot-a", "--instance-type", "m6i.large")
- stopped_with_instance(config_path, created["machine_id"])
- with patch("switch_hosted_controller.cli.Ec2Cloud") as cloud:
- code, error = upgrade(capsys, config_path, fingerprint)
- assert code == 2
- assert "runtimeFingerprint" in error["error"]
- cloud.assert_not_called()
-
-
-def test_upgrade_writes_the_disk_marker_fingerprint_to_user_data(capsys, config_path):
- _, created = run(capsys, config_path, "create", "slot-a", "--instance-type", "m6i.large")
+def test_upgrade_moves_the_machine_onto_the_new_image(capsys, config_path):
+ _, created = run(capsys, config_path, "create", "--instance-type", "m6i.large")
stopped_with_instance(config_path, created["machine_id"])
raw = json.loads(config_path.read_text())
raw["image_id"] = "ami-11111111111111111"
@@ -164,20 +160,20 @@ def test_upgrade_writes_the_disk_marker_fingerprint_to_user_data(capsys, config_
):
cloud.return_value.get_instance.return_value = {"State": {"Name": "terminated"}}
cloud.return_value.get_volume.return_value = {"State": "available", "Attachments": []}
- code, _ = upgrade(capsys, config_path, FINGERPRINT)
+ code, upgraded = upgrade(capsys, config_path, created["machine_id"])
assert code == 0
+ assert upgraded["instance_id"] is None
+ assert upgraded["instance_seq"] == created["instance_seq"] + 1
config = ControllerConfig.load(config_path)
store = MachineStore(config.state_db_path, config.fingerprint())
try:
machine = store.get(created["machine_id"])
finally:
store.close()
- assert machine.previous_runtime_fingerprint == FINGERPRINT
- user_data = Ec2Cloud(Mock(), config)._user_data(machine)
- encoded = next(
- line.split("content: ", 1)[1] for line in user_data.splitlines() if "content: " in line
- )
- assert json.loads(base64.b64decode(encoded))["previousRuntimeFingerprint"] == FINGERPRINT
+ assert machine.image_id == "ami-11111111111111111"
+ assert machine.previous_instance_id == "i-0123456789abcdef0"
+ assert machine.data_volume_id == "vol-0123456789abcdef0"
+ assert machine.instance_bundle is None
class OneIteration:
@@ -209,7 +205,6 @@ def test_serve_lists_core_machines_once_per_poll(tmp_path, config_path):
request = Mock(return_value={"machines": []})
with (
patch("switch_hosted_controller.cli.boto3"),
- patch("switch_hosted_controller.cli.VerificationWorkers"),
patch("switch_hosted_controller.cli._touch_health"),
patch("switch_hosted_controller.cli.signal.signal"),
patch("switch_hosted_controller.cli.threading.Event", OneIteration),
diff --git a/deploy/hosted/controller/tests/test_controller.py b/deploy/hosted/controller/tests/test_controller.py
index 14c587f02..7e52aad7e 100644
--- a/deploy/hosted/controller/tests/test_controller.py
+++ b/deploy/hosted/controller/tests/test_controller.py
@@ -13,19 +13,21 @@ from botocore.stub import Stubber
from switch_hosted_controller.cloud import CloudCapacityError, CloudResourceError, Ec2Cloud
from switch_hosted_controller.config import ConfigError, ControllerConfig
from switch_hosted_controller.model import DesiredState, Machine, ObservedState
-from switch_hosted_controller.reconciler import Reconciler
-from switch_hosted_controller.store import CapacityError, MachineStore, SlotInUseError, StoreError
+from switch_hosted_controller.reconciler import Reconciler, needs_new_user_data
+from switch_hosted_controller.store import (
+ SLOT_ROWS_MESSAGE,
+ CapacityError,
+ MachineStore,
+ StoreError,
+)
MACHINE_ID = "3f1c2b4a-0000-4000-8000-000000000001"
+INSTANCE_PROFILE_ARN = "arn:aws:iam::123456789012:instance-profile/switch-machine"
-WORKER_TESTDATA = Path(__file__).parents[2] / "worker" / "testdata"
CORE_FIXTURES = (
Path(__file__).parents[4] / "core" / "tests" / "switch_core" / "fixtures" / "hosted_machines"
)
-FIXTURE_SECRET_ARN = (
- "arn:aws:secretsmanager:us-east-1:000000000000:secret:switch-hosted/inst-test/slot-a"
-)
def config(tmp_path: Path, *, max_machines: int = 1) -> ControllerConfig:
@@ -33,52 +35,15 @@ def config(tmp_path: Path, *, max_machines: int = 1) -> ControllerConfig:
def fixture_config(tmp_path: Path, instance_type: str) -> ControllerConfig:
- """The installation of the cross-stream fixtures: inst-test with the one slot slot-a."""
+ """The installation of the cross-stream fixtures: inst-test, with room for one machine."""
raw = config_dict(tmp_path, max_machines=1)
raw["installation_id"] = "inst-test"
raw["allowed_instance_types"] = [instance_type]
- raw["machine_slots"] = {
- "slot-a": {
- "instance_profile_arn": "arn:aws:iam::000000000000:instance-profile/slot-a",
- "assignment_secret_arn": FIXTURE_SECRET_ARN,
- }
- }
+ raw["instance_profile_arn"] = "arn:aws:iam::000000000000:instance-profile/switch-machine"
return ControllerConfig.from_dict(raw)
-def fixture_machine(
- tmp_path: Path, instance_type: str
-) -> tuple[ControllerConfig, MachineStore, Machine]:
- """The machine Core lists in its fixture, with its volume."""
- [listed] = json.loads((CORE_FIXTURES / "machines_response.json").read_text())["machines"]
- cfg = fixture_config(tmp_path, instance_type)
- store = MachineStore(cfg.state_db_path, cfg.fingerprint())
- slot = cfg.slot(listed["slot_id"])
- store.insert(
- machine_id=listed["machine_id"],
- slot_id=listed["slot_id"],
- generation=listed["generation"],
- core_revision=listed["revision"],
- instance_type=instance_type,
- image_id=cfg.image_id,
- assignment_secret_arn=slot.assignment_secret_arn,
- instance_profile_arn=slot.instance_profile_arn,
- max_machines=cfg.max_machines,
- )
- machine = store.record_volume(
- listed["machine_id"], listed["data_volume_id"], cfg.availability_zone
- )
- return cfg, store, machine
-
-
def config_dict(tmp_path: Path, *, max_machines: int) -> dict:
- slots = {
- f"slot-{number}": {
- "instance_profile_arn": f"arn:aws:iam::123456789012:instance-profile/worker-{number}",
- "assignment_secret_arn": f"arn:aws:secretsmanager:us-east-1:123456789012:secret:slot-{number}",
- }
- for number in range(1, max_machines + 1)
- }
return {
"installation_id": "test-installation",
"region": "us-east-1",
@@ -91,33 +56,48 @@ def config_dict(tmp_path: Path, *, max_machines: int) -> dict:
"max_machines": max_machines,
"root_volume_gib": 20,
"data_volume_gib": 40,
- "machine_slots": slots,
+ "instance_profile_arn": INSTANCE_PROFILE_ARN,
"state_db_path": str(tmp_path / "state.db"),
"lock_path": str(tmp_path / "controller.lock"),
"poll_interval_seconds": 1,
}
-def insert_machine(
- store: MachineStore, cfg: ControllerConfig, slot_id: str, generation: int, machine_id: str
-) -> Machine:
- slot = cfg.slot(slot_id)
+def insert_machine(store: MachineStore, cfg: ControllerConfig, machine_id: str) -> Machine:
return store.insert(
machine_id=machine_id,
- slot_id=slot_id,
- generation=generation,
core_revision=1,
instance_type="m6i.large",
image_id=cfg.image_id,
- assignment_secret_arn=slot.assignment_secret_arn,
- instance_profile_arn=slot.instance_profile_arn,
max_machines=cfg.max_machines,
)
+def bundle_json(machine_id: str, enrollment_code: str) -> str:
+ return json.dumps(
+ {
+ "version": 4,
+ "installationId": "test-installation",
+ "machineId": machine_id,
+ "dataVolumeId": "vol-0123456789abcdef0",
+ "apiEndpoint": "https://switch.example.test/agent-api",
+ "controller": {"id": None, "enrollmentCode": enrollment_code},
+ },
+ separators=(",", ":"),
+ )
+
+
+def with_bundle(store: MachineStore, machine_id: str, revision: int) -> Machine:
+ token = f"bundle-{revision}"
+ store.require_bundle(machine_id, revision, token)
+ return store.record_bundle(
+ machine_id, token, bundle_json(machine_id, f"swce_SyntheticCodeRevision{revision:04d}")
+ )
+
+
def store_and_machine(cfg: ControllerConfig) -> tuple[MachineStore, Machine]:
store = MachineStore(cfg.state_db_path, cfg.fingerprint())
- return store, insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ return store, insert_machine(store, cfg, MACHINE_ID)
def ec2_client():
@@ -152,7 +132,7 @@ def instance(cfg: ControllerConfig, machine, state: str):
"ImageId": cfg.image_id,
"InstanceType": machine.instance_type,
"State": {"Name": state, "Code": 16 if state == "running" else 80},
- "IamInstanceProfile": {"Arn": machine.instance_profile_arn, "Id": "AIPAEXAMPLE"},
+ "IamInstanceProfile": {"Arn": cfg.instance_profile_arn, "Id": "AIPAEXAMPLE"},
"Placement": {"AvailabilityZone": cfg.availability_zone},
"SubnetId": cfg.subnet_id,
"SecurityGroups": [{"GroupId": cfg.security_group_ids[0], "GroupName": "worker"}],
@@ -196,33 +176,15 @@ def test_capacity_counts_machines_not_deleted(tmp_path: Path):
cfg = config(tmp_path, max_machines=2)
store, first = store_and_machine(cfg)
second_id = "3f1c2b4a-0000-4000-8000-000000000002"
- second = insert_machine(store, cfg, "slot-2", 1, second_id)
+ third_id = "3f1c2b4a-0000-4000-8000-000000000003"
+ second = insert_machine(store, cfg, second_id)
with pytest.raises(CapacityError):
- store.insert(
- machine_id="3f1c2b4a-0000-4000-8000-000000000003",
- slot_id="slot-3",
- generation=1,
- core_revision=1,
- instance_type="m6i.large",
- image_id=cfg.image_id,
- assignment_secret_arn="arn:aws:secretsmanager:us-east-1:123456789012:secret:other",
- instance_profile_arn="arn:aws:iam::123456789012:instance-profile/other",
- max_machines=2,
- )
+ insert_machine(store, cfg, third_id)
+ assert store.find(third_id) is None
second = store.set_desired(second.machine_id, DesiredState.STOPPED, None)
second = store.set_observed(second, ObservedState.STOPPED, None)
store.set_desired(second.machine_id, DesiredState.DELETED, None)
- third = store.insert(
- machine_id="3f1c2b4a-0000-4000-8000-000000000003",
- slot_id="slot-3",
- generation=1,
- core_revision=1,
- instance_type="m6i.large",
- image_id=cfg.image_id,
- assignment_secret_arn="arn:aws:secretsmanager:us-east-1:123456789012:secret:other",
- instance_profile_arn="arn:aws:iam::123456789012:instance-profile/other",
- max_machines=2,
- )
+ third = insert_machine(store, cfg, third_id)
assert [machine.machine_id for machine in store.list()] == [
first.machine_id,
second_id,
@@ -231,26 +193,22 @@ def test_capacity_counts_machines_not_deleted(tmp_path: Path):
store.close()
-def test_slot_reuse_needs_a_newer_generation_after_the_old_one_is_deleted(tmp_path: Path):
- cfg = config(tmp_path)
- store, old = store_and_machine(cfg)
- reuse_id = "3f1c2b4a-0000-4000-8000-000000000002"
- old = store.set_desired(old.machine_id, DesiredState.STOPPED, None)
- old = store.set_observed(old, ObservedState.STOPPED, None)
- old = store.set_desired(old.machine_id, DesiredState.DELETED, None)
- with pytest.raises(SlotInUseError, match="generation 1"):
- insert_machine(store, cfg, "slot-1", 2, reuse_id)
- store.set_observed(old, ObservedState.DELETED, None)
- with pytest.raises(StoreError, match="not newer"):
- insert_machine(store, cfg, "slot-1", 1, reuse_id)
+def test_machines_are_keyed_by_machine_id(tmp_path: Path):
+ cfg = config(tmp_path, max_machines=2)
+ store, machine = store_and_machine(cfg)
+ other_id = "3f1c2b4a-0000-4000-8000-000000000002"
+ assert store.find(MACHINE_ID) == machine
+ assert store.find(other_id) is None
+ machine = store.set_desired(machine.machine_id, DesiredState.STOPPED, None)
+ machine = store.set_observed(machine, ObservedState.STOPPED, None)
+ machine = store.set_desired(machine.machine_id, DesiredState.DELETED, None)
+ store.set_observed(machine, ObservedState.DELETED, None)
with pytest.raises(StoreError, match="already exists"):
- insert_machine(store, cfg, "slot-1", 2, old.machine_id)
- assert store.next_generation("slot-1") == 2
- reused = insert_machine(store, cfg, "slot-1", 2, reuse_id)
- assert store.latest("slot-1") == reused
- assert store.find("slot-1", 1).machine_id == old.machine_id
+ insert_machine(store, cfg, MACHINE_ID)
+ other = insert_machine(store, cfg, other_id)
+ assert store.find(other_id) == other
cloud = Ec2Cloud(ec2_client(), cfg)
- assert cloud._token(old, "data-volume") != cloud._token(reused, "data-volume")
+ assert cloud._token(machine, "data-volume") != cloud._token(other, "data-volume")
store.close()
@@ -277,7 +235,7 @@ def test_retained_release_starts_the_next_instance_sequence(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", "bundle-1")
machine = store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
with pytest.raises(StoreError, match="terminated"):
store.release_terminated(machine)
@@ -288,6 +246,8 @@ def test_retained_release_starts_the_next_instance_sequence(tmp_path: Path):
assert released.data_volume_id == "vol-0123456789abcdef0"
assert released.instance_seq == machine.instance_seq + 1
assert released.recovery_count == 0
+ assert machine.instance_bundle == "bundle-1"
+ assert released.instance_bundle is None
store.close()
@@ -374,11 +334,11 @@ def test_cloud_capacity_counts_all_pages(tmp_path: Path):
def test_foreign_recorded_instance_fails_closed(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
foreign = instance(cfg, machine, "running")
foreign["Tags"] = [
{"Key": "switch:installation-id", "Value": "other-installation"},
- {"Key": "switch:slot-id", "Value": machine.slot_id},
+ {"Key": "switch:machine-id", "Value": machine.machine_id},
]
client = ec2_client()
with Stubber(client) as stubber:
@@ -395,7 +355,7 @@ def test_foreign_recorded_instance_fails_closed(tmp_path: Path):
def test_stop_and_start_use_only_recorded_instance(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
client = ec2_client()
with Stubber(client) as stubber:
stubber.add_response(
@@ -412,31 +372,201 @@ def test_stop_and_start_use_only_recorded_instance(tmp_path: Path):
store.close()
-def test_config_rejects_duplicate_assignment_credentials(tmp_path: Path):
- cfg = config(tmp_path, max_machines=2)
- raw = {
- **cfg.__dict__,
- "security_group_ids": list(cfg.security_group_ids),
- "allowed_instance_types": list(cfg.allowed_instance_types),
- "state_db_path": str(cfg.state_db_path),
- "lock_path": str(cfg.lock_path),
- "machine_slots": {
- slot_id: {
- "instance_profile_arn": slot.instance_profile_arn,
- "assignment_secret_arn": cfg.slot("slot-1").assignment_secret_arn,
- }
- for slot_id, slot in cfg.machine_slots.items()
- },
+REVISION_1_BUNDLE = bundle_json(MACHINE_ID, "swce_SyntheticCodeRevision0001")
+REVISION_2_BUNDLE = bundle_json(MACHINE_ID, "swce_SyntheticCodeRevision0002")
+
+
+def stopped_instance_with_bundle(cfg: ControllerConfig, instance_bundle: str | None):
+ store, machine = store_and_machine(cfg)
+ store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)
+ store.record_instance(machine.machine_id, "i-0123456789abcdef0", instance_bundle)
+ return store, with_bundle(store, machine.machine_id, 2)
+
+
[email protected]("instance_bundle", [REVISION_1_BUNDLE, None])
+def test_an_instance_with_a_stale_bundle_makes_way_for_one_launched_with_the_new_one(
+ tmp_path: Path, instance_bundle
+):
+ cfg = config(tmp_path)
+ store, machine = stopped_instance_with_bundle(cfg, instance_bundle)
+ attachment = {"InstanceId": machine.instance_id, "Device": "/dev/sdf", "State": "attached"}
+ client = ec2_client()
+ with Stubber(client) as stubber:
+ stubber.add_response(
+ "describe_volumes",
+ {"Volumes": [volume(cfg, machine, state="in-use", attachments=[attachment])]},
+ {"VolumeIds": [machine.data_volume_id]},
+ )
+ stubber.add_response(
+ "describe_instances",
+ {"Reservations": [{"Instances": [instance(cfg, machine, "stopped")]}]},
+ {"InstanceIds": [machine.instance_id]},
+ )
+ stubber.add_response(
+ "terminate_instances",
+ {"TerminatingInstances": []},
+ {"InstanceIds": [machine.instance_id]},
+ )
+ result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)
+ stubber.assert_no_pending_responses()
+ assert result.observed_state is ObservedState.PROVISIONING
+ assert result.instance_terminate_issued
+ assert result.instance_id == "i-0123456789abcdef0"
+ assert result.data_volume_id == "vol-0123456789abcdef0"
+ store.close()
+
+
+def test_starting_an_instance_with_the_current_bundle_only_starts_it(tmp_path: Path):
+ cfg = config(tmp_path)
+ store, machine = stopped_instance_with_bundle(cfg, REVISION_2_BUNDLE)
+ client = ec2_client()
+ with Stubber(client) as stubber:
+ stubber.add_response(
+ "describe_volumes",
+ {"Volumes": [volume(cfg, machine)]},
+ {"VolumeIds": [machine.data_volume_id]},
+ )
+ stubber.add_response(
+ "describe_instances",
+ {"Reservations": [{"Instances": [instance(cfg, machine, "stopped")]}]},
+ {"InstanceIds": [machine.instance_id]},
+ )
+ stubber.add_response(
+ "start_instances", {"StartingInstances": []}, {"InstanceIds": [machine.instance_id]}
+ )
+ result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)
+ stubber.assert_no_pending_responses()
+ assert result.observed_state is ObservedState.PROVISIONING
+ assert result.instance_bundle == REVISION_2_BUNDLE
+ store.close()
+
+
+def _with_controller(bundle: str, controller_id: str | None, enrollment_code: str | None) -> str:
+ value = json.loads(bundle)
+ value["controller"] = {"id": controller_id, "enrollmentCode": enrollment_code}
+ return json.dumps(value, separators=(",", ":"))
+
+
+CONTROLLER_A = "9a1c2b4a-0000-4000-8000-0000000000aa"
+CONTROLLER_B = "9a1c2b4a-0000-4000-8000-0000000000bb"
+
+
+ ("current", "wanted", "replaced"),
+ [
+ (None, REVISION_2_BUNDLE, True),
+ (REVISION_2_BUNDLE, REVISION_2_BUNDLE, False),
+ (REVISION_1_BUNDLE, REVISION_2_BUNDLE, True),
+ (REVISION_1_BUNDLE, _with_controller(REVISION_1_BUNDLE, CONTROLLER_A, None), False),
+ (
+ _with_controller(REVISION_1_BUNDLE, CONTROLLER_A, None),
+ _with_controller(REVISION_1_BUNDLE, CONTROLLER_A, None),
+ False,
+ ),
+ (
+ _with_controller(REVISION_1_BUNDLE, CONTROLLER_A, None),
+ _with_controller(REVISION_1_BUNDLE, CONTROLLER_B, None),
+ True,
+ ),
+ (_with_controller(REVISION_1_BUNDLE, CONTROLLER_A, None), REVISION_2_BUNDLE, True),
+ (
+ REVISION_1_BUNDLE,
+ REVISION_1_BUNDLE.replace("switch.example.test", "other.example.test"),
+ True,
+ ),
+ (
+ REVISION_1_BUNDLE,
+ _with_controller(
+ REVISION_1_BUNDLE.replace("switch.example.test", "other.example.test"),
+ CONTROLLER_A,
+ None,
+ ),
+ True,
+ ),
+ ],
+)
+def test_user_data_is_replaced_only_when_the_boot_would_need_it(current, wanted, replaced):
+ assert needs_new_user_data(current, wanted) is replaced
+
+
+def test_starting_an_enrolled_instance_keeps_the_user_data_it_enrolled_with(tmp_path: Path):
+ cfg = config(tmp_path)
+ store, machine = stopped_instance_with_bundle(cfg, REVISION_1_BUNDLE)
+ enrolled = _with_controller(REVISION_1_BUNDLE, CONTROLLER_A, None)
+ store.require_bundle(machine.machine_id, 3, "bundle-3")
+ machine = store.record_bundle(machine.machine_id, "bundle-3", enrolled)
+ client = ec2_client()
+ with Stubber(client) as stubber:
+ stubber.add_response(
+ "describe_volumes",
+ {"Volumes": [volume(cfg, machine)]},
+ {"VolumeIds": [machine.data_volume_id]},
+ )
+ stubber.add_response(
+ "describe_instances",
+ {"Reservations": [{"Instances": [instance(cfg, machine, "stopped")]}]},
+ {"InstanceIds": [machine.instance_id]},
+ )
+ stubber.add_response(
+ "start_instances", {"StartingInstances": []}, {"InstanceIds": [machine.instance_id]}
+ )
+ result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)
+ stubber.assert_no_pending_responses()
+ assert result.instance_bundle == REVISION_1_BUNDLE
+ store.close()
+
+
+def config_raw(tmp_path: Path, **overrides) -> dict:
+ return {**config_dict(tmp_path, max_machines=1), **overrides}
+
+
+ "arn",
+ [
+ "arn:aws:iam::123456789012:role/switch-machine",
+ "arn:aws:secretsmanager:us-east-1:123456789012:secret:switch-machine",
+ "arn:aws:iam::123456789012:instance-profile",
+ "instance-profile/switch-machine",
+ None,
+ ],
+)
+def test_config_requires_an_instance_profile_arn(tmp_path: Path, arn):
+ with pytest.raises(ConfigError, match="instance_profile_arn"):
+ ControllerConfig.from_dict(config_raw(tmp_path, instance_profile_arn=arn))
+
+
+def test_config_refuses_machine_slots(tmp_path: Path):
+ raw = config_raw(tmp_path)
+ del raw["instance_profile_arn"]
+ raw["machine_slots"] = {
+ "slot-a": {
+ "instance_profile_arn": INSTANCE_PROFILE_ARN,
+ "assignment_secret_arn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:a",
+ }
}
- with pytest.raises(ConfigError, match="secrets must be unique"):
+ with pytest.raises(
+ ConfigError, match=r"missing=\['instance_profile_arn'\], unknown=\['machine_slots'\]"
+ ):
ControllerConfig.from_dict(raw)
+ ("max_machines", "valid"), [(0, False), (1, True), (100, True), (101, False)]
+)
+def test_capacity_is_bounded_by_configuration_alone(tmp_path: Path, max_machines, valid):
+ raw = config_raw(tmp_path, max_machines=max_machines)
+ if valid:
+ assert ControllerConfig.from_dict(raw).max_machines == max_machines
+ else:
+ with pytest.raises(ConfigError, match="max_machines"):
+ ControllerConfig.from_dict(raw)
+
+
def test_recovery_requires_terminated_predecessor_and_retains_disk(tmp_path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", "bundle-1")
with pytest.raises(StoreError, match="terminated"):
store.replace_terminated(machine, unexpected=True)
machine = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)
@@ -446,11 +576,12 @@ def test_recovery_requires_terminated_predecessor_and_retains_disk(tmp_path):
assert replacement.data_volume_id == "vol-0123456789abcdef0"
assert replacement.recovery_count == 1
assert replacement.instance_seq == 1
+ assert replacement.instance_bundle is None
assert not replacement.instance_launch_issued
cloud = Ec2Cloud(ec2_client(), cfg)
assert cloud._token(machine, "instance-0") != cloud._token(replacement, "instance-1")
for index in range(2, 5):
- current = store.record_instance(machine.machine_id, f"i-{index:017x}")
+ current = store.record_instance(machine.machine_id, f"i-{index:017x}", None)
current = store.mark_instance_terminal_observed(machine.machine_id, current.instance_id)
if index == 4:
with pytest.raises(StoreError, match="limit"):
@@ -491,13 +622,94 @@ def test_store_opens_beside_deleted_legacy_rows_and_keeps_them(tmp_path):
cfg = config(tmp_path)
legacy_database(cfg.state_db_path, [("deleted", "deleted"), ("deleted", "deleted")])
store, machine = store_and_machine(cfg)
- assert machine.generation == 1
+ assert store.find(MACHINE_ID) == machine
store.close()
connection = sqlite3.connect(cfg.state_db_path)
assert connection.execute("SELECT COUNT(*) FROM agents").fetchone()[0] == 2
connection.close()
+def slot_era_database(cfg: ControllerConfig, rows: list[tuple[str, str]]) -> None:
+ cfg.state_db_path.parent.mkdir(parents=True, exist_ok=True)
+ connection = sqlite3.connect(cfg.state_db_path)
+ connection.executescript(
+ """
+ CREATE TABLE machines (
+ machine_id TEXT NOT NULL UNIQUE,
+ slot_id TEXT NOT NULL,
+ generation INTEGER NOT NULL,
+ desired_state TEXT NOT NULL,
+ observed_state TEXT NOT NULL,
+ assignment_secret_arn TEXT NOT NULL,
+ instance_profile_arn TEXT NOT NULL,
+ PRIMARY KEY (slot_id, generation)
+ );
+ CREATE UNIQUE INDEX machines_one_live_row_per_slot
+ ON machines (slot_id) WHERE observed_state <> 'deleted';
+ CREATE TABLE controller_metadata (key TEXT PRIMARY KEY, value TEXT NOT NULL);
+ """
+ )
+ connection.execute(
+ "INSERT INTO controller_metadata VALUES ('controller_fingerprint', ?)",
+ (cfg.fingerprint(),),
+ )
+ connection.executemany(
+ "INSERT INTO machines VALUES (?, 'slot-a', ?, ?, ?, 'secret-arn', 'profile-arn')",
+ [
+ (f"3f1c2b4a-0000-4000-8000-0000000000{generation:02d}", generation, *row)
+ for generation, row in enumerate(rows, start=1)
+ ],
+ )
+ connection.commit()
+ connection.close()
+
+
+def test_slot_era_database_with_only_deleted_machines_is_reopened_clean(tmp_path):
+ cfg = config(tmp_path)
+ slot_era_database(cfg, [("deleted", "deleted"), ("deleted", "deleted")])
+ store = MachineStore(cfg.state_db_path, cfg.fingerprint())
+ assert store.list() == []
+ machine = insert_machine(store, cfg, MACHINE_ID)
+ assert store.list() == [machine]
+ store.close()
+ connection = sqlite3.connect(cfg.state_db_path)
+ columns = {row[1] for row in connection.execute("PRAGMA table_info(machines)")}
+ indexes = connection.execute(
+ "SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'machines_one_live_row_per_slot'"
+ ).fetchall()
+ connection.close()
+ assert "slot_id" not in columns
+ assert {"bundle", "instance_bundle"} <= columns
+ assert "instance_bundle_token" not in columns
+ assert indexes == []
+ reopened = MachineStore(cfg.state_db_path, cfg.fingerprint())
+ assert reopened.list() == [machine]
+ reopened.close()
+ with pytest.raises(StoreError, match="immutable configuration"):
+ MachineStore(cfg.state_db_path, replace(cfg, installation_id="other").fingerprint())
+
+
+ "row",
+ [
+ ("running", "running"),
+ ("stopped", "stopped"),
+ ("retained", "retained"),
+ ("deleted", "deleting"),
+ ("running", "needs_attention"),
+ ],
+)
+def test_slot_era_database_with_a_live_machine_is_refused(tmp_path, row):
+ cfg = config(tmp_path)
+ slot_era_database(cfg, [("deleted", "deleted"), row])
+ with pytest.raises(StoreError) as raised:
+ MachineStore(cfg.state_db_path, cfg.fingerprint())
+ assert str(raised.value) == SLOT_ROWS_MESSAGE
+ connection = sqlite3.connect(cfg.state_db_path)
+ assert connection.execute("SELECT COUNT(*) FROM machines").fetchone()[0] == 2
+ connection.close()
+
+
def test_capacity_and_image_can_change_but_placement_cannot(tmp_path):
cfg = config(tmp_path)
MachineStore(cfg.state_db_path, cfg.fingerprint()).close()
@@ -515,22 +727,22 @@ def test_image_upgrade_requires_stopped_terminal_claim_and_preserves_disk(tmp_pa
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", "bundle-1")
with pytest.raises(StoreError, match="stopped"):
- store.upgrade_terminated(machine, "ami-11111111111111111", "sha256:" + "a" * 64)
+ store.upgrade_terminated(machine, "ami-11111111111111111")
machine = store.set_desired(machine.machine_id, DesiredState.STOPPED, None)
with pytest.raises(StoreError, match="terminated"):
- store.upgrade_terminated(machine, "ami-11111111111111111", "sha256:" + "a" * 64)
+ store.upgrade_terminated(machine, "ami-11111111111111111")
machine = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)
- upgraded = store.upgrade_terminated(machine, "ami-11111111111111111", "sha256:" + "a" * 64)
+ upgraded = store.upgrade_terminated(machine, "ami-11111111111111111")
assert upgraded.instance_id is None
assert upgraded.previous_instance_id == machine.instance_id
assert upgraded.data_volume_id == machine.data_volume_id
- assert upgraded.previous_runtime_fingerprint == "sha256:" + "a" * 64
assert upgraded.desired_state is DesiredState.STOPPED
assert upgraded.image_id != machine.image_id
+ assert upgraded.instance_bundle is None
with pytest.raises(StoreError, match="changed"):
- store.upgrade_terminated(machine, "ami-11111111111111111", "sha256:" + "a" * 64)
+ store.upgrade_terminated(machine, "ami-11111111111111111")
store.close()
@@ -541,7 +753,7 @@ def test_terminating_worker_waits_without_profile_or_replacement(tmp_path, desir
machine = store.record_volume(
machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone
)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
machine = store.set_desired(machine.machine_id, desired, None)
worker = instance(cfg, machine, "shutting-down")
worker.pop("IamInstanceProfile")
@@ -570,3 +782,32 @@ def test_terminating_worker_waits_without_profile_or_replacement(tmp_path, desir
assert result.error is None
stubber.assert_no_pending_responses()
store.close()
+
+
[email protected]("current", [True, False])
+def test_a_database_that_kept_bundle_tokens_keeps_the_current_instance_bundle(
+ tmp_path: Path, current: bool
+):
+ cfg = config(tmp_path)
+ store, machine = stopped_instance_with_bundle(cfg, None)
+ store.close()
+ connection = sqlite3.connect(cfg.state_db_path)
+ connection.executescript(
+ """
+ ALTER TABLE machines ADD COLUMN instance_bundle_token TEXT;
+ ALTER TABLE machines DROP COLUMN instance_bundle;
+ """
+ )
+ connection.execute(
+ "UPDATE machines SET instance_bundle_token = ?",
+ (machine.bundle_token if current else "bundle-1",),
+ )
+ connection.commit()
+ connection.close()
+ reopened = MachineStore(cfg.state_db_path, cfg.fingerprint())
+ assert reopened.get(machine.machine_id).instance_bundle == (machine.bundle if current else None)
+ reopened.close()
+ connection = sqlite3.connect(cfg.state_db_path)
+ columns = {row[1] for row in connection.execute("PRAGMA table_info(machines)")}
+ connection.close()
+ assert "instance_bundle_token" not in columns
diff --git a/deploy/hosted/controller/tests/test_gateway.py b/deploy/hosted/controller/tests/test_gateway.py
index 2593c5021..42da860ed 100644
--- a/deploy/hosted/controller/tests/test_gateway.py
+++ b/deploy/hosted/controller/tests/test_gateway.py
@@ -1,23 +1,18 @@
import json
-from dataclasses import replace
from datetime import UTC, datetime
-from pathlib import Path
-from types import SimpleNamespace
from unittest.mock import Mock, patch
from uuid import NAMESPACE_URL, uuid5
import pytest
from test_controller import (
CORE_FIXTURES,
- FIXTURE_SECRET_ARN,
MACHINE_ID,
- WORKER_TESTDATA,
config,
fixture_config,
insert_machine,
)
-from switch_hosted_controller.config import ConfigError, ControllerConfig
+from switch_hosted_controller.config import ConfigError
from switch_hosted_controller.gateway import (
CoreMachine,
Gateway,
@@ -28,17 +23,14 @@ from switch_hosted_controller.gateway import (
from switch_hosted_controller.model import DesiredState, ObservedState
from switch_hosted_controller.store import CapacityError, MachineStore
-FIXTURES = Path(__file__).parent / "fixtures"
VOLUME_ID = "vol-0123456789abcdef0"
-CAPABILITY = "SYNTHETIC-MACHINE-CAPABILITY-0123456789"
+CODE = "swce_SyntheticEnrollmentCode0000"
OTHER_MACHINE_ID = "3f1c2b4a-0000-4000-8000-000000000002"
def core_machine(**overrides) -> dict:
return {
"machine_id": MACHINE_ID,
- "slot_id": "slot-1",
- "generation": 1,
"state": "provisioning",
"desired_state": "running",
"revision": 1,
@@ -52,22 +44,19 @@ def core_machine(**overrides) -> dict:
def prepared_machine(**overrides) -> dict:
return {
"machine_id": MACHINE_ID,
- "slot_id": "slot-1",
- "generation": 1,
"revision": 1,
"bundle_revision": 1,
- "machine_capability": CAPABILITY,
"api_endpoint": "https://switch.example.test/agent-api",
+ "controller": {"id": None, "enrollment_code": CODE},
**overrides,
}
-def make_gateway(cfg, store, secrets=None, instance_type="m6i.large") -> Gateway:
+def make_gateway(cfg, store, instance_type="m6i.large") -> Gateway:
return Gateway(
GatewayConfig("https://switch.example.test", "SYNTHETIC-CONTROLLER", instance_type),
cfg,
store,
- secrets if secrets is not None else Mock(),
)
@@ -97,69 +86,34 @@ def delete_fully(store: MachineStore, machine_id: str) -> None:
store.set_observed(claim, ObservedState.DELETED, None)
-def test_launch_retry_reuses_the_slot_and_row(tmp_path):
+def test_launch_retry_reuses_the_row_and_prepares_once(tmp_path):
cfg = config(tmp_path)
store = open_store(cfg)
- saved = {}
-
- def put(**kwargs):
- saved[kwargs["ClientRequestToken"]] = kwargs["SecretString"]
- raise RuntimeError("Simulated crash after the secret write")
-
- secrets = SimpleNamespace(
- describe_secret=Mock(
- side_effect=lambda **_: {
- "VersionIdsToStages": {token: ["AWSCURRENT"] for token in saved}
- }
- ),
- put_secret_value=Mock(side_effect=put),
- )
- gateway = make_gateway(cfg, store, secrets)
+ gateway = make_gateway(cfg, store)
gateway.request = routed([core_machine()], prepared_machine())
gateway.sync_machines(gateway.machines())
- assert not saved
machine = store.get(MACHINE_ID)
- assert (machine.slot_id, machine.generation, machine.instance_type) == (
- "slot-1",
- 1,
- "m6i.large",
- )
- assert machine.image_id == cfg.image_id
- assert machine.assignment_secret_arn == cfg.slot("slot-1").assignment_secret_arn
- assert machine.instance_profile_arn == cfg.slot("slot-1").instance_profile_arn
+ assert (machine.instance_type, machine.image_id) == ("m6i.large", cfg.image_id)
+ assert (machine.bundle_token, machine.bundle) == (None, None)
+ assert not any(call.args[0].endswith("/prepare") for call in gateway.request.call_args_list)
store.record_volume(MACHINE_ID, VOLUME_ID, cfg.availability_zone)
for _ in range(3):
gateway.sync_machines(gateway.machines())
assert len(store.list()) == 1
- assert secrets.put_secret_value.call_count == 1
assert sum(call.args[0].endswith("/prepare") for call in gateway.request.call_args_list) == 1
- token = str(uuid5(NAMESPACE_URL, f"{MACHINE_ID}:1"))
- assert store.get(MACHINE_ID).bundle_token == token
- assert json.loads(saved[token]) == {
- "version": 2,
- "machineId": MACHINE_ID,
- "assignment": {
+ machine = store.get(MACHINE_ID)
+ assert machine.bundle_token == str(uuid5(NAMESPACE_URL, f"{MACHINE_ID}:1"))
+ assert machine.bundle == json.dumps(
+ {
+ "version": 4,
"installationId": cfg.installation_id,
- "slotId": "slot-1",
- "generation": 1,
+ "machineId": MACHINE_ID,
"dataVolumeId": VOLUME_ID,
+ "apiEndpoint": "https://switch.example.test/agent-api",
+ "controller": {"id": None, "enrollmentCode": CODE},
},
- "machineCapability": CAPABILITY,
- "apiEndpoint": "https://switch.example.test/agent-api",
- }
- store.close()
-
-
[email protected]("capability", [None, "", "short", "has space in it 0123", 7])
-def test_bundle_refuses_a_missing_or_malformed_capability(tmp_path, capability):
- cfg = config(tmp_path)
- store = open_store(cfg)
- machine = insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
- prepared = prepared_machine(machine_capability=capability)
- if capability is None:
- del prepared["machine_capability"]
- with pytest.raises(ConfigError, match="capability"):
- make_gateway(cfg, store).bundle(prepared, machine)
+ separators=(",", ":"),
+ )
store.close()
@@ -167,27 +121,26 @@ def test_bundle_refuses_a_missing_or_malformed_capability(tmp_path, capability):
def test_bundle_refuses_an_invalid_api_endpoint(tmp_path, endpoint):
cfg = config(tmp_path)
store = open_store(cfg)
- machine = insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ machine = insert_machine(store, cfg, MACHINE_ID)
with pytest.raises(ConfigError, match="API endpoint"):
make_gateway(cfg, store).bundle(prepared_machine(api_endpoint=endpoint), machine)
store.close()
-def test_machine_without_a_capability_writes_no_bundle(tmp_path):
+def test_machine_without_a_controller_writes_no_bundle(tmp_path):
cfg = config(tmp_path)
store = open_store(cfg)
gateway = make_gateway(cfg, store)
prepared = prepared_machine()
- del prepared["machine_capability"]
+ del prepared["controller"]
gateway.request = Mock(return_value=prepared)
- gateway.secrets.describe_secret.return_value = {"VersionIdsToStages": {}}
core = CoreMachine.parse(core_machine())
gateway.sync_machine(core)
store.record_volume(MACHINE_ID, VOLUME_ID, cfg.availability_zone)
with pytest.raises(ConfigError):
gateway.sync_machine(core)
- gateway.secrets.put_secret_value.assert_not_called()
- assert store.get(MACHINE_ID).bundle_token is None
+ machine = store.get(MACHINE_ID)
+ assert (machine.bundle_token, machine.bundle) == (None, None)
store.close()
@@ -195,8 +148,6 @@ def test_machine_without_a_capability_writes_no_bundle(tmp_path):
"prepared",
[
prepared_machine(machine_id=OTHER_MACHINE_ID),
- prepared_machine(slot_id="slot-2"),
- prepared_machine(generation=2),
prepared_machine(bundle_revision=0),
],
)
@@ -205,21 +156,19 @@ def test_prepare_for_a_different_machine_or_revision_fails_loud(tmp_path, prepar
store = open_store(cfg)
gateway = make_gateway(cfg, store)
gateway.request = Mock(return_value=prepared)
- gateway.secrets.describe_secret.return_value = {"VersionIdsToStages": {}}
with pytest.raises(ConfigError):
gateway.sync_machine(CoreMachine.parse(core_machine(data_volume_id=VOLUME_ID)))
- gateway.secrets.put_secret_value.assert_not_called()
+ machine = store.get(MACHINE_ID)
+ assert (machine.bundle_token, machine.bundle) == (None, None)
store.close()
[email protected]("failure", [GatewayError(500), RuntimeError("Secret write failed")])
[email protected]("failure", [GatewayError(500), RuntimeError("Bundle write failed")])
def test_one_failed_machine_does_not_block_other_machines(tmp_path, failure):
cfg = config(tmp_path, max_machines=2)
store = open_store(cfg)
gateway = make_gateway(cfg, store)
- gateway.request = routed(
- [core_machine(), core_machine(machine_id=OTHER_MACHINE_ID, slot_id="slot-2")]
- )
+ gateway.request = routed([core_machine(), core_machine(machine_id=OTHER_MACHINE_ID)])
gateway.sync_machine = Mock(side_effect=[failure, None])
gateway.sync_machines(gateway.machines())
assert gateway.sync_machine.call_count == 2
@@ -263,9 +212,8 @@ def test_rejected_prepare_stops_the_machine_and_reports_the_detail(tmp_path):
cfg = config(tmp_path)
store = open_store(cfg)
gateway = make_gateway(cfg, store)
- insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ insert_machine(store, cfg, MACHINE_ID)
store.record_volume(MACHINE_ID, VOLUME_ID, cfg.availability_zone)
- gateway.secrets.describe_secret.return_value = {"VersionIdsToStages": {}}
def request(path, body=None):
if path == "/machines":
@@ -295,14 +243,14 @@ def test_rejected_prepare_stops_the_machine_and_reports_the_detail(tmp_path):
def test_error_machine_being_deleted_is_still_reported(tmp_path):
cfg = config(tmp_path, max_machines=2)
store = open_store(cfg)
- insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ insert_machine(store, cfg, MACHINE_ID)
delete_fully(store, MACHINE_ID)
- insert_machine(store, cfg, "slot-2", 1, OTHER_MACHINE_ID)
+ insert_machine(store, cfg, OTHER_MACHINE_ID)
gateway = make_gateway(cfg, store)
gateway.request = routed(
[
core_machine(state="error", desired_state="deleted", revision=4),
- core_machine(machine_id=OTHER_MACHINE_ID, slot_id="slot-2", state="error", revision=4),
+ core_machine(machine_id=OTHER_MACHINE_ID, state="error", revision=4),
]
)
gateway.report_observations(gateway.machines())
@@ -332,53 +280,38 @@ def synced_from_core(tmp_path) -> tuple[MachineStore, Gateway, dict]:
[listed] = core_fixture("machines_response.json")["machines"]
cfg = fixture_config(tmp_path, "c7i.2xlarge")
store = open_store(cfg)
- secrets = Mock()
- secrets.describe_secret.return_value = {"VersionIdsToStages": {}}
- gateway = make_gateway(cfg, store, secrets, instance_type="c7i.2xlarge")
- gateway.request = routed([listed], core_fixture("prepare_response.json"))
+ gateway = make_gateway(cfg, store, instance_type="c7i.2xlarge")
+ gateway.request = routed([listed], core_fixture("prepare_controller_response.json"))
gateway.sync_machines(gateway.machines())
return store, gateway, listed
def test_core_machine_list_and_prepare_produce_the_bundle(tmp_path):
store, gateway, listed = synced_from_core(tmp_path)
- prepared = core_fixture("prepare_response.json")
+ prepared = core_fixture("prepare_controller_response.json")
core = CoreMachine.parse(listed)
- assert (core.machine_id, core.slot_id, core.desired_state) == (
- listed["machine_id"],
- "slot-a",
- "running",
- )
+ assert (core.machine_id, core.desired_state) == (listed["machine_id"], "running")
machine = store.get(listed["machine_id"])
- assert (machine.generation, machine.core_revision, machine.data_volume_id) == (
- listed["generation"],
+ assert (machine.core_revision, machine.data_volume_id) == (
listed["revision"],
listed["data_volume_id"],
)
- token = bundle_token(listed["machine_id"], listed["revision"])
- assert machine.bundle_token == token
- put = gateway.secrets.put_secret_value.call_args.kwargs
- assert (put["SecretId"], put["ClientRequestToken"]) == (FIXTURE_SECRET_ARN, token)
- bundle = json.loads(put["SecretString"])
- assert bundle == {
- "version": 2,
+ assert machine.bundle_token == bundle_token(listed["machine_id"], listed["revision"])
+ assert json.loads(machine.bundle) == {
+ "version": 4,
+ "installationId": "inst-test",
"machineId": prepared["machine_id"],
- "assignment": {
- "installationId": "inst-test",
- "slotId": prepared["slot_id"],
- "generation": prepared["generation"],
- "dataVolumeId": listed["data_volume_id"],
- },
- "machineCapability": prepared["machine_capability"],
+ "dataVolumeId": listed["data_volume_id"],
"apiEndpoint": prepared["api_endpoint"],
+ "controller": {"id": None, "enrollmentCode": prepared["controller"]["enrollment_code"]},
}
- assert bundle == json.loads((WORKER_TESTDATA / "bundle.json").read_text())
store.close()
def test_running_observation_matches_the_core_fixture(tmp_path):
store, gateway, listed = synced_from_core(tmp_path)
- store.record_instance(listed["machine_id"], "i-0123456789abcdef0")
+ machine = store.get(listed["machine_id"])
+ store.record_instance(machine.machine_id, "i-0123456789abcdef0", machine.bundle)
store.set_observed(store.get(listed["machine_id"]), ObservedState.RUNNING, None)
gateway.report_observations(gateway.machines())
path, body = gateway.request.call_args_list[-1].args
@@ -404,7 +337,7 @@ def test_running_observation_matches_the_core_fixture(tmp_path):
def test_observed_states_map_onto_core_states(tmp_path, observed, state):
cfg = config(tmp_path)
store = open_store(cfg)
- machine = insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ machine = insert_machine(store, cfg, MACHINE_ID)
store.set_observed(machine, observed, "internal detail")
gateway = make_gateway(cfg, store)
gateway.request = routed([core_machine()])
@@ -435,7 +368,7 @@ def test_observed_states_map_onto_core_states(tmp_path, observed, state):
def test_pending_is_reported_by_desired_state(tmp_path, desired, core_desired, state):
cfg = config(tmp_path)
store = open_store(cfg)
- insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ insert_machine(store, cfg, MACHINE_ID)
if desired is DesiredState.DELETED:
at_rest(store, MACHINE_ID, DesiredState.STOPPED, ObservedState.STOPPED)
store.set_desired(MACHINE_ID, desired, None)
@@ -450,7 +383,7 @@ def test_pending_is_reported_by_desired_state(tmp_path, desired, core_desired, s
def test_retained_on_the_way_to_deletion_is_reported_as_deleting(tmp_path):
cfg = config(tmp_path)
store = open_store(cfg)
- machine = insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ machine = insert_machine(store, cfg, MACHINE_ID)
store.set_observed(machine, ObservedState.RETAINED, None)
gateway = make_gateway(cfg, store)
gateway.request = routed([core_machine(desired_state="deleted")])
@@ -496,7 +429,7 @@ def test_core_desired_states_map_onto_the_row(tmp_path):
def test_deleting_a_running_machine_retains_it_first(tmp_path):
cfg = config(tmp_path)
store = open_store(cfg)
- insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ insert_machine(store, cfg, MACHINE_ID)
gateway = make_gateway(cfg, store)
gateway.sync_machine(CoreMachine.parse(core_machine(desired_state="deleted", revision=2)))
machine = store.get(MACHINE_ID)
@@ -511,7 +444,7 @@ def test_running_machine_in_error_is_stopped(tmp_path):
gateway = make_gateway(cfg, store)
gateway.sync_machine(CoreMachine.parse(core_machine(state="error")))
assert store.list() == []
- insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ insert_machine(store, cfg, MACHINE_ID)
gateway.sync_machine(CoreMachine.parse(core_machine(state="error", revision=2)))
assert store.get(MACHINE_ID).desired_state is DesiredState.STOPPED
store.close()
@@ -529,41 +462,20 @@ def test_stale_core_revision_changes_nothing(tmp_path):
store.close()
-def test_slot_reuse_waits_until_the_old_generation_is_deleted(tmp_path):
+def test_a_new_machine_takes_the_capacity_a_deleted_one_released(tmp_path):
cfg = config(tmp_path)
store = open_store(cfg)
gateway = make_gateway(cfg, store)
- insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
+ insert_machine(store, cfg, MACHINE_ID)
+ other = CoreMachine.parse(core_machine(machine_id=OTHER_MACHINE_ID))
+ with pytest.raises(CapacityError):
+ gateway.sync_machine(other)
at_rest(store, MACHINE_ID, DesiredState.STOPPED, ObservedState.STOPPED)
- old = store.set_desired(MACHINE_ID, DesiredState.DELETED, None)
- reuse = CoreMachine.parse(core_machine(machine_id=OTHER_MACHINE_ID, generation=2))
- gateway.sync_machine(reuse)
- assert [machine.generation for machine in store.list()] == [1]
- store.set_observed(old, ObservedState.DELETED, None)
- gateway.sync_machine(reuse)
- reused = store.get(OTHER_MACHINE_ID)
- assert (reused.slot_id, reused.generation) == ("slot-1", 2)
- assert reused.assignment_secret_arn == cfg.slot("slot-1").assignment_secret_arn
- store.close()
-
-
-def test_a_different_machine_at_a_stored_slot_generation_fails_loud(tmp_path):
- cfg = config(tmp_path)
- store = open_store(cfg)
- insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
- gateway = make_gateway(cfg, store)
- with pytest.raises(ConfigError, match="different machine"):
- gateway.sync_machine(CoreMachine.parse(core_machine(machine_id=OTHER_MACHINE_ID)))
- store.close()
-
-
-def test_unknown_slot_fails_loud(tmp_path):
- cfg = config(tmp_path)
- store = open_store(cfg)
- gateway = make_gateway(cfg, store)
- with pytest.raises(ConfigError, match="not in machine_slots"):
- gateway.sync_machine(CoreMachine.parse(core_machine(slot_id="slot-9")))
- assert store.list() == []
+ store.set_desired(MACHINE_ID, DesiredState.DELETED, None)
+ gateway.sync_machine(other)
+ assert [machine.machine_id for machine in store.list()] == [MACHINE_ID, OTHER_MACHINE_ID]
+ assert store.get(OTHER_MACHINE_ID).desired_state is DesiredState.RUNNING
+ assert store.get(MACHINE_ID).desired_state is DesiredState.DELETED
store.close()
@@ -572,7 +484,6 @@ def test_core_data_volume_is_adopted_and_then_cross_checked(tmp_path):
store = open_store(cfg)
gateway = make_gateway(cfg, store)
gateway.request = Mock(return_value=prepared_machine())
- gateway.secrets.describe_secret.return_value = {"VersionIdsToStages": {}}
gateway.sync_machine(CoreMachine.parse(core_machine(data_volume_id=VOLUME_ID)))
machine = store.get(MACHINE_ID)
assert machine.data_volume_id == VOLUME_ID
@@ -584,11 +495,11 @@ def test_core_data_volume_is_adopted_and_then_cross_checked(tmp_path):
def test_capacity_counts_machines_before_inserting(tmp_path):
- cfg = replace(config(tmp_path, max_machines=2), max_machines=1)
+ cfg = config(tmp_path)
store = open_store(cfg)
gateway = make_gateway(cfg, store)
gateway.sync_machine(CoreMachine.parse(core_machine(desired_state="stopped")))
- second = CoreMachine.parse(core_machine(machine_id=OTHER_MACHINE_ID, slot_id="slot-2"))
+ second = CoreMachine.parse(core_machine(machine_id=OTHER_MACHINE_ID))
with pytest.raises(CapacityError):
gateway.sync_machine(second)
store.close()
@@ -602,31 +513,11 @@ def test_instance_type_outside_the_allowed_list_is_rejected(tmp_path):
store.close()
-def test_a_live_slot_cannot_change_its_identity(tmp_path):
- cfg = config(tmp_path)
- store = open_store(cfg)
- insert_machine(store, cfg, "slot-1", 1, MACHINE_ID)
- moved = replace(
- cfg,
- machine_slots={
- "slot-1": replace(
- cfg.slot("slot-1"),
- assignment_secret_arn="arn:aws:secretsmanager:us-east-1:123456789012:secret:moved",
- )
- },
- )
- with pytest.raises(ConfigError, match="cannot change"):
- make_gateway(moved, store)
- delete_fully(store, MACHINE_ID)
- make_gateway(moved, store)
- store.close()
-
-
@pytest.mark.parametrize(
"overrides",
[
{"machine_id": "not-a-uuid"},
- {"generation": 0},
+ {"revision": 0},
{"revision": True},
{"desired_state": "restart"},
{"retain_until": "2026-01-08T00:00:00"},
@@ -638,26 +529,41 @@ def test_invalid_core_machines_are_refused(overrides):
CoreMachine.parse(core_machine(**overrides))
[email protected]("slot_id", ["ab", "Slot-1", "slot_1", "slot.1", "-slot", "s" * 41])
-def test_slot_ids_follow_the_core_and_terraform_pattern(tmp_path, slot_id):
- with pytest.raises(ConfigError, match="slot_id"):
- CoreMachine.parse(core_machine(slot_id=slot_id))
- raw = json.loads((FIXTURES / "controller.json").read_text())
- raw["machine_slots"] = {slot_id: raw["machine_slots"]["slot-a"]}
- with pytest.raises(ConfigError, match="machine_slots key"):
- ControllerConfig.from_dict(raw)
-
-
[email protected]("slot_id", ["abc", "slot-1", "0-a", "s" * 40])
-def test_valid_slot_ids_are_accepted(slot_id):
- assert CoreMachine.parse(core_machine(slot_id=slot_id)).slot_id == slot_id
-
-
def test_invalid_core_machine_does_not_block_the_others(tmp_path):
cfg = config(tmp_path)
store = open_store(cfg)
gateway = make_gateway(cfg, store)
- gateway.request = routed([core_machine(generation=0), core_machine(desired_state="stopped")])
+ gateway.request = routed([core_machine(revision=0), core_machine(desired_state="stopped")])
gateway.sync_machines(gateway.machines())
assert store.get(MACHINE_ID).desired_state is DesiredState.STOPPED
store.close()
+
+
+ "controller",
+ [
+ None,
+ {"id": None, "enrollment_code": None},
+ {"id": None, "enrollment_code": "swcc_not-a-code-at-all-0000"},
+ {"id": "not-a-uuid", "enrollment_code": None},
+ {"id": MACHINE_ID, "enrollment_code": "swce_SyntheticEnrollmentCode0000"},
+ ],
+)
+def test_a_controller_bundle_refuses_what_is_not_one(tmp_path, controller):
+ cfg = config(tmp_path)
+ store = open_store(cfg)
+ machine = insert_machine(store, cfg, MACHINE_ID)
+ prepared = {**prepared_machine(), "controller": controller}
+ with pytest.raises((ConfigError, ValueError)):
+ make_gateway(cfg, store).bundle(prepared, machine)
+ store.close()
+
+
+def test_an_enrolled_controller_machine_gets_its_controller_and_no_code(tmp_path):
+ cfg = config(tmp_path)
+ store = open_store(cfg)
+ machine = insert_machine(store, cfg, MACHINE_ID)
+ prepared = {**prepared_machine(), "controller": {"id": MACHINE_ID, "enrollment_code": None}}
+ bundle = make_gateway(cfg, store).bundle(prepared, machine)
+ assert bundle["controller"] == {"id": MACHINE_ID, "enrollmentCode": None}
+ store.close()
diff --git a/deploy/hosted/controller/tests/test_launch_request.py b/deploy/hosted/controller/tests/test_launch_request.py
index 9f9c0c1d7..2cec2188c 100644
--- a/deploy/hosted/controller/tests/test_launch_request.py
+++ b/deploy/hosted/controller/tests/test_launch_request.py
@@ -1,4 +1,3 @@
-import base64
import hashlib
import json
import re
@@ -9,32 +8,23 @@ import pytest
from botocore.stub import ANY, Stubber
from test_controller import (
MACHINE_ID,
- WORKER_TESTDATA,
config,
ec2_client,
- fixture_machine,
store_and_machine,
+ with_bundle,
)
-from switch_hosted_controller.cloud import Ec2Cloud
+from switch_hosted_controller.cloud import CloudResourceError, Ec2Cloud
from switch_hosted_controller.config import ConfigError, ControllerConfig
FIXTURES = Path(__file__).parent / "fixtures"
-def assignment(user_data: str) -> str:
- encoded = next(
- line.split("content: ", 1)[1] for line in user_data.splitlines() if "content: " in line
- )
- return base64.b64decode(encoded).decode()
-
-
-def test_run_request_is_valid_and_user_data_contains_only_assignment_refs(tmp_path: Path):
+def test_run_request_boots_from_the_stored_bundle_with_the_shared_profile(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = store.record_volume(
- machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone
- )
+ store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)
+ machine = with_bundle(store, machine.machine_id, 1)
client = ec2_client()
cloud = Ec2Cloud(client, cfg)
with Stubber(client) as stubber:
@@ -47,76 +37,59 @@ def test_run_request_is_valid_and_user_data_contains_only_assignment_refs(tmp_pa
"MinCount": 1,
"MaxCount": 1,
"ClientToken": cloud._token(machine, "instance-0"),
- "IamInstanceProfile": {"Arn": machine.instance_profile_arn},
+ "IamInstanceProfile": {"Arn": cfg.instance_profile_arn},
"Placement": ANY,
"NetworkInterfaces": ANY,
"MetadataOptions": ANY,
"BlockDeviceMappings": ANY,
"TagSpecifications": ANY,
- "UserData": cloud._user_data(machine),
+ "UserData": machine.bundle,
},
)
assert cloud.run_instance(machine) == "i-0123456789abcdef0"
- assert json.loads(assignment(cloud._user_data(machine))) == {
- "version": 2,
- "installationId": cfg.installation_id,
- "slotId": "slot-1",
- "generation": 1,
- "assignmentSecretId": machine.assignment_secret_arn,
- "dataVolumeId": machine.data_volume_id,
- "dataDevice": "/dev/sdf",
- "mountPath": "/data",
- }
+ bundle = json.loads(machine.bundle)
+ assert bundle["version"] == 4
+ assert bundle["machineId"] == MACHINE_ID
+ assert not {"slotId", "generation", "assignment", "assignmentSecretId"} & set(bundle)
store.close()
-def test_user_data_matches_the_worker_assignment_fixture(tmp_path: Path):
- cfg, store, machine = fixture_machine(tmp_path, "m6i.large")
- fixture = json.loads((WORKER_TESTDATA / "assignment.json").read_text())
- assert json.loads(assignment(Ec2Cloud(ec2_client(), cfg)._user_data(machine))) == fixture
- store.close()
-
-
-def test_user_data_carries_the_predecessor_only_when_set(tmp_path: Path):
+def test_run_request_refuses_a_machine_without_a_bundle(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- cloud = Ec2Cloud(ec2_client(), cfg)
- successor = replace(
- machine,
- previous_instance_id="i-0123456789abcdef0",
- previous_runtime_fingerprint="sha256:" + "a" * 64,
+ machine = store.record_volume(
+ machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone
)
- metadata = json.loads(assignment(cloud._user_data(successor)))
- assert metadata["previousInstanceId"] == "i-0123456789abcdef0"
- assert metadata["previousRuntimeFingerprint"] == "sha256:" + "a" * 64
- assert "previousInstanceId" not in json.loads(assignment(cloud._user_data(machine)))
+ client = ec2_client()
+ with Stubber(client) as stubber:
+ with pytest.raises(CloudResourceError, match="prepared bundle"):
+ Ec2Cloud(client, cfg).run_instance(machine)
+ stubber.assert_no_pending_responses()
store.close()
-def test_tags_filters_and_tokens_use_the_real_generation(tmp_path: Path):
+def test_tags_filters_and_tokens_are_keyed_by_machine(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = replace(machine, generation=3, instance_seq=2)
+ machine = replace(machine, instance_seq=2)
cloud = Ec2Cloud(ec2_client(), cfg)
assert cloud._tags(machine, "data") == [
{"Key": "switch:installation-id", "Value": cfg.installation_id},
- {"Key": "switch:slot-id", "Value": "slot-1"},
- {"Key": "switch:generation", "Value": "3"},
{"Key": "switch:machine-id", "Value": MACHINE_ID},
{"Key": "switch:purpose", "Value": "data"},
{"Key": "switch:managed-by", "Value": "switch-hosted-controller"},
]
assert cloud._resource_filters(machine, "worker") == [
{"Name": "tag:switch:installation-id", "Values": [cfg.installation_id]},
- {"Name": "tag:switch:slot-id", "Values": ["slot-1"]},
- {"Name": "tag:switch:generation", "Values": ["3"]},
+ {"Name": "tag:switch:machine-id", "Values": [MACHINE_ID]},
{"Name": "tag:switch:purpose", "Values": ["worker"]},
{"Name": "tag:switch:managed-by", "Values": ["switch-hosted-controller"]},
]
- material = f"{cfg.installation_id}:slot-1:3:instance-2"
+ material = f"{cfg.installation_id}:{MACHINE_ID}:instance-2"
expected = "switch-m-" + hashlib.sha256(material.encode()).hexdigest()[:48]
assert cloud._token(machine, "instance-2") == expected
+ assert cloud._launch_token(machine) == expected
store.close()
@@ -125,7 +98,7 @@ def test_machine_tokens_never_equal_per_agent_controller_tokens(tmp_path: Path,
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
token = Ec2Cloud(ec2_client(), cfg)._token(machine, resource)
- material = f"{cfg.installation_id}:{machine.slot_id}:{machine.generation}:{resource}"
+ material = f"{cfg.installation_id}:{machine.machine_id}:{resource}"
per_agent = "switch-" + hashlib.sha256(material.encode()).hexdigest()[:48]
assert re.fullmatch(r"switch-m-[0-9a-f]{48}", token)
assert len(token) <= 64
@@ -134,10 +107,24 @@ def test_machine_tokens_never_equal_per_agent_controller_tokens(tmp_path: Path,
store.close()
-def test_packaged_controller_config_uses_machine_slots():
+def test_packaged_controller_config_uses_one_instance_profile():
cfg = ControllerConfig.load(FIXTURES / "controller.json")
assert cfg.max_machines == 1
- assert list(cfg.machine_slots) == ["slot-a"]
+ assert cfg.instance_profile_arn == "arn:aws:iam::123456789012:instance-profile/test-worker"
raw = json.loads((FIXTURES / "controller.json").read_text())
+ assert ControllerConfig.from_dict({**raw, "max_machines": 2}).max_machines == 2
with pytest.raises(ConfigError, match="max_machines"):
- ControllerConfig.from_dict({**raw, "max_machines": 2})
+ ControllerConfig.from_dict({**raw, "max_machines": 101})
+
+
+def test_a_new_database_reusing_the_installation_gets_tokens_and_lookups_of_its_own(
+ tmp_path: Path,
+):
+ cfg = config(tmp_path)
+ store, machine = store_and_machine(cfg)
+ earlier = replace(machine, machine_id="00000000-0000-4000-8000-00000000beef")
+ cloud = Ec2Cloud(ec2_client(), cfg)
+ assert cloud._token(machine, "data-volume") != cloud._token(earlier, "data-volume")
+ assert cloud._launch_token(machine) != cloud._launch_token(earlier)
+ assert cloud._resource_filters(machine, "data") != cloud._resource_filters(earlier, "data")
+ store.close()
diff --git a/deploy/hosted/controller/tests/test_lifecycle_fencing.py b/deploy/hosted/controller/tests/test_lifecycle_fencing.py
index 04e75e5b2..ad6de9e27 100644
--- a/deploy/hosted/controller/tests/test_lifecycle_fencing.py
+++ b/deploy/hosted/controller/tests/test_lifecycle_fencing.py
@@ -6,7 +6,7 @@ from pathlib import Path
import pytest
from botocore.stub import Stubber
-from test_controller import config, ec2_client, store_and_machine, volume
+from test_controller import config, ec2_client, store_and_machine, volume, with_bundle
from switch_hosted_controller import reconciler as reconciler_module
from switch_hosted_controller.cloud import CloudResourceError, Ec2Cloud
@@ -28,7 +28,7 @@ def terminal_instance(cloud: Ec2Cloud, machine) -> dict:
def record_compute(store: MachineStore, machine, availability_zone: str):
machine = store.record_volume(machine.machine_id, "vol-0123456789abcdef0", availability_zone)
- return store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ return store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
def accept_delete(store: MachineStore, machine, retain_until: datetime | None):
@@ -40,7 +40,7 @@ def accept_delete(store: MachineStore, machine, retain_until: datetime | None):
def test_recorded_terminated_instance_accepts_sparse_aws_response(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
client = ec2_client()
cloud = Ec2Cloud(client, cfg)
terminated = terminal_instance(cloud, machine)
@@ -66,7 +66,7 @@ def test_recorded_terminated_instance_accepts_sparse_aws_response(tmp_path: Path
def test_recorded_terminated_instance_still_validates_identity(tmp_path: Path, corruption: str):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
client = ec2_client()
cloud = Ec2Cloud(client, cfg)
terminated = terminal_instance(cloud, machine)
@@ -74,8 +74,8 @@ def test_recorded_terminated_instance_still_validates_identity(tmp_path: Path, c
terminated["InstanceId"] = "i-fffffffffffffffff"
else:
terminated["Tags"] = copy.deepcopy(terminated["Tags"])
- next(tag for tag in terminated["Tags"] if tag["Key"] == "switch:slot-id")["Value"] = (
- "slot-foreign"
+ next(tag for tag in terminated["Tags"] if tag["Key"] == "switch:machine-id")["Value"] = (
+ "00000000-0000-4000-8000-00000000beef"
)
with Stubber(client) as stubber:
@@ -94,7 +94,7 @@ def test_terminated_then_not_found_remains_safely_stopped_and_deletable(
):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
claim = store.set_desired(machine.machine_id, DesiredState.STOPPED, None)
client = ec2_client()
cloud = Ec2Cloud(client, cfg)
@@ -169,9 +169,9 @@ class StartRaceCloud:
def test_start_callback_cannot_publish_stale_success_or_error(tmp_path: Path, start_fails: bool):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = record_compute(store, machine, cfg.availability_zone)
- store.require_bundle(machine.machine_id, 1, "bundle-1")
- store.record_bundle(machine.machine_id, "bundle-1")
+ store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)
+ bundled = with_bundle(store, machine.machine_id, 1)
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", bundled.bundle)
store.set_observed(machine, ObservedState.STOPPED)
cloud = StartRaceCloud(cfg.state_db_path, cfg.fingerprint(), fail=start_fails)
@@ -302,8 +302,7 @@ def test_running_after_retained_launches_the_next_instance_on_the_kept_volume(tm
first = record_compute(store, machine, cfg.availability_zone)
store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
Reconciler(store, DeletionCloud(["terminated"])).reconcile(machine.machine_id)
- store.require_bundle(machine.machine_id, 2, "bundle-2")
- store.record_bundle(machine.machine_id, "bundle-2")
+ bundled = with_bundle(store, machine.machine_id, 2)
store.set_desired(machine.machine_id, DesiredState.RUNNING, None)
cloud = LaunchCloud()
@@ -312,8 +311,10 @@ def test_running_after_retained_launches_the_next_instance_on_the_kept_volume(tm
assert relaunched.instance_id == "i-0000000000000000b"
assert relaunched.data_volume_id == first.data_volume_id
assert relaunched.previous_instance_id == first.instance_id
+ assert relaunched.instance_bundle == bundled.bundle
[launched] = cloud.launched
assert launched.instance_seq == 1
+ assert launched.bundle == store.get(machine.machine_id).bundle
tokens = Ec2Cloud(ec2_client(), cfg)
assert tokens._token(launched, f"instance-{launched.instance_seq}") != tokens._token(
first, f"instance-{first.instance_seq}"
@@ -372,7 +373,7 @@ def test_accepted_delete_waits_for_pending_and_deletes_volume_only_after_termina
def test_purged_recorded_instance_reads_as_not_found(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
client = ec2_client()
cloud = Ec2Cloud(client, cfg)
terminated = terminal_instance(cloud, machine)
diff --git a/deploy/hosted/controller/tests/test_retention.py b/deploy/hosted/controller/tests/test_retention.py
index 02e146dae..0f392305c 100644
--- a/deploy/hosted/controller/tests/test_retention.py
+++ b/deploy/hosted/controller/tests/test_retention.py
@@ -12,7 +12,7 @@ def test_data_volume_delete_on_termination_is_explicitly_disabled(tmp_path: Path
machine = store.record_volume(
machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone
)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
client = ec2_client()
with Stubber(client) as stubber:
stubber.add_response(
diff --git a/deploy/hosted/controller/tests/test_retries.py b/deploy/hosted/controller/tests/test_retries.py
index d7abfa41f..f5a453b35 100644
--- a/deploy/hosted/controller/tests/test_retries.py
+++ b/deploy/hosted/controller/tests/test_retries.py
@@ -5,7 +5,14 @@ from unittest.mock import Mock
import pytest
from botocore.exceptions import ClientError
from botocore.stub import Stubber
-from test_controller import config, ec2_client, instance, store_and_machine, volume
+from test_controller import (
+ config,
+ ec2_client,
+ instance,
+ store_and_machine,
+ volume,
+ with_bundle,
+)
from switch_hosted_controller.cloud import Ec2Cloud
from switch_hosted_controller.model import DesiredState, ObservedState
@@ -70,8 +77,7 @@ def ready_to_launch(tmp_path: Path):
machine = store.record_volume(
machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone
)
- machine = store.require_bundle(machine.machine_id, 1, "bundle-1")
- machine = store.record_bundle(machine.machine_id, "bundle-1")
+ machine = with_bundle(store, machine.machine_id, 1)
cloud = Mock(spec=Ec2Cloud)
cloud.get_volume.return_value = volume(cfg, machine)
cloud.get_instance.return_value = None
@@ -219,7 +225,7 @@ def launched_instance(tmp_path: Path):
machine = store.record_volume(
machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone
)
- machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
cloud = Mock()
cloud.get_volume.return_value = volume(cfg, machine)
return cfg, store, machine, cloud
@@ -249,7 +255,7 @@ def test_recovery_limit_applies_per_operation(tmp_path: Path):
for index in range(3):
replaced = reconciler.reconcile(machine.machine_id)
assert replaced.recovery_count == index + 1
- store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
reconciler.reconcile_all()
assert store.get(machine.machine_id).observed_state is ObservedState.NEEDS_ATTENTION
@@ -259,6 +265,6 @@ def test_recovery_limit_applies_per_operation(tmp_path: Path):
replaced = reconciler.reconcile(machine.machine_id)
assert replaced.instance_id is None
assert replaced.recovery_count == 0
- store.record_instance(machine.machine_id, "i-0123456789abcdef0")
+ store.record_instance(machine.machine_id, "i-0123456789abcdef0", None)
assert reconciler.reconcile(machine.machine_id).recovery_count == 1
store.close()
diff --git a/deploy/hosted/controller/tests/test_verification.py b/deploy/hosted/controller/tests/test_verification.py
deleted file mode 100644
index f3a2aea7a..000000000
--- a/deploy/hosted/controller/tests/test_verification.py
+++ /dev/null
@@ -1,133 +0,0 @@
-from types import SimpleNamespace
-from unittest.mock import Mock
-from uuid import uuid4
-
-import pytest
-from test_controller import config
-
-from switch_hosted_controller.verification import MANAGED_BY, VerificationWorkers
-
-
-def setup(tmp_path):
- cfg = config(tmp_path)
- job = dict(id=str(uuid4()), state="queued", result=None, instance_id=None, deadline=9999999999)
- gateway = SimpleNamespace(
- settings=SimpleNamespace(origin="https://switch.example.com", instance_type="m6i.large"),
- request=Mock(
- side_effect=lambda path, body=None, **kwargs: (
- [job] if path == "" else {"token": "placeholder"}
- )
- ),
- )
- ec2 = SimpleNamespace(
- describe_instances=Mock(return_value={"Reservations": []}),
- run_instances=Mock(return_value={"Instances": [{"InstanceId": "i-0123456789abcdef0"}]}),
- terminate_instances=Mock(),
- )
- return VerificationWorkers(ec2, cfg, gateway), job
-
-
-def instance(worker, job, state="running"):
- return {
- "InstanceId": "i-0123456789abcdef0",
- "State": {"Name": state},
- "Tags": [
- {"Key": key, "Value": value}
- for key, value in {
- "switch:managed-by": MANAGED_BY,
- "switch:installation-id": worker.config.installation_id,
- "switch:verification-id": job["id"],
- "switch:deadline": str(job["deadline"]),
- }.items()
- ],
- }
-
-
-def page(item):
- return {"Reservations": [{"Instances": [item]}]}
-
-
-def test_launch_has_no_iam_identity_or_persistent_disk(tmp_path):
- worker, job = setup(tmp_path)
- worker.reconcile()
- launch = worker.ec2.run_instances.call_args.kwargs
- assert "IamInstanceProfile" not in launch
- assert launch["InstanceInitiatedShutdownBehavior"] == "terminate"
- assert len(launch["BlockDeviceMappings"]) == 1
- disk = launch["BlockDeviceMappings"][0]["Ebs"]
- assert disk["Encrypted"] and disk["DeleteOnTermination"]
- assert launch["ClientToken"] == "switch-verify-" + job["id"]
- assert "--verify-credential" in launch["UserData"]
- assert len(launch["UserData"].encode()) < 16384
- assert launch["MetadataOptions"]["HttpTokens"] == "required"
-
-
-def test_restart_finds_worker_and_waits_for_termination(tmp_path):
- worker, job = setup(tmp_path)
- job["result"] = True
- item = instance(worker, job)
- worker.ec2.describe_instances.return_value = page(item)
- worker.reconcile()
- worker.ec2.run_instances.assert_not_called()
- worker.ec2.terminate_instances.assert_called_once()
- assert worker.gateway.request.call_args.args[1]["terminated"] is False
- item["State"]["Name"] = "terminated"
- worker.reconcile()
- assert worker.gateway.request.call_args.args[1]["terminated"] is True
-
-
-def test_expired_worker_is_reaped_even_when_gateway_is_down(tmp_path):
- worker, job = setup(tmp_path)
- job["deadline"] = 0
- worker.ec2.describe_instances.return_value = page(instance(worker, job))
- worker.gateway.request.side_effect = RuntimeError("Unavailable")
- with pytest.raises(RuntimeError, match="Unavailable"):
- worker.reconcile()
- worker.ec2.terminate_instances.assert_called_once()
-
-
-def test_paginated_instances_prevent_duplicate_launch(tmp_path):
- worker, job = setup(tmp_path)
- worker.ec2.describe_instances.side_effect = [
- {"Reservations": [], "NextToken": "next"},
- page(instance(worker, job)),
- ]
- worker.reconcile()
- worker.ec2.run_instances.assert_not_called()
- assert worker.ec2.describe_instances.call_args.kwargs["NextToken"] == "next"
-
-
-def test_recorded_instance_requires_matching_ownership(tmp_path, caplog):
- worker, job = setup(tmp_path)
- job["instance_id"] = "i-0123456789abcdef0"
- item = instance(worker, job)
- item["Tags"] = []
- worker.ec2.describe_instances.side_effect = [{"Reservations": []}, page(item)]
- worker.reconcile()
- assert "failed to reconcile" in caplog.text
- worker.ec2.terminate_instances.assert_not_called()
-
-
-def test_cancelled_job_terminates_worker_without_relaunch(tmp_path):
- worker, job = setup(tmp_path)
- job["state"] = "cancelled"
- worker.ec2.describe_instances.return_value = page(instance(worker, job))
- worker.reconcile()
- worker.ec2.terminate_instances.assert_called_once()
- worker.ec2.run_instances.assert_not_called()
-
-
-def test_uncertain_launch_does_not_exceed_capacity(tmp_path):
- worker, first = setup(tmp_path)
- jobs = [dict(first, id=str(uuid4())) for _ in range(3)]
-
- def request(path, body=None, **kwargs):
- if not path:
- return jobs
- if path.endswith("/prepare"):
- return {"token": "placeholder"}
- raise RuntimeError("Observation unavailable after launch")
-
- worker.gateway.request.side_effect = request
- worker.reconcile()
- assert worker.ec2.run_instances.call_count == 2
diff --git a/deploy/hosted/machine/README.md b/deploy/hosted/machine/README.md
new file mode 100644
index 000000000..3f63a3083
--- /dev/null
+++ b/deploy/hosted/machine/README.md
@@ -0,0 +1,104 @@
+# The Switch cloud machine image
+
+A Switch cloud machine runs `switch-agent-controller`, the program any machine
+runs agents with. The owner's managed agents placed on it run there, each as a
+Linux user of its own, and they use the provider logins the owner gives the
+machine.
+
+## How a machine comes up
+
+1. Core claims the machine for its owner.
+2. The hosted controller creates the data volume and asks Core to prepare the
+ machine. Until a controller enrolls for the machine, Core answers with a
+ one-time **enrollment code**. The code is bound to the machine and is valid
+ for 30 minutes. A retry of the same revision gets the same code.
+3. The hosted controller launches the instance from this image with the bundle
+ as its user data:
+
+ ```json
+ {"version": 4, "installationId": "…", "machineId": "…", "dataVolumeId": "vol-…",
+ "apiEndpoint": "https://…",
+ "controller": {"id": null, "enrollmentCode": "swce_…"}}
+ ```
+
+ Once the machine has a controller, `controller.id` names it and
+ `enrollmentCode` is null. The bundle never holds a long-lived credential.
+ An instance boots from the bundle it was launched with. When the machine
+ needs a new enrollment code or a new endpoint, the hosted controller
+ terminates the stopped instance and launches a new one with the new bundle,
+ on the same data volume. A bundle that only names the controller the
+ machine enrolled as is not one of those: the boot keeps an enrollment made
+ with the code it already has.
+4. `switch-machine-boot.service` runs as root at every boot
+ (`switch_machine_boot.py`):
+ - It reads the bundle from the instance's user data (IMDSv2) on every boot.
+ An instance with no user data was not launched by the hosted controller:
+ the boot stops there and is not retried.
+ - It waits for the data volume, which is attached after the instance
+ starts.
+ - It formats the volume only if it is blank, then mounts it on `/data`
+ (`nodev,nosuid`).
+ - It checks the volume's marker (`/data/.switch-machine.json`), so the disk
+ of one machine is never used by another.
+ - With a code, it enrolls the controller as `switch-controller` into
+ `/data/.switch-controller`. The credential and the sealing key are kept
+ there, on the machine's own volume. The SHA-256 of the code is kept in
+ `/data/.switch-controller-code`. A directory that still holds an
+ enrollment made with another code is set aside first: Switch handed over
+ a new code, so it holds no live controller for this machine. One made
+ with this same code is kept: an earlier attempt of this boot enrolled,
+ then failed before Switch linked the controller.
+ - Then, on every boot, it runs `switch-agent-controller install-service
+ --separate-users --no-block` with the agents' directories in
+ `/data/agents`. That command is idempotent, and the root volume can be
+ new after an image change. It queues the controller's start: the
+ controller's unit is ordered after the boot service, so systemd starts it
+ once the boot completes.
+5. Core links the controller to the machine as a Switch cloud controller
+ (kind `ec2`). The controller's status reports are the machine's heartbeat:
+ the first one after the instance was seen running makes the machine
+ `ready`.
+
+If the owner removes the machine's controller in the Machines page, the next
+start of the machine gets a new code, and the old controller is replaced. A
+machine whose volume holds no enrollment and whose bundle has no code cannot
+start, and its boot log says so. Removing its controller in the Machines page
+gives it a new code.
+
+## Baking the image
+
+Put Node.js 24 at `/opt/switch/node`, and the provider CLIs under
+`/opt/switch/<provider>/bin`. Agents cannot reach anything under a home
+directory. Then, as root:
+
+ install.sh <switch-agent-controller .tgz> <node-sha256> [agent-users]
+
+`install.sh` does the following:
+
+- Checks the commands the boot needs, and that polkit is present.
+- Installs the controller at `/opt/switch/controller`.
+- Creates the controller's user `switch-controller` (uid 2000), the agents'
+ group `switch-agents-2000` (gid 2001) and one user per agent,
+ `sa2000-01`… (uid 2101…).
+- Installs the boot service, and orders the controller's service after it.
+- Writes `/etc/switch-hosted/machine.json` with the controller user, Node.js, the CLI,
+ the `PATH` the controller and its agents run with, and the number of agent
+ users.
+
+The ids are fixed because the data volume outlives the root volume. Every
+instance of the image must give the files on it the same owners.
+
+`agent-users` (16 by default) is how many agents a machine can run at once.
+
+## Running it
+
+- Core needs `AGENT_MANAGEMENT_ENABLED` as well as the hosted settings; see
+ [../README.md](../README.md).
+- The hosted controller's `image_id` must be an image baked here.
+- A machine is stopped once its status reports say no session has
+ run for `HOSTED_IDLE_STOP_MINUTES`. A message addressed to one of its agents,
+ or placing an agent on it, starts it again; Switch keeps the message (up to
+ 15 minutes, in memory) until the controller is back and resumes from its
+ saved cursor.
+
+Tests: `uv run --with pytest python -m pytest tests` in this directory.
diff --git a/deploy/hosted/machine/controller-after-boot.conf b/deploy/hosted/machine/controller-after-boot.conf
new file mode 100644
index 000000000..eb4d699ac
--- /dev/null
+++ b/deploy/hosted/machine/controller-after-boot.conf
@@ -0,0 +1,5 @@
+# The controller's state is on the data volume the boot service mounts.
+[Unit]
+Requires=switch-machine-boot.service
+After=switch-machine-boot.service
+RequiresMountsFor=/data
diff --git a/deploy/hosted/machine/install.sh b/deploy/hosted/machine/install.sh
new file mode 100644
index 000000000..310dd3dc4
--- /dev/null
+++ b/deploy/hosted/machine/install.sh
@@ -0,0 +1,118 @@
+#!/bin/sh
+# Bakes a Switch cloud machine image that runs the agents controller.
+#
+# install.sh <switch-agent-controller .tgz> <node-sha256> [agent-users]
+#
+# Run as root while the AMI is built, after Node.js 24 is at /opt/switch/node
+# and the provider CLIs are under /opt/switch (never under a home directory:
+# agents run as users of their own and cannot reach one). It installs the
+# controller at /opt/switch/controller, makes the controller's user and the
+# agents' users and group with fixed ids (the data volume outlives the root
+# volume, so its files must keep meaning the same users on every instance),
+# and installs the boot service that mounts the data volume and starts the
+# controller (switch_machine_boot.py).
+set -eu
+
+if [ "$(id -u)" -ne 0 ]; then
+ echo "install.sh must run as root" >&2
+ exit 1
+fi
+if [ "$#" -lt 2 ] || [ "$#" -gt 3 ]; then
+ echo "usage: install.sh <switch-agent-controller .tgz> <node-sha256> [agent-users]" >&2
+ exit 1
+fi
+controller_package=$1
+expected_node_sha=$2
+agent_users=${3:-16}
+case "$agent_users" in
+ ''|*[!0-9]*) echo "agent-users must be a number from 1 to 99" >&2; exit 1 ;;
+esac
+if [ "$agent_users" -lt 1 ] || [ "$agent_users" -gt 99 ]; then
+ echo "agent-users must be a number from 1 to 99" >&2
+ exit 1
+fi
+case "$expected_node_sha" in
+ *[!0-9a-f]*|'') echo "node-sha256 must be a lowercase SHA-256" >&2; exit 1 ;;
+esac
+[ "${#expected_node_sha}" -eq 64 ] || { echo "node-sha256 must be a lowercase SHA-256" >&2; exit 1; }
+
+for command in python3 lsblk wipefs udevadm mkfs.ext4 findmnt systemd-mount systemctl runuser useradd groupadd getent find chown pkaction; do
+ command -v "$command" >/dev/null 2>&1 || {
+ echo "missing required image command: $command" >&2
+ exit 1
+ }
+done
+[ -d /etc/polkit-1/rules.d ] || { echo "polkit (polkitd) must be installed" >&2; exit 1; }
+
+node=/opt/switch/node/bin/node
+case "$("$node" --version)" in
+ v24.*) ;;
+ *) echo "the image must carry Node.js 24 at $node" >&2; exit 1 ;;
+esac
+[ "$(sha256sum "$node" | cut -d ' ' -f 1)" = "$expected_node_sha" ] || {
+ echo "Node.js does not match the image-build pin" >&2
+ exit 1
+}
+# npm and the controller's CLI (#!/usr/bin/env node) run on this Node.js.
+PATH="/opt/switch/node/bin:$PATH"
+export PATH
+
+# The ids every instance of this image uses: see the header.
+controller_uid=2000
+agents_gid=2001
+first_agent_uid=2101
+controller_user=switch-controller
+agents_group="switch-agents-$controller_uid"
+if ! getent passwd "$controller_user" >/dev/null; then
+ groupadd --system --gid "$controller_uid" "$controller_user"
+ useradd --system --uid "$controller_uid" --gid "$controller_uid" \
+ --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin \
+ --comment "Switch agents controller" "$controller_user"
+fi
+[ "$(id -u "$controller_user")" -eq "$controller_uid" ] || {
+ echo "$controller_user exists with another uid than $controller_uid" >&2
+ exit 1
+}
+if ! getent group "$agents_group" >/dev/null; then
+ groupadd --system --gid "$agents_gid" "$agents_group"
+fi
+slot=1
+while [ "$slot" -le "$agent_users" ]; do
+ name=$(printf 'sa%s-%02d' "$controller_uid" "$slot")
+ if ! getent passwd "$name" >/dev/null; then
+ useradd --system --uid "$((first_agent_uid + slot - 1))" --gid "$agents_group" \
+ --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin \
+ --comment "Switch agent" "$name"
+ fi
+ slot=$((slot + 1))
+done
+
+install -d -o root -g root -m 0755 /opt/switch/controller /usr/local/libexec /etc/switch-hosted /data
+npm install --global --prefix /opt/switch/controller "$controller_package"
+cli=/opt/switch/controller/bin/switch-agent-controller
+"$cli" --version >/dev/null
+
+source_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+install -o root -g root -m 0755 "$source_dir/switch_machine_boot.py" /usr/local/libexec/switch-machine-boot
+install -o root -g root -m 0644 "$source_dir/switch-machine-boot.service" /etc/systemd/system/switch-machine-boot.service
+install -d -o root -g root -m 0755 "/etc/systemd/system/switch-agent-controller-$controller_uid.service.d"
+install -o root -g root -m 0644 "$source_dir/controller-after-boot.conf" \
+ "/etc/systemd/system/switch-agent-controller-$controller_uid.service.d/after-boot.conf"
+
+provider_path=$(find /opt/switch -mindepth 2 -maxdepth 3 -type d -name bin \
+ ! -path '/opt/switch/controller/*' ! -path '/opt/switch/node/*' | sort | tr '\n' ':')
+cat > /etc/switch-hosted/machine.json <<EOF
+{
+ "version": 1,
+ "controllerUser": "$controller_user",
+ "node": "$node",
+ "cli": "$cli",
+ "path": "/opt/switch/node/bin:${provider_path}/usr/local/bin:/usr/bin:/bin",
+ "agentUsers": $agent_users
+}
+EOF
+chmod 0644 /etc/switch-hosted/machine.json
+
+systemctl daemon-reload
+systemctl enable switch-machine-boot.service
+echo "Baked: the controller $("$cli" --version), $agent_users agent users, the boot service."
diff --git a/deploy/hosted/machine/switch-machine-boot.service b/deploy/hosted/machine/switch-machine-boot.service
new file mode 100644
index 000000000..912bf53dd
--- /dev/null
+++ b/deploy/hosted/machine/switch-machine-boot.service
@@ -0,0 +1,18 @@
+[Unit]
+Description=Mount this Switch cloud machine's data volume and start its agents controller
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/bin/python3 -I /usr/local/libexec/switch-machine-boot
+TimeoutStartSec=20min
+Restart=on-failure
+RestartSec=30s
+RestartPreventExitStatus=3
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
diff --git a/deploy/hosted/machine/switch_machine_boot.py b/deploy/hosted/machine/switch_machine_boot.py
new file mode 100644
index 000000000..f5cd6f001
--- /dev/null
+++ b/deploy/hosted/machine/switch_machine_boot.py
@@ -0,0 +1,521 @@
+#!/usr/bin/env python3
+"""Boot a Switch cloud machine that runs the agents controller.
+
+Runs as root, once per boot, before the controller's service
+(`switch-machine-boot.service`). It reads the bundle the hosted controller
+gave this machine as its instance's user data, mounts the machine's own data
+volume on /data,
+enrolls `switch-agent-controller` with the one-time code in the bundle when
+the data volume holds no live enrollment yet, and sets it up to run every
+agent as a Linux user of its own (`install-service --separate-users`), its
+state and its agents' directories on /data.
+
+The bundle never holds a long-lived credential: the controller keeps the one
+it enrolls with in its data directory, on the machine's own volume.
+"""
+
+from __future__ import annotations
+
+import argparse
+import fcntl
+import hashlib
+import json
+import logging
+import os
+import re
+import stat
+import subprocess
+import sys
+import time
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+from urllib.error import HTTPError
+from urllib.parse import urlsplit
+from urllib.request import OpenerDirector, ProxyHandler, Request, build_opener
+
+logger = logging.getLogger("switch-machine-boot")
+
+DATA_MOUNT = Path("/data")
+MACHINE_CONFIG_PATH = Path("/etc/switch-hosted/machine.json")
+LOCK_PATH = Path("/run/lock/switch-machine-boot.lock")
+MARKER_NAME = ".switch-machine.json"
+CONTROLLER_DIR_NAME = ".switch-controller"
+# The SHA-256 of the code the enrollment in CONTROLLER_DIR_NAME was made with.
+CODE_RECORD_NAME = ".switch-controller-code"
+AGENTS_DIR_NAME = "agents"
+SYSTEMD_MOUNT = "/usr/bin/systemd-mount"
+VOLUME_RE = re.compile(r"^vol-[0-9a-f]{8,17}$")
+IDENTIFIER_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:@/+\-]{0,199}$")
+ENROLLMENT_CODE_RE = re.compile(r"^swce_[A-Za-z0-9_-]{16,128}$")
+UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
+MAX_JSON_BYTES = 128 * 1024
+VOLUME_WAIT_SECONDS = 600
+METADATA_WAIT_SECONDS = 300
+# Instance metadata, IMDSv2: agents cannot reach it, root can.
+METADATA_URL = "http://169.254.169.254/latest"
+# The exit status of an instance the hosted controller did not launch, which
+# has nothing to boot from: systemd does not run it again.
+NO_BUNDLE_EXIT = 3
+
+
+class BootError(RuntimeError):
+ """Booting cannot go on; the message says why."""
+
+
+class NoBundle(BootError):
+ """The instance's user data is no bundle: the hosted controller did not launch it."""
+
+
+@dataclass(frozen=True)
+class MachineConfig:
+ controller_user: str
+ node: str
+ cli: str
+ path: str
+ agent_users: int
+
+
+@dataclass(frozen=True)
+class Bundle:
+ installation_id: str
+ machine_id: str
+ volume_id: str
+ api_endpoint: str
+ controller_id: str | None
+ enrollment_code: str | None
+
+
+def _load_json(path: Path, what: str) -> Any:
+ try:
+ details = path.lstat()
+ if not stat.S_ISREG(details.st_mode) or details.st_size > MAX_JSON_BYTES:
+ raise ValueError()
+ return json.loads(path.read_text())
+ except (OSError, ValueError):
+ raise BootError(f"The {what} at {path} cannot be read.") from None
+
+
+def _strict(value: Any, keys: set[str], what: str) -> dict[str, Any]:
+ if not isinstance(value, dict) or set(value) != keys:
+ raise BootError(f"The {what} is not one this image reads.")
+ return value
+
+
+def _identifier(value: Any, what: str) -> str:
+ if not isinstance(value, str) or not IDENTIFIER_RE.fullmatch(value):
+ raise BootError(f"The {what} is invalid.")
+ return value
+
+
+def load_machine_config(path: Path = MACHINE_CONFIG_PATH) -> MachineConfig:
+ """What the image was baked with (`install.sh`)."""
+ value = _strict(
+ _load_json(path, "machine image configuration"),
+ {"version", "controllerUser", "node", "cli", "path", "agentUsers"},
+ "machine image configuration",
+ )
+ agent_users = value["agentUsers"]
+ if (
+ value["version"] != 1
+ or not isinstance(agent_users, int)
+ or isinstance(agent_users, bool)
+ or not 1 <= agent_users <= 99
+ or not isinstance(value["cli"], str)
+ or not value["cli"].startswith("/")
+ or not isinstance(value["node"], str)
+ or not value["node"].startswith("/")
+ or not isinstance(value["path"], str)
+ ):
+ raise BootError("The machine image configuration is not one this image reads.")
+ return MachineConfig(
+ controller_user=_identifier(value["controllerUser"], "controller user"),
+ node=value["node"],
+ cli=value["cli"],
+ path=value["path"],
+ agent_users=agent_users,
+ )
+
+
+def parse_bundle(raw: str) -> Bundle:
+ """The bundle for this machine, as `Gateway.bundle` in the hosted controller writes it."""
+ try:
+ value = json.loads(raw)
+ except ValueError:
+ raise NoBundle(
+ "This instance's user data is not a machine bundle: only an instance the "
+ "hosted controller launched can boot as a Switch cloud machine."
+ ) from None
+ value = _strict(
+ value,
+ {"version", "installationId", "machineId", "dataVolumeId", "apiEndpoint", "controller"},
+ "machine bundle",
+ )
+ if value["version"] != 4:
+ raise BootError("The machine bundle's version is not one this image reads.")
+ machine_id = value["machineId"]
+ if not isinstance(machine_id, str) or not UUID_RE.fullmatch(machine_id):
+ raise BootError("The machine bundle's machine id is invalid.")
+ volume_id = value["dataVolumeId"]
+ if not isinstance(volume_id, str) or not VOLUME_RE.fullmatch(volume_id):
+ raise BootError("The machine bundle's data volume is invalid.")
+ endpoint = value["apiEndpoint"]
+ url = urlsplit(endpoint) if isinstance(endpoint, str) else None
+ if url is None or url.scheme != "https" or not url.hostname:
+ raise BootError("The machine bundle's server is not an HTTPS URL.")
+ controller = _strict(
+ value["controller"], {"id", "enrollmentCode"}, "machine bundle's controller"
+ )
+ controller_id, code = controller["id"], controller["enrollmentCode"]
+ if controller_id is not None:
+ if (
+ code is not None
+ or not isinstance(controller_id, str)
+ or not UUID_RE.fullmatch(controller_id)
+ ):
+ raise BootError("The machine bundle's controller is invalid.")
+ elif not isinstance(code, str) or not ENROLLMENT_CODE_RE.fullmatch(code):
+ raise BootError("The machine bundle holds neither a controller nor an enrollment code.")
+ return Bundle(
+ installation_id=_identifier(value["installationId"], "installation id"),
+ machine_id=machine_id,
+ volume_id=volume_id,
+ api_endpoint=endpoint,
+ controller_id=controller_id,
+ enrollment_code=code,
+ )
+
+
+def _user_data(opener: OpenerDirector) -> str:
+ token_request = Request(
+ f"{METADATA_URL}/api/token",
+ method="PUT",
+ headers={"X-aws-ec2-metadata-token-ttl-seconds": "60"},
+ )
+ with opener.open(token_request, timeout=5) as response:
+ token = response.read().decode()
+ request = Request(f"{METADATA_URL}/user-data", headers={"X-aws-ec2-metadata-token": token})
+ try:
+ with opener.open(request, timeout=5) as response:
+ raw = response.read(MAX_JSON_BYTES + 1)
+ except HTTPError as error:
+ if error.code == 404:
+ raise NoBundle(
+ "This instance has no user data: only an instance the hosted controller "
+ "launched can boot as a Switch cloud machine."
+ ) from None
+ raise
+ if len(raw) > MAX_JSON_BYTES:
+ raise BootError("The instance's user data is too large to be a machine bundle.")
+ return raw.decode()
+
+
+def read_bundle(
+ opener: OpenerDirector | None = None,
+ sleep: Callable[[float], None] = time.sleep,
+ wait_seconds: float = METADATA_WAIT_SECONDS,
+) -> Bundle:
+ """The bundle in the instance's user data, read on every boot: the hosted
+ controller replaces it while the instance is stopped when the machine must
+ enroll again. Instance metadata is waited for a while."""
+ opener = opener or build_opener(ProxyHandler({}))
+ deadline = time.monotonic() + wait_seconds
+ while True:
+ try:
+ raw = _user_data(opener)
+ break
+ except BootError:
+ raise
+ except Exception as error:
+ if time.monotonic() > deadline:
+ raise BootError(f"The instance's user data cannot be read: {error}") from None
+ logger.warning("Waiting for instance metadata: %s", error)
+ sleep(5)
+ return parse_bundle(raw)
+
+
+class Commands:
+ """Runs host commands, never through a shell."""
+
+ def result(
+ self, arguments: list[str], *, env: dict[str, str] | None = None
+ ) -> subprocess.CompletedProcess[str]:
+ try:
+ return subprocess.run(
+ arguments,
+ check=False,
+ text=True,
+ capture_output=True,
+ env=env or {"PATH": "/usr/sbin:/usr/bin:/sbin:/bin", "LANG": "C"},
+ )
+ except OSError:
+ raise BootError(f"{arguments[0]} cannot be run.") from None
+
+ def run(self, arguments: list[str], *, env: dict[str, str] | None = None) -> str:
+ completed = self.result(arguments, env=env)
+ if completed.returncode != 0:
+ detail = (completed.stderr or completed.stdout).strip().splitlines()
+ raise BootError(
+ f"{arguments[0]} {arguments[1] if len(arguments) > 1 else ''} failed"
+ f"{': ' + detail[-1] if detail else ''}."
+ )
+ return completed.stdout
+
+
+def _data_device(commands: Commands, volume_id: str) -> dict[str, Any] | None:
+ """The attached disk whose serial is the volume's id, or None while it is not attached."""
+ listed = json.loads(
+ commands.run(
+ [
+ "/usr/bin/lsblk",
+ "--json",
+ "--paths",
+ "--output",
+ "PATH,TYPE,FSTYPE,UUID,SERIAL,MOUNTPOINTS",
+ ]
+ )
+ )
+ serial = volume_id.replace("-", "")
+ found = [
+ device
+ for device in listed.get("blockdevices", [])
+ if device.get("type") == "disk"
+ and str(device.get("serial") or "").lower().replace("-", "") == serial
+ ]
+ if len(found) > 1:
+ raise BootError("More than one attached disk claims to be the data volume.")
+ return found[0] if found else None
+
+
+def prepare_storage(
+ commands: Commands,
+ volume_id: str,
+ sleep: Callable[[float], None] = time.sleep,
+ wait_seconds: float = VOLUME_WAIT_SECONDS,
+) -> None:
+ """Mounts the machine's data volume on /data (nodev, nosuid), formatting it
+ only when it is blank. The volume is attached after the instance starts,
+ so it is waited for."""
+ deadline = time.monotonic() + wait_seconds
+ device = _data_device(commands, volume_id)
+ while device is None:
+ if time.monotonic() > deadline:
+ raise BootError(f"The data volume {volume_id} was not attached in time.")
+ sleep(5)
+ device = _data_device(commands, volume_id)
+ path = device.get("path")
+ if not isinstance(path, str) or not path.startswith("/dev/"):
+ raise BootError("The data volume's device path is invalid.")
+ if device.get("children"):
+ raise BootError("The data volume is partitioned; refusing to touch it.")
+ if not device.get("fstype"):
+ signatures = json.loads(
+ commands.run(["/usr/sbin/wipefs", "--json", "--no-act", path]) or "{}"
+ ).get("signatures", [])
+ if signatures or device.get("uuid"):
+ raise BootError("The data volume is not blank, but holds no filesystem.")
+ logger.info("Formatting the blank data volume %s", volume_id)
+ commands.run(["/usr/sbin/mkfs.ext4", "-q", "-m", "0", "-L", "switch-data", path])
+ commands.run(["/usr/bin/udevadm", "settle", "--timeout=30"])
+ device = _data_device(commands, volume_id)
+ if device is None:
+ raise BootError("The data volume went away while it was formatted.")
+ if device.get("fstype") != "ext4":
+ raise BootError("The data volume's filesystem is not ext4; refusing to mount it.")
+ DATA_MOUNT.mkdir(mode=0o755, exist_ok=True)
+ mounted = commands.result(
+ ["/usr/bin/findmnt", "--mountpoint", str(DATA_MOUNT), "--noheadings", "--output", "SOURCE"]
+ )
+ if mounted.returncode == 0:
+ if os.path.realpath(mounted.stdout.strip()) != os.path.realpath(path):
+ raise BootError("/data is another device's mountpoint.")
+ return
+ commands.run([SYSTEMD_MOUNT, "--type=ext4", "--options=nodev,nosuid", path, str(DATA_MOUNT)])
+
+
+def reconcile_marker(data: Path, bundle: Bundle) -> None:
+ """Records which machine the data volume belongs to, and refuses one that
+ belongs to another: an agent's data never reaches someone else's machine."""
+ marker = data / MARKER_NAME
+ wanted = {
+ "version": 1,
+ "installationId": bundle.installation_id,
+ "machineId": bundle.machine_id,
+ "volumeId": bundle.volume_id,
+ }
+ if marker.exists():
+ if marker.is_symlink():
+ raise BootError("The data volume's marker is a link.")
+ if json.loads(marker.read_text()) != wanted:
+ raise BootError("The data volume belongs to another machine; refusing to use it.")
+ return
+ temporary = data / f"{MARKER_NAME}.new"
+ temporary.write_text(json.dumps(wanted))
+ os.chmod(temporary, 0o600)
+ os.replace(temporary, marker)
+
+
+def enrolled(commands: Commands, config: MachineConfig, data_dir: Path) -> bool:
+ """Whether the data directory holds an enrollment that was not revoked."""
+ status = commands.result(
+ _as_controller(config, [*_cli(config), "status", "--data-dir", str(data_dir)]),
+ env=_environment(config),
+ )
+ if status.returncode != 0:
+ return False
+ return "Revoked at:" not in status.stdout
+
+
+def _cli(config: MachineConfig) -> list[str]:
+ """The controller's CLI, run on the image's Node.js by its path rather
+ than through `#!/usr/bin/env node` and whatever PATH there is."""
+ return [config.node, config.cli]
+
+
+def _environment(config: MachineConfig) -> dict[str, str]:
+ return {"PATH": config.path, "LANG": "C.UTF-8"}
+
+
+def _as_controller(config: MachineConfig, arguments: list[str]) -> list[str]:
+ return ["/usr/sbin/runuser", "-u", config.controller_user, "--", *arguments]
+
+
+def start_controller(
+ commands: Commands,
+ config: MachineConfig,
+ bundle: Bundle,
+ now: Callable[[], float] = time.time,
+) -> None:
+ """Enrolls the controller when it must, then sets it up and starts it."""
+ data_dir = DATA_MOUNT / CONTROLLER_DIR_NAME
+ if not data_dir.exists():
+ data_dir.mkdir(mode=0o700)
+ commands.run(["/usr/bin/chown", f"{config.controller_user}:", str(data_dir)])
+ record = DATA_MOUNT / CODE_RECORD_NAME
+ code = bundle.enrollment_code
+ if code is not None:
+ code_hash = hashlib.sha256(code.encode()).hexdigest()
+ if enrolled(commands, config, data_dir):
+ if _recorded(record) == code_hash:
+ # An earlier attempt of this boot enrolled with this very code
+ # and failed later on. Switch has not linked the controller
+ # yet, so it still hands the code over, but the code is spent.
+ logger.info("The controller is already enrolled with this code")
+ code = None
+ else:
+ # A new code, so Switch holds no live controller for this
+ # machine: whatever this directory holds was revoked there.
+ aside = DATA_MOUNT / f"{CONTROLLER_DIR_NAME}.replaced-{int(now())}"
+ logger.warning(
+ "Switch gave a new code; setting the old enrollment aside in %s", aside
+ )
+ os.replace(data_dir, aside)
+ data_dir.mkdir(mode=0o700)
+ commands.run(["/usr/bin/chown", f"{config.controller_user}:", str(data_dir)])
+ if code is not None:
+ # Recorded first: a crash right after enrolling must not leave an
+ # enrollment the next attempt would take for a revoked one.
+ _record(record, code_hash)
+ logger.info("Enrolling the controller")
+ commands.run(
+ _as_controller(
+ config,
+ [
+ *_cli(config),
+ "enroll",
+ "--server",
+ bundle.api_endpoint,
+ "--code",
+ code,
+ "--name",
+ "Switch cloud",
+ "--data-dir",
+ str(data_dir),
+ "--secret-store",
+ "file",
+ ],
+ ),
+ env=_environment(config),
+ )
+ elif bundle.enrollment_code is None and not enrolled(commands, config, data_dir):
+ raise BootError(
+ "The data volume holds no live enrollment, and Switch gave no code to enroll "
+ "with. Remove this machine's controller in the Machines page; the next start "
+ "gets a new code."
+ )
+ commands.run(
+ [
+ *_cli(config),
+ "install-service",
+ "--separate-users",
+ "--user",
+ config.controller_user,
+ "--data-dir",
+ str(data_dir),
+ "--agents-dir",
+ str(DATA_MOUNT / AGENTS_DIR_NAME),
+ "--agent-users",
+ str(config.agent_users),
+ # The controller's unit is ordered after this service: a restart
+ # waited for here would wait for this very boot to finish.
+ "--no-block",
+ ],
+ env=_environment(config),
+ )
+
+
+def _recorded(path: Path) -> str | None:
+ try:
+ if path.is_symlink():
+ return None
+ return path.read_text().strip()
+ except FileNotFoundError:
+ return None
+
+
+def _record(path: Path, code_hash: str) -> None:
+ temporary = path.with_name(f"{path.name}.new")
+ temporary.write_text(code_hash)
+ os.chmod(temporary, 0o600)
+ os.replace(temporary, path)
+
+
+def acquire_lock(path: Path = LOCK_PATH) -> Any:
+ path.parent.mkdir(mode=0o755, parents=True, exist_ok=True)
+ descriptor = os.open(path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600)
+ try:
+ fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except BlockingIOError:
+ os.close(descriptor)
+ raise BootError("Another boot of this machine is already running.") from None
+ return os.fdopen(descriptor, "r+")
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+ parser.parse_args(argv)
+ logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
+ if os.geteuid() != 0:
+ logger.error("switch-machine-boot must run as root.")
+ return 2
+ try:
+ with acquire_lock():
+ bundle = read_bundle()
+ config = load_machine_config()
+ commands = Commands()
+ prepare_storage(commands, bundle.volume_id)
+ reconcile_marker(DATA_MOUNT, bundle)
+ start_controller(commands, config, bundle)
+ except NoBundle as error:
+ logger.error("%s", error)
+ return NO_BUNDLE_EXIT
+ except BootError as error:
+ logger.error("%s", error)
+ return 1
+ logger.info("The controller is running")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/deploy/hosted/machine/tests/test_boot.py b/deploy/hosted/machine/tests/test_boot.py
new file mode 100644
index 000000000..887c03ecd
--- /dev/null
+++ b/deploy/hosted/machine/tests/test_boot.py
@@ -0,0 +1,307 @@
+"""The boot of a Switch cloud machine that runs the agents controller."""
+
+from __future__ import annotations
+
+import hashlib
+import importlib.util
+import io
+import json
+import subprocess
+import sys
+from pathlib import Path
+from urllib.error import HTTPError
+
+import pytest
+
+SCRIPT = Path(__file__).resolve().parents[1] / "switch_machine_boot.py"
+spec = importlib.util.spec_from_file_location("switch_machine_boot", SCRIPT)
+assert spec is not None and spec.loader is not None
+boot = importlib.util.module_from_spec(spec)
+sys.modules["switch_machine_boot"] = boot
+spec.loader.exec_module(boot)
+
+MACHINE_ID = "3f1c2b4a-0000-4000-8000-000000000001"
+CONTROLLER_ID = "9a1c2b4a-0000-4000-8000-0000000000aa"
+CODE = "swce_SyntheticEnrollmentCode0000"
+VOLUME_ID = "vol-0123456789abcdef0"
+CONFIG = boot.MachineConfig(
+ controller_user="switch-controller",
+ node="/opt/switch/node/bin/node",
+ cli="/opt/switch/controller/bin/switch-agent-controller",
+ path="/opt/switch/node/bin:/usr/bin:/bin",
+ agent_users=16,
+)
+
+
+def bundle(**controller) -> str:
+ return json.dumps(
+ {
+ "version": 4,
+ "installationId": "inst-test",
+ "machineId": MACHINE_ID,
+ "dataVolumeId": VOLUME_ID,
+ "apiEndpoint": "https://switch.example.test/agent-api",
+ "controller": {"id": None, "enrollmentCode": CODE, **controller},
+ }
+ )
+
+
+class FakeCommands:
+ """Answers host commands from a table, and records them."""
+
+ def __init__(self, answers: dict[str, list[tuple[int, str]]] | None = None) -> None:
+ self.calls: list[list[str]] = []
+ self.answers = answers or {}
+
+ def result(self, arguments, *, env=None):
+ self.calls.append(arguments)
+ key = next((name for name in self.answers if name in " ".join(arguments)), None)
+ code, out = (0, "")
+ if key is not None and self.answers[key]:
+ code, out = self.answers[key][0]
+ if len(self.answers[key]) > 1:
+ self.answers[key].pop(0)
+ return subprocess.CompletedProcess(arguments, code, out, "")
+
+ def run(self, arguments, *, env=None):
+ completed = self.result(arguments, env=env)
+ if completed.returncode != 0:
+ raise boot.BootError(f"{arguments[0]} failed")
+ return completed.stdout
+
+ def ran(self, word: str) -> list[list[str]]:
+ return [call for call in self.calls if word in call]
+
+
+def test_the_bundle_is_read_strictly():
+ parsed = boot.parse_bundle(bundle())
+ assert parsed == boot.Bundle(
+ installation_id="inst-test",
+ machine_id=MACHINE_ID,
+ volume_id=VOLUME_ID,
+ api_endpoint="https://switch.example.test/agent-api",
+ controller_id=None,
+ enrollment_code=CODE,
+ )
+ enrolled = boot.parse_bundle(bundle(id=CONTROLLER_ID, enrollmentCode=None))
+ assert (enrolled.controller_id, enrolled.enrollment_code) == (CONTROLLER_ID, None)
+
+
+ "raw",
+ [
+ bundle(enrollmentCode=None),
+ bundle(enrollmentCode="swcc_a-long-lived-credential-00"),
+ bundle(id=CONTROLLER_ID),
+ bundle().replace('"version": 4', '"version": 3'),
+ bundle().replace(VOLUME_ID, "vol-nope"),
+ bundle().replace("https://", "http://"),
+ json.dumps({**json.loads(bundle()), "slotId": "slot-a"}),
+ ],
+)
+def test_a_bundle_this_image_does_not_read_is_refused(raw):
+ with pytest.raises(boot.BootError):
+ boot.parse_bundle(raw)
+
+
+def test_user_data_that_is_not_a_bundle_is_not_a_cloud_machine():
+ with pytest.raises(boot.NoBundle, match="not a machine bundle"):
+ boot.parse_bundle("#cloud-config\nruncmd: []\n")
+
+
+class FakeMetadata:
+ """Instance metadata (IMDSv2), answering from a list of user data answers."""
+
+ def __init__(self, *answers) -> None:
+ self.answers = list(answers)
+ self.requests = []
+
+ def open(self, request, timeout):
+ self.requests.append(request)
+ if request.full_url.endswith("/api/token"):
+ assert request.get_method() == "PUT"
+ return io.BytesIO(b"metadata-token")
+ assert request.get_header("X-aws-ec2-metadata-token") == "metadata-token"
+ answer = self.answers.pop(0)
+ if isinstance(answer, Exception):
+ raise answer
+ return io.BytesIO(answer.encode())
+
+
+def test_the_bundle_is_read_from_the_instance_user_data_once_metadata_answers():
+ metadata = FakeMetadata(OSError("connection refused"), bundle())
+ waits = []
+ read = boot.read_bundle(metadata, sleep=waits.append, wait_seconds=60)
+ assert read.machine_id == MACHINE_ID
+ assert waits == [5]
+ assert [request.full_url for request in metadata.requests] == [
+ f"{boot.METADATA_URL}/api/token",
+ f"{boot.METADATA_URL}/user-data",
+ f"{boot.METADATA_URL}/api/token",
+ f"{boot.METADATA_URL}/user-data",
+ ]
+
+
+def test_an_instance_without_user_data_is_not_waited_for():
+ missing = HTTPError(f"{boot.METADATA_URL}/user-data", 404, "Not Found", {}, None)
+ with pytest.raises(boot.NoBundle, match="hosted controller"):
+ boot.read_bundle(FakeMetadata(missing), sleep=pytest.fail, wait_seconds=60)
+
+
+def test_metadata_that_never_answers_fails_the_boot():
+ with pytest.raises(boot.BootError, match="cannot be read"):
+ boot.read_bundle(
+ FakeMetadata(OSError("unreachable")), sleep=lambda _: None, wait_seconds=-1
+ )
+
+
+LSBLK_BLANK = json.dumps(
+ {
+ "blockdevices": [
+ {"path": "/dev/nvme0n1", "type": "disk", "serial": "vol0aaaaaaaaaaaaaaaa"},
+ {"path": "/dev/nvme1n1", "type": "disk", "serial": "vol0123456789abcdef0"},
+ ]
+ }
+)
+LSBLK_EXT4 = json.dumps(
+ {
+ "blockdevices": [
+ {
+ "path": "/dev/nvme1n1",
+ "type": "disk",
+ "serial": "vol0123456789abcdef0",
+ "fstype": "ext4",
+ "uuid": "1111",
+ }
+ ]
+ }
+)
+
+
+def test_a_blank_volume_is_formatted_and_mounted(monkeypatch, tmp_path):
+ monkeypatch.setattr(boot, "DATA_MOUNT", tmp_path / "data")
+ commands = FakeCommands(
+ {
+ "lsblk": [(0, LSBLK_BLANK), (0, LSBLK_EXT4)],
+ "wipefs": [(0, '{"signatures": []}')],
+ "findmnt": [(1, "")],
+ }
+ )
+ boot.prepare_storage(commands, VOLUME_ID, sleep=lambda _: None)
+ assert commands.ran("/usr/sbin/mkfs.ext4")[0][-1] == "/dev/nvme1n1"
+ assert commands.ran(boot.SYSTEMD_MOUNT)[0][-2:] == ["/dev/nvme1n1", str(tmp_path / "data")]
+
+
+def test_a_volume_attached_late_is_waited_for_and_a_formatted_one_kept(monkeypatch, tmp_path):
+ monkeypatch.setattr(boot, "DATA_MOUNT", tmp_path / "data")
+ commands = FakeCommands(
+ {
+ "lsblk": [(0, json.dumps({"blockdevices": []})), (0, LSBLK_EXT4)],
+ "findmnt": [(1, "")],
+ }
+ )
+ boot.prepare_storage(commands, VOLUME_ID, sleep=lambda _: None)
+ assert commands.ran("/usr/sbin/mkfs.ext4") == []
+ assert len(commands.ran(boot.SYSTEMD_MOUNT)) == 1
+
+
+def test_a_disk_with_unknown_contents_is_never_formatted(monkeypatch, tmp_path):
+ monkeypatch.setattr(boot, "DATA_MOUNT", tmp_path / "data")
+ commands = FakeCommands(
+ {"lsblk": [(0, LSBLK_BLANK)], "wipefs": [(0, '{"signatures": [{"type": "xfs"}]}')]}
+ )
+ with pytest.raises(boot.BootError, match="not blank"):
+ boot.prepare_storage(commands, VOLUME_ID, sleep=lambda _: None)
+ assert commands.ran("/usr/sbin/mkfs.ext4") == []
+
+
+def test_the_volume_marker_refuses_another_machines_disk(tmp_path):
+ boot.reconcile_marker(tmp_path, boot.parse_bundle(bundle()))
+ boot.reconcile_marker(
+ tmp_path, boot.parse_bundle(bundle(id=CONTROLLER_ID, enrollmentCode=None))
+ )
+ other = bundle().replace(MACHINE_ID, "3f1c2b4a-0000-4000-8000-000000000002")
+ with pytest.raises(boot.BootError, match="another machine"):
+ boot.reconcile_marker(tmp_path, boot.parse_bundle(other))
+
+
+def test_a_fresh_machine_enrolls_then_runs_its_agents_as_users_of_their_own(monkeypatch, tmp_path):
+ monkeypatch.setattr(boot, "DATA_MOUNT", tmp_path)
+ commands = FakeCommands({" status ": [(1, "Not enrolled.")]})
+ boot.start_controller(commands, CONFIG, boot.parse_bundle(bundle()))
+ [enroll] = commands.ran("enroll")
+ assert enroll[:6] == [
+ "/usr/sbin/runuser",
+ "-u",
+ "switch-controller",
+ "--",
+ "/opt/switch/node/bin/node",
+ "/opt/switch/controller/bin/switch-agent-controller",
+ ]
+ assert enroll[enroll.index("--code") + 1] == CODE
+ assert enroll[enroll.index("--data-dir") + 1] == str(tmp_path / ".switch-controller")
+ [setup] = commands.ran("install-service")
+ assert setup[:2] == [
+ "/opt/switch/node/bin/node",
+ "/opt/switch/controller/bin/switch-agent-controller",
+ ]
+ assert setup[setup.index("--agents-dir") + 1] == str(tmp_path / "agents")
+ assert setup[setup.index("--user") + 1] == "switch-controller"
+ assert "--no-block" in setup
+ assert commands.calls.index(enroll) < commands.calls.index(setup)
+
+
+def test_an_enrolled_machine_only_starts_its_controller(monkeypatch, tmp_path):
+ monkeypatch.setattr(boot, "DATA_MOUNT", tmp_path)
+ (tmp_path / ".switch-controller").mkdir()
+ commands = FakeCommands({" status ": [(0, "Controller: 9a1c…")]})
+ boot.start_controller(
+ commands,
+ CONFIG,
+ boot.parse_bundle(bundle(id=CONTROLLER_ID, enrollmentCode=None)),
+ )
+ assert commands.ran("enroll") == []
+ assert len(commands.ran("install-service")) == 1
+
+
+def test_a_new_code_sets_a_revoked_enrollment_aside(monkeypatch, tmp_path):
+ monkeypatch.setattr(boot, "DATA_MOUNT", tmp_path)
+ (tmp_path / ".switch-controller").mkdir()
+ (tmp_path / ".switch-controller" / "controller.db").write_text("old")
+ (tmp_path / ".switch-controller-code").write_text(hashlib.sha256(b"swce_old").hexdigest())
+ commands = FakeCommands({" status ": [(0, "Controller: old")]})
+ boot.start_controller(commands, CONFIG, boot.parse_bundle(bundle()), now=lambda: 7)
+ assert (tmp_path / ".switch-controller.replaced-7" / "controller.db").read_text() == "old"
+ assert not (tmp_path / ".switch-controller" / "controller.db").exists()
+ assert len(commands.ran("enroll")) == 1
+ assert (tmp_path / ".switch-controller-code").read_text() != hashlib.sha256(
+ b"swce_old"
+ ).hexdigest()
+
+
+def test_an_enrollment_made_with_this_code_is_kept_when_the_boot_runs_again(monkeypatch, tmp_path):
+ monkeypatch.setattr(boot, "DATA_MOUNT", tmp_path)
+ first = FakeCommands({" status ": [(1, "Not enrolled.")]})
+ boot.start_controller(first, CONFIG, boot.parse_bundle(bundle()))
+ assert len(first.ran("enroll")) == 1
+ # It failed after enrolling: systemd starts the boot again, and Switch,
+ # which has not linked the controller yet, still hands over the code.
+ (tmp_path / ".switch-controller" / "controller.db").write_text("enrolled")
+ again = FakeCommands({" status ": [(0, "Controller: new")]})
+ boot.start_controller(again, CONFIG, boot.parse_bundle(bundle()), now=lambda: 7)
+ assert again.ran("enroll") == []
+ assert not (tmp_path / ".switch-controller.replaced-7").exists()
+ assert (tmp_path / ".switch-controller" / "controller.db").read_text() == "enrolled"
+ assert len(again.ran("install-service")) == 1
+
+
+def test_no_enrollment_and_no_code_says_what_to_do(monkeypatch, tmp_path):
+ monkeypatch.setattr(boot, "DATA_MOUNT", tmp_path)
+ commands = FakeCommands({" status ": [(1, "Not enrolled.")]})
+ with pytest.raises(boot.BootError, match="Machines page"):
+ boot.start_controller(
+ commands,
+ CONFIG,
+ boot.parse_bundle(bundle(id=CONTROLLER_ID, enrollmentCode=None)),
+ )
+ assert commands.ran("install-service") == []
diff --git a/deploy/hosted/terraform/main.tf b/deploy/hosted/terraform/main.tf
index 081ca347b..5424a657b 100644
--- a/deploy/hosted/terraform/main.tf
+++ b/deploy/hosted/terraform/main.tf
@@ -23,32 +23,17 @@ resource "aws_vpc_security_group_egress_rule" "worker" {
to_port = tonumber(each.value)
ip_protocol = "tcp"
}
-resource "aws_iam_role" "worker" {
- for_each = var.machine_slots
- name = "${local.prefix}-${substr(each.key, 0, 12)}-${substr(sha256(each.key), 0, 8)}"
+resource "aws_iam_role" "machine" {
+ name = "${local.prefix}-machine"
assume_role_policy = jsonencode({ Version = "2012-10-17", Statement = [{
Effect = "Allow", Principal = { Service = "ec2.amazonaws.com" }, Action = "sts:AssumeRole"
}] })
- tags = merge(local.tags, { "switch:slot-id" = each.key })
-}
-resource "aws_iam_role_policy" "worker_secret" {
- for_each = var.machine_slots
- role = aws_iam_role.worker[each.key].id
- policy = jsonencode({ Version = "2012-10-17", Statement = [
- { Effect = "Allow", Action = ["secretsmanager:GetSecretValue"], Resource = each.value.secret_arn, Condition = { StringEquals = { "secretsmanager:VersionStage" = "AWSCURRENT" } } },
- { Effect = "Allow", Action = ["kms:Decrypt"], Resource = each.value.kms_key_arn,
- Condition = { StringEquals = {
- "kms:ViaService" = "secretsmanager.${data.aws_region.current.name}.${data.aws_partition.current.dns_suffix}"
- "kms:EncryptionContext:SecretARN" = each.value.secret_arn
- } }
- }
- ] })
+ tags = local.tags
}
-resource "aws_iam_instance_profile" "worker" {
- for_each = var.machine_slots
- name = aws_iam_role.worker[each.key].name
- role = aws_iam_role.worker[each.key].name
- tags = merge(local.tags, { "switch:slot-id" = each.key })
+resource "aws_iam_instance_profile" "machine" {
+ name = aws_iam_role.machine.name
+ role = aws_iam_role.machine.name
+ tags = local.tags
}
resource "aws_iam_role" "controller" {
name = "${local.prefix}-controller"
@@ -61,37 +46,7 @@ resource "aws_iam_role" "controller" {
}] })
tags = local.tags
}
-resource "aws_iam_policy" "controller_assignments" {
- lifecycle {
- postcondition {
- condition = length(self.policy) <= 6144
- error_message = "Assignment permissions exceed the managed IAM policy limit; reduce the number of machine slots."
- }
- }
- name = "${local.prefix}-assignments"
- tags = local.tags
- policy = jsonencode({ Version = "2012-10-17", Statement = [
- { Sid = "PopulateWorkerAssignments", Effect = "Allow", Action = ["secretsmanager:DescribeSecret", "secretsmanager:PutSecretValue", "secretsmanager:UpdateSecretVersionStage"], Resource = [for slot in values(var.machine_slots) : slot.secret_arn] },
- { Sid = "EncryptWorkerAssignments", Effect = "Allow", Action = ["kms:GenerateDataKey", "kms:Decrypt"], Resource = distinct([for slot in values(var.machine_slots) : slot.kms_key_arn]),
- Condition = { StringEquals = {
- "kms:ViaService" = "secretsmanager.${data.aws_region.current.name}.${data.aws_partition.current.dns_suffix}"
- "kms:EncryptionContext:SecretARN" = [for slot in values(var.machine_slots) : slot.secret_arn]
- } }
- }
- ] })
-}
-resource "aws_iam_role_policy_attachment" "controller_assignments" {
- role = aws_iam_role.controller.name
- policy_arn = aws_iam_policy.controller_assignments.arn
-}
resource "aws_iam_role_policy" "controller" {
- depends_on = [aws_iam_role_policy_attachment.controller_assignments]
- lifecycle {
- postcondition {
- condition = length(self.policy) <= 10240
- error_message = "Controller permissions exceed the inline IAM policy limit; reduce the number of machine slots."
- }
- }
role = aws_iam_role.controller.id
policy = jsonencode({ Version = "2012-10-17", Statement = [
{ Sid = "Observe", Effect = "Allow", Action = ["ec2:DescribeInstances", "ec2:DescribeVolumes", "ec2:DescribeImages", "ec2:DescribeSubnets", "ec2:DescribeInstanceTypes"], Resource = "*" },
@@ -100,30 +55,13 @@ resource "aws_iam_role_policy" "controller" {
"${local.ec2_arn_base}:subnet/${aws_subnet.worker.id}",
"${local.ec2_arn_base}:security-group/${aws_security_group.worker.id}"
] },
- { Sid = "CreateVerificationInstances", Effect = "Allow", Action = ["ec2:RunInstances"], Resource = ["${local.ec2_arn_base}:instance/*", "${local.ec2_arn_base}:network-interface/*"],
- Condition = { StringEquals = { "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-provider-verification" } }
- },
- { Sid = "CreateVerificationRoot", Effect = "Allow", Action = ["ec2:RunInstances"], Resource = "${local.ec2_arn_base}:volume/*",
- Condition = {
- StringEquals = { "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-provider-verification", "ec2:VolumeType" = "gp3" }
- Bool = { "ec2:Encrypted" = "true" }
- NumericEquals = { "ec2:VolumeSize" = var.root_volume_gib }
- }
- },
- { Sid = "TagVerificationResources", Effect = "Allow", Action = ["ec2:CreateTags"], Resource = ["${local.ec2_arn_base}:instance/*", "${local.ec2_arn_base}:volume/*", "${local.ec2_arn_base}:network-interface/*"],
- Condition = { StringEquals = { "ec2:CreateAction" = "RunInstances", "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-provider-verification" } }
- },
- { Sid = "TerminateVerificationInstances", Effect = "Allow", Action = ["ec2:TerminateInstances"], Resource = "${local.ec2_arn_base}:instance/*",
- Condition = { StringEquals = { "ec2:ResourceTag/switch:installation-id" = var.installation_id, "ec2:ResourceTag/switch:managed-by" = "switch-provider-verification" } }
- },
{ Sid = "CreateManagedLaunchResources", Effect = "Allow", Action = ["ec2:RunInstances"], Resource = ["${local.ec2_arn_base}:instance/*", "${local.ec2_arn_base}:network-interface/*"],
- Condition = { StringEquals = { "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-hosted-controller" }, StringLike = { "aws:RequestTag/switch:slot-id" = keys(var.machine_slots) }, Null = { "aws:RequestTag/switch:generation" = "false" } }
+ Condition = { StringEquals = { "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-hosted-controller" }, Null = { "aws:RequestTag/switch:machine-id" = "false" } }
},
{ Sid = "CreateManagedRoot", Effect = "Allow", Action = ["ec2:RunInstances"], Resource = "${local.ec2_arn_base}:volume/*",
Condition = {
StringEquals = { "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-hosted-controller", "aws:RequestTag/switch:purpose" = "root", "ec2:VolumeType" = "gp3" }
- StringLike = { "aws:RequestTag/switch:slot-id" = keys(var.machine_slots) }
- Null = { "aws:RequestTag/switch:generation" = "false" }
+ Null = { "aws:RequestTag/switch:machine-id" = "false" }
Bool = { "ec2:Encrypted" = "true" }
NumericEquals = { "ec2:VolumeSize" = var.root_volume_gib, "ec2:VolumeIops" = 3000, "ec2:VolumeThroughput" = 125 }
}
@@ -131,22 +69,21 @@ resource "aws_iam_role_policy" "controller" {
{ Sid = "CreateManagedData", Effect = "Allow", Action = ["ec2:CreateVolume"], Resource = "${local.ec2_arn_base}:volume/*",
Condition = {
StringEquals = { "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-hosted-controller", "aws:RequestTag/switch:purpose" = "data", "ec2:VolumeType" = "gp3", "ec2:AvailabilityZone" = var.availability_zone }
- StringLike = { "aws:RequestTag/switch:slot-id" = keys(var.machine_slots) }
- Null = { "aws:RequestTag/switch:generation" = "false" }
+ Null = { "aws:RequestTag/switch:machine-id" = "false" }
Bool = { "ec2:Encrypted" = "true" }
NumericEquals = { "ec2:VolumeSize" = var.data_volume_gib, "ec2:VolumeIops" = 3000, "ec2:VolumeThroughput" = 125 }
}
},
{ Sid = "TagOnCreate", Effect = "Allow", Action = ["ec2:CreateTags"], Resource = ["${local.ec2_arn_base}:instance/*", "${local.ec2_arn_base}:volume/*", "${local.ec2_arn_base}:network-interface/*"],
- Condition = { StringEquals = { "ec2:CreateAction" = ["RunInstances", "CreateVolume"], "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-hosted-controller" }, StringLike = { "aws:RequestTag/switch:slot-id" = keys(var.machine_slots) }, Null = { "aws:RequestTag/switch:generation" = "false" } }
+ Condition = { StringEquals = { "ec2:CreateAction" = ["RunInstances", "CreateVolume"], "aws:RequestTag/switch:installation-id" = var.installation_id, "aws:RequestTag/switch:managed-by" = "switch-hosted-controller" }, Null = { "aws:RequestTag/switch:machine-id" = "false" } }
},
{ Sid = "ManageOwned", Effect = "Allow", Action = ["ec2:StartInstances", "ec2:StopInstances", "ec2:TerminateInstances", "ec2:AttachVolume", "ec2:DeleteVolume"], Resource = ["${local.ec2_arn_base}:instance/*", "${local.ec2_arn_base}:volume/*"],
- Condition = { StringEquals = { "ec2:ResourceTag/switch:installation-id" = var.installation_id, "ec2:ResourceTag/switch:managed-by" = "switch-hosted-controller", "ec2:ResourceTag/switch:purpose" = ["worker", "data"] }, StringLike = { "ec2:ResourceTag/switch:slot-id" = keys(var.machine_slots) }, Null = { "ec2:ResourceTag/switch:generation" = "false" } }
+ Condition = { StringEquals = { "ec2:ResourceTag/switch:installation-id" = var.installation_id, "ec2:ResourceTag/switch:managed-by" = "switch-hosted-controller", "ec2:ResourceTag/switch:purpose" = ["worker", "data"] }, Null = { "ec2:ResourceTag/switch:machine-id" = "false" } }
},
{ Sid = "PreserveAttachedData", Effect = "Allow", Action = ["ec2:ModifyInstanceAttribute"], Resource = "${local.ec2_arn_base}:instance/*",
- Condition = { StringEquals = { "ec2:ResourceTag/switch:installation-id" = var.installation_id, "ec2:ResourceTag/switch:managed-by" = "switch-hosted-controller", "ec2:ResourceTag/switch:purpose" = ["worker", "data"], "ec2:Attribute" = "blockDeviceMapping" }, StringLike = { "ec2:ResourceTag/switch:slot-id" = keys(var.machine_slots) }, Null = { "ec2:ResourceTag/switch:generation" = "false" } }
+ Condition = { StringEquals = { "ec2:ResourceTag/switch:installation-id" = var.installation_id, "ec2:ResourceTag/switch:managed-by" = "switch-hosted-controller", "ec2:ResourceTag/switch:purpose" = ["worker", "data"], "ec2:Attribute" = "blockDeviceMapping" }, Null = { "ec2:ResourceTag/switch:machine-id" = "false" } }
},
- { Sid = "PassOnlyWorkerRoles", Effect = "Allow", Action = ["iam:PassRole"], Resource = [for role in aws_iam_role.worker : role.arn], Condition = { StringEquals = { "iam:PassedToService" = "ec2.amazonaws.com" } } },
+ { Sid = "PassOnlyMachineRole", Effect = "Allow", Action = ["iam:PassRole"], Resource = aws_iam_role.machine.arn, Condition = { StringEquals = { "iam:PassedToService" = "ec2.amazonaws.com" } } },
{ Sid = "DenyUnapprovedType", Effect = "Deny", Action = ["ec2:RunInstances"], Resource = "${local.ec2_arn_base}:instance/*", Condition = { StringNotEquals = { "ec2:InstanceType" = var.allowed_instance_types } } },
{ Sid = "RequireMetadataTokens", Effect = "Deny", Action = ["ec2:RunInstances"], Resource = "${local.ec2_arn_base}:instance/*", Condition = { StringNotEquals = { "ec2:MetadataHttpTokens" = "required" } } }
] })
diff --git a/deploy/hosted/terraform/outputs.tf b/deploy/hosted/terraform/outputs.tf
index 804584119..225d596e8 100644
--- a/deploy/hosted/terraform/outputs.tf
+++ b/deploy/hosted/terraform/outputs.tf
@@ -1,11 +1,5 @@
output "controller_role_arn" { value = aws_iam_role.controller.arn }
output "worker_security_group_id" { value = aws_security_group.worker.id }
output "availability_zone" { value = aws_subnet.worker.availability_zone }
-output "machine_slots" {
- value = { for id, slot in var.machine_slots : id => {
- instance_profile_arn = aws_iam_instance_profile.worker[id].arn
- assignment_secret_arn = slot.secret_arn
- } }
-}
-
+output "machine_instance_profile_arn" { value = aws_iam_instance_profile.machine.arn }
output "worker_subnet_id" { value = aws_subnet.worker.id }
diff --git a/deploy/hosted/terraform/tests/plan.tftest.hcl b/deploy/hosted/terraform/tests/plan.tftest.hcl
index c615d7757..9e6b5c29a 100644
--- a/deploy/hosted/terraform/tests/plan.tftest.hcl
+++ b/deploy/hosted/terraform/tests/plan.tftest.hcl
@@ -1,6 +1,9 @@
mock_provider "aws" {
- mock_resource "aws_iam_policy" {
- defaults = { arn = "arn:aws:iam::000000000000:policy/example-assignments" }
+ mock_resource "aws_iam_role" {
+ defaults = { arn = "arn:aws:iam::000000000000:role/example-machine" }
+ }
+ mock_resource "aws_iam_instance_profile" {
+ defaults = { arn = "arn:aws:iam::000000000000:instance-profile/example-machine" }
}
mock_data "aws_caller_identity" {
defaults = { account_id = "000000000000" }
@@ -24,12 +27,6 @@ variables {
oidc_issuer_url = "https://example.invalid"
namespace = "example-hosted"
service_account = "switch-hosted-controller"
- machine_slots = {
- "example-slot" = {
- secret_arn = "arn:aws:secretsmanager:us-east-1:000000000000:secret:example-slot-ABCDEF"
- kms_key_arn = "arn:aws:kms:us-east-1:000000000000:key/00000000-0000-0000-0000-000000000000"
- }
- }
}
run "isolated_plan" {
@@ -39,16 +36,8 @@ run "isolated_plan" {
error_message = "Worker subnet must not assign public IPs."
}
assert {
- condition = aws_iam_role_policy.worker_secret["example-slot"].policy != ""
- error_message = "Assignment role must have an explicit scoped policy."
- }
- assert {
- condition = jsondecode(aws_iam_role_policy.worker_secret["example-slot"].policy).Statement[0].Resource == var.machine_slots["example-slot"].secret_arn
- error_message = "Worker secret access must be scoped to exactly its assignment."
- }
- assert {
- condition = jsondecode(aws_iam_role_policy.worker_secret["example-slot"].policy).Statement[1].Condition.StringEquals["kms:EncryptionContext:SecretARN"] == var.machine_slots["example-slot"].secret_arn
- error_message = "KMS decrypt must be restricted to the assignment secret."
+ condition = jsondecode(aws_iam_role.machine.assume_role_policy).Statement[0].Principal.Service == "ec2.amazonaws.com"
+ error_message = "The machine role must be assumable by EC2 only."
}
assert {
condition = alltrue([for rule in aws_network_acl_rule.deny_private_egress : rule.rule_action == "deny" && rule.rule_number < 100])
@@ -58,41 +47,28 @@ run "isolated_plan" {
run "rendered_permissions" {
command = apply
- assert {
- condition = jsondecode(aws_iam_policy.controller_assignments.policy).Statement[0].Resource == [var.machine_slots["example-slot"].secret_arn]
- error_message = "Controller secret writes must remain scoped to assigned secrets."
- }
- assert {
- condition = contains(jsondecode(aws_iam_policy.controller_assignments.policy).Statement[0].Action, "secretsmanager:UpdateSecretVersionStage")
- error_message = "Controller must be able to promote a written bundle to AWSCURRENT on its assigned secrets."
- }
assert {
condition = alltrue([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement :
- statement.Condition.Null[startswith(statement.Sid, "CreateManaged") || statement.Sid == "TagOnCreate" ? "aws:RequestTag/switch:generation" : "ec2:ResourceTag/switch:generation"] == "false"
- && !contains(keys(statement.Condition.StringEquals), "aws:RequestTag/switch:generation")
- && !contains(keys(statement.Condition.StringEquals), "ec2:ResourceTag/switch:generation")
+ statement.Condition.Null[startswith(statement.Sid, "CreateManaged") || statement.Sid == "TagOnCreate" ? "aws:RequestTag/switch:machine-id" : "ec2:ResourceTag/switch:machine-id"] == "false"
if contains(["CreateManagedLaunchResources", "CreateManagedRoot", "CreateManagedData", "TagOnCreate", "ManageOwned", "PreserveAttachedData"], statement.Sid)
- ]) && length([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement : statement if contains(["CreateManagedLaunchResources", "CreateManagedRoot", "CreateManagedData", "TagOnCreate", "ManageOwned", "PreserveAttachedData"], statement.Sid)]) == length(["CreateManagedLaunchResources", "CreateManagedRoot", "CreateManagedData", "TagOnCreate", "ManageOwned", "PreserveAttachedData"])
- error_message = "Managed resources must carry a generation tag of any value."
+ ]) && length([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement : statement if contains(["CreateManagedLaunchResources", "CreateManagedRoot", "CreateManagedData", "TagOnCreate", "ManageOwned", "PreserveAttachedData"], statement.Sid)]) == 6
+ error_message = "Managed resources must carry a machine id tag."
}
assert {
- condition = alltrue([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement :
- jsonencode(statement.Condition.StringLike) == jsonencode({ (contains(["ManageOwned", "PreserveAttachedData"], statement.Sid) ? "ec2:ResourceTag/switch:slot-id" : "aws:RequestTag/switch:slot-id") = keys(var.machine_slots) })
- if contains(["CreateManagedLaunchResources", "CreateManagedRoot", "CreateManagedData", "TagOnCreate", "ManageOwned", "PreserveAttachedData"], statement.Sid)
- ])
- error_message = "Managed resources must be scoped to the declared machine slots."
+ condition = one([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement : statement if statement.Sid == "PassOnlyMachineRole"]).Resource == aws_iam_role.machine.arn
+ error_message = "The controller may pass only the machine role."
}
assert {
- condition = output.machine_slots["example-slot"].assignment_secret_arn == var.machine_slots["example-slot"].secret_arn
- error_message = "The machine_slots output must map each slot to its assignment secret."
+ condition = output.machine_instance_profile_arn == aws_iam_instance_profile.machine.arn
+ error_message = "The machine instance profile must be an output for the controller configuration."
}
assert {
- condition = var.data_volume_gib == 200 && one([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement : statement if statement.Sid == "DenyUnapprovedType"]).Condition.StringNotEquals["ec2:InstanceType"] == ["c7i.2xlarge"]
- error_message = "Defaults must allow c7i.2xlarge workers with 200 GiB data disks."
+ condition = !strcontains(aws_iam_role_policy.controller.policy, "secretsmanager") && !strcontains(aws_iam_role_policy.controller.policy, "kms:")
+ error_message = "The controller needs no secrets: a machine's bundle is its user data."
}
assert {
- condition = aws_iam_role_policy_attachment.controller_assignments.role == aws_iam_role.controller.name && aws_iam_role_policy_attachment.controller_assignments.policy_arn == aws_iam_policy.controller_assignments.arn
- error_message = "Controller assignment policy must be attached to the controller role."
+ condition = var.data_volume_gib == 200 && one([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement : statement if statement.Sid == "DenyUnapprovedType"]).Condition.StringNotEquals["ec2:InstanceType"] == ["c7i.2xlarge"]
+ error_message = "Defaults must allow c7i.2xlarge workers with 200 GiB data disks."
}
assert {
condition = alltrue([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement :
@@ -102,11 +78,8 @@ run "rendered_permissions" {
error_message = "Lifecycle permissions require the full controller ownership marker."
}
assert {
- condition = contains(
- flatten([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement : statement.Action]),
- "ec2:ModifyInstanceAttribute"
- )
- error_message = "Controller must be able to set explicit data retention."
+ condition = one([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement : statement if statement.Sid == "PreserveAttachedData"]).Condition.StringEquals["ec2:Attribute"] == "blockDeviceMapping"
+ error_message = "Controller may modify only the data retention of its instances: a machine's user data is set when its instance launches."
}
assert {
condition = one([for statement in jsondecode(aws_iam_role_policy.controller.policy).Statement : statement if statement.Sid == "CreateManagedRoot"]).Condition.NumericEquals["ec2:VolumeSize"] == var.root_volume_gib
@@ -121,7 +94,7 @@ run "rendered_permissions" {
error_message = "Controller launch must require IMDSv2."
}
assert {
- condition = alltrue([for role in aws_iam_role.worker : length(role.name) <= 64]) && length(aws_iam_role.controller.name) <= 64
+ condition = length(aws_iam_role.machine.name) <= 64 && length(aws_iam_role.controller.name) <= 64
error_message = "IAM role names must stay within AWS limits."
}
assert {
diff --git a/deploy/hosted/terraform/variables.tf b/deploy/hosted/terraform/variables.tf
index b2167eb64..c5c9fa964 100644
--- a/deploy/hosted/terraform/variables.tf
+++ b/deploy/hosted/terraform/variables.tf
@@ -22,19 +22,6 @@ variable "allowed_instance_types" {
error_message = "At least one explicitly approved worker type is required."
}
}
-variable "machine_slots" {
- description = "Pre-created per-machine-slot secrets; never put secret values in Terraform."
- type = map(object({ secret_arn = string, kms_key_arn = string }))
- validation {
- condition = length(var.machine_slots) > 0 && alltrue([for id in keys(var.machine_slots) : can(regex("^[a-z0-9][a-z0-9-]{2,39}$", id))])
- error_message = "Declare at least one machine slot using lowercase 3–40 character IDs."
- }
- validation {
- condition = length(distinct([for slot in values(var.machine_slots) : slot.secret_arn])) == length(var.machine_slots)
- error_message = "Each machine slot must have a distinct assignment secret."
- }
-}
-
variable "root_volume_gib" {
type = number
default = 20
diff --git a/deploy/hosted/worker/README.md b/deploy/hosted/worker/README.md
deleted file mode 100644
index ac1a51eb8..000000000
--- a/deploy/hosted/worker/README.md
+++ /dev/null
@@ -1,228 +0,0 @@
-# Hosted EC2 machine supervisor
-
-This directory holds the trusted supervisor for hosted agents. One EC2 VM
-serves one user. The VM runs all agents of that user. The supervisor runs as
-root. Each agent runs as the unprivileged `switch-agent` account in its own
-systemd unit, `switch-agent@<agent-id>.service`.
-
-The AMI is built ahead of time. It pins Node.js 24, the provider CLIs and the
-built `@switch-console/agent-providers` bootstrap artifacts. The instance
-profile can call only `secretsmanager:GetSecretValue` on the assignment secret
-(and the KMS decrypt operation for that secret). The supervisor makes no EC2,
-IAM, KMS, S3 or secret-list calls.
-
-## Install
-
-Build the Node entrypoints:
-
- node deploy/hosted/build-runtime.mjs /path/to/runtime-build
-
-Run the installer while you bake the AMI:
-
- install.sh /path/to/runtime-build <node-sha256> <provider-sha256>
-
-The installer:
-
-- Verifies the runtime manifest and the SHA256 pins of Node.js and the provider.
-- Makes sure that the host commands the supervisor calls are present at their
- absolute paths, including `flock`, `systemctl` and `systemd-mount`.
-- Creates the `switch-agent` account.
-- Installs the supervisor, `switch-hosted-worker.service`,
- `[email protected]` and `switch-agents.slice`.
-- Writes the root-only `/etc/switch-hosted/runtime.json` with all artifact
- digests.
-- Enables the supervisor unit.
-
-The checked-in `runtime.json` shows the schema. Its zero digests are examples.
-`nodePath` and `bootstrapPath` must agree with `ExecStart` in
-`[email protected]`. The supervisor refuses to start if they do not.
-
-For more providers, put a `providers.json` beside the bundles. It maps
-`codex`, `cursor`, `opencode` and `antigravity` to
-`{ "path": "/opt/switch/providers/<provider>", "sha256": "<digest>" }`.
-Antigravity uses `/opt/switch/providers/antigravity-acp`. Each entry must be a
-root-owned executable, not a symlink, with no group or world write access. On
-AppArmor hosts, a Codex install also installs a profile for its bundled `bwrap`.
-
-## Assignment metadata
-
-The controller writes the root-owned, mode 0600
-`/etc/switch-hosted/assignment.json`:
-
-```json
-{
- "version": 2,
- "installationId": "inst-test",
- "slotId": "slot-a",
- "generation": 2,
- "assignmentSecretId": "arn:aws:secretsmanager:eu-west-1:000000000000:secret:example",
- "dataVolumeId": "vol-0123456789abcdef0",
- "dataDevice": "/dev/sdf"
-}
-```
-
-`previousInstanceId` and `previousRuntimeFingerprint` can be present. The
-supervisor ignores them. Other keys are refused. `assignmentSecretId` must be
-the full Secrets Manager ARN. The supervisor gets the region from the ARN and
-gives it to boto3.
-
-No credential can be in user-data, in this file, in an environment variable or
-in a command argument.
-
-## Machine bundle
-
-The assignment secret holds one JSON document, the machine bundle:
-
-```json
-{
- "version": 2,
- "assignment": {
- "installationId": "inst-test",
- "slotId": "slot-a",
- "generation": 2,
- "dataVolumeId": "vol-0123456789abcdef0"
- },
- "machineId": "<machine-uuid>",
- "apiEndpoint": "https://switch.example.test/agent-api",
- "machineCapability": "<machine-capability>"
-}
-```
-
-- If `version` is not 2, the supervisor writes `obsolete bundle` to stderr and
- exits with code 75. It does this before it changes the host.
-- `assignment` must agree with `assignment.json`.
-- `apiEndpoint` must be HTTPS, with no user information, query or fragment.
-- `machineCapability` is 16 to 4096 printable ASCII characters with no
- whitespace. The supervisor never logs it.
-
-The supervisor writes the machine fields to `/run/switch-hosted/machine/bundle.json`
-(root, mode 0600) after it makes sure that `/run/switch-hosted` is tmpfs.
-
-## Data volume
-
-The supervisor finds the data volume by its EBS serial. It uses `dataDevice`
-only as a hint. It formats a blank volume as ext4 only if `runtime.json` allows
-it and the volume has no filesystem, partition or other signature. It mounts
-the volume on `/data` with
-`systemd-mount --type=ext4 --options=nodev,nosuid`, so that the mount is also
-visible outside the supervisor's mount namespace.
-
-The layout is `per-user-v1`:
-
-| Path | Owner | Contents |
-| --- | --- | --- |
-| `/data/.switch-hosted/machine.json` | root, 0600 | Machine marker |
-| `/data/.switch-hosted/agents.json` | root, 0600 | OOM kill count for each agent |
-| `/data/.switch-hosted/quarantine/` | root, 0700 | Stale ownership records |
-| `/data/.switch-hosted/ownership-blocked.json` | root, 0600 | Agents whose stale ownership records could not be moved |
-| `/data/agents/<agent-id>/` | agent, 0700 | Agent state, `home/` and `tmp/` |
-| `/data/repos/<owner>/<repo>.git` | agent, 0700 | Shared repository mirror |
-| `/data/worktrees/<agent-id>/` | agent, 0700 | The agent's worktree |
-
-The marker has version 2 and records the installation, slot, generation,
-filesystem UUID, instance ID, boot ID and runtime fingerprint
-(`sha256:<hex>`). The supervisor refuses the volume if:
-
-- The marker has version 1. The message is `data volume uses the one-agent
- layout; see 'Moving to one machine per user' in deploy/hosted/README.md`.
-- The installation, slot, generation or filesystem UUID is different.
-- The volume has data but no marker.
-
-The supervisor writes the instance ID, boot ID and runtime fingerprint again at
-each start. On a new boot, it moves each agent's stale ownership records to
-`quarantine/<agent-id>/<old-boot>--<new-boot>/`. If an agent's records are not
-valid, that agent is not started and is reported as `failed` with the result
-`ownership-invalid`. Its records stay in place and the other agents start. The
-blocked agents are listed in `ownership-blocked.json`, and a supervisor restart
-in the same boot tries to move their records again. Journals, provider homes
-and worktrees stay in place.
-
-## Agent loop
-
-The supervisor holds a root flock under `/run/lock` for its lifetime. It then
-does these steps in a loop:
-
-1. It gets `GET <apiEndpoint>/hosted/machines/<machineId>/agents` at start, when
- `agents_version` in a heartbeat response changes, and when an agent exits
- with code 75. A failed request is tried again after 15 seconds.
-2. It reconciles each agent in the list:
- - A new or changed `revision` gets new runtime files, `reset-failed` and a
- restart (or a stop if `desired_state` is `stopped`). The revision counts
- as installed only when the restart or stop succeeds. A new revision
- resets the agent's OOM kill count.
- - The same revision only corrects the running state.
- - Each stop is followed by `reset-failed`, so a stopped unit is not left
- in the `failed` state.
- - An entry with an `unavailable` code (for example `agent_key_missing`) has
- no credentials. Its unit is stopped, its data stays on disk, and it is
- reported as `stopped` with the result `invalid-config`.
- - An agent that is not valid is stopped and reported as `failed` with the
- result `invalid-config`. Its data stays on disk.
- - A setup error is reported as `failed` with the result `setup-failed`.
-3. It removes each agent that is on disk but not in the list. It stops the
- unit, removes the runtime files, runs `git worktree remove --force` on the
- mirror, and removes the agent's state and worktree directories. It then
- runs `git worktree prune` on each mirror it touched. It keeps the mirror and
- the agent's branch.
-
- If any agent in steps 2 or 3 fails to set up, stop or be removed, or a
- prune fails, the list is not marked as applied. The supervisor gets the
- list and reconciles it again after 15 seconds, doubling the wait after each
- failure up to 5 minutes. A failed prune is kept and tried again.
-4. It reads unit state every 3 seconds and sends
- `POST <apiEndpoint>/hosted/machines/<machineId>/heartbeat` when a state
- changes, and at the interval that core returns (15 seconds by default).
- Each agent's `since` is a UTC time with a `Z` offset. Core refuses a time
- without an offset.
-
-Each request sends `Authorization: Bearer <machineCapability>` and the host
-instance and boot IDs. Redirects are not followed.
-
-- HTTP 401 stops the supervisor. systemd starts it again.
-- HTTP 410 means that the machine is retired. The supervisor stops all agent
- units, sends a heartbeat every 60 seconds and gets the list again when core
- accepts a heartbeat.
-- Other errors are logged and tried again.
-
-## Runtime files
-
-For each agent, the supervisor writes `/run/switch-hosted/agents/<agent-id>/`
-(root, group `switch-agent`, mode 0750). It replaces the directory atomically.
-Each file has mode 0440:
-
-- `deployment.json`: the deployment document, version 2.
-- `env`: the unit's environment file (`PATH`, `HOME`, `TMPDIR` and the host
- identity).
-- `switch.json`: the Switch credentials of the agent.
-- `worker-capability`: the capability for the current revision.
-
-The provider and GitHub credentials are not written here. The bootstrap gets
-them over authenticated HTTPS. Code that runs as the agent can read the
-credentials of the agents on that machine. The boundary is the VM of one user.
-
-## Units
-
-`[email protected]` runs the bootstrap as `switch-agent` in
-`switch-agents.slice`, with no capabilities. It restarts on failure, at most 5
-times in 10 minutes. Exit code 75 does not restart. Each unit is limited to 75%
-of memory. At start, the supervisor sets the slice limit to the total memory
-minus 1 GiB. An OOM kill stops the unit. The supervisor counts each OOM kill
-once and keeps the count in `agents.json`.
-
-`switch-hosted-worker.service` uses `Restart=always` and
-`RuntimeDirectoryPreserve=yes`, so agent runtime files stay in place when the
-supervisor restarts. The supervisor runs git as the agent through `setpriv`,
-which removes all capabilities, and under `flock --no-fork <mirror>.lock`, so it
-does not change a mirror while the bootstrap uses it. Each git command runs in
-its own process group. If it runs longer than 5 minutes, the supervisor kills
-the whole group and waits until every process in it is gone; while any is left,
-it does not delete the agent's directories. A git error is logged with the
-operation and the exit status only, because a repository's configuration could
-make git print a secret.
-
-## Tests
-
- python3 -m unittest deploy/hosted/worker/test_switch_hosted_worker.py
-
-Worker-only fixtures are in `testdata/`. The fixtures shared with core are in
-`core/tests/switch_core/fixtures/hosted_machines/`.
diff --git a/deploy/hosted/worker/install.sh b/deploy/hosted/worker/install.sh
deleted file mode 100755
index c0467d6f3..000000000
--- a/deploy/hosted/worker/install.sh
+++ /dev/null
@@ -1,178 +0,0 @@
-#!/bin/sh
-set -eu
-
-if [ "$(id -u)" -ne 0 ]; then
- echo "install.sh must run as root" >&2
- exit 1
-fi
-if [ "$#" -ne 3 ]; then
- echo "usage: install.sh <runtime-build-dir> <node-sha256> <provider-sha256>" >&2
- exit 1
-fi
-runtime_build=$1
-expected_node_sha=$2
-expected_provider_sha=$3
-if [ "${#expected_node_sha}" -ne 64 ] || [ "${#expected_provider_sha}" -ne 64 ]; then
- echo "node and provider SHA256 values must be lowercase 64-character hashes" >&2
- exit 1
-fi
-case "$expected_node_sha$expected_provider_sha" in
- *[!0-9a-f]*) echo "node and provider SHA256 values must be lowercase 64-character hashes" >&2; exit 1 ;;
-esac
-
-for command in python3 setpriv lsblk wipefs udevadm mkfs.ext4 mount findmnt sha256sum git gh flock systemctl systemd-mount; do
- command -v "$command" >/dev/null 2>&1 || {
- echo "missing required AMI command: $command" >&2
- exit 1
- }
-done
-for executable in /usr/bin/setpriv /usr/bin/lsblk /usr/sbin/wipefs /usr/bin/udevadm /usr/sbin/mkfs.ext4 /usr/bin/findmnt /usr/bin/git /usr/bin/flock /usr/bin/systemctl /usr/bin/systemd-mount; do
- [ -x "$executable" ] || {
- echo "the supervisor requires $executable" >&2
- exit 1
- }
-done
-python3 -c 'import boto3' >/dev/null 2>&1 || {
- echo "missing required AMI Python module: boto3" >&2
- exit 1
-}
-python3 - "$runtime_build/manifest.json" "$runtime_build" <<'PY'
-import hashlib
-import json
-import pathlib
-import sys
-
-manifest_path = pathlib.Path(sys.argv[1])
-directory = pathlib.Path(sys.argv[2])
-manifest = json.loads(manifest_path.read_text())
-if (
- set(manifest) != {"version", "nodeMajor", "files"}
- or manifest["version"] != 1
- or manifest["nodeMajor"] != 24
- or set(manifest["files"])
- != {
- "hosted-bootstrap.mjs",
- "shared-host-daemon.mjs",
- }
-):
- raise SystemExit("hosted runtime manifest is invalid")
-for name, expected in manifest["files"].items():
- if not isinstance(expected, str) or len(expected) != 64:
- raise SystemExit("hosted runtime manifest digest is invalid")
- actual = hashlib.sha256((directory / name).read_bytes()).hexdigest()
- if actual != expected:
- raise SystemExit(f"hosted runtime digest mismatch: {name}")
-PY
-
-if ! id switch-agent >/dev/null 2>&1; then
- useradd --system --user-group --home-dir /nonexistent --shell /usr/sbin/nologin switch-agent
-fi
-agent_uid=$(id -u switch-agent)
-agent_gid=$(id -g switch-agent)
-if [ "$agent_uid" -eq 0 ] || [ "$agent_gid" -eq 0 ] || [ "$(id -G switch-agent)" != "$agent_gid" ]; then
- echo "switch-agent must have one non-root primary group and no supplementary groups" >&2
- exit 1
-fi
-install -d -o root -g root -m 0755 /usr/local/libexec
-install -d -o root -g root -m 0755 /etc/switch-hosted
-install -d -o root -g root -m 0755 /opt/switch/agent-providers
-install -d -o root -g root -m 0755 /data
-install -o root -g root -m 0755 "$runtime_build/hosted-bootstrap.mjs" /opt/switch/agent-providers/hosted-bootstrap.mjs
-install -o root -g root -m 0755 "$runtime_build/shared-host-daemon.mjs" /opt/switch/agent-providers/shared-host-daemon.mjs
-install -o root -g root -m 0444 "$runtime_build/manifest.json" /opt/switch/agent-providers/manifest.json
-
-node_path=/opt/switch/node/bin/node
-provider_path=/opt/switch/claude/bin/claude
-node_version=$("$node_path" --version)
-case "$node_version" in
- v24.*) ;;
- *) echo "pinned AMI must contain Node.js 24, got $node_version" >&2; exit 1 ;;
-esac
-actual_node_sha=$(sha256sum "$node_path" | cut -d ' ' -f 1)
-actual_provider_sha=$(sha256sum "$provider_path" | cut -d ' ' -f 1)
-[ "$actual_node_sha" = "$expected_node_sha" ] || {
- echo "Node.js checksum does not match the image-build pin" >&2
- exit 1
-}
-[ "$actual_provider_sha" = "$expected_provider_sha" ] || {
- echo "provider checksum does not match the image-build pin" >&2
- exit 1
-}
-bootstrap_sha=$(sha256sum /opt/switch/agent-providers/hosted-bootstrap.mjs | cut -d ' ' -f 1)
-shared_sha=$(sha256sum /opt/switch/agent-providers/shared-host-daemon.mjs | cut -d ' ' -f 1)
-
-source_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
-install -o root -g root -m 0755 "$source_dir/switch_hosted_worker.py" /usr/local/libexec/switch-hosted-worker
-install -o root -g root -m 0644 "$source_dir/switch-hosted-worker.service" /etc/systemd/system/switch-hosted-worker.service
-install -o root -g root -m 0644 "$source_dir/[email protected]" /etc/systemd/system/[email protected]
-install -o root -g root -m 0644 "$source_dir/switch-agents.slice" /etc/systemd/system/switch-agents.slice
-python3 - "$actual_node_sha" "$bootstrap_sha" "$shared_sha" "$actual_provider_sha" "$runtime_build" <<'PY'
-import hashlib
-import json
-import os
-from pathlib import Path
-import stat
-import sys
-import tempfile
-
-node_sha, bootstrap_sha, shared_sha, provider_sha, runtime_build = sys.argv[1:]
-value = {
- "version": 1,
- "nodePath": "/opt/switch/node/bin/node",
- "bootstrapPath": "/opt/switch/agent-providers/hosted-bootstrap.mjs",
- "sharedHostDaemonPath": "/opt/switch/agent-providers/shared-host-daemon.mjs",
- "providerBinaryPath": "/opt/switch/claude/bin/claude",
- "agentUser": "switch-agent",
- "agentGroup": "switch-agent",
- "path": "/opt/switch/node/bin:/opt/switch/claude/bin:/usr/local/bin:/usr/bin:/bin",
- "allowInitialFormat": True,
- "artifactSha256": {
- "node": node_sha,
- "bootstrap": bootstrap_sha,
- "sharedHostDaemon": shared_sha,
- "provider": provider_sha,
- },
-}
-providers_path = Path(runtime_build) / "providers.json"
-if providers_path.exists():
- providers = json.loads(providers_path.read_text())
- if not isinstance(providers, dict) or not set(providers).issubset({"codex", "cursor", "opencode", "antigravity"}):
- raise SystemExit("Invalid provider manifest")
- for provider, artifact in providers.items():
- if not isinstance(artifact, dict) or set(artifact) != {"path", "sha256"} or artifact["path"] != f"/opt/switch/providers/{'antigravity-acp' if provider == 'antigravity' else provider}":
- raise SystemExit("Invalid provider artifact path")
- path = Path(artifact["path"])
- details = path.lstat()
- if not stat.S_ISREG(details.st_mode) or details.st_uid != 0 or details.st_mode & 0o022 or not details.st_mode & 0o111:
- raise SystemExit("Provider artifact must be a root-owned executable, not a symlink")
- if hashlib.sha256(path.read_bytes()).hexdigest() != artifact["sha256"]:
- raise SystemExit("Provider artifact digest mismatch")
- value["providers"] = providers
-directory = "/etc/switch-hosted"
-descriptor, temporary = tempfile.mkstemp(prefix=".runtime.", dir=directory)
-try:
- os.fchmod(descriptor, 0o600)
- with os.fdopen(descriptor, "w") as handle:
- json.dump(value, handle, indent=2)
- handle.write("\n")
- handle.flush()
- os.fsync(handle.fileno())
- os.replace(temporary, f"{directory}/runtime.json")
-finally:
- try:
- os.unlink(temporary)
- except FileNotFoundError:
- pass
-PY
-
-if [ -x /opt/switch/provider-runtime/codex/codex-resources/bwrap ] && [ -d /sys/kernel/security/apparmor ]; then
- command -v apparmor_parser >/dev/null 2>&1 || {
- echo "Codex sandbox requires apparmor_parser on an AppArmor host" >&2
- exit 1
- }
- install -o root -g root -m 0644 "$source_dir/switch-codex-bwrap.apparmor" /etc/apparmor.d/switch-codex-bwrap
- apparmor_parser --replace /etc/apparmor.d/switch-codex-bwrap
-fi
-
-systemctl daemon-reload
-systemctl enable switch-hosted-worker.service
diff --git a/deploy/hosted/worker/runtime.json b/deploy/hosted/worker/runtime.json
deleted file mode 100644
index bd661bb7e..000000000
--- a/deploy/hosted/worker/runtime.json
+++ /dev/null
@@ -1,17 +0,0 @@
-{
- "version": 1,
- "nodePath": "/opt/switch/node/bin/node",
- "bootstrapPath": "/opt/switch/agent-providers/hosted-bootstrap.mjs",
- "sharedHostDaemonPath": "/opt/switch/agent-providers/shared-host-daemon.mjs",
- "providerBinaryPath": "/opt/switch/claude/bin/claude",
- "agentUser": "switch-agent",
- "agentGroup": "switch-agent",
- "path": "/opt/switch/node/bin:/opt/switch/claude/bin:/usr/local/bin:/usr/bin:/bin",
- "allowInitialFormat": true,
- "artifactSha256": {
- "node": "0000000000000000000000000000000000000000000000000000000000000000",
- "bootstrap": "0000000000000000000000000000000000000000000000000000000000000000",
- "sharedHostDaemon": "0000000000000000000000000000000000000000000000000000000000000000",
- "provider": "0000000000000000000000000000000000000000000000000000000000000000"
- }
-}
diff --git a/deploy/hosted/worker/[email protected] b/deploy/hosted/worker/[email protected]
deleted file mode 100644
index 040712c77..000000000
--- a/deploy/hosted/worker/[email protected]
+++ /dev/null
@@ -1,38 +0,0 @@
-[Unit]
-Description=Switch agent %i
-StartLimitIntervalSec=600
-StartLimitBurst=5
-AssertPathIsMountPoint=/data
-
-[Service]
-Type=simple
-User=switch-agent
-Group=switch-agent
-Slice=switch-agents.slice
-EnvironmentFile=/run/switch-hosted/agents/%i/env
-ExecStart=/opt/switch/node/bin/node /opt/switch/agent-providers/hosted-bootstrap.mjs /data/agents/%i /run/switch-hosted/agents/%i/deployment.json
-WorkingDirectory=/data/agents/%i
-Restart=on-failure
-RestartSec=10s
-RestartPreventExitStatus=75
-MemoryMax=75%
-OOMPolicy=stop
-UMask=0077
-NoNewPrivileges=yes
-CapabilityBoundingSet=
-AmbientCapabilities=
-ProtectSystem=full
-ProtectHome=true
-PrivateTmp=true
-PrivateDevices=false
-ProtectKernelTunables=true
-ProtectKernelModules=true
-ProtectControlGroups=true
-ProtectClock=true
-ProtectHostname=true
-RestrictRealtime=true
-RestrictSUIDSGID=true
-LockPersonality=true
-RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
-StandardOutput=journal
-StandardError=journal
diff --git a/deploy/hosted/worker/switch-agents.slice b/deploy/hosted/worker/switch-agents.slice
deleted file mode 100644
index 833b3fc7d..000000000
--- a/deploy/hosted/worker/switch-agents.slice
+++ /dev/null
@@ -1,5 +0,0 @@
-[Unit]
-Description=Switch hosted agents
-
-[Slice]
-MemoryAccounting=yes
diff --git a/deploy/hosted/worker/switch-codex-bwrap.apparmor b/deploy/hosted/worker/switch-codex-bwrap.apparmor
deleted file mode 100644
index 33a4f0871..000000000
--- a/deploy/hosted/worker/switch-codex-bwrap.apparmor
+++ /dev/null
@@ -1,6 +0,0 @@
-abi <abi/4.0>,
-include <tunables/global>
-
-profile switch-codex-bwrap /opt/switch/provider-runtime/codex/codex-resources/bwrap flags=(unconfined) {
- userns,
-}
diff --git a/deploy/hosted/worker/switch-hosted-worker.service b/deploy/hosted/worker/switch-hosted-worker.service
deleted file mode 100644
index 85d5f0008..000000000
--- a/deploy/hosted/worker/switch-hosted-worker.service
+++ /dev/null
@@ -1,42 +0,0 @@
-[Unit]
-Description=Switch hosted machine supervisor
-Wants=network-online.target
-After=network-online.target cloud-final.service
-ConditionPathExists=/etc/switch-hosted/assignment.json
-ConditionPathExists=/etc/switch-hosted/runtime.json
-
-[Service]
-Type=simple
-User=root
-Group=root
-UMask=0077
-ExecStart=/usr/local/libexec/switch-hosted-worker --config /etc/switch-hosted/assignment.json --runtime-config /etc/switch-hosted/runtime.json
-Restart=always
-RestartSec=15s
-TimeoutStopSec=45s
-KillMode=control-group
-RuntimeDirectory=switch-hosted
-RuntimeDirectoryMode=0750
-RuntimeDirectoryPreserve=yes
-ProtectSystem=full
-ProtectHome=true
-PrivateTmp=true
-PrivateDevices=false
-NoNewPrivileges=true
-ProtectKernelTunables=true
-ProtectKernelModules=true
-ProtectControlGroups=true
-ProtectClock=true
-ProtectHostname=true
-RestrictRealtime=true
-RestrictSUIDSGID=true
-LockPersonality=true
-RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
-CapabilityBoundingSet=CAP_SYS_ADMIN CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_SETUID CAP_SETGID CAP_SETPCAP CAP_KILL
-# setpriv drops every capability before git runs as the agent.
-AmbientCapabilities=CAP_SETUID
-StandardOutput=journal
-StandardError=journal
-
-[Install]
-WantedBy=cloud-init.target
diff --git a/deploy/hosted/worker/switch_hosted_worker.py b/deploy/hosted/worker/switch_hosted_worker.py
deleted file mode 100755
index 8fd79c3ca..000000000
--- a/deploy/hosted/worker/switch_hosted_worker.py
+++ /dev/null
@@ -1,2612 +0,0 @@
-#!/usr/bin/env python3
-"""Trusted root supervisor for one hosted Switch machine and its agents."""
-
-from __future__ import annotations
-
-import argparse
-import fcntl
-import grp
-import hashlib
-import json
-import logging
-import os
-import pwd
-import re
-import shutil
-import signal
-import stat
-import subprocess
-import sys
-import tempfile
-import time
-import urllib.error
-import urllib.request
-import uuid
-from collections.abc import Callable
-from dataclasses import dataclass, field
-from datetime import UTC, datetime
-from pathlib import Path
-from typing import Any
-from urllib.parse import quote, urlsplit
-
-logger = logging.getLogger("switch-hosted-worker")
-
-DATA_MOUNT = Path("/data")
-RUNTIME_DIRECTORY = Path("/run/switch-hosted")
-LOCK_PATH = Path("/run/lock/switch-hosted-worker.lock")
-BOOT_ID_PATH = Path("/proc/sys/kernel/random/boot_id")
-MEMINFO_PATH = Path("/proc/meminfo")
-IMDS_BASE = "http://169.254.169.254/latest"
-SYSTEMCTL = "/usr/bin/systemctl"
-SYSTEMD_MOUNT = "/usr/bin/systemd-mount"
-SETPRIV = "/usr/bin/setpriv"
-FLOCK = "/usr/bin/flock"
-GIT = "/usr/bin/git"
-UNIT_NODE_PATH = "/opt/switch/node/bin/node"
-UNIT_BOOTSTRAP_PATH = "/opt/switch/agent-providers/hosted-bootstrap.mjs"
-AGENT_SLICE = "switch-agents.slice"
-SUPERVISOR_VERSION = "2.0.0"
-MARKER_LAYOUT = "per-user-v1"
-ONE_AGENT_LAYOUT_MESSAGE = "data volume uses the one-agent layout; see 'Moving to one machine per user' in deploy/hosted/README.md"
-OBSOLETE_EXIT_CODE = 75
-OBSERVE_SECONDS = 3
-DEFAULT_HEARTBEAT_SECONDS = 15
-RETIRED_HEARTBEAT_SECONDS = 60
-LIST_RETRY_SECONDS = 15
-RECONCILE_RETRY_MAX_SECONDS = 300
-HTTP_TIMEOUT_SECONDS = 30
-GIT_TIMEOUT_SECONDS = 300
-GIT_KILL_WAIT_SECONDS = 10
-SLICE_RESERVE_BYTES = 1024**3
-MAX_SECRET_BYTES = 128 * 1024
-MAX_RESPONSE_BYTES = 16 * 1024 * 1024
-ROOT_UID = 0
-INSTANCE_RE = re.compile(r"^i-[0-9a-f]{8,17}$")
-VOLUME_RE = re.compile(r"^vol-[0-9a-f]{8,17}$")
-SECRET_ARN_RE = re.compile(
- r"^arn:(aws|aws-us-gov|aws-cn):secretsmanager:([a-z]{2}(?:-gov)?-[a-z]+-\d):([0-9]{12}):secret:([A-Za-z0-9/_+=.@-]+)$"
-)
-IDENTIFIER_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:@/+\-]{0,199}$")
-CAPABILITY_RE = re.compile(r"^[\x21-\x7e]{16,4096}$")
-REPOSITORY_RE = re.compile(
- r"^[A-Za-z0-9][A-Za-z0-9-]{0,38}/(?!\.{1,2}$)[A-Za-z0-9_.-]{1,100}$"
-)
-DEFINITION_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,127}$")
-FINGERPRINT_RE = re.compile(r"^sha256:[0-9a-f]{1,64}$")
-OWNERSHIP_TICKET_RE = re.compile(r"^\d+-[0-9a-f-]+\.json$")
-OWNERSHIP_TEMPORARY_RE = re.compile(r"^\d+-[0-9a-f-]+\.json\.[0-9a-f-]+\.tmp$")
-SKILL_SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$")
-SKILL_PATH_RE = re.compile(
- r"^[A-Za-z0-9_-][A-Za-z0-9._-]{0,99}(/[A-Za-z0-9_-][A-Za-z0-9._-]{0,99}){0,7}$"
-)
-MAX_SKILL_BYTES = 32 * 1024
-MAX_SKILLS = 16
-CREDENTIAL_KINDS = {"api-key", "setup-token", "auth-json"}
-SKILL_PROVIDERS = {"claude", "codex", "opencode"}
-PROVIDER_CONTEXT = (
- "Use the Switch tools to read room context and post replies to the room.\n"
-)
-INVALID_CONFIG = "invalid-config"
-SETUP_FAILED = "setup-failed"
-OWNERSHIP_INVALID = "ownership-invalid"
-
-
-class WorkerError(RuntimeError):
- pass
-
-
-class ObsoleteBundle(WorkerError):
- pass
-
-
-class GitAbandoned(WorkerError):
- pass
-
-
-class ReconcileIncomplete(WorkerError):
- pass
-
-
-class MachineRetired(Exception):
- pass
-
-
-class CoreUnavailable(Exception):
- pass
-
-
-def _strict(
- value: Any, required: set[str], optional: set[str], label: str
-) -> dict[str, Any]:
- if not isinstance(value, dict):
- raise WorkerError(f"{label} must be an object.")
- keys = set(value)
- missing = required - keys
- unexpected = keys - required - optional
- if missing or unexpected:
- raise WorkerError(f"{label} has missing or unexpected fields.")
- return value
-
-
-def _text(value: Any, label: str, *, maximum: int = 4096) -> str:
- if (
- not isinstance(value, str)
- or not value
- or len(value) > maximum
- or "\x00" in value
- ):
- raise WorkerError(f"{label} is invalid.")
- return value
-
-
-def _identifier(value: Any, label: str) -> str:
- value = _text(value, label, maximum=200)
- if not IDENTIFIER_RE.fullmatch(value):
- raise WorkerError(f"{label} is invalid.")
- return value
-
-
-def _positive_integer(value: Any, label: str) -> int:
- if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
- raise WorkerError(f"{label} is invalid.")
- return value
-
-
-def _agent_id(value: Any) -> str:
- try:
- if isinstance(value, str) and str(uuid.UUID(value)) == value:
- return value
- except ValueError:
- pass
- raise WorkerError("Agent ID must be a lowercase UUID.")
-
-
-def _unavailable_code(entry: dict[str, Any]) -> str | None:
- code = entry.get("unavailable")
- if code is None:
- return None
- _text(code, "Unavailable code", maximum=128)
- _agent_id(entry["agent_id"])
- _identifier(entry["launch_id"], "Launch ID")
- return code
-
-
-def _is_agent_id(value: str) -> bool:
- try:
- _agent_id(value)
- except WorkerError:
- return False
- return True
-
-
-def _https_endpoint(value: Any, label: str) -> str:
- endpoint = _text(value, label)
- parsed = urlsplit(endpoint)
- if (
- parsed.scheme != "https"
- or not parsed.netloc
- or parsed.username
- or parsed.password
- or parsed.query
- or parsed.fragment
- ):
- raise WorkerError(f"{label} is invalid.")
- return endpoint
-
-
-@dataclass(frozen=True)
-class RuntimeConfig:
- node_path: str
- bootstrap_path: str
- shared_host_daemon_path: str
- provider_binary_path: str
- agent_user: str
- agent_group: str
- path: str
- allow_initial_format: bool
- artifact_sha256: dict[str, str]
- providers: dict[str, dict[str, str]] = field(default_factory=dict)
-
-
-@dataclass(frozen=True)
-class WorkerConfig:
- installation_id: str
- slot_id: str
- generation: int
- secret_id: str
- secret_region: str
- volume_id: str
- device_path: str
- runtime: RuntimeConfig
-
-
-@dataclass(frozen=True)
-class MachineBundle:
- machine_id: str
- api_endpoint: str
- machine_capability: str = field(repr=False)
-
-
-@dataclass(frozen=True)
-class MachineIdentity:
- instance_id: str
- boot_id: str
- assignment_generation: int
-
- def json(self) -> dict[str, Any]:
- return {
- "instanceId": self.instance_id,
- "bootId": self.boot_id,
- "assignmentGeneration": self.assignment_generation,
- }
-
-
-@dataclass(frozen=True)
-class StorageObservation:
- device_path: str
- volume_id: str
- filesystem_type: str | None
- filesystem_uuid: str | None
- has_children: bool
- signatures: tuple[str, ...]
-
-
-@dataclass(frozen=True)
-class Paths:
- data: Path
- runtime: Path
-
- @property
- def marker_directory(self) -> Path:
- return self.data / ".switch-hosted"
-
- @property
- def marker(self) -> Path:
- return self.marker_directory / "machine.json"
-
- @property
- def agent_records(self) -> Path:
- return self.marker_directory / "agents.json"
-
- @property
- def quarantine(self) -> Path:
- return self.marker_directory / "quarantine"
-
- @property
- def ownership_blocked(self) -> Path:
- return self.marker_directory / "ownership-blocked.json"
-
- @property
- def agents(self) -> Path:
- return self.data / "agents"
-
- @property
- def repos(self) -> Path:
- return self.data / "repos"
-
- @property
- def worktrees(self) -> Path:
- return self.data / "worktrees"
-
- @property
- def bundle(self) -> Path:
- return self.runtime / "machine" / "bundle.json"
-
- @property
- def agents_runtime(self) -> Path:
- return self.runtime / "agents"
-
- def pending_install(self, agent_id: str) -> Path:
- return self.bundle.parent / f"pending-{agent_id}.json"
-
-
-def load_worker_config(assignment_path: Path, runtime_path: Path) -> WorkerConfig:
- value = _load_json(
- assignment_path, "Worker assignment configuration is invalid.", MAX_SECRET_BYTES
- )
- value = _strict(
- value,
- {
- "version",
- "installationId",
- "slotId",
- "generation",
- "assignmentSecretId",
- "dataVolumeId",
- "dataDevice",
- "mountPath",
- },
- {"previousInstanceId", "previousRuntimeFingerprint"},
- "worker assignment",
- )
- if value["version"] != 2:
- raise WorkerError("Worker assignment version is unsupported.")
- if value["mountPath"] != str(DATA_MOUNT):
- raise WorkerError("Worker data mount path must be /data.")
- volume_id = _text(value["dataVolumeId"], "worker data volume ID", maximum=32)
- if not VOLUME_RE.fullmatch(volume_id):
- raise WorkerError("Worker data volume ID is invalid.")
- runtime_value = _load_json(
- runtime_path, "Pinned AMI runtime configuration is invalid.", MAX_SECRET_BYTES
- )
- runtime_value = _strict(
- runtime_value,
- {
- "version",
- "nodePath",
- "bootstrapPath",
- "sharedHostDaemonPath",
- "providerBinaryPath",
- "agentUser",
- "agentGroup",
- "path",
- "allowInitialFormat",
- "artifactSha256",
- },
- {"providers"},
- "worker runtime",
- )
- if runtime_value["version"] != 1:
- raise WorkerError("Pinned AMI runtime version is unsupported.")
- if not isinstance(runtime_value["allowInitialFormat"], bool):
- raise WorkerError("Worker initial-format policy is invalid.")
- artifact_sha256 = _artifact_hashes(runtime_value["artifactSha256"])
- runtime = RuntimeConfig(
- node_path=_absolute_path(runtime_value["nodePath"], "Node executable"),
- bootstrap_path=_absolute_path(
- runtime_value["bootstrapPath"], "bootstrap entrypoint"
- ),
- shared_host_daemon_path=_absolute_path(
- runtime_value["sharedHostDaemonPath"], "shared host daemon"
- ),
- provider_binary_path=_absolute_path(
- runtime_value["providerBinaryPath"], "provider executable"
- ),
- agent_user=_identifier(runtime_value["agentUser"], "agent user"),
- agent_group=_identifier(runtime_value["agentGroup"], "agent group"),
- path=_text(runtime_value["path"], "runtime PATH"),
- allow_initial_format=runtime_value["allowInitialFormat"],
- artifact_sha256=artifact_sha256,
- providers=_provider_runtimes(runtime_value.get("providers", {})),
- )
- if (
- runtime.node_path != UNIT_NODE_PATH
- or runtime.bootstrap_path != UNIT_BOOTSTRAP_PATH
- ):
- raise WorkerError(
- "Pinned runtime paths do not match the switch-agent@ unit ExecStart."
- )
- secret_id = _text(value["assignmentSecretId"], "worker secret ID")
- return WorkerConfig(
- installation_id=_identifier(value["installationId"], "installation ID"),
- slot_id=_identifier(value["slotId"], "slot ID"),
- generation=_positive_integer(
- value["generation"], "worker assignment generation"
- ),
- secret_id=secret_id,
- secret_region=_secret_arn_region(secret_id),
- volume_id=volume_id,
- device_path=_absolute_path(value["dataDevice"], "worker data device"),
- runtime=runtime,
- )
-
-
-def _provider_runtimes(value: Any) -> dict[str, dict[str, str]]:
- if not isinstance(value, dict) or not set(value).issubset(
- {"codex", "cursor", "opencode", "antigravity"}
- ):
- raise WorkerError("Pinned provider runtimes are invalid.")
- for provider, runtime in value.items():
- if (
- not isinstance(runtime, dict)
- or set(runtime) != {"path", "sha256"}
- or runtime["path"]
- != f"/opt/switch/providers/{'antigravity-acp' if provider == 'antigravity' else provider}"
- or not re.fullmatch(r"[0-9a-f]{64}", str(runtime["sha256"]))
- ):
- raise WorkerError("Pinned provider runtime path or checksum is invalid.")
- return value
-
-
-def _secret_arn_region(value: str) -> str:
- match = SECRET_ARN_RE.fullmatch(value)
- if not match:
- raise WorkerError(
- "Worker assignment secret ID must be a full Secrets Manager ARN."
- )
- partition, region, _account, _name = match.groups()
- if (
- partition == "aws-cn"
- and not region.startswith("cn-")
- or partition == "aws-us-gov"
- and not region.startswith("us-gov-")
- or partition == "aws"
- and (region.startswith("cn-") or region.startswith("us-gov-"))
- ):
- raise WorkerError(
- "Worker assignment secret ARN partition and region do not match."
- )
- return region
-
-
-def _artifact_hashes(value: Any) -> dict[str, str]:
- value = _strict(
- value,
- {"node", "bootstrap", "sharedHostDaemon", "provider"},
- set(),
- "runtime artifact hashes",
- )
- for name, digest in value.items():
- if not isinstance(digest, str) or not re.fullmatch(r"[0-9a-f]{64}", digest):
- raise WorkerError(f"Runtime artifact hash {name} is invalid.")
- return value
-
-
-def parse_bundle(raw: str, config: WorkerConfig) -> MachineBundle:
- if len(raw.encode()) > MAX_SECRET_BYTES:
- raise WorkerError("Machine bundle is invalid.")
- try:
- value = json.loads(raw)
- except (json.JSONDecodeError, UnicodeError):
- raise WorkerError("Machine bundle is invalid.") from None
- if not isinstance(value, dict) or "version" not in value:
- raise WorkerError("Machine bundle is invalid.")
- if value["version"] != 2:
- raise ObsoleteBundle("obsolete bundle")
- value = _strict(
- value,
- {"version", "machineId", "assignment", "machineCapability", "apiEndpoint"},
- set(),
- "machine bundle",
- )
- assignment = _strict(
- value["assignment"],
- {"installationId", "slotId", "generation", "dataVolumeId"},
- set(),
- "bundle assignment",
- )
- if (
- assignment["installationId"] != config.installation_id
- or assignment["slotId"] != config.slot_id
- or assignment["generation"] != config.generation
- or assignment["dataVolumeId"] != config.volume_id
- ):
- raise WorkerError("Machine bundle does not match the worker assignment.")
- capability = value["machineCapability"]
- if not isinstance(capability, str) or not CAPABILITY_RE.fullmatch(capability):
- raise WorkerError("Machine capability is invalid.")
- return MachineBundle(
- machine_id=_identifier(value["machineId"], "machine ID"),
- api_endpoint=_https_endpoint(value["apiEndpoint"], "Switch API endpoint"),
- machine_capability=capability,
- )
-
-
-def _validate_skills(value: Any) -> None:
- if not isinstance(value, list) or not 0 < len(value) <= MAX_SKILLS:
- raise WorkerError("Deployment skills are invalid.")
- slugs: set[str] = set()
- total = 0
- for item in value:
- skill = _strict(item, {"slug", "files"}, set(), "deployment skill")
- slug = skill["slug"]
- if (
- not isinstance(slug, str)
- or not SKILL_SLUG_RE.fullmatch(slug)
- or slug in slugs
- ):
- raise WorkerError("Deployment skill name is invalid or repeated.")
- slugs.add(slug)
- files = skill["files"]
- if not isinstance(files, dict) or "SKILL.md" not in files:
- raise WorkerError("Deployment skill must include SKILL.md.")
- for path, content in files.items():
- if (
- not isinstance(path, str)
- or not SKILL_PATH_RE.fullmatch(path)
- or any(part in {".", ".."} for part in path.split("/"))
- ):
- raise WorkerError("Deployment skill file path is unsafe.")
- if not isinstance(content, str) or "\x00" in content:
- raise WorkerError("Deployment skill file content is invalid.")
- total += len(content.encode())
- if total > MAX_SKILL_BYTES:
- raise WorkerError("Deployment skills exceed the size limit.")
-
-
-def _validate_switch_credentials(value: Any, agent_id: str) -> dict[str, Any]:
- value = _strict(value, {"env"}, set(), "Switch credentials")
- env = _strict(
- value["env"],
- {"SWITCH_API_ENDPOINT", "SWITCH_API_TOKEN", "SWITCH_AGENT_ID"},
- set(),
- "Switch credential environment",
- )
- _https_endpoint(env["SWITCH_API_ENDPOINT"], "Switch API endpoint")
- _text(env["SWITCH_API_TOKEN"], "Switch API token", maximum=16 * 1024)
- if env["SWITCH_AGENT_ID"] != agent_id:
- raise WorkerError("Switch credentials belong to a different agent.")
- return value
-
-
-def _absolute_path(value: Any, label: str) -> str:
- value = _text(value, label)
- if not os.path.isabs(value) or os.path.normpath(value) != value:
- raise WorkerError(f"{label} must be a normalized absolute path.")
- return value
-
-
-def _load_json(path: Path, failure: str, maximum: int) -> Any:
- try:
- if path.is_symlink() or path.stat().st_size > maximum:
- raise WorkerError(failure)
- return json.loads(path.read_text(encoding="utf-8"))
- except (OSError, json.JSONDecodeError, UnicodeError):
- raise WorkerError(failure) from None
-
-
-@dataclass(frozen=True)
-class AgentPlan:
- launch_id: str
- agent_id: str
- revision: int
- desired_state: str
- deployment: dict[str, Any]
- switch_credentials: dict[str, Any] = field(repr=False)
- worker_capability: str = field(repr=False)
- workspace: Path
- worktree_owner: Path | None
-
-
-AGENT_FIELDS = {
- "launch_id",
- "agent_id",
- "name",
- "revision",
- "desired_state",
- "provider",
- "provider_credential_kind",
- "worker_capability",
- "switch_credentials",
- "repository",
- "spec",
- "skills",
-}
-
-
-def build_agent_plan(
- value: dict[str, Any], runtime: RuntimeConfig, paths: Paths
-) -> AgentPlan:
- missing = AGENT_FIELDS - set(value)
- if missing:
- raise WorkerError(f"Agent entry is missing {', '.join(sorted(missing))}.")
- agent_id = _agent_id(value["agent_id"])
- launch_id = _identifier(value["launch_id"], "launch ID")
- revision = _positive_integer(value["revision"], "launch revision")
- if value["desired_state"] not in {"running", "stopped"}:
- raise WorkerError("Agent desired state is invalid.")
- provider = value["provider"]
- if provider == "claude":
- binary = runtime.provider_binary_path
- elif isinstance(provider, str) and provider in runtime.providers:
- binary = runtime.providers[provider]["path"]
- else:
- raise WorkerError(f"Provider {provider!r} is not installed on this machine.")
- credential_kind = value["provider_credential_kind"]
- if credential_kind not in CREDENTIAL_KINDS:
- raise WorkerError(f"Provider credential kind {credential_kind!r} is invalid.")
- capability = value["worker_capability"]
- if not isinstance(capability, str) or not CAPABILITY_RE.fullmatch(capability):
- raise WorkerError("Worker capability is invalid.")
- switch_credentials = _validate_switch_credentials(
- value["switch_credentials"], agent_id
- )
- repository = value["repository"]
- if repository is not None and (
- not isinstance(repository, str) or not REPOSITORY_RE.fullmatch(repository)
- ):
- raise WorkerError("Hosted GitHub repository must be an owner/repository name.")
- skills = value["skills"]
- if not isinstance(skills, list):
- raise WorkerError("Deployment skills are invalid.")
- if skills:
- _validate_skills(skills)
- if provider not in SKILL_PROVIDERS:
- raise WorkerError("This provider has no skills directory.")
- spec = value["spec"]
- if not isinstance(spec, dict):
- raise WorkerError("Launch spec must be an object.")
- for key in (
- "instructions",
- "auto_session",
- "auto_approve",
- "definition_attributes",
- ):
- if key not in spec:
- raise WorkerError(f"Launch spec is missing {key}.")
- if not isinstance(spec["instructions"], str):
- raise WorkerError("Launch instructions are invalid.")
- if not isinstance(spec["auto_session"], bool) or not isinstance(
- spec["auto_approve"], bool
- ):
- raise WorkerError("Launch session flags are invalid.")
- if not isinstance(spec["definition_attributes"], dict):
- raise WorkerError("Launch definition attributes are invalid.")
- agent_runtime = paths.agents_runtime / agent_id
- provider_spec: dict[str, Any] = {
- "kind": provider,
- "credential": {
- "kind": credential_kind,
- "path": str(agent_runtime / "provider"),
- "refresh": True,
- },
- "binaryPath": binary,
- "context": _text(
- PROVIDER_CONTEXT + spec["instructions"],
- "deployment provider context",
- maximum=64 * 1024,
- ),
- }
- if provider == "claude":
- name = spec.get("name")
- if not isinstance(name, str) or not DEFINITION_NAME_RE.fullmatch(name):
- raise WorkerError("Hosted agent definition name is invalid.")
- provider_spec["definition"] = {
- "name": name,
- "content": _text(
- spec.get("definition"), "agent definition", maximum=64 * 1024
- ),
- }
- model = spec["definition_attributes"].get("model")
- if model:
- provider_spec["model"] = {"id": _text(model, "deployment model ID")}
- deployment: dict[str, Any] = {
- "version": 2,
- "revision": revision,
- "session": {"sessionId": "watcher-" + agent_id, "agentId": agent_id},
- "provider": provider_spec,
- }
- worktree_owner = None
- if repository is not None:
- owner, name = repository.lower().split("/")
- worktree_owner = paths.worktrees / agent_id / owner
- workspace = worktree_owner / name
- deployment["github"] = {
- "credentialPath": str(agent_runtime / "github"),
- "repository": repository,
- "refresh": True,
- "mirrorPath": str(paths.repos / owner / f"{name}.git"),
- }
- else:
- workspace = paths.worktrees / agent_id / "workspace"
- deployment.update(
- {
- "workspacePath": str(workspace),
- "watch": spec["auto_session"],
- "runtimeMode": "full-access"
- if spec["auto_approve"]
- else "approval-required",
- "switchCredentialsPath": str(agent_runtime / "switch.json"),
- "workerCapabilityPath": str(agent_runtime / "worker-capability"),
- }
- )
- if skills:
- deployment["skills"] = skills
- return AgentPlan(
- launch_id=launch_id,
- agent_id=agent_id,
- revision=revision,
- desired_state=value["desired_state"],
- deployment=deployment,
- switch_credentials=switch_credentials,
- worker_capability=capability,
- workspace=workspace,
- worktree_owner=worktree_owner,
- )
-
-
-def agent_environment(
- runtime: RuntimeConfig,
- identity: MachineIdentity,
- machine_id: str,
- paths: Paths,
- agent_id: str,
-) -> str:
- state = paths.agents / agent_id
- values = {
- "PATH": runtime.path,
- "USER": runtime.agent_user,
- "LOGNAME": runtime.agent_user,
- "SHELL": "/bin/bash",
- "LANG": "C.UTF-8",
- "HOME": str(state / "home"),
- "TMPDIR": str(state / "tmp"),
- "SWITCH_HOST_INSTANCE_ID": identity.instance_id,
- "SWITCH_HOST_BOOT_ID": identity.boot_id,
- "SWITCH_HOST_ASSIGNMENT_GENERATION": str(identity.assignment_generation),
- "SWITCH_HOST_MACHINE_ID": machine_id,
- }
- for name, value in values.items():
- if any(character in value for character in "\r\n\x00\\\"'"):
- raise WorkerError(f"Agent environment value {name} is invalid.")
- return "".join(f"{name}={value}\n" for name, value in values.items())
-
-
-class ImdsV2:
- def __init__(self, opener: Callable[..., Any] = urllib.request.urlopen) -> None:
- self._opener = opener
-
- def instance_id(self) -> str:
- token_request = urllib.request.Request(
- f"{IMDS_BASE}/api/token",
- method="PUT",
- headers={"X-aws-ec2-metadata-token-ttl-seconds": "60"},
- )
- try:
- with self._opener(token_request, timeout=2) as response:
- token = response.read(4096).decode().strip()
- if not token:
- raise ValueError()
- identity_request = urllib.request.Request(
- f"{IMDS_BASE}/meta-data/instance-id",
- headers={"X-aws-ec2-metadata-token": token},
- )
- with self._opener(identity_request, timeout=2) as response:
- instance_id = response.read(128).decode().strip()
- except Exception:
- raise WorkerError("IMDSv2 instance identity is unavailable.") from None
- if not INSTANCE_RE.fullmatch(instance_id):
- raise WorkerError("IMDSv2 returned an invalid instance identity.")
- return instance_id
-
-
-class SecretsManager:
- def __init__(self, region: str, client: Any = None) -> None:
- if client is None:
- try:
- import boto3 # type: ignore[import-not-found]
- except ImportError:
- raise WorkerError("The pinned AMI is missing boto3.") from None
- client = boto3.client("secretsmanager", region_name=region)
- self._client = client
-
- def read(self, secret_id: str) -> str:
- try:
- response = self._client.get_secret_value(
- SecretId=secret_id, VersionStage="AWSCURRENT"
- )
- value = response.get("SecretString")
- except Exception:
- raise WorkerError("The assignment secret could not be read.") from None
- if not isinstance(value, str):
- raise WorkerError("The assignment secret is not a JSON string.")
- return value
-
-
-class _NoRedirect(urllib.request.HTTPRedirectHandler):
- def redirect_request(self, req, fp, code, msg, headers, newurl): # type: ignore[no-untyped-def]
- return None
-
-
-class CoreClient:
- def __init__(
- self,
- bundle: MachineBundle,
- identity: MachineIdentity,
- opener: Callable[[urllib.request.Request, float], Any],
- ) -> None:
- self._base = (
- bundle.api_endpoint.rstrip("/")
- + "/hosted/machines/"
- + quote(bundle.machine_id, safe="")
- )
- self._headers = {
- "Authorization": "Bearer " + bundle.machine_capability,
- "X-Switch-Host-Boot-Id": identity.boot_id,
- "X-Switch-Host-Instance-Id": identity.instance_id,
- "Accept": "application/json",
- }
- self._opener = opener
-
- def agents(self) -> dict[str, Any]:
- value = self._request("/agents", None)
- if (
- not isinstance(value, dict)
- or isinstance(value.get("agents_version"), bool)
- or not isinstance(value.get("agents_version"), int)
- or not isinstance(value.get("agents"), list)
- ):
- raise CoreUnavailable("Switch returned an invalid agent list.")
- return value
-
- def heartbeat(self, body: dict[str, Any]) -> dict[str, Any]:
- value = self._request("/heartbeat", body)
- if (
- not isinstance(value, dict)
- or isinstance(value.get("agents_version"), bool)
- or not isinstance(value.get("agents_version"), int)
- ):
- raise CoreUnavailable("Switch returned an invalid heartbeat response.")
- return value
-
- def _request(self, suffix: str, body: dict[str, Any] | None) -> Any:
- headers = dict(self._headers)
- data = None
- if body is not None:
- headers["Content-Type"] = "application/json"
- data = json.dumps(body, separators=(",", ":")).encode()
- request = urllib.request.Request(
- self._base + suffix,
- data=data,
- headers=headers,
- method="GET" if body is None else "POST",
- )
- try:
- with self._opener(request, HTTP_TIMEOUT_SECONDS) as response:
- raw = response.read(MAX_RESPONSE_BYTES + 1)
- except urllib.error.HTTPError as error:
- error.close()
- if error.code == 401:
- raise WorkerError(
- "Switch rejected the machine capability; restarting to re-read the bundle."
- ) from None
- if error.code == 410:
- raise MachineRetired() from None
- raise CoreUnavailable(
- f"Switch returned HTTP {error.code} for {suffix}."
- ) from None
- except (urllib.error.URLError, OSError, ValueError) as error:
- raise CoreUnavailable(
- f"Switch is unreachable for {suffix}: {type(error).__name__}."
- ) from None
- if len(raw) > MAX_RESPONSE_BYTES:
- raise CoreUnavailable(f"Switch response for {suffix} is too large.")
- try:
- return json.loads(raw)
- except (json.JSONDecodeError, UnicodeError):
- raise CoreUnavailable(
- f"Switch response for {suffix} is not JSON."
- ) from None
-
-
-def default_opener() -> Callable[[urllib.request.Request, float], Any]:
- opener = urllib.request.build_opener(_NoRedirect())
- return lambda request, timeout: opener.open(request, timeout=timeout)
-
-
-class Commands:
- def result(
- self, arguments: list[str], *, capture: bool = True
- ) -> subprocess.CompletedProcess[str]:
- try:
- return subprocess.run(
- arguments,
- check=False,
- text=True,
- stdout=subprocess.PIPE if capture else subprocess.DEVNULL,
- stderr=subprocess.PIPE,
- env={"PATH": "/usr/sbin:/usr/bin:/sbin:/bin", "LANG": "C"},
- )
- except OSError:
- raise WorkerError(
- f"Required host operation failed: {arguments[0]}."
- ) from None
-
- def run(self, arguments: list[str], *, capture: bool = True) -> str:
- completed = self.result(arguments, capture=capture)
- if completed.returncode != 0:
- raise WorkerError(f"Required host operation failed: {arguments[0]}.")
- return completed.stdout if capture else ""
-
-
-def agent_unit(agent_id: str) -> str:
- return f"switch-agent@{_agent_id(agent_id)}.service"
-
-
-SHOW_PROPERTIES = (
- "ActiveState",
- "SubState",
- "Result",
- "NRestarts",
- "ExecMainStatus",
- "ExecMainCode",
- "ExecMainExitTimestampMonotonic",
-)
-
-
-class Systemd:
- def __init__(self, commands: Commands) -> None:
- self._commands = commands
-
- def show(self, agent_id: str) -> dict[str, str]:
- output = self._commands.run(
- [SYSTEMCTL, "show", "-p", ",".join(SHOW_PROPERTIES), agent_unit(agent_id)]
- )
- values = dict(line.split("=", 1) for line in output.splitlines() if "=" in line)
- if not set(SHOW_PROPERTIES) <= set(values):
- raise WorkerError(f"systemctl show returned no state for {agent_id}.")
- return values
-
- def start(self, agent_id: str) -> None:
- self._commands.run(
- [SYSTEMCTL, "--no-block", "start", agent_unit(agent_id)], capture=False
- )
-
- def restart(self, agent_id: str) -> None:
- self._commands.run(
- [SYSTEMCTL, "--no-block", "restart", agent_unit(agent_id)], capture=False
- )
-
- def stop(self, agent_id: str, *, wait: bool) -> None:
- self._commands.run(
- [
- SYSTEMCTL,
- *([] if wait else ["--no-block"]),
- "stop",
- agent_unit(agent_id),
- ],
- capture=False,
- )
-
- def reset_failed(self, agent_id: str) -> None:
- self._commands.result(
- [SYSTEMCTL, "reset-failed", agent_unit(agent_id)], capture=False
- )
-
- def stop_all(self) -> None:
- self._commands.run([SYSTEMCTL, "stop", "switch-agent@*.service"], capture=False)
-
- def limit_slice(self, memory_max: int) -> None:
- self._commands.run(
- [
- SYSTEMCTL,
- "set-property",
- "--runtime",
- AGENT_SLICE,
- f"MemoryMax={memory_max}",
- ],
- capture=False,
- )
-
-
-def setpriv_prefix(uid: int, gid: int) -> list[str]:
- return [
- SETPRIV,
- f"--reuid={uid}",
- f"--regid={gid}",
- "--clear-groups",
- "--no-new-privs",
- "--inh-caps=-all",
- "--ambient-caps=-all",
- "--bounding-set=-all",
- ]
-
-
-class GitRunner:
- def __init__(self, prefix: list[str], flock: str, git: str) -> None:
- self._prefix = prefix
- self._flock = flock
- self._git = git
-
- def run(self, mirror: Path, arguments: list[str]) -> None:
- command = [
- *self._prefix,
- self._flock,
- "--no-fork",
- f"{mirror}.lock",
- self._git,
- "-C",
- str(mirror),
- *arguments,
- ]
- label = f"git {' '.join(arguments[:2])} on {mirror}"
- try:
- process = subprocess.Popen(
- command,
- stdin=subprocess.DEVNULL,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
- start_new_session=True,
- env={
- "PATH": "/usr/bin:/bin",
- "LANG": "C",
- "HOME": "/nonexistent",
- "GIT_CONFIG_NOSYSTEM": "1",
- "GIT_CONFIG_GLOBAL": "/dev/null",
- "GIT_TERMINAL_PROMPT": "0",
- },
- )
- except OSError:
- raise WorkerError(f"{label} could not run.") from None
- try:
- returncode = process.wait(timeout=GIT_TIMEOUT_SECONDS)
- except subprocess.TimeoutExpired:
- _kill_process_group(process, label)
- raise WorkerError(f"{label} timed out and was killed.") from None
- if returncode != 0:
- raise WorkerError(f"{label} failed with exit status {returncode}.")
-
-
-def _kill_process_group(process: subprocess.Popen[bytes], label: str) -> None:
- try:
- os.killpg(process.pid, signal.SIGKILL)
- except (ProcessLookupError, PermissionError):
- pass
- process.wait()
- deadline = time.monotonic() + GIT_KILL_WAIT_SECONDS
- while True:
- try:
- os.killpg(process.pid, signal.SIGKILL)
- except ProcessLookupError:
- return
- except PermissionError:
- pass
- if time.monotonic() >= deadline:
- raise GitAbandoned(f"{label} timed out and its processes did not exit.")
- time.sleep(0.1)
-
-
-def inspect_storage(
- commands: Commands, device_path: str, volume_id: str
-) -> StorageObservation:
- try:
- value = json.loads(
- commands.run(
- [
- "/usr/bin/lsblk",
- "--json",
- "--paths",
- "--output",
- "PATH,TYPE,FSTYPE,UUID,SERIAL,MOUNTPOINTS",
- ]
- )
- )
- serial_id = volume_id.replace("-", "")
- devices = [
- item
- for item in value["blockdevices"]
- if item.get("type") == "disk"
- and str(item.get("serial") or "").lower().replace("-", "") == serial_id
- ]
- if len(devices) != 1:
- raise ValueError()
- device = devices[0]
- except (KeyError, TypeError, ValueError, json.JSONDecodeError):
- raise WorkerError(
- "Data volume inspection returned an invalid result."
- ) from None
- resolved_path = _absolute_path(device.get("path"), "resolved data device")
- serial = str(device.get("serial") or "").lower().replace("-", "")
- if device.get("type") != "disk" or serial != volume_id.replace("-", ""):
- raise WorkerError(
- "Attached data device does not match the assigned EBS volume."
- )
- children = bool(device.get("children"))
- signatures: tuple[str, ...] = ()
- if not device.get("fstype") and not children:
- try:
- signatures_value = json.loads(
- commands.run(["/usr/sbin/wipefs", "--json", "--no-act", resolved_path])
- )
- signatures = tuple(
- str(item.get("type") or item.get("usage") or "unknown")
- for item in signatures_value.get("signatures", [])
- )
- except (AttributeError, TypeError, ValueError, json.JSONDecodeError):
- raise WorkerError("Data volume signature inspection failed.") from None
- return StorageObservation(
- device_path=resolved_path,
- volume_id=volume_id,
- filesystem_type=device.get("fstype") or None,
- filesystem_uuid=device.get("uuid") or None,
- has_children=children,
- signatures=signatures,
- )
-
-
-def _validate_root_directory(path: Path, *, create: bool) -> None:
- if create:
- path.mkdir(mode=0o755, parents=True, exist_ok=True)
- try:
- details = path.lstat()
- except OSError:
- raise WorkerError(f"{path} is unavailable.") from None
- if (
- not stat.S_ISDIR(details.st_mode)
- or stat.S_ISLNK(details.st_mode)
- or details.st_uid != ROOT_UID
- or details.st_mode & 0o022
- ):
- raise WorkerError(f"{path} must be a root-owned non-writable directory.")
-
-
-def prepare_storage(
- commands: Commands, config: WorkerConfig
-) -> tuple[StorageObservation, bool]:
- observation = inspect_storage(commands, config.device_path, config.volume_id)
- formatted = False
- if observation.filesystem_type is None:
- if (
- not config.runtime.allow_initial_format
- or observation.filesystem_uuid is not None
- or observation.has_children
- or observation.signatures
- ):
- raise WorkerError("Data volume is not a safely initializable blank disk.")
- commands.run(
- [
- "/usr/sbin/mkfs.ext4",
- "-q",
- "-m",
- "0",
- "-L",
- "switch-hosted-data",
- observation.device_path,
- ],
- capture=False,
- )
- commands.run(
- ["/usr/bin/udevadm", "trigger", "--action=change", observation.device_path]
- )
- commands.run(["/usr/bin/udevadm", "settle", "--timeout=30"])
- observation = inspect_storage(commands, config.device_path, config.volume_id)
- formatted = True
- if (
- observation.filesystem_type != "ext4"
- or not observation.filesystem_uuid
- or observation.has_children
- ):
- raise WorkerError(
- "Data volume filesystem is unexpected; refusing to format or mount it."
- )
- _validate_root_directory(DATA_MOUNT, create=True)
- mounted = commands.result(
- [
- "/usr/bin/findmnt",
- "--mountpoint",
- str(DATA_MOUNT),
- "--noheadings",
- "--output",
- "SOURCE",
- ]
- )
- if mounted.returncode == 0:
- mounted_source = mounted.stdout.strip()
- if os.path.realpath(mounted_source) != os.path.realpath(
- observation.device_path
- ):
- raise WorkerError("The data mountpoint is occupied by another device.")
- elif mounted.returncode == 1:
- commands.run(
- [
- SYSTEMD_MOUNT,
- "--type=ext4",
- "--options=nodev,nosuid",
- observation.device_path,
- str(DATA_MOUNT),
- ],
- capture=False,
- )
- else:
- raise WorkerError("Data mountpoint inspection failed.")
- _validate_root_directory(DATA_MOUNT, create=False)
- return observation, formatted
-
-
-def _read_boot_id(path: Path = BOOT_ID_PATH) -> str:
- try:
- value = path.read_text(encoding="ascii").strip()
- return str(uuid.UUID(value))
- except (OSError, ValueError):
- raise WorkerError("Kernel boot identity is unavailable.") from None
-
-
-def acquire_root_lock(path: Path = LOCK_PATH) -> Any:
- path.parent.mkdir(mode=0o755, parents=True, exist_ok=True)
- descriptor = os.open(path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600)
- details = os.fstat(descriptor)
- if (
- not stat.S_ISREG(details.st_mode)
- or details.st_uid != ROOT_UID
- or details.st_mode & 0o077
- ):
- os.close(descriptor)
- raise WorkerError("Worker root lock is not a private root-owned file.")
- try:
- fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
- except BlockingIOError:
- os.close(descriptor)
- raise WorkerError(
- "Another trusted worker supervisor already owns this instance."
- ) from None
- return os.fdopen(descriptor, "r+")
-
-
-def reconcile_marker(
- identity: MachineIdentity,
- config: WorkerConfig,
- filesystem_uuid: str,
- runtime_fingerprint: str,
- paths: Paths,
-) -> set[str]:
- marker_directory = paths.marker_directory
- marker_directory.mkdir(mode=0o700, parents=False, exist_ok=True)
- os.chown(marker_directory, 0, 0)
- os.chmod(marker_directory, 0o700)
- if paths.marker.exists() or paths.marker.is_symlink():
- marker = _read_root_marker(paths.marker)
- if (
- marker["installationId"] != config.installation_id
- or marker["slotId"] != config.slot_id
- ):
- raise WorkerError("Retained disk belongs to another installation or slot.")
- if marker["generation"] != config.generation:
- raise WorkerError("Retained disk belongs to another assignment generation.")
- if marker["filesystemUuid"] != filesystem_uuid:
- raise WorkerError("Retained disk filesystem identity changed.")
- if marker["bootId"] != identity.boot_id:
- blocked = _quarantine_agents(
- paths,
- [path.name for path in _agent_state_directories(paths.agents)],
- marker["bootId"],
- identity.boot_id,
- )
- else:
- blocked = _retry_blocked_ownership(paths, identity.boot_id)
- if (
- marker["instanceId"] == identity.instance_id
- and marker["runtimeFingerprint"] == runtime_fingerprint
- ):
- return blocked
- else:
- blocked = set()
- unexpected = {
- child.name
- for child in paths.data.iterdir()
- if child.name not in {"lost+found", marker_directory.name}
- }
- if unexpected:
- raise WorkerError(
- "Retained disk has no trusted machine marker; refusing legacy state."
- )
- _write_root_json(
- paths.marker,
- {
- "version": 2,
- "installationId": config.installation_id,
- "slotId": config.slot_id,
- "generation": config.generation,
- "instanceId": identity.instance_id,
- "bootId": identity.boot_id,
- "filesystemUuid": filesystem_uuid,
- "runtimeFingerprint": runtime_fingerprint,
- "layout": MARKER_LAYOUT,
- },
- )
- return blocked
-
-
-def _quarantine_agents(
- paths: Paths, agent_ids: list[str], previous_boot_id: str, current_boot_id: str
-) -> set[str]:
- blocked: set[str] = set()
- for agent_id in agent_ids:
- try:
- _quarantine_stale_ownership(
- paths.agents / agent_id,
- paths.quarantine,
- agent_id,
- previous_boot_id,
- current_boot_id,
- )
- except (WorkerError, OSError) as error:
- logger.error(
- "Agent %s ownership could not be quarantined; it will not start: %s",
- agent_id,
- error,
- )
- blocked.add(agent_id)
- if blocked:
- _write_root_json(
- paths.ownership_blocked,
- {
- "version": 1,
- "bootId": current_boot_id,
- "previousBootId": previous_boot_id,
- "agents": sorted(blocked),
- },
- )
- else:
- paths.ownership_blocked.unlink(missing_ok=True)
- return blocked
-
-
-def _retry_blocked_ownership(paths: Paths, boot_id: str) -> set[str]:
- path = paths.ownership_blocked
- if not path.exists() and not path.is_symlink():
- return set()
- try:
- details = path.lstat()
- if (
- not stat.S_ISREG(details.st_mode)
- or details.st_uid != ROOT_UID
- or details.st_mode & 0o077
- ):
- raise ValueError()
- value = _strict(
- _read_json_nofollow(path),
- {"version", "bootId", "previousBootId", "agents"},
- set(),
- "blocked ownership",
- )
- if (
- value["version"] != 1
- or value["bootId"] != boot_id
- or not isinstance(value["previousBootId"], str)
- or not isinstance(value["agents"], list)
- ):
- raise ValueError()
- agent_ids = [_agent_id(agent_id) for agent_id in value["agents"]]
- except (OSError, ValueError, WorkerError):
- raise WorkerError("Blocked ownership record is invalid.") from None
- return _quarantine_agents(paths, agent_ids, value["previousBootId"], boot_id)
-
-
-def _read_root_marker(path: Path) -> dict[str, Any]:
- try:
- details = path.lstat()
- if (
- not stat.S_ISREG(details.st_mode)
- or details.st_uid != ROOT_UID
- or details.st_mode & 0o077
- or details.st_size > 4096
- ):
- raise ValueError()
- value = json.loads(path.read_text(encoding="utf-8"))
- except (OSError, ValueError, json.JSONDecodeError):
- raise WorkerError("Retained disk machine marker is invalid.") from None
- if isinstance(value, dict) and value.get("version") == 1:
- raise WorkerError(ONE_AGENT_LAYOUT_MESSAGE)
- try:
- value = _strict(
- value,
- {
- "version",
- "installationId",
- "slotId",
- "generation",
- "instanceId",
- "bootId",
- "filesystemUuid",
- "runtimeFingerprint",
- "layout",
- },
- set(),
- "machine marker",
- )
- if value["version"] != 2 or value["layout"] != MARKER_LAYOUT:
- raise ValueError()
- _identifier(value["installationId"], "marker installation ID")
- _identifier(value["slotId"], "marker slot ID")
- _positive_integer(value["generation"], "marker generation")
- if not INSTANCE_RE.fullmatch(_text(value["instanceId"], "marker instance")):
- raise ValueError()
- if str(uuid.UUID(_text(value["bootId"], "marker boot ID"))) != value["bootId"]:
- raise ValueError()
- str(uuid.UUID(_text(value["filesystemUuid"], "filesystem UUID")))
- if not FINGERPRINT_RE.fullmatch(
- _text(value["runtimeFingerprint"], "runtime fingerprint")
- ):
- raise ValueError()
- return value
- except (ValueError, WorkerError):
- raise WorkerError("Retained disk machine marker is invalid.") from None
-
-
-def _agent_state_directories(agents: Path) -> list[Path]:
- if not agents.exists() and not agents.is_symlink():
- return []
- _validate_root_directory(agents, create=False)
- result = []
- for child in sorted(agents.iterdir()):
- if not _is_agent_id(child.name):
- logger.warning("Ignoring unexpected entry %s in %s.", child.name, agents)
- continue
- result.append(child)
- return result
-
-
-def _validated_directory(path: Path, *, root: Path) -> None:
- try:
- path.relative_to(root)
- details = path.lstat()
- except (OSError, ValueError):
- raise WorkerError("Saved ownership path is invalid.") from None
- if (
- not stat.S_ISDIR(details.st_mode)
- or stat.S_ISLNK(details.st_mode)
- or details.st_mode & 0o022
- ):
- raise WorkerError("Saved ownership directory is invalid.")
-
-
-def _read_json_nofollow(path: Path, maximum: int = 16 * 1024) -> Any:
- descriptor = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)
- try:
- details = os.fstat(descriptor)
- if not stat.S_ISREG(details.st_mode) or details.st_size > maximum:
- raise WorkerError("Saved ownership record is invalid.")
- with os.fdopen(descriptor, "r", encoding="utf-8", closefd=False) as handle:
- return json.load(handle)
- except (OSError, UnicodeError, json.JSONDecodeError):
- raise WorkerError("Saved ownership record is invalid.") from None
- finally:
- os.close(descriptor)
-
-
-def _ownership_paths(state_path: Path) -> list[Path]:
- if not state_path.exists():
- return []
- _validated_directory(state_path, root=state_path)
- paths = [state_path / "shared-owner.lock"]
- supervisor = state_path / "supervisor"
- if supervisor.exists() or supervisor.is_symlink():
- _validated_directory(supervisor, root=state_path)
- paths.append(supervisor / "owner.json")
- launch = state_path / "launch"
- if launch.exists() or launch.is_symlink():
- _validated_directory(launch, root=state_path)
- for directory in [
- state_path / "ownership",
- supervisor / "ownership",
- launch / "ownership",
- ]:
- if directory.exists() or directory.is_symlink():
- _validated_directory(directory, root=state_path)
- paths.extend(sorted(directory.iterdir()))
- sessions = state_path
- for component in ("home", ".local", "state", "switch", "sdk-sessions"):
- sessions = sessions / component
- if not sessions.exists() and not sessions.is_symlink():
- break
- _validated_directory(sessions, root=state_path)
- else:
- for root in sorted(sessions.iterdir()):
- if not re.fullmatch(r"[0-9a-f]{64}", root.name):
- raise WorkerError("Saved SDK session directory is invalid.")
- _validated_directory(root, root=state_path)
- paths.extend(_ownership_paths(root))
- return [path for path in paths if path.exists() or path.is_symlink()]
-
-
-def _known_owner_relative(relative: Path) -> bool:
- parts = relative.parts
- if (
- len(parts) > 6
- and parts[:5] == ("home", ".local", "state", "switch", "sdk-sessions")
- and re.fullmatch(r"[0-9a-f]{64}", parts[5])
- ):
- return _known_owner_relative(Path(*parts[6:]))
- if parts in {("shared-owner.lock",), ("supervisor", "owner.json")}:
- return True
- return (
- len(parts) == 2
- and parts[0] == "ownership"
- or len(parts) == 3
- and parts[:2] in {("supervisor", "ownership"), ("launch", "ownership")}
- )
-
-
-def _validate_owner_record(relative: Path, path: Path) -> None:
- name = relative.name
- if relative.parent.name == "ownership" and OWNERSHIP_TEMPORARY_RE.fullmatch(name):
- return
- value = _read_json_nofollow(path)
- try:
- if name == "shared-owner.lock":
- record = _strict(value, {"pid", "token"}, {"group"}, "owner record")
- if record.get("group") is not None:
- _positive_integer(record["group"], "owner process group")
- elif name == "owner.json":
- record = _strict(value, {"pid", "token", "build"}, set(), "owner record")
- _text(record["build"], "owner build")
- elif OWNERSHIP_TICKET_RE.fullmatch(name):
- record = _strict(value, {"choosing", "ticket"}, set(), "ownership ticket")
- ticket = record["ticket"]
- if (
- not isinstance(record["choosing"], bool)
- or isinstance(ticket, bool)
- or not isinstance(ticket, int)
- or ticket < 0
- ):
- raise WorkerError("ownership ticket is invalid.")
- return
- else:
- raise WorkerError("ownership record name is invalid.")
- _positive_integer(record["pid"], "owner PID")
- _text(record["token"], "owner token")
- except WorkerError:
- raise WorkerError("Saved ownership record is invalid.") from None
-
-
-def _validate_quarantine_tree(directory: Path) -> dict[Path, Path]:
- result: dict[Path, Path] = {}
- if not directory.exists() and not directory.is_symlink():
- return result
- _validated_directory(directory, root=directory)
- if (
- directory.lstat().st_uid != ROOT_UID
- or stat.S_IMODE(directory.lstat().st_mode) != 0o700
- ):
- raise WorkerError("Ownership quarantine is not a private root-owned directory.")
- for root, names, files in os.walk(directory, followlinks=False):
- root_path = Path(root)
- _validated_directory(root_path, root=directory)
- if root_path.lstat().st_uid != ROOT_UID:
- raise WorkerError("Ownership quarantine directory is not root-owned.")
- for name in names:
- child = root_path / name
- if child.is_symlink():
- raise WorkerError("Ownership quarantine contains a symlink.")
- for name in files:
- path = root_path / name
- relative = path.relative_to(directory)
- if not _known_owner_relative(relative) or path.is_symlink():
- raise WorkerError("Ownership quarantine contains an unknown record.")
- _validate_owner_record(relative, path)
- result[relative] = path
- return result
-
-
-def _private_root_directory(path: Path) -> None:
- path.mkdir(mode=0o700, exist_ok=True)
- os.chown(path, 0, 0)
- os.chmod(path, 0o700)
-
-
-def _quarantine_stale_ownership(
- state_path: Path,
- quarantine_root: Path,
- agent_id: str,
- previous_boot_id: str,
- current_boot_id: str,
-) -> None:
- sources: dict[Path, Path] = {}
- for path in _ownership_paths(state_path):
- if path.is_symlink() or not path.is_file():
- raise WorkerError("Saved ownership record is invalid.")
- relative = path.relative_to(state_path)
- _validate_owner_record(relative, path)
- sources[relative] = path
- if not sources:
- return
- agent_quarantine = quarantine_root / agent_id
- quarantine = agent_quarantine / f"{previous_boot_id}--{current_boot_id}"
- for directory in (quarantine_root, agent_quarantine, quarantine):
- _private_root_directory(directory)
- existing = _validate_quarantine_tree(quarantine)
- collisions = set(sources) & set(existing)
- if collisions:
- raise WorkerError(
- "Ownership quarantine collides with a saved ownership record."
- )
- for relative, path in sources.items():
- target = quarantine / relative
- target.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
- os.chown(target.parent, 0, 0)
- os.chmod(target.parent, 0o700)
- os.replace(path, target)
- _fsync_directory(target.parent)
- _fsync_directory(quarantine)
- _fsync_directory(agent_quarantine)
-
-
-def _write_root_json(path: Path, value: Any) -> None:
- path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
- descriptor, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
- try:
- os.fchmod(descriptor, 0o600)
- with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
- json.dump(value, handle, separators=(",", ":"))
- handle.flush()
- os.fsync(handle.fileno())
- os.replace(temporary, path)
- _fsync_directory(path.parent)
- finally:
- try:
- os.unlink(temporary)
- except FileNotFoundError:
- pass
-
-
-def _fsync_directory(path: Path) -> None:
- descriptor = os.open(path, os.O_RDONLY | os.O_DIRECTORY)
- try:
- os.fsync(descriptor)
- finally:
- os.close(descriptor)
-
-
-def resolve_agent_account(user: str, group: str) -> tuple[int, int]:
- try:
- account = pwd.getpwnam(user)
- gid = grp.getgrnam(group).gr_gid
- except KeyError:
- raise WorkerError(
- "Pinned AMI is missing the unprivileged agent account."
- ) from None
- if account.pw_uid == 0 or gid == 0 or account.pw_gid != gid:
- raise WorkerError(
- "Pinned AMI agent account must use its non-root primary group."
- )
- return account.pw_uid, gid
-
-
-def _root_directory(path: Path, mode: int, gid: int) -> None:
- if not path.exists() and not path.is_symlink():
- path.mkdir(mode=mode)
- details = path.lstat()
- if (
- not stat.S_ISDIR(details.st_mode)
- or stat.S_ISLNK(details.st_mode)
- or details.st_uid != ROOT_UID
- ):
- raise WorkerError(f"{path} must be a root-owned directory.")
- os.chown(path, 0, gid)
- os.chmod(path, mode)
-
-
-DIRECTORY_FLAGS = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW
-
-
-def _open_trusted_root(path: Path) -> int:
- try:
- descriptor = os.open(path, DIRECTORY_FLAGS)
- except OSError:
- raise WorkerError(f"{path} is unavailable.") from None
- details = os.fstat(descriptor)
- if details.st_uid != ROOT_UID or details.st_mode & 0o022:
- os.close(descriptor)
- raise WorkerError(f"{path} must be a root-owned non-writable directory.")
- return descriptor
-
-
-def _agent_directories(root: Path, parts: tuple[str, ...], uid: int, gid: int) -> None:
- """Create or repair each directory of root/parts as a private agent directory.
-
- Every component is opened relative to its already-open parent with
- O_NOFOLLOW and changed through its descriptor, so an agent that swaps a
- component for a symlink cannot redirect root's chown or chmod.
- """
- descriptor = _open_trusted_root(root)
- path = root
- try:
- for name in parts:
- if name in {"", ".", ".."} or "/" in name:
- raise WorkerError(f"{root} has an unsafe path component.")
- path = path / name
- try:
- os.mkdir(name, 0o700, dir_fd=descriptor)
- except FileExistsError:
- pass
- try:
- child = os.open(name, DIRECTORY_FLAGS, dir_fd=descriptor)
- except OSError:
- raise WorkerError(
- f"{path} is not a private agent-owned directory."
- ) from None
- os.close(descriptor)
- descriptor = child
- details = os.fstat(descriptor)
- if details.st_uid not in {ROOT_UID, uid}:
- raise WorkerError(f"{path} is not a private agent-owned directory.")
- if details.st_uid != uid or details.st_gid != gid:
- os.fchown(descriptor, uid, gid)
- if stat.S_IMODE(details.st_mode) != 0o700:
- os.fchmod(descriptor, 0o700)
- finally:
- os.close(descriptor)
-
-
-def prepare_layout(paths: Paths, uid: int, gid: int) -> None:
- _root_directory(paths.agents, 0o755, 0)
- _root_directory(paths.worktrees, 0o755, 0)
- _agent_directories(paths.data, (paths.repos.name,), uid, gid)
-
-
-def prepare_runtime_directory(commands: Commands, paths: Paths, gid: int) -> None:
- filesystem = commands.run(
- [
- "/usr/bin/findmnt",
- "--noheadings",
- "--output",
- "FSTYPE",
- "--target",
- str(paths.runtime),
- ]
- ).strip()
- if filesystem != "tmpfs":
- raise WorkerError("Runtime secret directory is not backed by tmpfs.")
- _validate_root_directory(paths.runtime, create=False)
- os.chown(paths.runtime, 0, gid)
- os.chmod(paths.runtime, 0o750)
- _root_directory(paths.bundle.parent, 0o700, 0)
- _root_directory(paths.agents_runtime, 0o750, gid)
-
-
-def write_bundle(paths: Paths, bundle: MachineBundle) -> None:
- _write_root_json(
- paths.bundle,
- {
- "version": 2,
- "machineId": bundle.machine_id,
- "machineCapability": bundle.machine_capability,
- "apiEndpoint": bundle.api_endpoint,
- },
- )
-
-
-def _validate_secret_tree(directory: Path) -> None:
- details = directory.lstat()
- if (
- not stat.S_ISDIR(details.st_mode)
- or stat.S_ISLNK(details.st_mode)
- or details.st_uid != ROOT_UID
- or details.st_mode & 0o027
- ):
- raise WorkerError("Runtime secret directory is unsafe.")
- for root, names, files in os.walk(directory, followlinks=False):
- root_path = Path(root)
- root_details = root_path.lstat()
- if (
- not stat.S_ISDIR(root_details.st_mode)
- or stat.S_ISLNK(root_details.st_mode)
- or root_details.st_uid != ROOT_UID
- or root_details.st_mode & 0o027
- ):
- raise WorkerError("Runtime secret directory contains an unsafe directory.")
- for name in names:
- if (root_path / name).is_symlink():
- raise WorkerError("Runtime secret directory contains a symlink.")
- for name in files:
- path = root_path / name
- file_details = path.lstat()
- if (
- not stat.S_ISREG(file_details.st_mode)
- or stat.S_ISLNK(file_details.st_mode)
- or file_details.st_uid != ROOT_UID
- or file_details.st_mode & 0o022
- ):
- raise WorkerError("Runtime secret directory contains an unsafe file.")
-
-
-def _remove_secret_tree(directory: Path) -> None:
- _validate_secret_tree(directory)
- shutil.rmtree(directory)
-
-
-def remove_runtime_orphans(parent: Path) -> None:
- for orphan in parent.iterdir():
- if orphan.name.startswith("."):
- _remove_secret_tree(orphan)
-
-
-def install_runtime_files(
- parent: Path, name: str, files: dict[str, str], gid: int
-) -> None:
- target = parent / name
- temporary = parent / f".tmp-{uuid.uuid4()}"
- previous = parent / f".old-{uuid.uuid4()}"
- try:
- temporary.mkdir(mode=0o750)
- directory_fd = os.open(temporary, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
- try:
- os.fchown(directory_fd, 0, gid)
- os.fchmod(directory_fd, 0o750)
- for file_name, value in files.items():
- descriptor = os.open(
- file_name,
- os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW,
- 0o440,
- dir_fd=directory_fd,
- )
- try:
- os.fchown(descriptor, 0, gid)
- os.fchmod(descriptor, 0o440)
- os.write(descriptor, value.encode())
- os.fsync(descriptor)
- finally:
- os.close(descriptor)
- os.fsync(directory_fd)
- finally:
- os.close(directory_fd)
- if target.exists() or target.is_symlink():
- _validate_secret_tree(target)
- os.replace(target, previous)
- os.replace(temporary, target)
- if previous.exists():
- _remove_secret_tree(previous)
- except Exception:
- for path in (temporary, previous):
- if path.exists() and not path.is_symlink():
- try:
- _remove_secret_tree(path)
- except WorkerError:
- pass
- raise
-
-
-def _remove_tree(path: Path) -> None:
- if path.is_symlink():
- path.unlink()
- elif path.exists():
- shutil.rmtree(path)
-
-
-def _sha256_file(path: str) -> str:
- digest = hashlib.sha256()
- try:
- with open(path, "rb") as handle:
- for chunk in iter(lambda: handle.read(1024 * 1024), b""):
- digest.update(chunk)
- except OSError:
- raise WorkerError("Pinned runtime artifact is unreadable.") from None
- return digest.hexdigest()
-
-
-def verify_pinned_runtime(config: WorkerConfig) -> str:
- artifacts = {
- "node": config.runtime.node_path,
- "bootstrap": config.runtime.bootstrap_path,
- "sharedHostDaemon": config.runtime.shared_host_daemon_path,
- "provider": config.runtime.provider_binary_path,
- }
- hashes = dict(config.runtime.artifact_sha256)
- for provider, runtime in config.runtime.providers.items():
- artifacts["provider-" + provider] = runtime["path"]
- hashes["provider-" + provider] = runtime["sha256"]
- for name, path in artifacts.items():
- try:
- details = os.stat(path, follow_symlinks=False)
- except OSError:
- raise WorkerError("Pinned runtime artifact is missing.") from None
- if (
- not stat.S_ISREG(details.st_mode)
- or details.st_uid != ROOT_UID
- or details.st_mode & 0o022
- ):
- raise WorkerError("Pinned runtime artifact is missing or not root-owned.")
- if _sha256_file(path) != hashes[name]:
- raise WorkerError(
- "Pinned runtime artifact checksum does not match the AMI manifest."
- )
- try:
- version = subprocess.run(
- [config.runtime.node_path, "--version"],
- check=True,
- text=True,
- capture_output=True,
- env={"PATH": config.runtime.path, "LANG": "C"},
- ).stdout.strip()
- except (OSError, subprocess.CalledProcessError):
- raise WorkerError(
- "Pinned Node.js executable failed its version check."
- ) from None
- if not re.fullmatch(r"v24\.\d+\.\d+", version):
- raise WorkerError("Pinned AMI does not provide Node.js 24.")
- fingerprint = hashlib.sha256(
- json.dumps(
- {
- "artifacts": hashes,
- "nodeVersion": version,
- },
- sort_keys=True,
- separators=(",", ":"),
- ).encode()
- ).hexdigest()
- return fingerprint
-
-
-def read_meminfo(path: Path) -> tuple[int, int]:
- values: dict[str, int] = {}
- try:
- for line in path.read_text(encoding="ascii").splitlines():
- name, _, rest = line.partition(":")
- if name in {"MemTotal", "MemAvailable"}:
- number, unit = rest.split()
- if unit != "kB":
- raise ValueError()
- values[name] = int(number) * 1024
- return values["MemTotal"], values["MemAvailable"]
- except (OSError, ValueError, KeyError):
- raise WorkerError("Memory information is unavailable.") from None
-
-
-@dataclass(frozen=True)
-class Failure:
- process_state: str
- result: str
-
-
-@dataclass
-class HeldAgent:
- launch_id: str
- agent_id: str
- revision: int
- desired_state: str | None
- failure: Failure | None
-
-
-@dataclass
-class ObservedState:
- process_state: str
- restarts: int
- exit: dict[str, Any] | None
- since: str
-
-
-class Supervisor:
- def __init__(
- self,
- *,
- runtime: RuntimeConfig,
- identity: MachineIdentity,
- machine_id: str,
- runtime_fingerprint: str,
- paths: Paths,
- uid: int,
- gid: int,
- client: CoreClient,
- systemd: Systemd,
- git: GitRunner,
- clock: Callable[[], datetime],
- monotonic: Callable[[], float],
- sleep: Callable[[float], None],
- statvfs: Callable[[str], Any],
- meminfo: Path,
- ownership_blocked: set[str],
- ) -> None:
- self._ownership_blocked = set(ownership_blocked)
- self._runtime = runtime
- self._identity = identity
- self._machine_id = machine_id
- self._fingerprint = runtime_fingerprint
- self._paths = paths
- self._uid = uid
- self._gid = gid
- self._client = client
- self._systemd = systemd
- self._git = git
- self._clock = clock
- self._monotonic = monotonic
- self._sleep = sleep
- self._statvfs = statvfs
- self._meminfo = meminfo
- self._held: list[HeldAgent] = []
- self._states: dict[str, ObservedState] = {}
- self._touched: set[str] = set()
- self._obsolete_exits: dict[str, str] = {}
- self._agents_version: int | None = None
- self._reconcile_failures = 0
- self._pending_prunes: set[Path] = set()
- self._need_list = True
- self._next_list = 0.0
- self._next_heartbeat = 0.0
- self._heartbeat_every = DEFAULT_HEARTBEAT_SECONDS
- self._retired = False
- self._records = self._load_records()
-
- def run(self) -> None:
- self.limit_slice()
- while True:
- self.tick()
- self._sleep(OBSERVE_SECONDS)
-
- def limit_slice(self) -> None:
- total, _available = read_meminfo(self._meminfo)
- if total <= SLICE_RESERVE_BYTES:
- raise WorkerError("The machine has too little memory for agents.")
- self._systemd.limit_slice(total - SLICE_RESERVE_BYTES)
-
- def tick(self) -> None:
- now = self._monotonic()
- if self._retired:
- if now >= self._next_heartbeat:
- self._observe()
- self._send_heartbeat()
- return
- if self._need_list and now >= self._next_list:
- self._refresh()
- if self._retired:
- return
- changed = self._observe()
- if changed or self._monotonic() >= self._next_heartbeat:
- self._send_heartbeat()
- if (
- self._need_list
- and self._monotonic() >= self._next_list
- and not self._retired
- ):
- self._refresh()
-
- def _refresh(self) -> None:
- try:
- listing = self._client.agents()
- except MachineRetired:
- self._retire()
- return
- except CoreUnavailable as error:
- logger.warning("Agent list fetch failed; retrying: %s", error)
- self._next_list = self._monotonic() + LIST_RETRY_SECONDS
- return
- try:
- self.reconcile(listing["agents"])
- except (WorkerError, OSError) as error:
- self._reconcile_failures += 1
- delay = min(
- LIST_RETRY_SECONDS * 2 ** (self._reconcile_failures - 1),
- RECONCILE_RETRY_MAX_SECONDS,
- )
- logger.error(
- "Agents version %s was not fully applied; retrying in %s seconds: %s",
- listing["agents_version"],
- delay,
- error,
- )
- self._next_list = self._monotonic() + delay
- return
- self._reconcile_failures = 0
- self._need_list = False
- self._agents_version = listing["agents_version"]
-
- def _send_heartbeat(self) -> None:
- interval = RETIRED_HEARTBEAT_SECONDS if self._retired else self._heartbeat_every
- try:
- response = self._client.heartbeat(self.heartbeat_body())
- except MachineRetired:
- self._retire()
- return
- except CoreUnavailable as error:
- logger.warning("Heartbeat failed; retrying: %s", error)
- self._next_heartbeat = self._monotonic() + interval
- return
- every = response.get("heartbeat_every_s", DEFAULT_HEARTBEAT_SECONDS)
- if (
- isinstance(every, bool)
- or not isinstance(every, int)
- or not 0 < every <= 3600
- ):
- logger.warning("Heartbeat interval from Switch is invalid; using default.")
- every = DEFAULT_HEARTBEAT_SECONDS
- self._heartbeat_every = every
- if self._retired:
- logger.warning("Switch accepts this machine again; resuming agents.")
- self._retired = False
- self._need_list = True
- self._next_list = 0.0
- if response["agents_version"] != self._agents_version:
- self._need_list = True
- self._next_heartbeat = self._monotonic() + self._heartbeat_every
-
- def _retire(self) -> None:
- if not self._retired:
- logger.warning("Switch retired this machine; stopping all agents.")
- self._systemd.stop_all()
- self._touched.update(held.agent_id for held in self._held)
- self._retired = True
- self._next_heartbeat = self._monotonic() + RETIRED_HEARTBEAT_SECONDS
-
- def reconcile(self, entries: list[Any]) -> None:
- failures: list[str] = []
- remove_runtime_orphans(self._paths.agents_runtime)
- counts: dict[str, int] = {}
- for entry in entries:
- if isinstance(entry, dict) and isinstance(entry.get("agent_id"), str):
- counts[entry["agent_id"]] = counts.get(entry["agent_id"], 0) + 1
- held: list[HeldAgent] = []
- for entry in entries:
- if (
- not isinstance(entry, dict)
- or not isinstance(entry.get("launch_id"), str)
- or not isinstance(entry.get("agent_id"), str)
- or isinstance(entry.get("revision"), bool)
- or not isinstance(entry.get("revision"), int)
- ):
- logger.error(
- "Ignoring an agent entry without launch, agent and revision."
- )
- continue
- agent_id = entry["agent_id"]
- try:
- if counts[agent_id] > 1:
- raise WorkerError("The agent is listed more than once.")
- if agent_id in self._ownership_blocked:
- logger.error(
- "Agent %s has invalid saved ownership; it stays stopped.",
- agent_id,
- )
- held.append(
- HeldAgent(
- entry["launch_id"],
- agent_id,
- entry["revision"],
- None,
- Failure("failed", OWNERSHIP_INVALID),
- )
- )
- self._disable(agent_id, failures)
- continue
- unavailable = _unavailable_code(entry)
- if unavailable is not None:
- logger.warning(
- "Agent %s is unavailable (%s); stopping it and keeping its data.",
- agent_id,
- unavailable,
- )
- held.append(
- HeldAgent(
- entry["launch_id"],
- agent_id,
- entry["revision"],
- "stopped",
- Failure("stopped", INVALID_CONFIG),
- )
- )
- self._disable(agent_id, failures)
- continue
- plan = build_agent_plan(entry, self._runtime, self._paths)
- except WorkerError as error:
- logger.error(
- "Agent %s has an invalid configuration: %s", agent_id, error
- )
- held.append(
- HeldAgent(
- entry["launch_id"],
- agent_id,
- entry["revision"],
- None,
- Failure("failed", INVALID_CONFIG),
- )
- )
- self._disable(agent_id, failures)
- continue
- failure: Failure | None = None
- try:
- self._apply(plan)
- except (WorkerError, OSError) as error:
- logger.error("Agent %s could not be set up: %s", agent_id, error)
- failure = Failure("failed", SETUP_FAILED)
- failures.append(agent_id)
- held.append(
- HeldAgent(
- plan.launch_id,
- plan.agent_id,
- plan.revision,
- plan.desired_state,
- failure,
- )
- )
- self._held = held
- listed = {agent.agent_id for agent in held} | set(counts)
- for agent_id in sorted(self._agents_on_disk() - listed):
- try:
- self.remove_agent(agent_id)
- except (WorkerError, OSError) as error:
- logger.error("Agent %s removal failed; will retry: %s", agent_id, error)
- failures.append(agent_id)
- self._prune_mirrors()
- if self._pending_prunes:
- failures.append("worktree prune")
- for agent_id in set(self._states) - {agent.agent_id for agent in held}:
- del self._states[agent_id]
- if failures:
- raise ReconcileIncomplete(
- "Work is left to retry for: " + ", ".join(failures) + "."
- )
-
- def _disable(self, agent_id: str, failures: list[str]) -> None:
- if not _is_agent_id(agent_id):
- return
- try:
- self._stop(agent_id, wait=False)
- target = self._paths.agents_runtime / agent_id
- if target.exists() or target.is_symlink():
- _remove_secret_tree(target)
- self._paths.pending_install(agent_id).unlink(missing_ok=True)
- except (WorkerError, OSError) as error:
- logger.error("Agent %s could not be stopped: %s", agent_id, error)
- failures.append(agent_id)
-
- def _stop(self, agent_id: str, *, wait: bool) -> None:
- self._systemd.stop(agent_id, wait=wait)
- self._systemd.reset_failed(agent_id)
- self._touched.add(agent_id)
-
- def _prune_mirrors(self) -> None:
- for mirror in sorted(self._pending_prunes):
- if mirror.is_dir() and not mirror.is_symlink():
- try:
- self._git.run(mirror, ["worktree", "prune"])
- except WorkerError as error:
- logger.error("Worktree prune failed; will retry: %s", error)
- continue
- self._pending_prunes.discard(mirror)
-
- def _apply(self, plan: AgentPlan) -> None:
- agent_id = plan.agent_id
- for leaf in ("home", "tmp"):
- _agent_directories(
- self._paths.agents, (agent_id, leaf), self._uid, self._gid
- )
- _agent_directories(
- self._paths.worktrees,
- plan.workspace.relative_to(self._paths.worktrees).parts,
- self._uid,
- self._gid,
- )
- if self._installed_revision(agent_id) != plan.revision:
- pending = self._paths.pending_install(agent_id)
- _write_root_json(pending, {"revision": plan.revision})
- install_runtime_files(
- self._paths.agents_runtime,
- agent_id,
- {
- "switch.json": json.dumps(
- plan.switch_credentials, separators=(",", ":")
- ),
- "deployment.json": json.dumps(
- plan.deployment, separators=(",", ":")
- ),
- "worker-capability": plan.worker_capability,
- "env": agent_environment(
- self._runtime,
- self._identity,
- self._machine_id,
- self._paths,
- agent_id,
- ),
- },
- self._gid,
- )
- if plan.desired_state == "running":
- self._systemd.reset_failed(agent_id)
- self._systemd.restart(agent_id)
- self._touched.add(agent_id)
- else:
- self._stop(agent_id, wait=False)
- self._reset_oom_kills(agent_id, plan.revision)
- pending.unlink()
- return
- active = self._systemd.show(agent_id)["ActiveState"]
- if plan.desired_state == "running" and active == "inactive":
- self._systemd.start(agent_id)
- self._touched.add(agent_id)
- elif plan.desired_state == "stopped" and active in {
- "active",
- "activating",
- "reloading",
- "failed",
- }:
- self._stop(agent_id, wait=False)
-
- def _reset_oom_kills(self, agent_id: str, revision: int) -> None:
- record = self._records.get(agent_id)
- if record is not None and record.get("revision") != revision:
- self._records[agent_id] = {
- "oomKills": 0,
- "lastOomExit": record["lastOomExit"],
- "revision": revision,
- }
- self._save_records()
-
- def _installed_revision(self, agent_id: str) -> int | None:
- pending = self._paths.pending_install(agent_id)
- if pending.exists() or pending.is_symlink():
- return None
- try:
- value = _read_json_nofollow(
- self._paths.agents_runtime / agent_id / "deployment.json",
- maximum=MAX_SECRET_BYTES,
- )
- except (OSError, WorkerError):
- return None
- revision = value.get("revision") if isinstance(value, dict) else None
- return revision if isinstance(revision, int) else None
-
- def _agents_on_disk(self) -> set[str]:
- found: set[str] = set()
- for parent in (
- self._paths.agents_runtime,
- self._paths.agents,
- self._paths.worktrees,
- ):
- if not parent.is_dir():
- continue
- for child in parent.iterdir():
- if parent == self._paths.agents_runtime and child.name.startswith("."):
- continue
- if not _is_agent_id(child.name):
- logger.warning(
- "Ignoring unexpected entry %s in %s.", child.name, parent
- )
- continue
- found.add(child.name)
- return found
-
- def remove_agent(self, agent_id: str) -> None:
- agent_id = _agent_id(agent_id)
- self._stop(agent_id, wait=True)
- runtime_files = self._paths.agents_runtime / agent_id
- if runtime_files.exists() or runtime_files.is_symlink():
- _remove_secret_tree(runtime_files)
- self._paths.pending_install(agent_id).unlink(missing_ok=True)
- worktree_root = self._paths.worktrees / agent_id
- mirrors: list[Path] = []
- if worktree_root.is_dir() and not worktree_root.is_symlink():
- for owner in sorted(worktree_root.iterdir()):
- if owner.is_symlink() or not owner.is_dir():
- continue
- for repository in sorted(owner.iterdir()):
- if repository.is_symlink() or not repository.is_dir():
- continue
- mirror = self._paths.repos / owner.name / f"{repository.name}.git"
- if mirror.is_symlink() or not mirror.is_dir():
- continue
- try:
- self._git.run(
- mirror, ["worktree", "remove", "--force", str(repository)]
- )
- except GitAbandoned:
- raise
- except WorkerError as error:
- logger.warning("Agent %s: %s", agent_id, error)
- mirrors.append(mirror)
- _remove_tree(self._paths.agents / agent_id)
- _remove_tree(worktree_root)
- self._pending_prunes.update(mirrors)
- if self._records.pop(agent_id, None) is not None:
- self._save_records()
- self._states.pop(agent_id, None)
- self._touched.discard(agent_id)
- self._obsolete_exits.pop(agent_id, None)
- self._ownership_blocked.discard(agent_id)
- logger.warning("Removed agent %s from this machine.", agent_id)
-
- def _load_records(self) -> dict[str, dict[str, Any]]:
- path = self._paths.agent_records
- if not path.exists() and not path.is_symlink():
- return {}
- try:
- details = path.lstat()
- if (
- not stat.S_ISREG(details.st_mode)
- or details.st_uid != ROOT_UID
- or details.st_mode & 0o077
- ):
- raise ValueError()
- value = _strict(
- _read_json_nofollow(path, maximum=MAX_SECRET_BYTES),
- {"version", "agents"},
- set(),
- "agent records",
- )
- if value["version"] != 1 or not isinstance(value["agents"], dict):
- raise ValueError()
- for agent_id, record in value["agents"].items():
- _agent_id(agent_id)
- record = _strict(
- record, {"oomKills", "lastOomExit"}, {"revision"}, "record"
- )
- if isinstance(record["oomKills"], bool) or not isinstance(
- record["oomKills"], int
- ):
- raise ValueError()
- if "revision" in record:
- _positive_integer(record["revision"], "record revision")
- return value["agents"]
- except (OSError, ValueError, WorkerError):
- raise WorkerError("Agent records on the data volume are invalid.") from None
-
- def _save_records(self) -> None:
- _write_root_json(
- self._paths.agent_records, {"version": 1, "agents": self._records}
- )
-
- def _observe(self) -> bool:
- changed = False
- now = self._clock().astimezone(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
- for held in self._held:
- observed: tuple[str, int, dict[str, Any] | None]
- if held.failure is not None:
- observed = (
- held.failure.process_state,
- 0,
- {"code": None, "signal": None, "result": held.failure.result},
- )
- else:
- try:
- observed = self._unit_state(held)
- except (WorkerError, ValueError) as error:
- logger.warning(
- "Agent %s state is unavailable: %s", held.agent_id, error
- )
- continue
- process_state, restarts, exit_value = observed
- previous = self._states.get(held.agent_id)
- since = now
- if previous is not None and previous.process_state == process_state:
- since = previous.since
- else:
- changed = True
- self._states[held.agent_id] = ObservedState(
- process_state, restarts, exit_value, since
- )
- return changed
-
- def _unit_state(self, held: HeldAgent) -> tuple[str, int, dict[str, Any] | None]:
- agent_id = held.agent_id
- properties = self._systemd.show(agent_id)
- active = properties["ActiveState"]
- sub = properties["SubState"]
- result = properties["Result"]
- restarts = int(properties["NRestarts"] or 0)
- code = int(properties["ExecMainCode"] or 0)
- status = int(properties["ExecMainStatus"] or 0)
- exited_at = properties["ExecMainExitTimestampMonotonic"] or "0"
- ran = exited_at != "0"
- if result == "oom-kill" and ran:
- key = f"{self._identity.boot_id}:{exited_at}"
- record = self._records.get(agent_id, {"oomKills": 0, "lastOomExit": ""})
- if record["lastOomExit"] != key:
- self._records[agent_id] = {
- "oomKills": record["oomKills"] + 1,
- "lastOomExit": key,
- "revision": held.revision,
- }
- self._save_records()
- if code == 1 and status == OBSOLETE_EXIT_CODE and ran:
- if self._obsolete_exits.get(agent_id) != exited_at:
- self._obsolete_exits[agent_id] = exited_at
- logger.warning(
- "Agent %s reported an obsolete worker; refetching.", agent_id
- )
- self._need_list = True
- self._next_list = 0.0
- if active in {"active", "reloading"}:
- process_state = "running"
- elif active == "activating":
- process_state = (
- "restarting" if sub.startswith("auto-restart") else "starting"
- )
- elif active == "deactivating":
- process_state = "stopping"
- elif active == "failed":
- process_state = "crashed" if result == "start-limit-hit" else "failed"
- elif active == "inactive":
- process_state = (
- "stopped"
- if held.desired_state == "stopped" or agent_id in self._touched or ran
- else "pending"
- )
- else:
- raise WorkerError(f"unknown unit state {active}/{sub}")
- exit_value: dict[str, Any] | None = None
- if process_state != "running":
- if code == 1:
- exit_value = {"code": status, "signal": None, "result": result}
- elif code in {2, 3}:
- exit_value = {"code": None, "signal": status, "result": result}
- return process_state, restarts, exit_value
-
- def heartbeat_body(self) -> dict[str, Any]:
- disk = self._statvfs(str(self._paths.data))
- total_memory, available_memory = read_meminfo(self._meminfo)
- agents = []
- for held in self._held:
- observed = self._states.get(held.agent_id)
- if observed is None:
- continue
- agents.append(
- {
- "launch_id": held.launch_id,
- "agent_id": held.agent_id,
- "revision": held.revision,
- "process_state": observed.process_state,
- "restarts": observed.restarts,
- "oom_kills": self._records.get(held.agent_id, {}).get(
- "oomKills", 0
- ),
- "exit": observed.exit,
- "since": observed.since,
- }
- )
- return {
- "boot_id": self._identity.boot_id,
- "instance_id": self._identity.instance_id,
- "supervisor_version": SUPERVISOR_VERSION,
- "runtime_fingerprint": self._fingerprint,
- "disk": {
- "path": str(self._paths.data),
- "total_bytes": disk.f_blocks * disk.f_frsize,
- "used_bytes": (disk.f_blocks - disk.f_bfree) * disk.f_frsize,
- "available_bytes": disk.f_bavail * disk.f_frsize,
- },
- "memory": {
- "total_bytes": total_memory,
- "available_bytes": available_memory,
- },
- "agents": agents,
- }
-
-
-def main(argv: list[str] | None = None) -> int:
- parser = argparse.ArgumentParser()
- parser.add_argument(
- "--config", default="/etc/switch-hosted/assignment.json", type=Path
- )
- parser.add_argument(
- "--runtime-config", default="/etc/switch-hosted/runtime.json", type=Path
- )
- arguments = parser.parse_args(argv)
- logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
- if os.geteuid() != 0:
- raise WorkerError("Trusted worker supervisor must run as root.")
- root_lock = acquire_root_lock()
- try:
- config = load_worker_config(arguments.config, arguments.runtime_config)
- runtime_fingerprint = "sha256:" + verify_pinned_runtime(config)
- uid, gid = resolve_agent_account(
- config.runtime.agent_user, config.runtime.agent_group
- )
- identity = MachineIdentity(
- ImdsV2().instance_id(), _read_boot_id(), config.generation
- )
- try:
- bundle = parse_bundle(
- SecretsManager(config.secret_region).read(config.secret_id), config
- )
- except ObsoleteBundle:
- print("obsolete bundle", file=sys.stderr, flush=True)
- return OBSOLETE_EXIT_CODE
- commands = Commands()
- paths = Paths(DATA_MOUNT, RUNTIME_DIRECTORY)
- prepare_runtime_directory(commands, paths, gid)
- write_bundle(paths, bundle)
- storage, _formatted = prepare_storage(commands, config)
- ownership_blocked = reconcile_marker(
- identity,
- config,
- storage.filesystem_uuid or "",
- runtime_fingerprint,
- paths,
- )
- prepare_layout(paths, uid, gid)
- Supervisor(
- runtime=config.runtime,
- identity=identity,
- machine_id=bundle.machine_id,
- runtime_fingerprint=runtime_fingerprint,
- paths=paths,
- uid=uid,
- gid=gid,
- client=CoreClient(bundle, identity, default_opener()),
- systemd=Systemd(commands),
- git=GitRunner(setpriv_prefix(uid, gid), FLOCK, GIT),
- clock=lambda: datetime.now(UTC),
- monotonic=time.monotonic,
- sleep=time.sleep,
- statvfs=os.statvfs,
- meminfo=MEMINFO_PATH,
- ownership_blocked=ownership_blocked,
- ).run()
- return 0
- finally:
- root_lock.close()
-
-
-if __name__ == "__main__":
- try:
- raise SystemExit(main())
- except WorkerError as error:
- print(str(error), file=sys.stderr)
- raise SystemExit(1)
diff --git a/deploy/hosted/worker/test_switch_hosted_worker.py b/deploy/hosted/worker/test_switch_hosted_worker.py
deleted file mode 100644
index d93d19767..000000000
--- a/deploy/hosted/worker/test_switch_hosted_worker.py
+++ /dev/null
@@ -1,2494 +0,0 @@
-from __future__ import annotations
-
-import configparser
-import contextlib
-import fcntl
-import hashlib
-import http.server
-import importlib.util
-import io
-import json
-import os
-import shutil
-import stat
-import subprocess
-import sys
-import tempfile
-import threading
-import unittest
-import urllib.error
-from datetime import UTC, datetime
-from pathlib import Path
-from types import SimpleNamespace
-from unittest import mock
-
-HERE = Path(__file__).parent
-MODULE_PATH = HERE / "switch_hosted_worker.py"
-SPEC = importlib.util.spec_from_file_location("switch_hosted_worker", MODULE_PATH)
-assert SPEC and SPEC.loader
-worker = importlib.util.module_from_spec(SPEC)
-sys.modules[SPEC.name] = worker
-SPEC.loader.exec_module(worker)
-
-TESTDATA = HERE / "testdata"
-CORE_FIXTURES = HERE.parents[2] / "core/tests/switch_core/fixtures/hosted_machines"
-INSTANCE = "i-0123456789abcdef0"
-OTHER_INSTANCE = "i-0fedcba9876543210"
-VOLUME = "vol-0123456789abcdef0"
-MACHINE = "3f1c2b4a-0000-4000-8000-000000000001"
-AGENT = "3f1c2b4a-0000-4000-8000-0000000000a1"
-AGENT_2 = "3f1c2b4a-0000-4000-8000-0000000000a2"
-BOOT_1 = "00000000-0000-4000-8000-00000000b001"
-BOOT_2 = "00000000-0000-4000-8000-00000000b002"
-FS_UUID = "00000000-0000-4000-8000-00000000f001"
-FINGERPRINT = "sha256:0000"
-SECRET_ARN = "arn:aws:secretsmanager:eu-west-1:000000000000:secret:switch-hosted/inst-test/slot-a"
-CAPABILITY = "wcap-test-0000000000000000"
-MACHINE_CAPABILITY = "mcap-test-00000000000000000000000000000000"
-GENERATION = 1
-ONE_AGENT_MESSAGE = "data volume uses the one-agent layout; see 'Moving to one machine per user' in deploy/hosted/README.md"
-INACTIVE = {
- "ActiveState": "inactive",
- "SubState": "dead",
- "Result": "success",
- "NRestarts": "0",
- "ExecMainStatus": "0",
- "ExecMainCode": "0",
- "ExecMainExitTimestampMonotonic": "0",
-}
-
-
-def fixture(name: str):
- return json.loads((TESTDATA / name).read_text())
-
-
-def core_fixture(name: str):
- return json.loads((CORE_FIXTURES / name).read_text())
-
-
-def core_agent(**overrides) -> dict:
- agent = core_fixture("agents_response.json")["agents"][0]
- agent.update(overrides)
- return agent
-
-
-def structure(value):
- if isinstance(value, dict):
- return {key: structure(field) for key, field in value.items()}
- if isinstance(value, list):
- return [structure(item) for item in value]
- return type(value).__name__
-
-
-def runtime_config() -> worker.RuntimeConfig:
- return worker.RuntimeConfig(
- node_path="/opt/switch/node/bin/node",
- bootstrap_path="/opt/switch/agent-providers/hosted-bootstrap.mjs",
- shared_host_daemon_path="/opt/switch/agent-providers/shared-host-daemon.mjs",
- provider_binary_path="/opt/switch/claude/bin/claude",
- agent_user="switch-agent",
- agent_group="switch-agent",
- path="/opt/switch/node/bin:/usr/bin:/bin",
- allow_initial_format=True,
- artifact_sha256={
- "node": "1" * 64,
- "bootstrap": "2" * 64,
- "sharedHostDaemon": "3" * 64,
- "provider": "4" * 64,
- },
- providers={
- "codex": {"path": "/opt/switch/providers/codex", "sha256": "5" * 64}
- },
- )
-
-
-def worker_config(runtime: worker.RuntimeConfig | None = None) -> worker.WorkerConfig:
- return worker.WorkerConfig(
- installation_id="inst-test",
- slot_id="slot-a",
- generation=GENERATION,
- secret_id=SECRET_ARN,
- secret_region="eu-west-1",
- volume_id=VOLUME,
- device_path="/dev/sdf",
- runtime=runtime or runtime_config(),
- )
-
-
-def valid_agent(**overrides) -> dict:
- agent = core_agent()
- agent["worker_capability"] = CAPABILITY
- agent["skills"] = []
- agent["spec"] = {
- "name": "reviewer",
- "instructions": "Review pull requests.",
- "provider": "claude",
- "definition": "You review code.",
- "definition_attributes": {"model": "claude-test-model"},
- "auto_session": True,
- "auto_approve": False,
- "session_limit": 8,
- }
- agent.update(overrides)
- return agent
-
-
-def second_agent(**overrides) -> dict:
- agent = valid_agent(
- agent_id=AGENT_2,
- launch_id="req-0000000000000002",
- repository=None,
- switch_credentials={
- "env": {
- "SWITCH_API_ENDPOINT": "https://switch.example.test/agent-api",
- "SWITCH_API_TOKEN": "test-token-placeholder",
- "SWITCH_AGENT_ID": AGENT_2,
- }
- },
- )
- agent.update(overrides)
- return agent
-
-
-def listing(*agents: dict, version: int = 3) -> dict:
- value = core_fixture("agents_response.json")
- value["agents"] = list(agents)
- value["agents_version"] = version
- return value
-
-
-class Response(io.BytesIO):
- def __enter__(self):
- return self
-
- def __exit__(self, *_args):
- self.close()
-
-
-def http_error(code: int) -> urllib.error.HTTPError:
- return urllib.error.HTTPError(
- "https://switch.example.test", code, "error", {}, io.BytesIO(b"{}")
- )
-
-
-def core_client(requests: list, agents_status: int | None = None) -> worker.CoreClient:
- def opener(request, timeout):
- requests.append(request)
- if request.full_url.endswith("/agents"):
- if agents_status is not None:
- raise http_error(agents_status)
- return Response((CORE_FIXTURES / "agents_response.json").read_bytes())
- return Response((CORE_FIXTURES / "heartbeat_response.json").read_bytes())
-
- return worker.CoreClient(
- worker.MachineBundle(
- MACHINE,
- "https://switch.example.test/agent-api",
- MACHINE_CAPABILITY,
- ),
- worker.MachineIdentity(INSTANCE, BOOT_1, GENERATION),
- opener,
- )
-
-
-class FakeSystemctl:
- def __init__(self):
- self.calls: list[list[str]] = []
- self.units: dict[str, dict[str, str]] = {}
-
- def run(self, arguments, capture=True):
- self.calls.append(arguments)
- if arguments[1] == "show":
- agent = arguments[-1].removeprefix("switch-agent@").removesuffix(".service")
- properties = {**INACTIVE, **self.units.get(agent, {})}
- return "".join(f"{name}={value}\n" for name, value in properties.items())
- return ""
-
- def result(self, arguments, capture=True):
- self.calls.append(arguments)
- return subprocess.CompletedProcess(arguments, 1, "", "")
-
- def actions(self) -> list[list[str]]:
- return [call[1:] for call in self.calls if call[1] != "show"]
-
- def clear(self) -> None:
- self.calls.clear()
-
-
-class FakeClient:
- def __init__(self):
- self.listings: list = []
- self.heartbeats: list = []
- self.bodies: list[dict] = []
- self.list_calls = 0
-
- def agents(self):
- self.list_calls += 1
- value = self.listings.pop(0)
- if isinstance(value, BaseException):
- raise value
- return value
-
- def heartbeat(self, body):
- self.bodies.append(body)
- value = (
- self.heartbeats.pop(0)
- if self.heartbeats
- else core_fixture("heartbeat_response.json")
- )
- if isinstance(value, BaseException):
- raise value
- return value
-
-
-class FakeGit:
- def __init__(self):
- self.calls: list[tuple[Path, list[str]]] = []
-
- def run(self, mirror, arguments):
- self.calls.append((mirror, arguments))
-
-
-class Harness:
- def __init__(self, root: Path, git=None):
- root.mkdir(exist_ok=True)
- self.paths = worker.Paths(root / "data", root / "run")
- self.paths.data.mkdir(mode=0o755)
- self.paths.marker_directory.mkdir(mode=0o700)
- self.paths.runtime.mkdir(mode=0o750)
- self.paths.agents_runtime.mkdir(mode=0o750)
- worker.prepare_layout(self.paths, os.getuid(), os.getgid())
- self.meminfo = root / "meminfo"
- self.meminfo.write_text(
- "MemTotal: 16777216 kB\n"
- "MemFree: 1000 kB\n"
- "MemAvailable: 12582912 kB\n"
- )
- self.commands = FakeSystemctl()
- self.client = FakeClient()
- self.git = git or FakeGit()
- self.ownership_blocked: set[str] = set()
- self.now = 0.0
- self.clock_value = datetime(2026, 1, 1, tzinfo=UTC)
- self.supervisor = self.build()
-
- def build(self) -> worker.Supervisor:
- return worker.Supervisor(
- runtime=runtime_config(),
- identity=worker.MachineIdentity(INSTANCE, BOOT_1, GENERATION),
- machine_id=MACHINE,
- runtime_fingerprint=FINGERPRINT,
- paths=self.paths,
- uid=os.getuid(),
- gid=os.getgid(),
- client=self.client,
- systemd=worker.Systemd(self.commands),
- git=self.git,
- clock=lambda: self.clock_value,
- monotonic=lambda: self.now,
- sleep=lambda _seconds: None,
- statvfs=lambda _path: SimpleNamespace(
- f_frsize=4096, f_blocks=52428800, f_bfree=49807360, f_bavail=49807360
- ),
- meminfo=self.meminfo,
- ownership_blocked=self.ownership_blocked,
- )
-
-
-class RootPatched(unittest.TestCase):
- def setUp(self):
- for patcher in (
- mock.patch.object(worker, "ROOT_UID", os.getuid()),
- mock.patch.object(worker.os, "chown"),
- mock.patch.object(worker.os, "fchown"),
- ):
- patcher.start()
- self.addCleanup(patcher.stop)
- self.temporary = Path(tempfile.mkdtemp()).resolve()
- self.addCleanup(shutil.rmtree, self.temporary, True)
-
-
-class ConfigTests(unittest.TestCase):
- def write(self, value) -> Path:
- directory = Path(tempfile.mkdtemp())
- self.addCleanup(shutil.rmtree, directory, True)
- path = directory / "assignment.json"
- path.write_text(json.dumps(value))
- return path
-
- def test_assignment_v2_fixture_with_secret_arn(self):
- value = fixture("assignment.json")
- value["assignmentSecretId"] = SECRET_ARN
- value["previousInstanceId"] = INSTANCE
- value["previousRuntimeFingerprint"] = "sha256:ffff"
- config = worker.load_worker_config(self.write(value), HERE / "runtime.json")
- self.assertEqual(
- (config.installation_id, config.slot_id, config.generation),
- ("inst-test", "slot-a", GENERATION),
- )
- self.assertEqual(config.volume_id, VOLUME)
- self.assertEqual(config.device_path, "/dev/sdf")
- self.assertEqual(config.secret_region, "eu-west-1")
-
- def test_verbatim_assignment_fixture_parses(self):
- config = worker.load_worker_config(
- self.write(fixture("assignment.json")), HERE / "runtime.json"
- )
- self.assertEqual(
- config.secret_id,
- "arn:aws:secretsmanager:us-east-1:000000000000:secret:switch-hosted/inst-test/slot-a",
- )
- self.assertEqual(config.secret_region, "us-east-1")
- self.assertEqual(
- (config.installation_id, config.slot_id, config.generation),
- ("inst-test", "slot-a", GENERATION),
- )
- self.assertEqual(config.volume_id, VOLUME)
- self.assertEqual(config.device_path, "/dev/sdf")
-
- def test_assignment_with_a_plain_secret_name_is_rejected(self):
- value = fixture("assignment.json")
- value["assignmentSecretId"] = "switch-hosted/inst-test/slot-a"
- with self.assertRaisesRegex(worker.WorkerError, "full Secrets Manager ARN"):
- worker.load_worker_config(self.write(value), HERE / "runtime.json")
-
- def test_assignment_rejects_v1_unknown_fields_and_other_mounts(self):
- base = fixture("assignment.json")
- base["assignmentSecretId"] = SECRET_ARN
- for change in (
- {"version": 1},
- {"agentId": "agent-1"},
- {"mountPath": "/mnt"},
- {"generation": 0},
- ):
- with self.subTest(change=change):
- with self.assertRaises(worker.WorkerError):
- worker.load_worker_config(
- self.write({**base, **change}), HERE / "runtime.json"
- )
-
- def test_secret_arn_rejects_malformed_and_partition_mismatched_regions(self):
- with self.assertRaisesRegex(worker.WorkerError, "full Secrets Manager ARN"):
- worker._secret_arn_region("secret-id")
- with self.assertRaisesRegex(worker.WorkerError, "partition and region"):
- worker._secret_arn_region(
- "arn:aws-cn:secretsmanager:eu-west-1:000000000000:secret:assignment"
- )
- self.assertEqual(
- worker._secret_arn_region(
- "arn:aws-us-gov:secretsmanager:us-gov-west-1:000000000000:secret:assignment"
- ),
- "us-gov-west-1",
- )
-
-
-class BundleTests(unittest.TestCase):
- def raw(self, **changes) -> str:
- return json.dumps({**fixture("bundle.json"), **changes})
-
- def test_fixture_bundle_parses(self):
- bundle = worker.parse_bundle(self.raw(), worker_config())
- self.assertEqual(bundle.machine_id, MACHINE)
- self.assertEqual(bundle.api_endpoint, "https://switch.example.test/agent-api")
- self.assertEqual(bundle.machine_capability, MACHINE_CAPABILITY)
- self.assertNotIn("mcap-test", repr(bundle))
-
- def test_other_version_is_obsolete(self):
- for version in (1, 3, "2"):
- with self.subTest(version=version):
- with self.assertRaises(worker.ObsoleteBundle):
- worker.parse_bundle(self.raw(version=version), worker_config())
-
- def test_bundle_must_match_the_assignment(self):
- for key, value in (
- ("installationId", "inst-other"),
- ("slotId", "slot-b"),
- ("generation", 3),
- ("dataVolumeId", "vol-0fedcba9876543210"),
- ):
- assignment = {**fixture("bundle.json")["assignment"], key: value}
- with self.subTest(key=key):
- with self.assertRaisesRegex(worker.WorkerError, "does not match"):
- worker.parse_bundle(
- self.raw(assignment=assignment), worker_config()
- )
-
- def test_bundle_rejects_unsafe_endpoints_capabilities_and_extra_keys(self):
- for change in (
- {"apiEndpoint": "http://switch.example.test/agent-api"},
- {"apiEndpoint": "https://user:[email protected]/agent-api"},
- {"apiEndpoint": "https://switch.example.test/agent-api?x=1"},
- {"apiEndpoint": "https://switch.example.test/agent-api#x"},
- {"machineCapability": "short"},
- {"machineCapability": "has a space in it 0000"},
- {"machineCapability": "x" * 4097},
- {"agentId": AGENT},
- ):
- with self.subTest(change=change):
- with self.assertRaises(worker.WorkerError) as raised:
- worker.parse_bundle(self.raw(**change), worker_config())
- self.assertNotIsInstance(raised.exception, worker.ObsoleteBundle)
-
- def test_main_exits_75_on_an_obsolete_bundle_before_touching_the_host(self):
- secrets = mock.Mock()
- secrets.read.return_value = self.raw(version=1)
- imds = mock.Mock()
- imds.return_value.instance_id.return_value = INSTANCE
- stderr = io.StringIO()
- with (
- mock.patch.object(worker.os, "geteuid", return_value=0),
- mock.patch.object(worker, "acquire_root_lock"),
- mock.patch.object(
- worker, "load_worker_config", return_value=worker_config()
- ),
- mock.patch.object(worker, "verify_pinned_runtime", return_value="0" * 64),
- mock.patch.object(
- worker, "resolve_agent_account", return_value=(1000, 1000)
- ),
- mock.patch.object(worker, "ImdsV2", imds),
- mock.patch.object(worker, "_read_boot_id", return_value=BOOT_1),
- mock.patch.object(worker, "SecretsManager", return_value=secrets),
- mock.patch.object(worker, "prepare_runtime_directory") as runtime,
- mock.patch.object(worker, "prepare_storage") as storage,
- contextlib.redirect_stderr(stderr),
- ):
- self.assertEqual(worker.main([]), 75)
- self.assertEqual(stderr.getvalue(), "obsolete bundle\n")
- secrets.read.assert_called_once_with(SECRET_ARN)
- runtime.assert_not_called()
- storage.assert_not_called()
-
-
-class BundleFileTests(RootPatched):
- def test_bundle_file_is_private_and_holds_only_machine_fields(self):
- paths = worker.Paths(self.temporary / "data", self.temporary / "run")
- paths.runtime.mkdir()
- worker.write_bundle(
- paths,
- worker.parse_bundle(json.dumps(fixture("bundle.json")), worker_config()),
- )
- self.assertEqual(paths.bundle, self.temporary / "run/machine/bundle.json")
- self.assertEqual(stat.S_IMODE(paths.bundle.stat().st_mode), 0o600)
- self.assertEqual(
- json.loads(paths.bundle.read_text()),
- {
- "version": 2,
- "machineId": MACHINE,
- "machineCapability": MACHINE_CAPABILITY,
- "apiEndpoint": "https://switch.example.test/agent-api",
- },
- )
-
- def test_runtime_directory_must_be_tmpfs(self):
- class Commands:
- def __init__(self, filesystem):
- self.filesystem = filesystem
-
- def run(self, arguments, capture=True):
- return self.filesystem + "\n"
-
- paths = worker.Paths(self.temporary / "data", self.temporary / "run")
- paths.runtime.mkdir(mode=0o750)
- with self.assertRaisesRegex(worker.WorkerError, "tmpfs"):
- worker.prepare_runtime_directory(Commands("ext4"), paths, os.getgid())
- worker.prepare_runtime_directory(Commands("tmpfs"), paths, os.getgid())
- self.assertEqual(stat.S_IMODE(paths.bundle.parent.stat().st_mode), 0o700)
- self.assertEqual(stat.S_IMODE(paths.agents_runtime.stat().st_mode), 0o750)
-
-
-class MarkerTests(RootPatched):
- def setUp(self):
- super().setUp()
- self.paths = worker.Paths(self.temporary / "data", self.temporary / "run")
- self.paths.data.mkdir(mode=0o755)
- self.identity = worker.MachineIdentity(INSTANCE, BOOT_1, GENERATION)
-
- def reconcile(self, identity=None, fingerprint=FINGERPRINT):
- return worker.reconcile_marker(
- identity or self.identity,
- worker_config(),
- FS_UUID,
- fingerprint,
- self.paths,
- )
-
- def marker(self) -> dict:
- return json.loads(self.paths.marker.read_text())
-
- def test_blank_disk_gets_a_v2_marker(self):
- (self.paths.data / "lost+found").mkdir()
- self.reconcile()
- self.assertEqual(self.marker(), fixture("machine.json"))
- self.assertEqual(stat.S_IMODE(self.paths.marker.stat().st_mode), 0o600)
-
- def test_unmarked_disk_with_data_is_refused(self):
- (self.paths.data / "state").mkdir()
- with self.assertRaisesRegex(worker.WorkerError, "no trusted machine marker"):
- self.reconcile()
-
- def test_fixture_marker_is_accepted_unchanged(self):
- worker._write_root_json(self.paths.marker, fixture("machine.json"))
- before = self.paths.marker.stat().st_ino
- self.reconcile()
- self.assertEqual(self.paths.marker.stat().st_ino, before)
-
- def test_v1_marker_is_refused_with_the_migration_message(self):
- worker._write_root_json(
- self.paths.marker,
- {
- "version": 1,
- "installationId": "inst-test",
- "agentId": "agent-1",
- "generation": 2,
- "instanceId": INSTANCE,
- "bootId": BOOT_1,
- "filesystemUuid": FS_UUID,
- "runtimeFingerprint": "a" * 64,
- },
- )
- with self.assertRaises(worker.WorkerError) as raised:
- self.reconcile()
- self.assertEqual(str(raised.exception), ONE_AGENT_MESSAGE)
-
- def test_identity_mismatches_are_refused(self):
- for key, value in (
- ("installationId", "inst-other"),
- ("slotId", "slot-b"),
- ("generation", 3),
- ("filesystemUuid", "00000000-0000-4000-8000-00000000f002"),
- ):
- with self.subTest(key=key):
- worker._write_root_json(
- self.paths.marker, {**fixture("machine.json"), key: value}
- )
- with self.assertRaises(worker.WorkerError):
- self.reconcile()
-
- def test_malformed_marker_is_refused(self):
- for change in (
- {"layout": "one-agent"},
- {"runtimeFingerprint": "0000"},
- {"extra": 1},
- ):
- with self.subTest(change=change):
- worker._write_root_json(
- self.paths.marker, {**fixture("machine.json"), **change}
- )
- with self.assertRaisesRegex(worker.WorkerError, "marker is invalid"):
- self.reconcile()
-
- def test_new_instance_and_fingerprint_are_rewritten_without_pinning(self):
- worker._write_root_json(self.paths.marker, fixture("machine.json"))
- self.reconcile(
- worker.MachineIdentity(OTHER_INSTANCE, BOOT_2, GENERATION),
- fingerprint="sha256:" + "b" * 64,
- )
- marker = self.marker()
- self.assertEqual(marker["instanceId"], OTHER_INSTANCE)
- self.assertEqual(marker["bootId"], BOOT_2)
- self.assertEqual(marker["runtimeFingerprint"], "sha256:" + "b" * 64)
- self.assertEqual(marker["layout"], "per-user-v1")
-
- def owner_files(self, agent_id: str) -> Path:
- state = self.paths.agents / agent_id
- for relative, record in fixture("owner-records.json").items():
- path = state / relative
- path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
- path.write_text(json.dumps(record, separators=(",", ":")))
- self.paths.agents.chmod(0o755)
- (state / "shared-state.jsonl").write_text('{"journal":"preserve"}\n')
- return state
-
- def test_new_boot_quarantines_ownership_per_agent(self):
- worker._write_root_json(self.paths.marker, fixture("machine.json"))
- first = self.owner_files(AGENT)
- second = self.owner_files(AGENT_2)
- (self.paths.agents / "not-a-uuid").mkdir()
- with self.assertLogs(worker.logger, "WARNING"):
- self.reconcile(worker.MachineIdentity(INSTANCE, BOOT_2, GENERATION))
- for agent_id, state in ((AGENT, first), (AGENT_2, second)):
- quarantine = self.paths.quarantine / agent_id / f"{BOOT_1}--{BOOT_2}"
- for relative, record in fixture("owner-records.json").items():
- self.assertFalse((state / relative).exists())
- self.assertEqual(
- json.loads((quarantine / relative).read_text()), record
- )
- self.assertEqual(
- (state / "shared-state.jsonl").read_text(), '{"journal":"preserve"}\n'
- )
- self.assertEqual(self.marker()["bootId"], BOOT_2)
-
- def test_torn_ticket_write_is_quarantined(self):
- worker._write_root_json(self.paths.marker, fixture("machine.json"))
- state = self.owner_files(AGENT)
- torn = (
- "ownership/4108-00000000-0000-4000-8000-0000000000c8.json."
- "00000000-0000-4000-8000-0000000000c9.tmp"
- )
- (state / torn).write_text("")
- self.reconcile(worker.MachineIdentity(INSTANCE, BOOT_2, GENERATION))
- self.assertFalse((state / torn).exists())
- self.assertTrue(
- (self.paths.quarantine / AGENT / f"{BOOT_1}--{BOOT_2}" / torn).exists()
- )
- self.assertEqual(self.marker()["bootId"], BOOT_2)
-
- def test_same_boot_does_not_quarantine(self):
- worker._write_root_json(self.paths.marker, fixture("machine.json"))
- state = self.owner_files(AGENT)
- self.reconcile(self.identity, fingerprint="sha256:" + "c" * 64)
- self.assertTrue((state / "supervisor/owner.json").exists())
- self.assertFalse(self.paths.quarantine.exists())
- self.assertEqual(self.marker()["runtimeFingerprint"], "sha256:" + "c" * 64)
-
- def test_corrupt_owner_record_fails_closed(self):
- ticket = "ownership/4103-00000000-0000-4000-8000-0000000000c3.json"
- for relative, content in (
- ("supervisor/owner.json", "{"),
- ("supervisor/owner.json", '{"pid":1,"token":"t"}'),
- ("shared-owner.lock", '{"pid":0,"token":"t"}'),
- ("shared-owner.lock", '{"pid":1,"token":"t","group":true}'),
- (ticket, '{"choosing":1,"ticket":1}'),
- (ticket, '{"choosing":false,"ticket":-1}'),
- ("ownership/unexpected.json", '{"choosing":false,"ticket":1}'),
- ):
- with self.subTest(relative=relative, content=content):
- shutil.rmtree(self.paths.data)
- self.paths.data.mkdir(mode=0o755)
- worker._write_root_json(self.paths.marker, fixture("machine.json"))
- state = self.owner_files(AGENT)
- (state / relative).write_text(content)
- with self.assertLogs(worker.logger, "ERROR") as logs:
- blocked = self.reconcile(
- worker.MachineIdentity(INSTANCE, BOOT_2, GENERATION)
- )
- self.assertEqual(blocked, {AGENT})
- self.assertIn("ownership record is invalid", logs.output[0])
- self.assertTrue((state / relative).exists())
- self.assertTrue((state / "shared-owner.lock").exists())
- self.assertEqual(self.marker()["bootId"], BOOT_2)
-
- def test_corrupt_owner_record_blocks_only_its_agent(self):
- worker._write_root_json(self.paths.marker, fixture("machine.json"))
- first = self.owner_files(AGENT)
- second = self.owner_files(AGENT_2)
- (first / "supervisor/owner.json").write_text("{")
- boot_2 = worker.MachineIdentity(INSTANCE, BOOT_2, GENERATION)
- with self.assertLogs(worker.logger, "ERROR"):
- self.assertEqual(self.reconcile(boot_2), {AGENT})
- self.assertFalse((second / "supervisor/owner.json").exists())
- self.assertTrue(
- (
- self.paths.quarantine
- / AGENT_2
- / f"{BOOT_1}--{BOOT_2}"
- / "supervisor/owner.json"
- ).exists()
- )
- self.assertEqual(
- json.loads(self.paths.ownership_blocked.read_text()),
- {
- "version": 1,
- "bootId": BOOT_2,
- "previousBootId": BOOT_1,
- "agents": [AGENT],
- },
- )
- with self.assertLogs(worker.logger, "ERROR"):
- self.assertEqual(self.reconcile(boot_2), {AGENT})
- (first / "supervisor/owner.json").unlink()
- self.assertEqual(self.reconcile(boot_2), set())
- self.assertFalse(self.paths.ownership_blocked.exists())
- self.assertFalse((first / "shared-owner.lock").exists())
- self.assertTrue(
- (
- self.paths.quarantine
- / AGENT
- / f"{BOOT_1}--{BOOT_2}"
- / "shared-owner.lock"
- ).exists()
- )
-
- def test_partial_quarantine_is_resumed(self):
- worker._write_root_json(self.paths.marker, fixture("machine.json"))
- state = self.owner_files(AGENT)
- quarantine = self.paths.quarantine / AGENT / f"{BOOT_1}--{BOOT_2}"
- quarantine.mkdir(parents=True, mode=0o700)
- self.paths.quarantine.chmod(0o700)
- (self.paths.quarantine / AGENT).chmod(0o700)
- os.replace(state / "shared-owner.lock", quarantine / "shared-owner.lock")
- self.reconcile(worker.MachineIdentity(INSTANCE, BOOT_2, GENERATION))
- self.assertFalse((state / "supervisor/owner.json").exists())
- self.assertTrue((quarantine / "supervisor/owner.json").exists())
- self.assertTrue((quarantine / "shared-owner.lock").exists())
-
-
-class StorageTests(unittest.TestCase):
- def test_storage_resolves_nitro_device_by_ebs_serial(self):
- class FakeCommands:
- def __init__(self):
- self.calls = []
-
- def run(self, arguments, capture=True):
- self.calls.append(arguments)
- return json.dumps(
- {
- "blockdevices": [
- {
- "path": "/dev/nvme0n1",
- "type": "disk",
- "fstype": "ext4",
- "uuid": FS_UUID,
- "serial": VOLUME.replace("-", ""),
- "mountpoints": [None],
- }
- ]
- }
- )
-
- commands = FakeCommands()
- observed = worker.inspect_storage(commands, "/dev/sdf", VOLUME)
- self.assertEqual(observed.device_path, "/dev/nvme0n1")
- self.assertNotIn("/dev/sdf", commands.calls[0])
-
- def test_unexpected_signature_never_formats(self):
- blank = worker.StorageObservation(
- "/dev/nvme1n1", VOLUME, None, None, False, ("xfs",)
- )
- commands = mock.Mock()
- with mock.patch.object(worker, "inspect_storage", return_value=blank):
- with self.assertRaisesRegex(
- worker.WorkerError, "not a safely initializable"
- ):
- worker.prepare_storage(commands, worker_config())
- commands.run.assert_not_called()
-
- def test_unmounted_data_is_mounted_with_systemd_mount(self):
- observation = worker.StorageObservation(
- "/dev/nvme1n1", VOLUME, "ext4", FS_UUID, False, ()
- )
- commands = mock.Mock()
- commands.result.return_value = subprocess.CompletedProcess([], 1, "", "")
- with tempfile.TemporaryDirectory() as temporary:
- mount = Path(temporary) / "data"
- with (
- mock.patch.object(worker, "DATA_MOUNT", mount),
- mock.patch.object(worker, "inspect_storage", return_value=observation),
- mock.patch.object(worker, "ROOT_UID", os.getuid()),
- ):
- worker.prepare_storage(commands, worker_config())
- commands.run.assert_called_once_with(
- [
- "/usr/bin/systemd-mount",
- "--type=ext4",
- "--options=nodev,nosuid",
- "/dev/nvme1n1",
- str(mount),
- ],
- capture=False,
- )
-
- def test_mountpoint_held_by_another_device_is_refused(self):
- observation = worker.StorageObservation(
- "/dev/nvme1n1", VOLUME, "ext4", FS_UUID, False, ()
- )
- commands = mock.Mock()
- commands.result.return_value = subprocess.CompletedProcess(
- [], 0, "/dev/nvme9n1\n", ""
- )
- with tempfile.TemporaryDirectory() as temporary:
- with (
- mock.patch.object(worker, "DATA_MOUNT", Path(temporary) / "data"),
- mock.patch.object(worker, "inspect_storage", return_value=observation),
- mock.patch.object(worker, "ROOT_UID", os.getuid()),
- ):
- with self.assertRaisesRegex(worker.WorkerError, "occupied"):
- worker.prepare_storage(commands, worker_config())
-
- def test_new_filesystem_waits_for_device_metadata_before_mount(self):
- blank = worker.StorageObservation("/dev/nvme1n1", VOLUME, None, None, False, ())
- formatted = worker.StorageObservation(
- "/dev/nvme1n1", VOLUME, "ext4", FS_UUID, False, ()
- )
- settled = False
- calls = []
-
- def run(arguments, capture=True):
- nonlocal settled
- calls.append(arguments)
- if arguments[:2] == ["/usr/bin/udevadm", "settle"]:
- settled = True
- return ""
-
- commands = mock.Mock()
- commands.run.side_effect = run
- commands.result.return_value = subprocess.CompletedProcess([], 1, "", "")
- with tempfile.TemporaryDirectory() as temporary:
- with (
- mock.patch.object(worker, "DATA_MOUNT", Path(temporary) / "data"),
- mock.patch.object(worker, "ROOT_UID", os.getuid()),
- mock.patch.object(
- worker,
- "inspect_storage",
- side_effect=lambda *_: formatted if settled else blank,
- ),
- ):
- observation, did_format = worker.prepare_storage(
- commands, worker_config()
- )
- self.assertTrue(did_format)
- self.assertEqual(observation.filesystem_uuid, FS_UUID)
- self.assertEqual(sum(call[0].endswith("mkfs.ext4") for call in calls), 1)
- self.assertEqual(calls[-1][0], "/usr/bin/systemd-mount")
-
-
-class HostTests(unittest.TestCase):
- def test_imdsv2_requires_token_and_does_not_guess_identity(self):
- requests = []
-
- def opener(request, timeout):
- requests.append((request, timeout))
- if request.full_url.endswith("/api/token"):
- return Response(b"token")
- self.assertEqual(request.get_header("X-aws-ec2-metadata-token"), "token")
- return Response(INSTANCE.encode())
-
- self.assertEqual(worker.ImdsV2(opener).instance_id(), INSTANCE)
- self.assertEqual(requests[0][0].method, "PUT")
- self.assertEqual(len(requests), 2)
-
- def test_secret_store_calls_only_get_for_configured_secret(self):
- client = mock.Mock()
- client.get_secret_value.return_value = {"SecretString": "{}", "VersionId": "v"}
- self.assertEqual(
- worker.SecretsManager("eu-west-1", client).read(SECRET_ARN), "{}"
- )
- client.get_secret_value.assert_called_once_with(
- SecretId=SECRET_ARN, VersionStage="AWSCURRENT"
- )
-
- def test_secret_region_is_explicit(self):
- boto3 = mock.Mock()
- with (
- mock.patch.dict(sys.modules, {"boto3": boto3}),
- mock.patch.dict(os.environ, {}, clear=True),
- ):
- worker.SecretsManager("eu-west-1")
- boto3.client.assert_called_once_with("secretsmanager", region_name="eu-west-1")
-
- def test_agent_account_cannot_resolve_to_root(self):
- with (
- mock.patch.object(
- worker.pwd,
- "getpwnam",
- return_value=SimpleNamespace(pw_uid=0, pw_gid=0),
- ),
- mock.patch.object(
- worker.grp, "getgrnam", return_value=SimpleNamespace(gr_gid=0)
- ),
- ):
- with self.assertRaisesRegex(worker.WorkerError, "non-root"):
- worker.resolve_agent_account("switch-agent", "switch-agent")
-
- def test_tampered_baked_runtime_fails_checksum_verification(self):
- with tempfile.TemporaryDirectory() as temporary:
- paths = {}
- hashes = {}
- for name in ("node", "bootstrap", "sharedHostDaemon", "provider"):
- path = Path(temporary) / name
- path.write_bytes(f"trusted-{name}".encode())
- path.chmod(0o444)
- paths[name] = str(path)
- hashes[name] = hashlib.sha256(path.read_bytes()).hexdigest()
- runtime = worker.RuntimeConfig(
- node_path=paths["node"],
- bootstrap_path=paths["bootstrap"],
- shared_host_daemon_path=paths["sharedHostDaemon"],
- provider_binary_path=paths["provider"],
- agent_user="switch-agent",
- agent_group="switch-agent",
- path="/usr/bin:/bin",
- allow_initial_format=True,
- artifact_sha256=hashes,
- )
- completed = subprocess.CompletedProcess([], 0, "v24.1.0\n", "")
- with (
- mock.patch.object(worker, "ROOT_UID", os.getuid()),
- mock.patch.object(worker.subprocess, "run", return_value=completed),
- ):
- self.assertRegex(
- worker.verify_pinned_runtime(worker_config(runtime)),
- r"^[0-9a-f]{64}$",
- )
- Path(paths["bootstrap"]).chmod(0o644)
- Path(paths["bootstrap"]).write_text("tampered")
- Path(paths["bootstrap"]).chmod(0o444)
- with self.assertRaisesRegex(worker.WorkerError, "checksum"):
- worker.verify_pinned_runtime(worker_config(runtime))
-
- def test_meminfo_is_read_in_bytes(self):
- with tempfile.TemporaryDirectory() as temporary:
- path = Path(temporary) / "meminfo"
- path.write_text("MemTotal: 2048 kB\nMemAvailable: 1024 kB\n")
- self.assertEqual(worker.read_meminfo(path), (2097152, 1048576))
- path.write_text("MemTotal: 2048 kB\n")
- with self.assertRaisesRegex(worker.WorkerError, "Memory"):
- worker.read_meminfo(path)
-
-
-class DeploymentTests(unittest.TestCase):
- paths = worker.Paths(Path("/data"), Path("/run/switch-hosted"))
-
- def plan(self, agent: dict) -> worker.AgentPlan:
- return worker.build_agent_plan(agent, runtime_config(), self.paths)
-
- def test_core_agent_without_a_credential_kind_is_a_per_agent_config_error(self):
- with self.assertRaisesRegex(worker.WorkerError, "credential kind None"):
- self.plan(core_agent(provider_credential_kind=None))
-
- def test_fixture_agent_builds_deployment_v2(self):
- plan = self.plan(valid_agent())
- run = f"/run/switch-hosted/agents/{AGENT}"
- self.assertEqual(
- plan.deployment,
- {
- "version": 2,
- "revision": 1,
- "session": {"sessionId": "watcher-" + AGENT, "agentId": AGENT},
- "provider": {
- "kind": "claude",
- "credential": {
- "kind": "setup-token",
- "path": f"{run}/provider",
- "refresh": True,
- },
- "binaryPath": "/opt/switch/claude/bin/claude",
- "context": "Use the Switch tools to read room context and post replies to the room.\nReview pull requests.",
- "definition": {"name": "reviewer", "content": "You review code."},
- "model": {"id": "claude-test-model"},
- },
- "github": {
- "credentialPath": f"{run}/github",
- "repository": "example-org/example-repo",
- "refresh": True,
- "mirrorPath": "/data/repos/example-org/example-repo.git",
- },
- "workspacePath": f"/data/worktrees/{AGENT}/example-org/example-repo",
- "watch": True,
- "runtimeMode": "approval-required",
- "switchCredentialsPath": f"{run}/switch.json",
- "workerCapabilityPath": f"{run}/worker-capability",
- },
- )
- self.assertEqual(plan.worker_capability, CAPABILITY)
- self.assertNotIn(CAPABILITY, json.dumps(plan.deployment))
- self.assertNotIn("test-token-placeholder", json.dumps(plan.deployment))
-
- def test_repository_paths_are_lowercase_and_optional(self):
- plan = self.plan(valid_agent(repository="Example-Org/Example.Repo"))
- self.assertEqual(
- plan.deployment["github"]["repository"], "Example-Org/Example.Repo"
- )
- self.assertEqual(
- plan.deployment["github"]["mirrorPath"],
- "/data/repos/example-org/example.repo.git",
- )
- self.assertEqual(
- plan.worktree_owner, Path(f"/data/worktrees/{AGENT}/example-org")
- )
- plan = self.plan(valid_agent(repository=None))
- self.assertNotIn("github", plan.deployment)
- self.assertEqual(
- plan.deployment["workspacePath"], f"/data/worktrees/{AGENT}/workspace"
- )
- self.assertIsNone(plan.worktree_owner)
-
- def test_codex_uses_its_pinned_binary_and_has_no_definition(self):
- spec = {
- **valid_agent()["spec"],
- "definition_attributes": {},
- "auto_approve": True,
- }
- plan = self.plan(
- valid_agent(
- provider="codex", provider_credential_kind="auth-json", spec=spec
- )
- )
- provider = plan.deployment["provider"]
- self.assertEqual(provider["binaryPath"], "/opt/switch/providers/codex")
- self.assertNotIn("definition", provider)
- self.assertNotIn("model", provider)
- self.assertEqual(plan.deployment["runtimeMode"], "full-access")
-
- def test_skills_are_validated_and_included(self):
- skills = [{"slug": "review", "files": {"SKILL.md": "# Review\n"}}]
- self.assertEqual(
- self.plan(valid_agent(skills=skills)).deployment["skills"], skills
- )
- for bad in (
- [{"slug": "../x", "files": {"SKILL.md": "x"}}],
- [{"slug": "review", "files": {"notes.md": "x"}}],
- [{"slug": "review", "files": {"../SKILL.md": "x", "SKILL.md": "x"}}],
- ):
- with self.subTest(skills=bad):
- with self.assertRaises(worker.WorkerError):
- self.plan(valid_agent(skills=bad))
-
- def test_invalid_agent_fields_are_rejected(self):
- wrong_agent = {
- "env": {
- "SWITCH_API_ENDPOINT": "https://switch.example.test",
- "SWITCH_API_TOKEN": "x",
- "SWITCH_AGENT_ID": AGENT_2,
- }
- }
- for change in (
- {"agent_id": "agent-1"},
- {"agent_id": AGENT.upper()},
- {"provider": "opencode"},
- {"provider_credential_kind": "oauth"},
- {"worker_capability": "wcap-test-0000"},
- {"repository": "example-org/.."},
- {"repository": "-bad/repo"},
- {"desired_state": "deleted"},
- {"revision": 0},
- {"spec": {"session_limit": 8}},
- {"switch_credentials": wrong_agent},
- ):
- with self.subTest(change=change):
- with self.assertRaises(worker.WorkerError):
- self.plan(valid_agent(**change))
- without = valid_agent()
- del without["skills"]
- with self.assertRaisesRegex(worker.WorkerError, "missing skills"):
- self.plan(without)
-
- def test_environment_file_follows_the_contract(self):
- text = worker.agent_environment(
- runtime_config(),
- worker.MachineIdentity(INSTANCE, BOOT_1, GENERATION),
- MACHINE,
- self.paths,
- AGENT,
- )
- self.assertEqual(
- text,
- "PATH=/opt/switch/node/bin:/usr/bin:/bin\n"
- "USER=switch-agent\n"
- "LOGNAME=switch-agent\n"
- "SHELL=/bin/bash\n"
- "LANG=C.UTF-8\n"
- f"HOME=/data/agents/{AGENT}/home\n"
- f"TMPDIR=/data/agents/{AGENT}/tmp\n"
- f"SWITCH_HOST_INSTANCE_ID={INSTANCE}\n"
- f"SWITCH_HOST_BOOT_ID={BOOT_1}\n"
- f"SWITCH_HOST_ASSIGNMENT_GENERATION={GENERATION}\n"
- f"SWITCH_HOST_MACHINE_ID={MACHINE}\n",
- )
-
- def test_environment_values_with_newlines_are_refused(self):
- runtime = runtime_config()
- for path in ("/usr/bin\nEVIL=1", "/usr/bin\r", '/usr/bin"'):
- with self.subTest(path=path):
- with self.assertRaisesRegex(worker.WorkerError, "PATH"):
- worker.agent_environment(
- worker.RuntimeConfig(**{**runtime.__dict__, "path": path}),
- worker.MachineIdentity(INSTANCE, BOOT_1, GENERATION),
- MACHINE,
- self.paths,
- AGENT,
- )
-
-
-class CoreClientTests(unittest.TestCase):
- def client(self, opener, endpoint="https://switch.example.test/agent-api/"):
- bundle = worker.MachineBundle(MACHINE, endpoint, MACHINE_CAPABILITY)
- return worker.CoreClient(
- bundle, worker.MachineIdentity(INSTANCE, BOOT_1, GENERATION), opener
- )
-
- def test_requests_carry_capability_and_host_identity(self):
- requests = []
-
- def opener(request, timeout):
- requests.append((request, timeout))
- if request.full_url.endswith("/agents"):
- value = {**core_fixture("agents_response.json"), "future": 1}
- else:
- value = {**core_fixture("heartbeat_response.json"), "future": 1}
- return Response(json.dumps(value).encode())
-
- client = self.client(opener)
- self.assertEqual(client.agents()["agents_version"], 3)
- body = core_fixture("heartbeat_request.json")
- self.assertEqual(client.heartbeat(body)["heartbeat_every_s"], 15)
- listed, beat = requests
- self.assertEqual(
- listed[0].full_url,
- f"https://switch.example.test/agent-api/hosted/machines/{MACHINE}/agents",
- )
- self.assertEqual(listed[0].get_method(), "GET")
- self.assertIsNone(listed[0].data)
- self.assertEqual(
- beat[0].full_url,
- f"https://switch.example.test/agent-api/hosted/machines/{MACHINE}/heartbeat",
- )
- self.assertEqual(beat[0].get_method(), "POST")
- self.assertEqual(json.loads(beat[0].data), body)
- for request, timeout in requests:
- self.assertEqual(timeout, worker.HTTP_TIMEOUT_SECONDS)
- self.assertEqual(
- request.get_header("Authorization"),
- "Bearer " + MACHINE_CAPABILITY,
- )
- self.assertEqual(request.get_header("X-switch-host-boot-id"), BOOT_1)
- self.assertEqual(request.get_header("X-switch-host-instance-id"), INSTANCE)
-
- def test_status_codes_map_to_restart_retire_and_retry(self):
- for code, expected in (
- (401, worker.WorkerError),
- (410, worker.MachineRetired),
- (500, worker.CoreUnavailable),
- (404, worker.CoreUnavailable),
- ):
-
- def opener(request, timeout, code=code):
- raise http_error(code)
-
- with self.subTest(code=code):
- with self.assertRaises(expected):
- self.client(opener).heartbeat({})
-
- def test_network_errors_and_invalid_bodies_are_retryable(self):
- def unreachable(request, timeout):
- raise urllib.error.URLError("down")
-
- with self.assertRaises(worker.CoreUnavailable):
- self.client(unreachable).agents()
- for raw in (
- b"not json",
- b"[]",
- b'{"agents_version": true, "agents": []}',
- b'{"agents_version": 1}',
- ):
- with self.subTest(raw=raw):
- with self.assertRaises(worker.CoreUnavailable):
- self.client(
- lambda request, timeout, raw=raw: Response(raw)
- ).agents()
-
- def test_redirects_are_not_followed(self):
- hits = []
-
- class Handler(http.server.BaseHTTPRequestHandler):
- def do_GET(self):
- hits.append(self.path)
- if self.path.endswith("/agents"):
- self.send_response(302)
- self.send_header("Location", "/elsewhere")
- self.end_headers()
- else:
- self.send_response(200)
- self.end_headers()
- self.wfile.write(b'{"agents_version":1,"agents":[]}')
-
- def log_message(self, *_args):
- pass
-
- server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler)
- thread = threading.Thread(target=server.serve_forever, daemon=True)
- thread.start()
- self.addCleanup(server.server_close)
- self.addCleanup(server.shutdown)
- endpoint = f"http://127.0.0.1:{server.server_address[1]}/agent-api"
- with mock.patch.dict(os.environ, {"no_proxy": "*", "NO_PROXY": "*"}):
- with self.assertRaisesRegex(worker.CoreUnavailable, "302"):
- self.client(worker.default_opener(), endpoint).agents()
- self.assertEqual(hits, [f"/agent-api/hosted/machines/{MACHINE}/agents"])
-
-
-class SystemdTests(unittest.TestCase):
- def test_systemctl_argv(self):
- commands = FakeSystemctl()
- systemd = worker.Systemd(commands)
- systemd.start(AGENT)
- systemd.restart(AGENT)
- systemd.stop(AGENT, wait=False)
- systemd.stop(AGENT, wait=True)
- systemd.reset_failed(AGENT)
- systemd.stop_all()
- systemd.limit_slice(15 * 1024**3)
- unit = f"switch-agent@{AGENT}.service"
- self.assertEqual(
- commands.calls,
- [
- ["/usr/bin/systemctl", "--no-block", "start", unit],
- ["/usr/bin/systemctl", "--no-block", "restart", unit],
- ["/usr/bin/systemctl", "--no-block", "stop", unit],
- ["/usr/bin/systemctl", "stop", unit],
- ["/usr/bin/systemctl", "reset-failed", unit],
- ["/usr/bin/systemctl", "stop", "switch-agent@*.service"],
- [
- "/usr/bin/systemctl",
- "set-property",
- "--runtime",
- "switch-agents.slice",
- "MemoryMax=16106127360",
- ],
- ],
- )
- with self.assertRaises(worker.WorkerError):
- systemd.start("../../etc")
-
- def test_show_reads_the_contract_properties(self):
- commands = FakeSystemctl()
- self.assertEqual(worker.Systemd(commands).show(AGENT), INACTIVE)
- self.assertEqual(
- commands.calls[0],
- [
- "/usr/bin/systemctl",
- "show",
- "-p",
- "ActiveState,SubState,Result,NRestarts,ExecMainStatus,ExecMainCode,ExecMainExitTimestampMonotonic",
- f"switch-agent@{AGENT}.service",
- ],
- )
-
-
-class SupervisorTests(RootPatched):
- def setUp(self):
- super().setUp()
- self.harness = Harness(self.temporary / "machine")
- self.supervisor = self.harness.supervisor
- self.paths = self.harness.paths
- self.commands = self.harness.commands
-
- def state(self, agent_id=AGENT) -> dict:
- body = self.supervisor.heartbeat_body()
- return next(agent for agent in body["agents"] if agent["agent_id"] == agent_id)
-
- def test_new_running_agent_gets_files_directories_and_a_restart(self):
- self.supervisor.reconcile([valid_agent()])
- unit = f"switch-agent@{AGENT}.service"
- self.assertEqual(
- self.commands.actions(),
- [["reset-failed", unit], ["--no-block", "restart", unit]],
- )
- directory = self.paths.agents_runtime / AGENT
- self.assertEqual(stat.S_IMODE(directory.stat().st_mode), 0o750)
- self.assertEqual(
- sorted(path.name for path in directory.iterdir()),
- ["deployment.json", "env", "switch.json", "worker-capability"],
- )
- for path in directory.iterdir():
- self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o440)
- self.assertEqual((directory / "worker-capability").read_text(), CAPABILITY)
- self.assertEqual(
- json.loads((directory / "switch.json").read_text()),
- valid_agent()["switch_credentials"],
- )
- deployment = json.loads((directory / "deployment.json").read_text())
- self.assertEqual(deployment["version"], 2)
- self.assertEqual(
- deployment["workspacePath"],
- str(self.paths.worktrees / AGENT / "example-org/example-repo"),
- )
- self.assertIn(
- f"SWITCH_HOST_MACHINE_ID={MACHINE}\n", (directory / "env").read_text()
- )
- for path in (
- self.paths.agents / AGENT,
- self.paths.agents / AGENT / "home",
- self.paths.agents / AGENT / "tmp",
- self.paths.worktrees / AGENT,
- self.paths.worktrees / AGENT / "example-org",
- self.paths.worktrees / AGENT / "example-org/example-repo",
- self.paths.repos,
- ):
- self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o700, path)
- self.assertEqual(stat.S_IMODE(self.paths.agents.stat().st_mode), 0o755)
- self.assertEqual(stat.S_IMODE(self.paths.worktrees.stat().st_mode), 0o755)
- self.assertEqual(list(self.paths.agents_runtime.glob(".*")), [])
-
- def test_new_stopped_agent_is_stopped(self):
- self.supervisor.reconcile([valid_agent(desired_state="stopped")])
- unit = f"switch-agent@{AGENT}.service"
- self.assertEqual(
- self.commands.actions(),
- [["--no-block", "stop", unit], ["reset-failed", unit]],
- )
- self.supervisor._observe()
- self.assertEqual(self.state()["process_state"], "stopped")
-
- def test_same_revision_only_corrects_the_running_state(self):
- unit = f"switch-agent@{AGENT}.service"
- self.supervisor.reconcile([valid_agent()])
- deployment = self.paths.agents_runtime / AGENT / "deployment.json"
- inode = deployment.stat().st_ino
- self.commands.clear()
- self.supervisor.reconcile([valid_agent()])
- self.assertEqual(self.commands.actions(), [["--no-block", "start", unit]])
- self.commands.clear()
- self.commands.units[AGENT] = {"ActiveState": "active", "SubState": "running"}
- self.supervisor.reconcile([valid_agent()])
- self.assertEqual(self.commands.actions(), [])
- self.supervisor.reconcile([valid_agent(desired_state="stopped")])
- self.assertEqual(
- self.commands.actions(),
- [["--no-block", "stop", unit], ["reset-failed", unit]],
- )
- self.commands.clear()
- self.commands.units[AGENT] = {
- "ActiveState": "failed",
- "Result": "start-limit-hit",
- }
- self.supervisor.reconcile([valid_agent()])
- self.assertEqual(self.commands.actions(), [])
- self.assertEqual(deployment.stat().st_ino, inode)
-
- def test_stopping_a_crashed_agent_resets_the_failed_unit(self):
- unit = f"switch-agent@{AGENT}.service"
- self.supervisor.reconcile([valid_agent()])
- self.commands.clear()
- self.commands.units[AGENT] = {
- "ActiveState": "failed",
- "Result": "start-limit-hit",
- }
- self.supervisor.reconcile([valid_agent(desired_state="stopped")])
- self.assertEqual(
- self.commands.actions(),
- [["--no-block", "stop", unit], ["reset-failed", unit]],
- )
-
- def test_failed_restart_is_retried_on_the_next_reconcile(self):
- unit = f"switch-agent@{AGENT}.service"
- failing = [True]
- original = self.commands.run
-
- def run(arguments, capture=True):
- if "restart" in arguments and failing and failing.pop():
- original(arguments, capture)
- raise worker.WorkerError("Required host operation failed: systemctl.")
- return original(arguments, capture)
-
- self.commands.run = run
- with (
- self.assertLogs(worker.logger, "ERROR"),
- self.assertRaises(worker.ReconcileIncomplete),
- ):
- self.supervisor.reconcile([valid_agent()])
- self.commands.clear()
- self.supervisor.reconcile([valid_agent()])
- self.assertEqual(
- self.commands.actions(),
- [["reset-failed", unit], ["--no-block", "restart", unit]],
- )
- self.commands.clear()
- self.supervisor.reconcile([valid_agent()])
- self.assertEqual(self.commands.actions(), [["--no-block", "start", unit]])
-
- def test_revision_change_rewrites_files_and_resets_a_crashed_unit(self):
- unit = f"switch-agent@{AGENT}.service"
- self.supervisor.reconcile([valid_agent()])
- self.commands.clear()
- self.commands.units[AGENT] = {
- "ActiveState": "failed",
- "Result": "start-limit-hit",
- }
- self.supervisor.reconcile(
- [valid_agent(revision=4, worker_capability="wcap-test-1111111111111111")]
- )
- self.assertEqual(
- self.commands.actions(),
- [["reset-failed", unit], ["--no-block", "restart", unit]],
- )
- directory = self.paths.agents_runtime / AGENT
- self.assertEqual(
- json.loads((directory / "deployment.json").read_text())["revision"], 4
- )
- self.assertEqual(
- (directory / "worker-capability").read_text(),
- "wcap-test-1111111111111111",
- )
-
- def test_core_agent_list_configures_the_agent(self):
- requests: list = []
- self.harness.client = core_client(requests)
- supervisor = self.harness.build()
- supervisor.tick()
- unit = f"switch-agent@{AGENT}.service"
- self.assertEqual(
- self.commands.actions(),
- [["reset-failed", unit], ["--no-block", "restart", unit]],
- )
- self.assertEqual(
- [request.full_url.rsplit("/", 1)[-1] for request in requests],
- ["agents", "heartbeat"],
- )
- written = (self.paths.agents_runtime / AGENT / "deployment.json").read_text()
- written = written.replace(str(self.paths.data), "/data").replace(
- str(self.paths.runtime), "/run/switch-hosted"
- )
- self.assertEqual(json.loads(written), fixture("deployment.json"))
- body = json.loads(requests[1].data)
- agent = core_agent()
- self.assertEqual(
- [
- (state["launch_id"], state["revision"], state["exit"])
- for state in body["agents"]
- ],
- [(agent["launch_id"], agent["revision"], None)],
- )
- self.harness.now = 14
- supervisor.tick()
- self.assertEqual(len(requests), 2)
- self.harness.now = 15
- supervisor.tick()
- self.assertEqual(
- [request.full_url.rsplit("/", 1)[-1] for request in requests],
- ["agents", "heartbeat", "heartbeat"],
- )
-
- def test_core_agent_without_a_credential_kind_is_stopped_and_kept(self):
- self.supervisor.reconcile(core_fixture("agents_response.json")["agents"])
- self.commands.clear()
- with self.assertLogs(worker.logger, "ERROR"):
- self.supervisor.reconcile([core_agent(provider_credential_kind=None)])
- unit = f"switch-agent@{AGENT}.service"
- self.assertEqual(
- self.commands.actions(),
- [["--no-block", "stop", unit], ["reset-failed", unit]],
- )
- self.assertFalse((self.paths.agents_runtime / AGENT).exists())
- self.assertTrue((self.paths.agents / AGENT / "home").is_dir())
- self.assertTrue(
- (self.paths.worktrees / AGENT / "example-org/example-repo").is_dir()
- )
- self.supervisor._observe()
- self.assertEqual(
- self.state(),
- {
- "launch_id": core_agent()["launch_id"],
- "agent_id": AGENT,
- "revision": 1,
- "process_state": "failed",
- "restarts": 0,
- "oom_kills": 0,
- "exit": {"code": None, "signal": None, "result": "invalid-config"},
- "since": "2026-01-01T00:00:00Z",
- },
- )
-
- def test_unavailable_agent_is_stopped_and_kept(self):
- unit = f"switch-agent@{AGENT}.service"
- self.supervisor.reconcile([valid_agent()])
- self.commands.clear()
- entry = core_fixture("agents_response_unavailable.json")["agents"][0]
- self.assertNotIn("worker_capability", entry)
- self.assertNotIn("switch_credentials", entry)
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.reconcile([entry])
- self.assertEqual(
- self.commands.actions(),
- [["--no-block", "stop", unit], ["reset-failed", unit]],
- )
- self.assertFalse((self.paths.agents_runtime / AGENT).exists())
- self.assertTrue((self.paths.agents / AGENT / "home").is_dir())
- self.assertTrue(
- (self.paths.worktrees / AGENT / "example-org/example-repo").is_dir()
- )
- self.supervisor._observe()
- self.assertEqual(
- self.state(),
- {
- "launch_id": entry["launch_id"],
- "agent_id": AGENT,
- "revision": 1,
- "process_state": "stopped",
- "restarts": 0,
- "oom_kills": 0,
- "exit": {"code": None, "signal": None, "result": "invalid-config"},
- "since": "2026-01-01T00:00:00Z",
- },
- )
- self.commands.clear()
- self.supervisor.reconcile([valid_agent()])
- self.assertEqual(
- self.commands.actions(),
- [["reset-failed", unit], ["--no-block", "restart", unit]],
- )
-
- def test_agent_with_a_missing_identity_is_stopped_and_kept(self):
- self.supervisor.reconcile([valid_agent()])
- entry = core_fixture("agents_response_unavailable.json")["agents"][0]
- entry["unavailable"] = "agent_identity_missing"
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.reconcile([entry])
- self.assertTrue((self.paths.agents / AGENT / "home").is_dir())
- self.supervisor._observe()
- self.assertEqual(self.state()["process_state"], "stopped")
-
- def test_malformed_unavailable_code_is_an_invalid_config(self):
- entry = core_fixture("agents_response_unavailable.json")["agents"][0]
- entry["unavailable"] = ""
- with self.assertLogs(worker.logger, "ERROR"):
- self.supervisor.reconcile([entry])
- self.supervisor._observe()
- self.assertEqual(self.state()["process_state"], "failed")
- self.assertEqual(self.state()["exit"]["result"], "invalid-config")
-
- def test_invalid_revision_stops_a_running_agent_but_keeps_its_data(self):
- self.supervisor.reconcile([valid_agent()])
- self.commands.clear()
- with self.assertLogs(worker.logger, "ERROR"):
- self.supervisor.reconcile(
- [valid_agent(revision=4, provider_credential_kind="oauth")]
- )
- unit = f"switch-agent@{AGENT}.service"
- self.assertEqual(
- self.commands.actions(),
- [["--no-block", "stop", unit], ["reset-failed", unit]],
- )
- self.assertFalse((self.paths.agents_runtime / AGENT).exists())
- self.assertTrue((self.paths.agents / AGENT).exists())
-
- def test_duplicate_agents_are_all_invalid(self):
- with self.assertLogs(worker.logger, "ERROR"):
- self.supervisor.reconcile(
- [valid_agent(), valid_agent(launch_id="req-0000000000000009")]
- )
- self.supervisor._observe()
- states = self.supervisor.heartbeat_body()["agents"]
- self.assertEqual(
- [agent["exit"]["result"] for agent in states], ["invalid-config"] * 2
- )
- self.assertFalse((self.paths.agents_runtime / AGENT).exists())
-
- def test_entries_without_identity_are_left_out(self):
- with self.assertLogs(worker.logger, "ERROR"):
- self.supervisor.reconcile([{"agent_id": AGENT}, "junk"])
- self.supervisor._observe()
- self.assertEqual(self.supervisor.heartbeat_body()["agents"], [])
- self.assertEqual(self.commands.actions(), [])
-
- def test_setup_failure_is_reported(self):
- (self.paths.worktrees / AGENT).write_text("not a directory")
- with (
- self.assertLogs(worker.logger, "ERROR"),
- self.assertRaises(worker.ReconcileIncomplete),
- ):
- self.supervisor.reconcile([valid_agent()])
- self.supervisor._observe()
- self.assertEqual(self.state()["process_state"], "failed")
- self.assertEqual(self.state()["exit"]["result"], "setup-failed")
- self.assertEqual(self.commands.actions(), [])
-
- def test_directory_swapped_for_a_symlink_is_not_followed(self):
- victim = self.temporary / "victim"
- victim.mkdir(mode=0o755)
- real_mkdir = os.mkdir
-
- def racing_mkdir(path, mode=0o777, *, dir_fd=None):
- real_mkdir(path, mode, dir_fd=dir_fd)
- if os.path.basename(os.fspath(path)) == "home":
- os.rmdir(path, dir_fd=dir_fd)
- os.symlink(victim, path, dir_fd=dir_fd)
-
- with (
- mock.patch.object(worker.os, "mkdir", racing_mkdir),
- self.assertLogs(worker.logger, "ERROR"),
- self.assertRaises(worker.ReconcileIncomplete),
- ):
- self.supervisor.reconcile([valid_agent()])
- self.assertEqual(stat.S_IMODE(victim.stat().st_mode), 0o755)
- for call in worker.os.chown.call_args_list + worker.os.fchown.call_args_list:
- self.assertNotIn(str(victim), str(call))
- self.supervisor._observe()
- self.assertEqual(self.state()["exit"]["result"], "setup-failed")
- self.assertEqual(self.commands.actions(), [])
-
- def test_loosened_directory_modes_are_repaired(self):
- self.supervisor.reconcile([valid_agent()])
- loosened = (
- self.paths.agents / AGENT / "home",
- self.paths.worktrees / AGENT / "example-org/example-repo",
- self.paths.repos,
- )
- for path in loosened:
- path.chmod(0o755)
- worker.prepare_layout(self.paths, os.getuid(), os.getgid())
- self.supervisor.reconcile([valid_agent()])
- self.supervisor._observe()
- self.assertIsNone(self.state()["exit"])
- for path in loosened:
- self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o700, path)
-
- def test_absent_agent_is_removed_and_mirrors_are_kept(self):
- self.supervisor.reconcile([valid_agent(), second_agent()])
- mirror = self.paths.repos / "example-org/example-repo.git"
- mirror.mkdir(parents=True)
- (self.paths.agents / AGENT / "home/notes").write_text("x")
- (self.paths.agents / "not-a-uuid").mkdir()
- self.commands.clear()
- with self.assertLogs(worker.logger, "WARNING") as logs:
- self.supervisor.reconcile([second_agent()])
- actions = self.commands.actions()
- self.assertIn(["stop", f"switch-agent@{AGENT}.service"], actions)
- self.assertNotIn(["stop", f"switch-agent@{AGENT_2}.service"], actions)
- self.assertFalse((self.paths.agents_runtime / AGENT).exists())
- self.assertFalse((self.paths.agents / AGENT).exists())
- self.assertFalse((self.paths.worktrees / AGENT).exists())
- self.assertTrue((self.paths.agents / AGENT_2).exists())
- self.assertTrue((self.paths.agents / "not-a-uuid").exists())
- self.assertTrue(mirror.exists())
- worktree = self.paths.worktrees / AGENT / "example-org/example-repo"
- self.assertEqual(
- self.harness.git.calls,
- [
- (mirror, ["worktree", "remove", "--force", str(worktree)]),
- (mirror, ["worktree", "prune"]),
- ],
- )
- self.assertTrue(any("not-a-uuid" in line for line in logs.output))
-
- def test_failed_prune_is_retried(self):
- class FlakyPruneGit(FakeGit):
- def __init__(self):
- super().__init__()
- self.fail_prune = True
-
- def run(self, mirror, arguments):
- super().run(mirror, arguments)
- if arguments == ["worktree", "prune"] and self.fail_prune:
- self.fail_prune = False
- raise worker.WorkerError("git worktree prune failed.")
-
- harness = Harness(self.temporary / "prune", git=FlakyPruneGit())
- harness.supervisor.reconcile([valid_agent()])
- mirror = harness.paths.repos / "example-org/example-repo.git"
- mirror.mkdir(parents=True)
- with (
- self.assertLogs(worker.logger, "WARNING"),
- self.assertRaises(worker.ReconcileIncomplete),
- ):
- harness.supervisor.reconcile([])
- self.assertFalse((harness.paths.worktrees / AGENT).exists())
- harness.supervisor.reconcile([])
- self.assertEqual(
- [arguments for _mirror, arguments in harness.git.calls][-2:],
- [["worktree", "prune"], ["worktree", "prune"]],
- )
- harness.git.calls.clear()
- harness.supervisor.reconcile([])
- self.assertEqual(harness.git.calls, [])
-
- def test_owner_named_workspace_is_cleaned_up(self):
- self.supervisor.reconcile([valid_agent(repository="workspace/example-repo")])
- mirror = self.paths.repos / "workspace/example-repo.git"
- mirror.mkdir(parents=True)
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.reconcile([])
- worktree = self.paths.worktrees / AGENT / "workspace/example-repo"
- self.assertEqual(
- self.harness.git.calls,
- [
- (mirror, ["worktree", "remove", "--force", str(worktree)]),
- (mirror, ["worktree", "prune"]),
- ],
- )
-
- def test_agent_with_blocked_ownership_is_not_started(self):
- self.harness.ownership_blocked.add(AGENT)
- supervisor = self.harness.build()
- with self.assertLogs(worker.logger, "ERROR"):
- supervisor.reconcile([valid_agent(), second_agent()])
- self.assertEqual(
- self.commands.actions(),
- [
- ["--no-block", "stop", f"switch-agent@{AGENT}.service"],
- ["reset-failed", f"switch-agent@{AGENT}.service"],
- ["reset-failed", f"switch-agent@{AGENT_2}.service"],
- ["--no-block", "restart", f"switch-agent@{AGENT_2}.service"],
- ],
- )
- self.assertFalse((self.paths.agents_runtime / AGENT).exists())
- supervisor._observe()
- body = supervisor.heartbeat_body()
- states = {agent["agent_id"]: agent for agent in body["agents"]}
- self.assertEqual(states[AGENT]["process_state"], "failed")
- self.assertEqual(
- states[AGENT]["exit"],
- {"code": None, "signal": None, "result": "ownership-invalid"},
- )
- self.assertNotEqual(states[AGENT_2]["process_state"], "failed")
-
- def test_worktree_without_mirror_skips_git(self):
- self.supervisor.reconcile([valid_agent()])
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.reconcile([])
- self.assertEqual(self.harness.git.calls, [])
- self.assertFalse((self.paths.worktrees / AGENT).exists())
-
- def test_failed_removal_is_logged_and_retried(self):
- self.supervisor.reconcile([valid_agent()])
- failing = [True]
- original = self.commands.run
-
- def run(arguments, capture=True):
- if arguments[1] == "stop" and failing and failing.pop():
- raise worker.WorkerError("Required host operation failed: systemctl.")
- return original(arguments, capture)
-
- self.commands.run = run
- with (
- self.assertLogs(worker.logger, "ERROR"),
- self.assertRaises(worker.ReconcileIncomplete),
- ):
- self.supervisor.reconcile([])
- self.assertTrue((self.paths.agents / AGENT).exists())
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.reconcile([])
- self.assertFalse((self.paths.agents / AGENT).exists())
-
- def test_removal_stops_when_git_outlives_its_kill(self):
- class AbandoningGit(FakeGit):
- def run(self, mirror, arguments):
- super().run(mirror, arguments)
- raise worker.GitAbandoned("git worktree remove did not exit.")
-
- harness = Harness(self.temporary / "abandoned", git=AbandoningGit())
- harness.supervisor.reconcile([valid_agent()])
- (harness.paths.repos / "example-org/example-repo.git").mkdir(parents=True)
- with (
- self.assertLogs(worker.logger, "ERROR"),
- self.assertRaises(worker.ReconcileIncomplete),
- ):
- harness.supervisor.reconcile([])
- self.assertTrue(
- (harness.paths.worktrees / AGENT / "example-org/example-repo").is_dir()
- )
- self.assertTrue((harness.paths.agents / AGENT).is_dir())
- self.assertEqual([call[1][1] for call in harness.git.calls], ["remove"])
-
- def test_stale_temporary_runtime_directories_are_cleaned(self):
- stale = self.paths.agents_runtime / ".tmp-crashed"
- stale.mkdir(mode=0o750)
- (stale / "switch.json").write_text("{}")
- self.supervisor.reconcile([])
- self.assertFalse(stale.exists())
-
- def test_slice_memory_is_total_minus_one_gibibyte(self):
- self.supervisor.limit_slice()
- self.assertEqual(
- self.commands.actions(),
- [
- [
- "set-property",
- "--runtime",
- "switch-agents.slice",
- f"MemoryMax={16 * 1024**3 - 1024**3}",
- ]
- ],
- )
-
-
-class ProcessStateTests(RootPatched):
- def setUp(self):
- super().setUp()
- self.harness = Harness(self.temporary / "machine")
- self.supervisor = self.harness.supervisor
- self.commands = self.harness.commands
-
- def observe(self, desired="running", touched=False, **properties):
- held = worker.HeldAgent("req-1", AGENT, 3, desired, None)
- self.commands.units[AGENT] = properties
- if touched:
- self.supervisor._touched.add(AGENT)
- else:
- self.supervisor._touched.discard(AGENT)
- return self.supervisor._unit_state(held)
-
- def test_systemd_states_map_to_process_states(self):
- cases = [
- ({"ActiveState": "activating", "SubState": "start"}, "starting"),
- ({"ActiveState": "active", "SubState": "running"}, "running"),
- ({"ActiveState": "reloading", "SubState": "reload"}, "running"),
- ({"ActiveState": "activating", "SubState": "auto-restart"}, "restarting"),
- (
- {"ActiveState": "activating", "SubState": "auto-restart-queued"},
- "restarting",
- ),
- ({"ActiveState": "deactivating", "SubState": "stop-sigterm"}, "stopping"),
- ({"ActiveState": "failed", "Result": "start-limit-hit"}, "crashed"),
- ({"ActiveState": "failed", "Result": "exit-code"}, "failed"),
- ({"ActiveState": "failed", "Result": "oom-kill"}, "failed"),
- ({"ActiveState": "inactive"}, "pending"),
- ]
- for properties, expected in cases:
- with self.subTest(properties=properties):
- self.assertEqual(self.observe(**properties)[0], expected)
- self.assertEqual(self.observe(touched=True)[0], "stopped")
- self.assertEqual(self.observe(desired="stopped")[0], "stopped")
- self.assertEqual(self.observe(ExecMainExitTimestampMonotonic="5")[0], "stopped")
- with self.assertRaisesRegex(worker.WorkerError, "unknown unit state"):
- self.observe(ActiveState="maintenance")
-
- def test_restarts_and_exit_shape(self):
- state, restarts, exit_value = self.observe(
- ActiveState="failed",
- Result="start-limit-hit",
- NRestarts="5",
- ExecMainCode="2",
- ExecMainStatus="9",
- ExecMainExitTimestampMonotonic="100",
- )
- self.assertEqual((state, restarts), ("crashed", 5))
- self.assertEqual(
- exit_value, {"code": None, "signal": 9, "result": "start-limit-hit"}
- )
- _, _, exit_value = self.observe(
- ActiveState="activating",
- SubState="auto-restart",
- Result="exit-code",
- ExecMainCode="1",
- ExecMainStatus="3",
- ExecMainExitTimestampMonotonic="100",
- )
- self.assertEqual(exit_value, {"code": 3, "signal": None, "result": "exit-code"})
- _, _, exit_value = self.observe(
- ActiveState="active",
- SubState="running",
- ExecMainCode="1",
- ExecMainStatus="3",
- )
- self.assertIsNone(exit_value)
- self.assertIsNone(self.observe()[2])
-
- def test_oom_kills_are_counted_once_per_exit_and_persisted(self):
- self.supervisor.reconcile([valid_agent()])
- oom = {
- "ActiveState": "activating",
- "SubState": "auto-restart",
- "Result": "oom-kill",
- "ExecMainCode": "2",
- "ExecMainStatus": "9",
- }
- units = self.commands.units
- units[AGENT] = {**oom, "ExecMainExitTimestampMonotonic": "100"}
- self.supervisor._observe()
- self.supervisor._observe()
- units[AGENT] = {
- "ActiveState": "active",
- "SubState": "running",
- "ExecMainExitTimestampMonotonic": "100",
- }
- self.supervisor._observe()
- units[AGENT] = {**oom, "ExecMainExitTimestampMonotonic": "200"}
- self.supervisor._observe()
- records = self.harness.paths.agent_records
- self.assertEqual(stat.S_IMODE(records.stat().st_mode), 0o600)
- self.assertEqual(
- json.loads(records.read_text()),
- {
- "version": 1,
- "agents": {
- AGENT: {
- "oomKills": 2,
- "lastOomExit": f"{BOOT_1}:200",
- "revision": 1,
- }
- },
- },
- )
- restarted = self.harness.build()
- restarted.reconcile([valid_agent()])
- restarted._observe()
- self.assertEqual(restarted.heartbeat_body()["agents"][0]["oom_kills"], 2)
- with self.assertLogs(worker.logger, "WARNING"):
- restarted.reconcile([])
- self.assertEqual(json.loads(records.read_text()), {"version": 1, "agents": {}})
-
- def test_new_revision_resets_the_oom_count(self):
- self.supervisor.reconcile([valid_agent()])
- self.commands.units[AGENT] = {
- "ActiveState": "activating",
- "SubState": "auto-restart",
- "Result": "oom-kill",
- "ExecMainCode": "2",
- "ExecMainStatus": "9",
- "ExecMainExitTimestampMonotonic": "100",
- }
- self.supervisor._observe()
- self.assertEqual(self.supervisor.heartbeat_body()["agents"][0]["oom_kills"], 1)
- self.supervisor.reconcile([valid_agent(revision=2)])
- self.supervisor._observe()
- self.assertEqual(self.supervisor.heartbeat_body()["agents"][0]["oom_kills"], 0)
- self.assertEqual(
- json.loads(self.harness.paths.agent_records.read_text())["agents"][AGENT],
- {"oomKills": 0, "lastOomExit": f"{BOOT_1}:100", "revision": 2},
- )
- restarted = self.harness.build()
- restarted.reconcile([valid_agent(revision=2)])
- restarted._observe()
- self.assertEqual(restarted.heartbeat_body()["agents"][0]["oom_kills"], 0)
-
- def test_invalid_records_file_fails_loud(self):
- worker._write_root_json(self.harness.paths.agent_records, {"version": 9})
- with self.assertRaisesRegex(worker.WorkerError, "records"):
- self.harness.build()
-
- def test_since_changes_only_with_the_process_state(self):
- self.supervisor.reconcile([valid_agent()])
- self.commands.units[AGENT] = {"ActiveState": "active", "SubState": "running"}
- self.assertTrue(self.supervisor._observe())
- self.harness.clock_value = datetime(2026, 1, 1, 0, 5, tzinfo=UTC)
- self.assertFalse(self.supervisor._observe())
- self.assertEqual(
- self.supervisor.heartbeat_body()["agents"][0]["since"],
- "2026-01-01T00:00:00Z",
- )
- self.commands.units[AGENT] = {"ActiveState": "deactivating"}
- self.assertTrue(self.supervisor._observe())
- self.assertEqual(
- self.supervisor.heartbeat_body()["agents"][0]["since"],
- "2026-01-01T00:05:00Z",
- )
-
- def test_heartbeat_body_matches_the_contract_fixture(self):
- worker._write_root_json(
- self.harness.paths.agent_records,
- {
- "version": 1,
- "agents": {
- AGENT: {
- "oomKills": 1,
- "lastOomExit": f"{BOOT_1}:50",
- "revision": 1,
- }
- },
- },
- )
- supervisor = self.harness.build()
- supervisor.reconcile(core_fixture("agents_response.json")["agents"])
- self.commands.units[AGENT] = {
- "ActiveState": "failed",
- "SubState": "failed",
- "Result": "start-limit-hit",
- "NRestarts": "5",
- "ExecMainCode": "2",
- "ExecMainStatus": "9",
- "ExecMainExitTimestampMonotonic": "100",
- }
- supervisor._observe()
- body = supervisor.heartbeat_body()
- expected = core_fixture("heartbeat_request.json")
- self.assertEqual(structure(body), structure(expected))
- self.assertEqual(expected["disk"]["path"], "/data")
- expected["disk"]["path"] = str(self.harness.paths.data)
- self.assertEqual(body, expected)
-
-
-class LoopTests(RootPatched):
- def setUp(self):
- super().setUp()
- self.harness = Harness(self.temporary / "machine")
- self.supervisor = self.harness.supervisor
- self.client = self.harness.client
- self.commands = self.harness.commands
-
- def test_boot_fetches_reconciles_and_heartbeats(self):
- self.client.listings.append(listing(valid_agent()))
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 1)
- self.assertEqual(len(self.client.bodies), 1)
- self.assertEqual(self.client.bodies[0]["agents"][0]["process_state"], "stopped")
- self.harness.now = 3
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 1)
- self.commands.units[AGENT] = {"ActiveState": "active", "SubState": "running"}
- self.harness.now = 6
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 2)
- self.assertEqual(self.client.bodies[1]["agents"][0]["process_state"], "running")
- self.harness.now = 6 + 15
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 3)
- self.assertEqual(self.client.list_calls, 1)
-
- def test_heartbeat_interval_comes_from_the_response(self):
- self.client.listings.append(listing())
- self.client.heartbeats.append(
- {
- "agents_version": 3,
- "machine_desired_state": "stopped",
- "heartbeat_every_s": 30,
- }
- )
- self.supervisor.tick()
- self.harness.now = 20
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 1)
- self.assertEqual(self.commands.actions(), [])
- self.harness.now = 30
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 2)
-
- def test_changed_agents_version_refetches(self):
- self.client.listings += [listing(), listing(valid_agent(), version=8)]
- self.client.heartbeats.append(
- {
- "agents_version": 8,
- "machine_desired_state": "running",
- "heartbeat_every_s": 15,
- }
- )
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 2)
- self.assertEqual(
- self.commands.actions()[-1],
- ["--no-block", "restart", f"switch-agent@{AGENT}.service"],
- )
-
- def test_exit_75_refetches_the_list(self):
- self.client.listings += [listing(valid_agent()), listing(valid_agent())]
- self.supervisor.tick()
- self.commands.units[AGENT] = {
- "ActiveState": "failed",
- "Result": "exit-code",
- "ExecMainCode": "1",
- "ExecMainStatus": "75",
- "ExecMainExitTimestampMonotonic": "100",
- }
- self.harness.now = 3
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 2)
- self.harness.now = 6
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 2)
-
- def test_failed_fetch_is_retried(self):
- self.client.listings += [worker.CoreUnavailable("down"), listing()]
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.tick()
- self.harness.now = 3
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 1)
- self.harness.now = 15
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 2)
-
- def test_failed_reconcile_is_retried_with_backoff(self):
- blocker = self.harness.paths.worktrees / AGENT
- blocker.write_text("not a directory")
- self.client.listings += [listing(valid_agent()) for _ in range(3)]
- with self.assertLogs(worker.logger, "ERROR"):
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 1)
- self.assertIsNone(self.supervisor._agents_version)
- self.harness.now = 14
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 1)
- self.harness.now = 15
- with self.assertLogs(worker.logger, "ERROR"):
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 2)
- self.harness.now = 44
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 2)
- blocker.unlink()
- self.harness.now = 45
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 3)
- self.assertEqual(self.supervisor._agents_version, 3)
- self.assertEqual(
- self.commands.actions()[-1],
- ["--no-block", "restart", f"switch-agent@{AGENT}.service"],
- )
- self.harness.now = 200
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 3)
-
- def test_failed_heartbeat_is_retried_next_interval(self):
- self.client.listings.append(listing())
- self.client.heartbeats.append(worker.CoreUnavailable("down"))
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.tick()
- self.harness.now = 15
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 2)
-
- def test_rejected_agent_list_leaves_agents_and_their_data_alone(self):
- for status, retired in (
- (400, False),
- (409, False),
- (500, False),
- (503, False),
- (410, True),
- ):
- with self.subTest(status=status):
- harness = Harness(self.temporary / f"machine-{status}")
- for directory in (
- harness.paths.agents / AGENT / "home",
- harness.paths.worktrees / AGENT / "workspace",
- ):
- directory.mkdir(parents=True)
- (directory / "notes").write_text("kept")
- requests: list = []
- harness.client = core_client(requests, status)
- supervisor = harness.build()
- with self.assertLogs(worker.logger, "WARNING"):
- supervisor.tick()
- self.assertEqual(
- harness.commands.actions(),
- [["stop", "switch-agent@*.service"]] if retired else [],
- )
- self.assertEqual(supervisor.heartbeat_body()["agents"], [])
- self.assertEqual(
- (harness.paths.agents / AGENT / "home/notes").read_text(), "kept"
- )
- self.assertEqual(
- (harness.paths.worktrees / AGENT / "workspace/notes").read_text(),
- "kept",
- )
- self.assertEqual(requests[0].full_url.rsplit("/", 1)[-1], "agents")
-
- def test_401_stops_the_supervisor(self):
- self.client.listings.append(worker.WorkerError("rejected"))
- with self.assertRaises(worker.WorkerError):
- self.supervisor.tick()
-
- def test_410_stops_all_agents_idles_and_resumes(self):
- stop_all = ["stop", "switch-agent@*.service"]
- self.client.listings += [listing(valid_agent()), listing(valid_agent())]
- self.client.heartbeats += [
- worker.MachineRetired(),
- worker.MachineRetired(),
- core_fixture("heartbeat_response.json"),
- ]
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.tick()
- self.assertEqual(self.commands.actions()[-1], stop_all)
- self.assertEqual(len(self.client.bodies), 1)
- self.harness.now = 59
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 1)
- self.harness.now = 60
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 2)
- self.assertEqual(self.commands.actions().count(stop_all), 1)
- self.assertEqual(self.client.bodies[1]["agents"][0]["process_state"], "stopped")
- self.harness.now = 120
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.tick()
- self.assertEqual(len(self.client.bodies), 3)
- self.assertEqual(self.client.list_calls, 1)
- self.harness.now = 123
- self.supervisor.tick()
- self.assertEqual(self.client.list_calls, 2)
- self.assertEqual(
- self.commands.actions()[-1],
- ["--no-block", "start", f"switch-agent@{AGENT}.service"],
- )
-
- def test_410_on_the_list_retires_and_401_while_retired_exits(self):
- self.client.listings.append(worker.MachineRetired())
- self.client.heartbeats.append(worker.WorkerError("rejected"))
- with self.assertLogs(worker.logger, "WARNING"):
- self.supervisor.tick()
- self.assertEqual(self.commands.actions(), [["stop", "switch-agent@*.service"]])
- self.assertEqual(self.client.bodies, [])
- self.harness.now = 60
- with self.assertRaises(worker.WorkerError):
- self.supervisor.tick()
-
-
-class GitTests(RootPatched):
- def setUp(self):
- super().setUp()
- real_git = shutil.which("git")
- if real_git is None:
- self.skipTest("git is not installed")
- self.real_git = real_git
- tools = self.temporary / "tools"
- tools.mkdir()
- self.flock = tools / "flock"
- self.flock.write_text(
- f"#!{sys.executable}\n"
- "import fcntl, os, sys\n"
- "if sys.argv[1] != '--no-fork':\n"
- " sys.exit('flock must not fork')\n"
- "fd = os.open(sys.argv[2], os.O_RDWR | os.O_CREAT, 0o600)\n"
- "fcntl.flock(fd, fcntl.LOCK_EX)\n"
- "os.set_inheritable(fd, True)\n"
- "os.execv(sys.argv[3], sys.argv[3:])\n"
- )
- self.record = tools / "record"
- self.git = tools / "git"
- self.git.write_text(
- f"#!{sys.executable}\n"
- "import fcntl, os, sys\n"
- "arguments = sys.argv[1:]\n"
- "mirror = arguments[arguments.index('-C') + 1]\n"
- "fd = os.open(mirror + '.lock', os.O_RDWR)\n"
- "try:\n"
- " fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)\n"
- " held = 'free'\n"
- "except BlockingIOError:\n"
- " held = 'held'\n"
- "os.close(fd)\n"
- f"with open({str(self.record)!r}, 'a') as handle:\n"
- " handle.write(held + ' ' + ' '.join(arguments[2:4]) + '\\n')\n"
- f"os.execv({real_git!r}, [{real_git!r}] + arguments)\n"
- )
- for path in (self.flock, self.git):
- path.chmod(0o755)
- self.runner = worker.GitRunner([], str(self.flock), str(self.git))
-
- def git_setup(self, *arguments):
- subprocess.run(
- [self.real_git, *arguments],
- check=True,
- capture_output=True,
- env={
- "PATH": os.environ.get("PATH", "/usr/bin:/bin"),
- "HOME": str(self.temporary),
- "GIT_CONFIG_NOSYSTEM": "1",
- "GIT_CONFIG_GLOBAL": "/dev/null",
- },
- )
-
- def git_output(self, *arguments) -> str:
- return subprocess.run(
- [self.real_git, *arguments], check=True, capture_output=True, text=True
- ).stdout
-
- def test_removal_runs_real_git_under_the_mirror_lock(self):
- harness = Harness(self.temporary / "machine", git=self.runner)
- harness.supervisor.reconcile([valid_agent()])
- source = self.temporary / "source"
- self.git_setup("init", "-q", str(source))
- self.git_setup(
- "-C",
- str(source),
- "-c",
- "user.name=Test",
- "-c",
- "commit",
- "-q",
- "--allow-empty",
- "-m",
- "init",
- )
- mirror = harness.paths.repos / "example-org/example-repo.git"
- mirror.parent.mkdir(mode=0o700)
- self.git_setup("clone", "-q", "--bare", str(source), str(mirror))
- worktree = harness.paths.worktrees / AGENT / "example-org/example-repo"
- self.git_setup(
- "-C",
- str(mirror),
- "worktree",
- "add",
- "-q",
- "-b",
- f"switch/{AGENT}",
- str(worktree),
- )
- (worktree / "work.txt").write_text("uncommitted")
- self.assertIn(
- str(worktree), self.git_output("-C", str(mirror), "worktree", "list")
- )
- harness.commands.clear()
- with self.assertLogs(worker.logger, "WARNING"):
- harness.supervisor.reconcile([])
- unit = f"switch-agent@{AGENT}.service"
- self.assertEqual(
- harness.commands.actions(), [["stop", unit], ["reset-failed", unit]]
- )
- self.assertFalse(worktree.exists())
- self.assertFalse((harness.paths.worktrees / AGENT).exists())
- self.assertFalse((harness.paths.agents / AGENT).exists())
- self.assertTrue(mirror.exists())
- self.assertEqual(
- self.record.read_text().splitlines(),
- ["held worktree remove", "held worktree prune"],
- )
- self.assertNotIn(
- str(worktree), self.git_output("-C", str(mirror), "worktree", "list")
- )
- self.assertIn(
- f"switch/{AGENT}",
- self.git_output("-C", str(mirror), "branch", "--list", f"switch/{AGENT}"),
- )
-
- def test_git_waits_for_a_concurrent_lock_holder(self):
- mirror = self.temporary / "mirror.git"
- self.git_setup("init", "-q", "--bare", str(mirror))
- descriptor = os.open(f"{mirror}.lock", os.O_RDWR | os.O_CREAT, 0o600)
- fcntl.flock(descriptor, fcntl.LOCK_EX)
- done = threading.Event()
- errors: list[BaseException] = []
-
- def run():
- try:
- self.runner.run(mirror, ["worktree", "prune"])
- except BaseException as error:
- errors.append(error)
- done.set()
-
- thread = threading.Thread(target=run)
- thread.start()
- try:
- self.assertFalse(done.wait(0.5))
- finally:
- fcntl.flock(descriptor, fcntl.LOCK_UN)
- os.close(descriptor)
- self.assertTrue(done.wait(30))
- thread.join()
- self.assertEqual(errors, [])
-
- def script(self, name: str, body: str) -> Path:
- path = self.temporary / "tools" / name
- path.write_text(f"#!{sys.executable}\n" + body)
- path.chmod(0o755)
- return path
-
- def test_git_failure_reports_the_exit_status_but_not_stderr(self):
- secret = "ghs_placeholder-secret-from-repo-config"
- git = self.script(
- "noisy-git", f"import sys\nsys.stderr.write({secret!r})\nsys.exit(3)\n"
- )
- runner = worker.GitRunner([], str(self.flock), str(git))
- mirror = self.temporary / "mirror.git"
- with self.assertRaises(worker.WorkerError) as raised:
- runner.run(mirror, ["worktree", "prune"])
- self.assertEqual(
- str(raised.exception),
- f"git worktree prune on {mirror} failed with exit status 3.",
- )
- self.assertNotIn(secret, str(raised.exception))
-
- def test_timeout_kills_every_process_holding_the_lock(self):
- child_record = self.temporary / "child-pid"
- git = self.script(
- "hanging-git",
- "import os, time\n"
- "pid = os.fork()\n"
- "if pid == 0:\n"
- " time.sleep(60)\n"
- " os._exit(0)\n"
- f"with open({str(child_record)!r}, 'w') as handle:\n"
- " handle.write(str(pid))\n"
- "time.sleep(60)\n",
- )
- runner = worker.GitRunner([], str(self.flock), str(git))
- mirror = self.temporary / "mirror.git"
- with mock.patch.object(worker, "GIT_TIMEOUT_SECONDS", 1):
- with self.assertRaises(worker.WorkerError) as raised:
- runner.run(mirror, ["worktree", "prune"])
- child = int(child_record.read_text())
- self.addCleanup(self.kill_quietly, child)
- with self.assertRaises(ProcessLookupError):
- os.kill(child, 0)
- descriptor = os.open(f"{mirror}.lock", os.O_RDWR)
- try:
- fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
- finally:
- os.close(descriptor)
- self.assertIn("timed out", str(raised.exception))
-
- @staticmethod
- def kill_quietly(pid: int) -> None:
- with contextlib.suppress(ProcessLookupError):
- os.kill(pid, 9)
-
- def test_setpriv_prefix_drops_everything(self):
- self.assertEqual(
- worker.setpriv_prefix(1001, 1002),
- [
- "/usr/bin/setpriv",
- "--reuid=1001",
- "--regid=1002",
- "--clear-groups",
- "--no-new-privs",
- "--inh-caps=-all",
- "--ambient-caps=-all",
- "--bounding-set=-all",
- ],
- )
-
-
-def unit_file(name: str) -> configparser.ConfigParser:
- parser = configparser.ConfigParser(interpolation=None, strict=False)
- parser.optionxform = str # type: ignore[assignment,method-assign]
- parser.read_string((HERE / name).read_text())
- return parser
-
-
-class UnitFileTests(unittest.TestCase):
- def test_agent_unit(self):
- unit = unit_file("[email protected]")
- self.assertEqual(unit.sections(), ["Unit", "Service"])
- self.assertEqual(
- dict(unit["Unit"]),
- {
- "Description": "Switch agent %i",
- "StartLimitIntervalSec": "600",
- "StartLimitBurst": "5",
- "AssertPathIsMountPoint": "/data",
- },
- )
- service = unit["Service"]
- expected = {
- "Type": "simple",
- "User": "switch-agent",
- "Group": "switch-agent",
- "Slice": "switch-agents.slice",
- "EnvironmentFile": "/run/switch-hosted/agents/%i/env",
- "ExecStart": f"{worker.UNIT_NODE_PATH} {worker.UNIT_BOOTSTRAP_PATH} /data/agents/%i /run/switch-hosted/agents/%i/deployment.json",
- "WorkingDirectory": "/data/agents/%i",
- "Restart": "on-failure",
- "RestartSec": "10s",
- "RestartPreventExitStatus": "75",
- "MemoryMax": "75%",
- "OOMPolicy": "stop",
- "NoNewPrivileges": "yes",
- "CapabilityBoundingSet": "",
- "AmbientCapabilities": "",
- "UMask": "0077",
- }
- for key, value in expected.items():
- self.assertEqual(service[key], value, key)
- self.assertNotIn("PartOf", service)
-
- def test_slice_and_supervisor_units(self):
- slice_unit = unit_file("switch-agents.slice")
- self.assertEqual(slice_unit["Slice"]["MemoryAccounting"], "yes")
- self.assertNotIn("MemoryMax", slice_unit["Slice"])
- supervisor = unit_file("switch-hosted-worker.service")["Service"]
- self.assertEqual(supervisor["User"], "root")
- self.assertEqual(supervisor["Restart"], "always")
- self.assertEqual(supervisor["RuntimeDirectory"], "switch-hosted")
- self.assertEqual(supervisor["RuntimeDirectoryPreserve"], "yes")
- self.assertIn("CAP_SETUID", supervisor["CapabilityBoundingSet"].split())
- self.assertIn("CAP_SETGID", supervisor["CapabilityBoundingSet"].split())
-
- def test_install_ships_units_and_runtime_matches_the_unit(self):
- install = (HERE / "install.sh").read_text()
- for name in (
- "switch-agents.slice",
- "switch-hosted-worker.service",
- ):
- self.assertIn(f'"$source_dir/{name}" /etc/systemd/system/{name}', install)
- for command in ("flock", "systemctl", "systemd-mount"):
- self.assertRegex(install, rf"for command in [^\n]* {command}[ ;]")
- self.assertIn("systemctl daemon-reload", install)
- runtime = json.loads((HERE / "runtime.json").read_text())
- self.assertEqual(runtime["nodePath"], worker.UNIT_NODE_PATH)
- self.assertEqual(runtime["bootstrapPath"], worker.UNIT_BOOTSTRAP_PATH)
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/deploy/hosted/worker/testdata/assignment.json b/deploy/hosted/worker/testdata/assignment.json
deleted file mode 100644
index ee8c4bccd..000000000
--- a/deploy/hosted/worker/testdata/assignment.json
+++ /dev/null
@@ -1 +0,0 @@
-{"version":2,"installationId":"inst-test","slotId":"slot-a","generation":1,"assignmentSecretId":"arn:aws:secretsmanager:us-east-1:000000000000:secret:switch-hosted/inst-test/slot-a","dataVolumeId":"vol-0123456789abcdef0","dataDevice":"/dev/sdf","mountPath":"/data"}
diff --git a/deploy/hosted/worker/testdata/bundle.json b/deploy/hosted/worker/testdata/bundle.json
deleted file mode 100644
index a87ac6cf7..000000000
--- a/deploy/hosted/worker/testdata/bundle.json
+++ /dev/null
@@ -1 +0,0 @@
-{"version":2,"machineId":"3f1c2b4a-0000-4000-8000-000000000001","assignment":{"installationId":"inst-test","slotId":"slot-a","generation":1,"dataVolumeId":"vol-0123456789abcdef0"},"machineCapability":"mcap-test-00000000000000000000000000000000","apiEndpoint":"https://switch.example.test/agent-api"}
diff --git a/deploy/hosted/worker/testdata/deployment.json b/deploy/hosted/worker/testdata/deployment.json
deleted file mode 100644
index 631fd5520..000000000
--- a/deploy/hosted/worker/testdata/deployment.json
+++ /dev/null
@@ -1,44 +0,0 @@
-{
- "version": 2,
- "revision": 1,
- "session": {
- "sessionId": "watcher-3f1c2b4a-0000-4000-8000-0000000000a1",
- "agentId": "3f1c2b4a-0000-4000-8000-0000000000a1"
- },
- "provider": {
- "kind": "claude",
- "credential": {
- "kind": "setup-token",
- "path": "/run/switch-hosted/agents/3f1c2b4a-0000-4000-8000-0000000000a1/provider",
- "refresh": true
- },
- "binaryPath": "/opt/switch/claude/bin/claude",
- "context": "Use the Switch tools to read room context and post replies to the room.\nReview each pull request you are asked about and post your findings in the room.",
- "definition": {
- "name": "reviewer",
- "content": "---\nname: reviewer\ndescription: Reviews pull requests\n---\nYou review pull requests for correctness and clarity."
- },
- "model": {
- "id": "sonnet"
- }
- },
- "github": {
- "credentialPath": "/run/switch-hosted/agents/3f1c2b4a-0000-4000-8000-0000000000a1/github",
- "repository": "example-org/example-repo",
- "refresh": true,
- "mirrorPath": "/data/repos/example-org/example-repo.git"
- },
- "workspacePath": "/data/worktrees/3f1c2b4a-0000-4000-8000-0000000000a1/example-org/example-repo",
- "watch": true,
- "runtimeMode": "approval-required",
- "switchCredentialsPath": "/run/switch-hosted/agents/3f1c2b4a-0000-4000-8000-0000000000a1/switch.json",
- "workerCapabilityPath": "/run/switch-hosted/agents/3f1c2b4a-0000-4000-8000-0000000000a1/worker-capability",
- "skills": [
- {
- "slug": "github",
- "files": {
- "SKILL.md": "---\nname: github\ndescription: How to use GitHub from this cloud agent. Load before running git or gh against the granted repository — cloning, fetching, pushing, branches, pull requests or reviews.\n---\n\n# GitHub\n\nYour owner granted this agent access to one GitHub repository. It is already\ncloned into your workspace, and `git` and `gh` are already signed in for it.\n\n## Access\n\n- `gh` on your `PATH` is a wrapper that fetches a fresh, short-lived token for\n every call. The `git` credential helper does the same for HTTPS pushes and\n fetches to github.com. Both refresh automatically; there is nothing to renew.\n- The token reaches only the granted repository, with exactly these\n permissions:\n - **Contents** read/write: clone, fetch, branch, commit, push.\n - **Pull requests** read/write: create, list, view, diff, comment, review.\n - **Metadata** read.\n- Nothing else is granted. Issues, Actions, check runs, commit statuses and\n workflow files are out of reach: a push that adds or changes a file under\n `.github/workflows/` is rejected, and API operations requiring additional permissions are unavailable.\n Other repositories, organization settings and your owner's personal account\n are out of reach too. That is the scope, not a bug — tell the user what you\n could not do rather than retrying or looking for other credentials.\n- Never print, echo, log or persist a token. Do not run `gh auth token`,\n `gh auth login` or `gh auth setup-git`, do not set `GH_TOKEN` or\n `GITHUB_TOKEN`, do not put a token in a remote URL, `.git/config`, a file, a\n commit or a message. The wrapper and helper already supply it.\n\n## Everyday commands\n\nRun these from the workspace; `gh` picks the repository from its `origin`.\n\n```sh\ngit switch -c <branch> # work on a branch, not the default one\ngit push -u origin <branch>\n\ngh pr create --base <default-branch> --head <branch> --title \"...\" --body-file <path>\ngh pr list --json number,title,state,headRefName,url\ngh pr view <number> --json number,title,body,state,author,baseRefName,headRefName,url\ngh pr diff <number>\ngh pr review <number> --comment --body \"...\" # or --approve / --request-changes\ngh pr comment <number> --body \"...\"\n\ngh api repos/{owner}/{repo}/issues/<number>/comments # conversation on a pull request\ngh api repos/{owner}/{repo}/pulls/<number>/comments # inline review comments\ngh api repos/{owner}/{repo}/pulls/<number>/reviews\n```\n\nPass `--json` with only the fields you need to `gh pr view` and `gh pr list`.\nPlain `gh pr view` also asks for checks, projects and other data this token\ncannot read, and fails; so do fields such as `statusCheckRollup`,\n`projectItems` or `closingIssuesReferences`. Do not use `gh pr checks`, `gh issue`,\n`gh run` or `gh workflow`; they need permissions the token does not have.\n\nUse `--body-file <path>` for long text instead of shell-quoting it. `gh` never\nprompts in this environment, so pass every value it would ask for as a flag.\n\n## Etiquette\n\n- Push branches and open pull requests; do not push to the default branch or\n force-push a branch someone else is working on unless asked.\n- Link the pull request in the room when you open one, and keep its\n description short: what changed and how you checked it.\n"
- }
- }
- ]
-}
diff --git a/deploy/hosted/worker/testdata/machine.json b/deploy/hosted/worker/testdata/machine.json
deleted file mode 100644
index 695d853d9..000000000
--- a/deploy/hosted/worker/testdata/machine.json
+++ /dev/null
@@ -1 +0,0 @@
-{"version":2,"installationId":"inst-test","slotId":"slot-a","generation":1,"instanceId":"i-0123456789abcdef0","bootId":"00000000-0000-4000-8000-00000000b001","filesystemUuid":"00000000-0000-4000-8000-00000000f001","runtimeFingerprint":"sha256:0000","layout":"per-user-v1"}
diff --git a/deploy/hosted/worker/testdata/owner-records.json b/deploy/hosted/worker/testdata/owner-records.json
deleted file mode 100644
index 033061e53..000000000
--- a/deploy/hosted/worker/testdata/owner-records.json
+++ /dev/null
@@ -1,9 +0,0 @@
-{
- "shared-owner.lock": {"pid": 4101, "token": "00000000-0000-4000-8000-0000000000c1"},
- "supervisor/owner.json": {"pid": 4102, "token": "00000000-0000-4000-8000-0000000000c2", "build": "/opt/switch/agent-providers/shared-host-daemon.mjs"},
- "ownership/4103-00000000-0000-4000-8000-0000000000c3.json": {"choosing": false, "ticket": 1},
- "supervisor/ownership/4104-00000000-0000-4000-8000-0000000000c4.json": {"choosing": true, "ticket": 0},
- "home/.local/state/switch/sdk-sessions/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/shared-owner.lock": {"pid": 4105, "group": 4105, "token": "00000000-0000-4000-8000-0000000000c5"},
- "home/.local/state/switch/sdk-sessions/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/supervisor/owner.json": {"pid": 4106, "token": "00000000-0000-4000-8000-0000000000c6", "build": "/opt/switch/agent-providers/shared-host-daemon.mjs"},
- "home/.local/state/switch/sdk-sessions/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/launch/ownership/4107-00000000-0000-4000-8000-0000000000c7.json": {"choosing": false, "ticket": 2}
-}
diff --git a/deploy/remote/helm/switch/samples/ingress.example.yaml b/deploy/remote/helm/switch/samples/ingress.example.yaml
index 7c0286e38..93045386b 100644
--- a/deploy/remote/helm/switch/samples/ingress.example.yaml
+++ b/deploy/remote/helm/switch/samples/ingress.example.yaml
@@ -37,9 +37,6 @@ spec:
- path: /agents
pathType: Prefix
backend: { service: { name: RELEASE-switch-core, port: { number: 8000 } } }
- - path: /hosted
- pathType: Prefix
- backend: { service: { name: RELEASE-switch-core, port: { number: 8000 } } }
# Session hosts report session starts, turn activity and approval
# requests here, and poll for approval answers. Without it a
# session fails on the HTML the gateway serves instead.
diff --git a/deploy/remote/helm/switch/templates/switch-core/deployment.yaml b/deploy/remote/helm/switch/templates/switch-core/deployment.yaml
index 650281acd..c637f546f 100644
--- a/deploy/remote/helm/switch/templates/switch-core/deployment.yaml
+++ b/deploy/remote/helm/switch/templates/switch-core/deployment.yaml
@@ -70,8 +70,6 @@ spec:
{{- if .Values.switchCore.hostedControllerSecret }}
- name: HOSTED_CONTROLLER_CONFIG_PATH
value: /etc/switch-hosted/controller.json
- - name: HOSTED_PROVIDER_VERIFICATION_ENABLED
- value: {{ .Values.switchCore.hostedProviderVerificationEnabled | quote }}
- name: HOSTED_LAUNCH_CAPACITY
value: {{ .Values.switchCore.hostedLaunchCapacity | quote }}
- name: HOSTED_DISK_RETENTION_DAYS
diff --git a/deploy/remote/helm/switch/values.yaml b/deploy/remote/helm/switch/values.yaml
index c0d0c9f1e..0ecdec4b8 100644
--- a/deploy/remote/helm/switch/values.yaml
+++ b/deploy/remote/helm/switch/values.yaml
@@ -54,7 +54,6 @@ ingress:
# gateway: the Bearer-auth agent bridge + MCP, and public OAuth/MCP discovery.
agentApiPaths:
- /agents
- - /hosted
# Session hosts report session starts, turn activity and approval requests
# here, and poll for approval answers.
- /agent-sessions
@@ -175,10 +174,11 @@ switchCore:
# Existing Secret with github.json (client_id, client_secret, slug, origin).
githubConnectionsSecret: ""
hostedControllerSecret: ""
+ # How many cloud machines may run at once. Cloud machines run the agents
+ # controller and their owner's managed agents, so they need agentManagement.
hostedLaunchCapacity: 0
hostedDiskRetentionDays: 7
hostedIdleStopMinutes: 30
- hostedProviderVerificationEnabled: false
image: switch-core:latest
imagePullPolicy: Always
logLevel: INFO
diff --git a/docs/TELEMETRY.md b/docs/TELEMETRY.md
index 6b8409d0f..0ec3363a9 100644
--- a/docs/TELEMETRY.md
+++ b/docs/TELEMETRY.md
@@ -122,7 +122,7 @@ id, no IP field, no Switch identity.
| `onboarding_step_started` | `step_id`, one of exactly 4: `addServer`, `agentProviders`, `onboardAgents`, `createRoom` |
| `onboarding_checklist_dismissed` | *(no fields)* |
| `onboarding_completed` | *(no fields)* |
-| `add_server_step` | `step`: `choose` / `managedReady` / `managedClaude` / `managedGitHub` / `managedAgent` / `local` / `remoteHost` / `external` / `signIn` / `linkAccounts` · `choice`: `none` / `local` / `remoteHost` / `external` / `cloud` · `first_run`: `true` / `false` — whether the same step was reached on a fresh install rather than from the Add server dialog |
+| `add_server_step` | `step`: `choose` / `managedGitHub` / `managedAgent` / `local` / `remoteHost` / `external` / `signIn` / `linkAccounts` · `choice`: `none` / `local` / `remoteHost` / `external` / `cloud` · `first_run`: `true` / `false` — whether the same step was reached on a fresh install rather than from the Add server dialog |
**Agents and sessions**
diff --git a/docs/design/agent-controllers-v1.md b/docs/design/agent-controllers-v1.md
index 5a3ce37f5..1fba52eb9 100644
--- a/docs/design/agent-controllers-v1.md
+++ b/docs/design/agent-controllers-v1.md
@@ -300,12 +300,14 @@ exactly the answer a missing one does.
- CLI `switch-agent-controller`, released as its own npm package on
`switch-agent-controller-v*` GitHub releases, with an `install.sh` (see RELEASING.md):
- `enroll --server <agent-bridge-url> --code <code> [--name] [--description] [--data-dir] [--secret-store]`
- - `run [--data-dir] [--env-file]`
+ - `run [--data-dir] [--env-file] [--agent-runtime default|separate-user]`
- `set-info [--name] [--description] [--data-dir]`: renames the machine and/or changes its
description on the server, with the controller's own credential, and records the new name
locally.
- `status [--data-dir]`
- `install-service` / `uninstall-service`: a systemd user unit or a launchd agent
+ - `install-service --separate-users` / `uninstall-service --separate-users`: as root,
+ once, on Linux with systemd and polkit; see "Each agent as a Linux user of its own" below
- `doctor`: what the machine lacks to run agents
- `update [--check]`: the newest release, from GitHub
- **Data dir:** `SWITCH_CONTROLLER_DATA_DIR`, otherwise the OS default.
@@ -338,6 +340,34 @@ exactly the answer a missing one does.
- Providers: installed via a PATH lookup, auth via the bundle's `--probe`, cached for 10 min. `provider.recheck` forces a probe.
- Agents: read from each running agent host's state and `supervisor/failure.json`, mapped to the contract's process states and reason codes, with `directory`, the working directory its agent host was configured with.
- **Operations:** `agent.restart` restarts the agent host. `provider.recheck` forces a probe and reports.
+- **Provider logins given to the machine** (`sealed-logins.ts`, contract §5): `enroll` makes an
+ X25519 keypair and sends the public half; a controller enrolled before registers it once
+ (`PATCH .../controllers/{id} {public_key}`). The owner's Console seals a login to it; Core
+ stores the ciphertext (`sealed_provider_logins`) and queues `provider.login {provider, method:
+ "sealed"}`, which the controller answers after opening the login and probing the provider with
+ it. `ProviderStatuses` prefers the machine's own login and falls back to a given one
+ (`auth_source: "sealed"`); `ensureRelayCredentials` writes it into the agent's credentials
+ (`providerLogin`), `startAgent` puts its environment in the template and runs the agent isolated,
+ and the agent host writes its login file (`readProviderLogin`, `materializeHostedProvider`).
+- **Each agent as a Linux user of its own** (`--agent-runtime separate-user`, `separate-users.ts`,
+ `systemd-runtime.ts`): root sets up once a pool of system users `sa<uid>-NN` in a group
+ `switch-agents-<uid>` that only the controller also joins, a template unit
+ `switch-agent-<uid>@.service`, a polkit rule letting the controller's user start, stop,
+ restart and reset only those units, and the controller as a system service of its user. The
+ controller then needs no root. Each agent claims a free pool user (kept in the store's
+ `agent_users`; none free is `capacity_exceeded`) and runs as that user's unit, with systemd
+ restarting it, seeing only its own directory, always at `<agents dir>/agent`, so the paths it
+ records stay valid when it later runs as another user. The controller writes its
+ `config.json` and `watch.json` there (sticky directory, group-readable files) and the relay
+ credentials and provider environment the unit loads (`LoadCredential=`, `EnvironmentFile=`);
+ the agent host writes its health and records group-readable (`SWITCH_HOST_SHARED_GROUP=1`).
+ The unit hides the home directories, the controller's data and the other agents (their
+ processes too, `ProtectProc=invisible`), blocks the instance metadata address, and makes the
+ rest of the system read-only. A removed agent frees its user, and its directory moves to
+ `released/<agent id>` until it comes back; before an agent starts, root hands it the files in
+ its directory that another pool user owns. Provider logins are the provider settings in the
+ controller's environment only: the controller user's own logins are in a home the agents
+ cannot see.
---
diff --git a/docs/design/controller-contract-v1.md b/docs/design/controller-contract-v1.md
index af33bd6d1..f7dff8f83 100644
--- a/docs/design/controller-contract-v1.md
+++ b/docs/design/controller-contract-v1.md
@@ -210,13 +210,23 @@ POST /v1/management/controllers/{id}/connector-token
→ 403 not_assigned | 404 connector_not_connected
GET /v1/management/controllers/{id}/provider-credentials/{provider}
- → 200 { revision: string, sealed: { alg: "X25519-XChaCha20Poly1305", key_id: string, ciphertext: base64 } }
+ → 200 { provider: string, revision: int, sealed: SealedLogin } // Cache-Control: no-store
→ 404 provider_login_missing
+
+type SealedLogin = {
+ alg: "X25519-HKDF-SHA256-A256GCM",
+ key_id: string, // first 16 hex digits of SHA-256(controller public key)
+ ephemeral_key: base64, // 32 bytes
+ nonce: base64, // 12 bytes
+ ciphertext: base64, // AES-256-GCM, tag appended
+}
```
-- The Console seals a provider login to the controller's `public_key`, and the server stores only the ciphertext.
-- On Switch EC2, the blob is sealed with a KMS key that only that VM's role can decrypt.
-- The controller **prefers a local login** (`auth_source: "local"`) and fetches a sealed one only when none exists.
+- **The key.** The controller makes an X25519 keypair at enrollment and sends `public_key`. A controller enrolled before then registers it once with `PATCH /v1/management/controllers/{id}` `{ public_key }`. A different key afterwards is refused (409): the machine is enrolled again. A controller that keeps no secrets across starts (its credential handed over by a parent) registers none, and uses only the machine's own logins.
+- **Sealing.** The owner's client (Console) seals `{ kind: "api-key" | "setup-token" | "auth-json", credential }` to that key: an ephemeral X25519 key, `HKDF-SHA256(shared, salt = ephemeral public ‖ controller public, info = "switch provider login v1")`, and AES-256-GCM with `aad = "switch-provider-login-v1\n<controller id>\n<provider>"`. The server checks the envelope's shape and that `key_id` names the machine's current key, stores only the ciphertext, and can never open it. The same construction is in `core/switch_core/management/sealed_logins.py` (checks) and `@switch-console/agent-providers` `sealed-login.ts` (seal and open).
+- **On demand.** The owner gives a machine a login with `PUT /gateway/management/controllers/{id}/provider-logins/{provider}` `{ sealed }`, which answers `{ login: { provider, revision, key_id, updated_at }, operation }`. The operation is `provider.login` `{ provider, method: "sealed", revision }`: the controller fetches the login, opens it, checks the provider signs in with it, and reports `succeeded` with its provider status, or `failed` with `provider_login_missing`, `provider_login_expired` or `internal`. `GET …/provider-logins` lists what was given (never the ciphertext). `DELETE …/provider-logins/{provider}` takes it back and queues `provider.recheck`. Revoking the machine deletes its logins.
+- The controller **prefers a local login** (`auth_source: "local"`). It uses a sealed one (`auth_source: "sealed"`) only for a provider it has no local login for, and only once the provider signs in with it. An agent using one gets it with its relay credentials, and is restarted when the login changes or is withdrawn.
+- On Switch EC2, the same sealed logins apply: no KMS is involved.
- When a CLI refreshes and rotates a login on the machine, the controller doesn't upload it back.
---
diff --git a/docs/hosted-activity-contracts.md b/docs/hosted-activity-contracts.md
deleted file mode 100644
index 9cea43aa8..000000000
--- a/docs/hosted-activity-contracts.md
+++ /dev/null
@@ -1,1593 +0,0 @@
-# Hosted agents on the session-activity model: contracts
-
-Port of PR #538 (`feat/hosted-idle-autostop`, cloud "hosted" agents on EC2)
-onto main after #543 ("Sessions live with the host", `df0aa35e`). Sibling of
-`docs/session-activity-handoff.md`; read that first. Nothing here is built.
-Paths are on `origin/main` unless marked *(#538)* (`758bc5ae`); claims not
-checked against code are marked **unverified**.
-
-## Why
-
-#538 was written against the 0.27 model: switch-core held every session
-(`sdk_sessions.snapshot`), and the hosted pieces read it. Idle autostop
-(`HostedLaunchStore.idle_busy` *(#538)*), the session limit and start/restart
-checks (`gateway/hosted_launches.py` `session_operation` *(#538)*), credential
-recovery (`/hosted/provider-credential` returns `sessions` *(#538)*), Console's
-cloud session list (`rpc.sdkHost.sharedList`, polled every 2 s *(#538)*),
-room-message submission (`sessions/service.py` `submit_room_message` *(#538)*)
-and failure notices (`SessionAuthority.failure_notice_thread` *(#538)*) all go
-through tables that main's `b9e4d2a71c05` drops. On main the host owns the
-session and Core only relays (`protocol/connections.py`
-`relay_session_command`, in-memory `EventBuffer`, 2000 events / 15 min).
-
-Main's local and SSH paths both assume that something can reach the watcher
-directly: Console's own child process, or a loopback control port opened over
-SSH (`host/control.ts` `serveControl`, desktop `sdk-host/sidecar-control.ts`).
-Terraform permits **no inbound access** to a hosted worker, so for the cloud
-the only link is the one the worker opens itself: the watcher's outbound
-stream to Switch.
-
-## Target design
-
-- **The worker's watcher owns its sessions**, the same as a sidecar does:
- `shared-daemon.ts --watch-worker` running `runSharedWatcher` plus
- `serveControl`'s message handling. `hosted-bootstrap.ts` *(#538)* already
- starts the daemon with `--watch-worker`. Session tool calls are forwarded to
- the process that holds the agent's connection (`switch-agent-runtime`
- `hosted.ts`), so on the worker only the watcher talks to Switch.
-- **Core relays Console to the watcher** over the watcher's own SSE stream
- (down) and HTTP up-calls (up), in dedicated frames. The relayed vocabulary
- is `control.ts`'s `clientMessageSchema` / `SessionRequest`, so local, SSH and
- cloud share one set of message types. Core never stores or logs a relayed
- body.
-- **Only the current worker can be the watcher.** Attach needs a capability
- the controller mints per launch revision; any other holder of the agent key
- is refused the watcher's stream, placements and relays.
-- **Core keeps durably only what must outlive the worker or Core itself**: the
- launch and its revision (`hosted_launches`), infrastructure commands
- (`hosted_operations`), and a **wake mailbox** of every addressed event until
- the watcher has it on disk (new).
-- **The controller stops a VM only on evidence**: a fresh idle report from the
- current worker that acknowledges every mutating relay Core dispatched, after
- the grace window, rechecked under the launch lock. Missing, stale or foreign
- reports count as busy.
-- **Session existence and slot admission are the watcher's**, in one serial
- section that covers auto-created and Console-started sessions alike.
-
-## Identity, fencing and the worker capability
-
-### Levels
-
-| Level | Where | Bumped by |
-|---|---|---|
-| `hosted_launches.revision` *(#538)* | Postgres | every start, wake, restart, stop, autostop (`observe`, `note_addressed`, lifecycle routes) |
-| worker capability | `hosted_launches.worker_capability_hash`, `worker_capability_encrypted`, `worker_capability_revision` (new) | first `prepare` at each running revision |
-| `Connection.stream_generation` | memory (`ConnectionRegistry._new_incarnation`, random seed per boot) | every stream open or reattach |
-| Core boot | `agent_event_boot` sequence *(#538, migration `51bc94a017d2`)* | every Core start |
-
-A durable record that names a connection (`offered_to`, `claimed_by`) stores
-`core_boot:connection_id:generation`. Generations are only unique within one
-Core boot, so a record from another boot is treated as foreign, never
-compared.
-
-### Issuance: one capability per running revision
-
-The capability has to follow the revision, and the revision changes on every
-wake, while the `request_id` and the retained VM stay the same. Today's
-controller never gets a new bundle to a woken VM
-(`deploy/hosted/controller/.../gateway.py` `accept_launch` *(#538)*):
-
-- it returns early once `instance_launch_issued` is set, which stays true for
- the retained instance;
-- otherwise it calls `prepare` only when the assignment secret has no version
- whose `ClientRequestToken` is the `request_id`, which is true after the
- first launch;
-- the reconciler starts a stopped instance as soon as desired is `running`
- (`reconciler.py` `_running`: `state == "stopped"` → `start_instance`),
- without waiting for any bundle.
-
-The worker reads the assignment secret's `AWSCURRENT` version each time its
-service starts (`deploy/hosted/worker/switch_hosted_worker.py` `main` →
-`SecretsManager.read` *(#538)*). The service is `switch-hosted-worker.service`,
-`Restart=always`, `RestartSec=15s`. So a secret refreshed before the start
-reaches the VM on boot, and a service restart re-reads it without a reboot.
-The contract promises **safe rejection plus recovery**, not an atomic Core
-and EC2 start: a worker that boots on an obsolete bundle is refused by Core,
-then restarts onto the current one.
-
-**Core, `prepare` (idempotent by revision).** For a launch at revision R that
-is `desired_state = running` (checked already):
-
-- if `worker_capability_revision = R`, return the **same** capability,
- decrypted from `worker_capability_encrypted`;
-- otherwise mint 32 random bytes, store `encrypt_token(capability,
- jwt_secret_key)` in `worker_capability_encrypted` (the way the agent key's
- `ApiKey.encrypted_key` is stored, `crypto.py`), `sha256` in
- `worker_capability_hash` and R in `worker_capability_revision`, in the
- transaction that sets `state = provisioning`;
-- return `{..., "revision": R, "worker_capability": ...}`. The capability is a
- top-level field, not inside `switch_credentials.env`, with `Cache-Control:
- no-store` as today.
-
-Minting overwrites the previous revision's values, so at most one capability
-is valid. A retry at the same revision returns the same bytes, so a lost
-response costs nothing.
-
-**Controller, bundle per revision.** `accept_launch` builds the bundle version
-token as `uuid5(request_id, str(revision))` and adds two columns to the
-controller's agent store (`store.py`, SQLite): `required_bundle_token` and
-`bundle_token`, plus `required_bundle_revision` so that a job older than the
-required revision is ignored rather than rolling the requirement (and, through
-step 2, `AWSCURRENT`) back to an obsolete bundle.
-
-1. For a job with desired `running`, set `required_bundle_token` to the
- revision's token before anything else. The `instance_launch_issued` early
- return is removed from this step; it still guards `run_instance`.
-2. If the secret has a version with that token **and its stages include
- `AWSCURRENT`** (`describe_secret` → `VersionIdsToStages[token]`), the
- bundle is persisted: set `bundle_token` and stop. If the version exists
- without `AWSCURRENT`, move the stage to it
- (`update_secret_version_stage`, `MoveToVersionId = token`,
- `RemoveFromVersionId` = the current holder), then set `bundle_token`.
- Mere presence of the version is never enough.
-3. Otherwise call `prepare`, check `prepared.revision == job.revision` (a
- mismatch is a stale job: stop and wait for the next poll), build the
- bundle, and `put_secret_value` with that `ClientRequestToken`. The put
- moves `AWSCURRENT` to the new version. Then set `bundle_token`.
-
-The reconciler neither calls `run_instance` nor `start_instance` while
-`bundle_token != required_bundle_token`; it reports `provisioning`. So a
-retained VM never boots on the previous revision's bundle.
-
-**Crash and loss recovery**:
-
-- A controller restart after `prepare` but before the put: the next poll
- reaches step 3 again, `prepare` returns the same capability (same
- revision), and the put goes through.
-- A lost `put_secret_value` response: either the version exists (step 2 on
- the next poll) or it does not (step 3 again). Secrets Manager commits a put
- atomically. A retry with the same token and different content (the GitHub
- token is issued fresh on each `prepare`) is refused `ResourceExistsException`, which
- the controller treats as step 2. Step 2 is checked first, so this only
- happens when the check and the put race.
-- A restart between the put and `bundle_token`: step 2 on the next poll.
-- A revision bump after `prepare(R)` returns: the controller can still
- write R's bundle, set both tokens to R and start the instance before its
- next poll. The bundle gate does not prevent that boot. What makes it safe
- is Core: the R capability is refused on attach (`worker_capability_obsolete`),
- so that worker never gets a wake, relay, placement or operation. The next
- poll then writes R+1's bundle (steps 1–3), and the obsolete-bundle restart
- below moves the running worker onto it.
-
-**Obsolete boot bundle on a running instance.** A worker that is refused
-`worker_capability_obsolete`, or evicted `launch_superseded`, restarts its
-service onto the current secret. It never retries with the same bundle:
-
-1. The daemon exits with code 75 (`EX_TEMPFAIL`), reserved for this case.
-2. `switch_hosted_worker.py` records the secret `VersionId` it booted with
- (returned by `get_secret_value`) as obsolete in
- `<state root>/obsolete-bundle`, then exits. systemd restarts the service
- 15 s later.
-3. On start, the worker reads `AWSCURRENT`. If its `VersionId` is the recorded
- obsolete one, it does **not** start the daemon. It polls the secret every
- 30 s and logs a warning every 5 minutes, and starts the daemon only once a
- different version is current. So there is no daemon restart loop and no
- repeated refused attach, only one cheap poll.
-4. With a new version current, it deletes the marker and boots normally. If
- that bundle is obsolete as well (another bump in between), the same steps
- run again, once per revision, which is bounded by how often the owner or
- autostop changes the launch.
-
-If the launch was stopping (autostop, Stop, restart), EC2 stops the instance
-while the worker waits, and the marker has no effect on the next boot. That
-boot reads a newer version, because `start_instance` is gated on the bundle.
-While a worker waits, `observe` sees no attached worker and the launch stays
-`provisioning`. Its existing 10-minute `error` path is the visible failure if
-the controller never supplies a bundle.
-
-**Validity.** A capability is valid iff its hash matches **and**
-`worker_capability_revision = launch.revision` **and** `desired_state =
-running`, `state != error`, owner still a tenant member (`worker_launch`
-*(#538)*). Any revision bump makes it obsolete without a write.
-
-**On the worker.** `switch_hosted_worker.py` accepts a `workerCapability`
-field in the bundle (its parser rejects unknown fields today). The bootstrap
-writes it to `<state root>/worker-capability` with mode 0600 by atomic rename
-on every boot. It must not use `writeNewJson` (`hosted-bootstrap.ts`
-*(#538)*), which is write-once. The capability is never put in the provider or
-session environment and never logged.
-
-### Attaching
-
-Attach is the stream open itself: `GET /agents/{agent_id}/events` (main's
-`_open_event_stream`) with headers `X-Switch-Worker-Capability`,
-`X-Switch-Host-Boot-Id`, `X-Switch-Host-Instance-Id` (from
-`SWITCH_HOST_BOOT_ID` / `SWITCH_HOST_INSTANCE_ID`, which
-`prepareHostedDeployment` *(#538)* already exports). For an agent whose
-metadata carries `hosted_launch_id` *(#538)*, Core takes the launch advisory
-lock (`hosted-launch:{tenant}:{id}`, as `operation_launch` *(#538)*) and:
-
-| Outcome | HTTP / code | When |
-|---|---|---|
-| refused | 403 `worker_capability_required` | no header |
-| refused, terminal | 403 `worker_capability_obsolete` | hash or revision mismatch, launch not running. The watcher exits cleanly and does not restart itself (like `EVICTION_TAKEN_OVER` in `shared-watcher.ts`). |
-| refused | 426 `upgrade_required` | `speaks < HOSTED_PROTOCOL_REVISION` (7, new; `artifacts.yaml` `agent-protocol` goes 6 → 7) |
-| refused | 409 `worker_already_attached` | another connection of this agent is attached as worker, alive (beat within `HEARTBEAT_TTL_SECONDS`, 6 s), with a different `boot_id` |
-| takeover | 200 | same `boot_id`, different connection id (daemon restart on the same boot): the old connection is evicted `taken_over` |
-| reattach | 200 | same connection id, under main's existing `expected_generation` rule (`SupersededReattachError`) |
-
-There is exactly one attached worker per agent. Two VMs holding a valid
-capability cannot happen by construction; a refused second worker retries
-with backoff and wins only once the first has lapsed.
-
-On success the connection gets in-memory `launch_id`, `launch_revision`,
-`boot_id`, `instance_id`, `worker = true`. The first frame after
-`connection_state` is `worker_attached` (see Frames), and every `pending` wake
-row is sent before the first buffered event.
-
-### What a non-worker connection of a hosted agent may do
-
-Any holder of the agent key (a local Console, an old VM, a leaked key) can
-still call the agent's tool operations; that is the agent key's power and is
-unchanged. It is **refused 403 `hosted_worker_only`** for:
-
-- opening or reattaching any connection of a hosted agent without a valid
- capability. This covers the watcher's own connection id, so a generic
- "reopen to take over" is gated too;
-- `connection/subscribe`, `connection/placements`, `place_session` and room
- claims;
-- every hosted up-call below (relay reply and push, idle, mailbox ack,
- operations, notices).
-
-The frames of protocol 7 go only to the attached worker.
-`relay_session_command` (room controls) is also narrowed to it for hosted
-agents; it no longer broadcasts to every `scope: all` connection.
-`observe`'s `listening` check *(#538)*, which counts any `spawn_capable`
-connection as ready, counts only the attached worker at the current revision.
-
-### Revision bump
-
-Any bump evicts that agent's attached worker if it is bound to a lower
-revision, with code `launch_superseded` (new, terminal like `taken_over`). It
-also fails that connection's pending relays with `generation_changed`,
-returns its wake offers to `pending` (D3), and runs `fail_stale_operations`
-*(#538)*.
-
-### Core restart and the sequence floor
-
-The per-boot sequence floor (`agent_event_boot`,
-`EventBuffer(sequence_base=event_boot << 32)` *(#538)*) is ported, and with it
-the `ac2ffa1d` fix *(#538)*: build the `event_stream` generator before
-`record_client_declaration` can yield, in `handlers._open_event_stream`. Main
-lacks the fix.
-
-With the floor, a new boot's sequences are **above** every cursor from an
-earlier boot. Main's restart detection (`stream.py`: `conn.cursor > head`,
-then `buffer.mark_restarted` and a `gap` with `all_rooms`) then never fires,
-and without more the restart is silent. The port carries #538's second branch
-(`0 < conn.cursor < floor - 1`, `protocol/stream.py` *(#538)*) into main's
-form: `mark_restarted`, a `gap` frame with `all_rooms: true` and `resumed_at =
-floor - 1`.
-
-Because `resumed_at` is above the client's cursor, `event-stream.ts`
-computes `cursorReset: false`, and the watcher's `onGap` does **not** append a
-`restarted` record (`SharedWatchAssignments.restart`). That is correct:
-sequence numbers are never reused across boots, so the watcher's
-sequence-based dedupe stays valid. Events lost with the old buffer are
-recovered by the mailbox (D3) for hosted agents. For local and SSH watchers
-the gap warning is the signal, as today. Main's `cursor > head` branch stays
-for a database whose sequence was reset.
-
-### Tests
-
-- `test_worker_attach_requires_current_capability`: an old worker reconnects
- after a revision bump → 403 `worker_capability_obsolete`, and it receives no
- wake, relay or placement.
-- `test_local_console_cannot_take_over_hosted_stream`: an agent-key client
- opens `scope: all`, reopens the worker's connection id, posts placements →
- all 403, and the worker's generation is unchanged.
-- `test_second_worker_refused_while_first_alive`, and
- `test_same_boot_takeover_evicts_old`.
-- `test_prepare_idempotent_by_revision` (core): two `prepare` calls at R
- return the same capability; after a bump to R+1 a third returns a new one
- and the R capability is refused on attach.
-- `test_sleep_wake_same_request_id_refreshes_bundle` (controller, with the
- fake cloud of `tests/test_controller.py`): run, autostop, wake with the same
- `request_id` and retained instance. `prepare` is called for the new
- revision, a new secret version becomes `AWSCURRENT`, and `start_instance`
- is not called before `bundle_token` matches. The woken worker attaches with
- the new capability.
-- `test_revision_bump_between_prepare_and_start_recovers` (controller, Core
- and worker, fake cloud): `prepare(R)` returns, Core bumps to R+1, and the
- controller writes R and starts the instance. The R worker is refused 403 and
- exits 75, and no wake, relay or placement reaches it. The next poll writes
- R+1 as `AWSCURRENT`, the service restarts onto it and attaches at R+1. The
- worker waits (no daemon start) while `AWSCURRENT` is still the obsolete
- version.
-- `test_bundle_present_but_not_current_is_promoted` (controller): a version
- with the token but no `AWSCURRENT` stage is promoted before `bundle_token`
- is set.
-- `test_controller_restart_between_issuance_and_persistence` (controller):
- kill after `prepare` returns, before `put_secret_value`; restart. One secret
- version for that revision, holding the capability `prepare` first returned.
- The same test with the put response lost and a `ResourceExistsException`
- retry.
-- `test_stream_reports_restart_below_floor` (core) and
- `test_gap_above_cursor_is_not_cursor_reset` (switch-agent-runtime).
-
-## Frames (protocol 7)
-
-One frame per purpose, as main already has (`session_command` 5,
-`approval_outcome` 4, `room_released` 6). All are sent only to the attached
-worker. `session_command` stays room controls only.
-
-| Frame | Data | Body? |
-|---|---|---|
-| `worker_attached` | `{launch_revision, limits: {sessions_per_agent}, idle: {report_every_s, fresh_for_s}, credential_revision, queued_operations: [id], relay_fence, cancelled: [{room_id, message_id, reason}]}` (`reason` is `stopped` or `expired`) | no |
-| `relay` | `{id, deadline_ms, relay_seq \| null, message}` | yes (Console request) |
-| `relay_cancel` | `{id}` | no |
-| `wake` | `{entries: [{room_id, message_id, thread_id, event}]}`, ≤ 50 entries | yes (room event) |
-| `mailbox_cancel` | `{entries: [{room_id, message_id}]}` | no |
-| `operation` | `{id}` | no: a doorbell to claim |
-| `credential` | `{revision \| null}` | no: a doorbell to fetch |
-
-Frames that carry a body are queued per connection (as `session_commands`),
-bounded at 64 frames or 8 MiB. A mutating relay reserves its slot before it
-takes a `relay_seq` (D2), so a full queue refuses it `worker_busy` (503)
-without consuming a sequence number. A wake entry that does not fit stays
-`pending`. Nothing is dropped silently.
-
-A relay's `deadline_ms` is an absolute deadline in Unix epoch milliseconds
-(Core's wall clock when it sent the relay plus the request's timeout). Core
-fails the relay `relay_timeout` at that moment whatever the worker does; a
-worker may use it to abandon work nobody will read, and must not treat it as
-a duration.
-
-## D1 — Console ⇄ cloud relay
-
-### Vocabulary
-
-The relayed message is exactly one element of `control.ts`
-`clientMessageSchema` minus `id` and `token`, plus additions the sidecar path
-gains too, so the three paths stay one vocabulary:
-
-| Message | Answer | Notes |
-|---|---|---|
-| `{sessionId, request: SessionRequest}` | the host's reply | `command` (prompt, `session.stop`, `turn.interrupt`, `session.reset`, …; `origin.surface = "console"`, as `submitSessionCommand`). `snapshot` is paged (below). `room` and `approvals` are **refused**: only the watcher sends those. |
-| `{subscribe: sessionId}` / `{unsubscribe}` | ack, then pushed `ServerEvent`s | As `ControlClient.subscribe`, including the standing failure on first subscribe. |
-| `{place: {sessionId, roomId}}` | `PlaceOutcome` | "Reconnect to room". |
-| `{forget: sessionId}` | ack | |
-| `{health: true}` / `{watchHealth: bool}` | `WatcherHealth` / pushed `{health}` | |
-| `{list: true}` **new** | `Session[]` (`sessionSchema`), paged | What `host-sessions.ts` `LIST_SCRIPT` reads over SSH, computed in-process. Extract to `agent-providers/src/host/session-list.ts` and build `LIST_SCRIPT` from it, with a parity test. |
-| `{journal: sessionId}` **new** | `Snapshot`, paged | A stopped or parked session read from its `events.jsonl`; `JournalTail.snapshot`'s replay (desktop `sdk-host/host-journal.ts`) moves into `agent-providers`. |
-| `{page: {snapshotId, index}}` **new** | one page | See Paged answers. |
-| `{attachment: …}`, `{attachmentCancel: transferId}` **new** | see Attachments | Closes main's "Console attachments have no path to the host" for all three paths. |
-
-`ensure` is **refused**: it carries a full `SharedHostConfig` (worker paths
-and credential locations) that Console must not dictate for a cloud worker.
-Starting and restarting a cloud session stay `HostedOperation`s.
-
-Mutating messages are `command`, `place`, `forget`, `attachment`,
-`attachmentCancel`. Everything else is read-only; that split drives D2.
-
-The loopback socket and the relay call one function: factor `serveControl`'s
-per-message body into `handleControlMessage(links, ensure, control, message,
-send)`; `serveControl` keeps the token handshake and the socket.
-
-### Console → Core (gateway, user-authenticated)
-
-**Owner only**, as #538 (`HostedLaunchStore.owned`). A launch that is not the
-caller's, or not in the caller's tenant, is **404** (not 403), so existence
-does not leak. Tenant admins cannot relay.
-
-`POST /gateway/hosted-launches/{request_id}/relay`
-
-```
-request: {"message": <relayed message>, "timeout_ms": int <= 30000} // body <= 2 MiB
-response: {"ok": bool, "value"?: any, "error"?: {"code", "message"},
- "worker": {"launch_revision": int, "boot_id": str, "generation": int}}
-```
-
-For a mutating message the route takes the launch lock and checks that the
-launch is awake and the worker is attached at the current revision. It then
-reserves a slot in that connection's frame queue (or fails `worker_busy`),
-bumps `hosted_launches.relay_seq` (new, durable) and `active_at`, registers
-the pending relay, commits, and only then puts the frame in its reserved
-slot. Putting a frame in a reserved slot cannot fail. What can still lose the
-frame is the connection dying or Core crashing; both end in a reattach, and
-the reattach fence settles the sequence (D2). Read-only messages take no
-lock and carry `relay_seq: null`.
-
-Core checks the refusals in this order and returns the first that applies:
-
-| Code | HTTP | When |
-|---|---|---|
-| `machine_stopped` | 409 | The owner stopped the machine. Console does not retry; the user starts the machine. |
-| `machine_error` | 409 | The machine is in `error`. Console does not retry; the user retries the machine. |
-| `worker_sleeping` | 409 | Read-only message and the machine is idle-sleeping; body `wake_available: true`. Console offers an explicit wake. A mutating message to an idle-sleeping machine starts it instead and returns `worker_waking`. |
-| `worker_waking` | 409 | The machine is starting, or the launch is `queued` or `provisioning`. Console retries with backoff; nothing is queued. |
-| `agent_stopped` | 409 | The launch's desired state is `stopped`. Console does not retry; the user starts the agent. |
-| `agent_crashed` | 409 | The launch is in `error` with `error_code` `agent_crashed`. Console does not retry; the user retries the agent. |
-| `worker_not_attached` | 409 | No worker attached for the launch's current revision, or the launch is gone. Console retries with backoff; nothing is queued. |
-| `generation_changed` | 409 | The reply came from, or the dispatch target became, another generation or revision. Console drops its live views and resnapshots. |
-| `worker_busy` | 503 | Frame queue full. |
-| `relay_timeout` | 504 | No reply by the deadline. Outcome unknown: Console reconciles through `snapshot` → `commandStatuses` (as `reconcileSessionCommand`), never by resending blind. |
-| `refused_message` | 400 | `room`, `approvals`, `ensure`. |
-| `too_large` | 413 | Request > 2 MiB. Replies are never too large: they are paged. |
-
-The machine checks also apply outside the relay. `POST
-/gateway/hosted-launches/{request_id}/sessions` returns 409 `{"detail":
-"<message>", "code": "machine_error" | "machine_stopped" | "worker_waking"}`;
-for an idle-sleeping machine it starts the machine and returns `worker_waking`.
-`POST /gateway/hosted-launches/{request_id}/lifecycle` with action `restart` or
-`retry` returns the same body with code `machine_stopped` when the owner stopped
-the machine.
-
-Core sets these launch `error_code` values (launch `state` becomes `error`):
-
-- `agent_key_missing`: the agents list finds no stored API key for the agent.
- The entry is listed as `unavailable`. The user removes and creates the agent
- again.
-- `agent_identity_missing`: the agents list finds no agent row. A queued launch
- gets one minute of grace, because launch creation commits `agent_id` before
- it registers the agent. Retry registers the id again.
-- `worker_attach_timeout`: the supervisor reports the agent `running`, but no
- worker attached for the current revision within 10 minutes while the launch
- is `provisioning`.
-- `agent_stop_timeout`: the launch stayed `stopping` for more than 10 minutes
- while its machine is `ready`.
-
-`GET /gateway/hosted-launches/{request_id}/relay/stream?subscribe=<sessionId>[&subscribe=…][&watchHealth=1]`
-is an SSE stream to Console with these frames:
-
-- `event` `{sessionId, event}` and `failure` `{sessionId, failure}`;
-- `health` `{health}`;
-- `resync` `{sessionId | null, reason}`;
-- `worker` `{launch_revision, boot_id, generation}`, sent first and again on
- change.
-
-Core holds **one** worker subscription per `(launch, session)`, reference
-counted across Console streams, and unsubscribes on the last close.
-Per-Console buffer: 1000 frames or 4 MiB. Opening this stream is read-only
-(D2).
-
-### Core → watcher, watcher → Core
-
-`relay` frames carry a uuid4 `id`. Pending relays live in an in-memory map
-`id → {tenant, agent, launch_id, launch_revision, relay_seq, core_boot,
-connection_id, generation, future, deadline}`; nothing is written to the
-database but `relay_seq`. A detach, eviction or revision bump fails every
-pending relay of that connection with `generation_changed`.
-
-- `POST /agents/{agent_id}/connection/relay/{relay_id}`
- `{connection_id, generation, ok, value?, error?}`. The reply is 404 for an
- unknown relay or one registered to another agent. It is 409
- `generation_changed` when the caller is not the connection and generation
- the relay was dispatched to (`require_current`, as
- `connection_placements`). A second reply is 409 and ignored. Replies are ≤
- 1 MiB plus envelope; a larger value is refused 413 and the watcher must
- page it.
-- `POST /agents/{agent_id}/connection/relay/push`
- `{connection_id, generation, pushes: [{subscription: sessionId | "health",
- seq, event? | failure? | health?}]}`, batched by the watcher (≤ 250 ms or
- 64 KiB). `seq` is per `(generation, subscription)`; Core forwards in order
- and sends `resync` on a gap. A push for a subscription Core no longer holds
- is answered `{unsubscribe: [...]}`.
-
-### Paged answers and live views
-
-`snapshot`, `journal` and `list` answers are paged; nothing is capped.
-
-- **Pinning.** The first answer is `{snapshotId, epoch, throughSequence,
- bytes, sha256, pageCount, page: {index: 0, data}}`. The watcher serializes
- the answer once and pins the bytes under
- `<state root>/relay-pages/<snapshotId>`. `epoch` and `throughSequence` are
- the `Snapshot`'s (`session-v1` `snapshotSchema`). Pages are 1 MiB slices of
- that one serialization, so an entry larger than a page is simply split.
- Console reassembles, checks `bytes` and `sha256`, then parses.
-- **Cursor retention.** `{page: {snapshotId, index}}` is valid for 120 s after
- the previous page was served. At most 4 pinned answers per watcher, at most
- 64 MiB pinned bytes; beyond that a new request fails `snapshot_busy` (503,
- retry). An expired or unknown id is `snapshot_expired`: Console starts over.
- Pins are deleted on expiry, on generation change and at watcher start.
-- **Subscribe first.** Console opens the relay stream with `subscribe` before
- requesting the snapshot, buffers live events, then drops those with
- `sequence ≤ throughSequence` and applies the rest. If the first buffered
- event is above `throughSequence + 1`, or a buffered event's `epoch` differs,
- that is a gap: resync. `transcripts.ts` `openTranscript` already does
- snapshot then live; it gains the buffering.
-- **Resync** (Console discards the view, keeps the subscription, requests a
- new snapshot):
-
- | Reason | Signal |
- |---|---|
- | session reset (epoch changed) | a live event with a new `epoch`; a later `page` is `snapshot_superseded` |
- | generation or revision change | `worker` frame changes; pending pages fail `generation_changed` |
- | Console buffer overflow | `resync {reason: overflow}` |
- | push sequence gap | `resync {reason: gap}` |
- | page cursor expired | `snapshot_expired` |
-
- Three resyncs of one session within 30 s stop the loop and show an error.
-
-### Attachments
-
-- `{attachment: {transferId, sessionId, name, mimeType, size, sha256, index,
- count, chunk}}`: `chunk` is base64 of ≤ 1 MiB, and the whole file is at most
- `MAX_ATTACHMENT_BYTES` (10 MiB, `host/attachments.ts`).
-- **Idempotent and ordered**: `(transferId, index)` identifies a chunk. An
- index below the next expected one is re-acknowledged without effect. An
- index above it is refused `out_of_order` with the expected index.
-- **Digest**: on the last chunk the watcher checks `size` and `sha256`, then
- stages with `stageAttachment` and answers `{staged: {transferId, ref}}`. A
- mismatch deletes the transfer and fails `digest_mismatch`.
-- **Bounds**: at most 4 transfers and 32 MiB staged per watcher; beyond that
- it fails `staging_full` (503).
-- **Cleanup**: `attachmentCancel`, 120 s without a chunk, generation change,
- and watcher start all delete partial transfers. A staged `ref` is consumed
- once by the `command` that names it, and deleted unconsumed after 10 min.
-
-### Tests that prove D1
-
-- Relay round trip for each message against a real watcher over a fake stream
- (agent-providers) and the Core routes (core), including `list` / `journal`
- parity with `LIST_SCRIPT` / `JournalTail`.
-- `room`, `approvals`, `ensure` refused; a foreign tenant, a non-owner, a
- tenant admin and a deleted launch all get 404.
-- A reply from a superseded generation is refused and the Console request
- fails `generation_changed`; a revision bump mid-request does the same.
-- `test_snapshot_pages_pinned_across_events`: events and a reset land between
- pages; the pages still form the pinned snapshot, and the reset yields
- `snapshot_superseded` and a resync.
-- `test_subscribe_before_snapshot_no_loss`: events land while the
- subscription is being set up; no event is lost or applied twice.
-- `snapshot_expired` after 120 s idle; `snapshot_busy` at the fifth pin.
-- Attachments: a chunk resent after a lost reply, out of order, a digest
- mismatch, `staging_full`, and a disconnect mid-transfer (partial deleted).
-- Two Console streams on one session create one worker subscription; overflow
- yields `resync`.
-- Nothing relayed reaches the database or the log (captured logs and a table
- scan).
-
-## D2 — Idle report and autostop
-
-### Report (watcher → Core)
-
-`POST /agents/{agent_id}/connection/idle`, accepted only from the attached
-worker (`require_current`). Sent on every change and at least every
-`report_every_s` (30).
-
-```
-{
- "connection_id": str, "generation": int,
- "report_seq": int, // monotonic per watcher process; lower or equal is ignored
- "relays_through": int, // contiguous resolved watermark (below)
- "busy": bool,
- "reasons": [{"kind": str, "session_id": str | null, "count": int}],
- "sessions": {"total": int, "live": int, "parked": int, "failed": int}
-}
-response: {"queued_operations": [id], "credential_revision": str | null}
-```
-
-The response is the durable catch-up for lost `operation` and `credential`
-doorbells.
-
-The reason kinds are:
-
-- `turn_running`, `turn_starting`;
-- `room_pending`: the watcher's pump queue, or a held room for a session;
-- `approval_open`, `reset_waiting`, `operation_claimed`, `relay_inflight`, `host_unknown`;
-- `console_recent`: a **mutating** relay in the last 10 minutes. Read-only
- messages (`list`, `snapshot`, `journal`, `page`, `health`, `subscribe`) and
- an open live view do not count, so a forgotten window cannot pin a VM;
-- `failed_holding` (below).
-
-Per-session busy is one predicate shared with parking: extract
-`shared-host.ts`'s `idleEnough` conditions (status ready, no running turn, no
-open request, no `resetDecisionPending`, no pending room input) into
-`sessionBusy(snapshot, host)`. The host announces it to its parent as a new
-host → parent message `{kind: 'busy', busy, reasons}` (`session-channel.ts`
-`fromChildSchema`) on every change. A host that is not running and has
-nothing queued in the watcher is idle. Parking (`SWITCH_SESSION_PARK_AFTER_MS`,
-30 min) is unchanged; a parked host is idle for this report.
-
-**Bounded holds**: a session whose host failed to start
-(`SessionHostFailedError`, `pumps` entry `failed`) with queued room messages,
-or one waiting on a reset decision, reports `failed_holding` /
-`reset_waiting` as busy for at most 15 minutes after its failure notice was
-posted, then idle. When the hold ends, the watcher acks each of that session's
-mailbox rows `held` (D3), so Core stops counting them too. The messages stay in
-the watcher journal and are retried on the next address of that session,
-after a wake if need be.
-
-Core stores the latest accepted report on the `Connection` with its own
-monotonic `received_at`. It is **fresh** when the connection is alive, the
-current generation, bound to the launch's current revision, and `received_at`
-is within `fresh_for_s` (75). Core restart, detach, generation change and
-revision bump all make it absent. **Absent or stale = busy.**
-
-### Activity Core knows about
-
-A command can be dispatched and still be invisible to the next idle report.
-The watcher may have sent the report before it applied the command, or before
-the host posted `busy`. So Core does not wait to be told:
-
-- Under the launch lock, a mutating relay reserves its queue slot, bumps
- `relay_seq` and sets `active_at = now` before the frame is sent (D1).
-- While any mutating relay of the launch is pending (in the in-memory map),
- the launch is busy.
-- Once replies are in, idle is trusted again only from a report with
- `relays_through ≥ hosted_launches.relay_seq`, read under the lock at
- decision time. A report below that counts as busy. `relay_seq` is durable,
- so the rule survives a Core restart; the report is absent then anyway.
-- Read-only relays, the relay stream and snapshots take no lock and touch
- neither `active_at` nor `relay_seq`.
-
-**`relays_through` is a contiguous resolved watermark**: the largest N such
-that every sequence number up to N is *resolved*. It is never the highest
-number received, because a lower one may still be in flight. A sequence
-number is resolved when its effect, if any, is already reflected in the
-watcher's `busy`. Each number is resolved exactly one way:
-
-| Resolution | When |
-|---|---|
-| `taken` | `command`: the host replied to the command (accepted, or a status). The host posts `busy` before it replies whenever its busy state changed, and the channel is ordered, so the watcher has applied the change first. `place`, `forget`, `attachment`, `attachmentCancel`: the watcher's handler has returned its answer. |
-| `refused` | validation failed in the watcher, or the host refused (`STALE_EPOCH`, `UNSUPPORTED_CAPABILITY`, `HOST_STOPPING`, …). Nothing changed. |
-| `interrupted` | received before a watcher restart and not resolved. The hosts died with it (`KillMode=control-group` in `switch-hosted-worker.service`), so nothing can apply the command later. After the restart, host recovery has settled it (`unknown`, `HOST_RESTARTED`), so current `busy` already reflects it. |
-| `not_delivered` | at or below the reattach fence and never received (below). |
-| `abandoned` | the handler had not yet dispatched to the host, and the watcher cancelled it (below). Cancelling happens inside the watcher's serial section, and the handler checks the flag in that section right before dispatching, so a cancelled handler can never apply later. The relay reply, if Console still waits, is `relay_timeout`. |
-| `barrier` | the command was dispatched to the host, no reply came, and a later busy barrier to that host has completed (below). |
-
-**Elapsed time alone never resolves a relay.** An unresolved relay is busy,
-reported as reason `relay_inflight` with its session, however old it is and
-whatever idle report Core last cached. Its handler may still be waiting to
-dispatch (the session's host starting, the watcher's serial chain), or the
-host may still be about to act on it. After 5 minutes unresolved (logged as a
-warning), the watcher tries to close it in one of two ways:
-
-- **Not yet dispatched**: cancel the handler, then resolve `abandoned`.
-- **Dispatched, no reply**: send the host a busy barrier, a new parent → host
- message `{kind: 'busyBarrier', id}` (`session-channel.ts`
- `toChildSchema`). The host runs it on the same serial chain as
- `accept` (`session-host.ts` `this.serial`), so it completes only after every
- earlier command has been accepted or refused. It answers `{kind: 'busy',
- busy, reasons, barrier: id}` with its state at that point. The watcher
- applies that state, then resolves `barrier`.
-
-If the barrier gets no answer (the host is hung, or its serial chain is
-stuck behind the command), the host's state is **unknown**, and unknown
-counts as busy (reason `host_unknown`). The relay stays unresolved. The watcher
-retries the barrier every minute. It gives up only if the host exits or is
-stopped; a host that has exited cannot apply the command, and the host's
-recovery on its next start settles it (`interrupted`). A permanently hung
-host therefore keeps the VM up. That is visible (`host_unknown` in the
-report, and in Console's health view), and stopping the session from Console
-or the room releases it.
-
-The watcher keeps `relays.jsonl` beside `assignments.jsonl`, with records
-`received {seq, id}` (fsynced before the message is applied) and `resolved
-{seq, how}`. It computes `relays_through` from them: completion out of order
-advances nothing until the gap below it resolves.
-
-**Gateway request cancellation.** Console can drop the request (closed
-window, network, its own timeout) at any await in the route. Once the
-sequence number is about to be committed, the rest of the route runs as one
-task under `asyncio.shield`, in D1's order: register the pending relay, commit,
-then put the frame in its reserved slot. Registration comes first so that an
-immediate worker reply always finds its relay. The task opens and closes its
-own database session (`tenant_session`), never the request's: `asyncio.shield`
-does not stop the request middleware from closing the request's session when
-the request is torn down. Cancelling the request cancels only the wait for the
-reply, never the send. A healthy worker stream therefore
-gets the frame, and neither side reconnects because Console left. The reply
-is discarded when it arrives, and the watermark advances as usual. If the
-shielded step itself fails after the commit (Core crash, the connection
-gone), that is the fenced reconciliation's case below.
-
-**Fenced reconciliation on attach.** A sequence number can be committed and
-never reach the watcher: the connection dies between commit and send, or Core
-crashes. Both end in a new attach, which runs under the launch lock:
-
-1. Every pending relay of the old connection or boot has already failed
- `generation_changed`. A frame for a gone connection has no queue to land
- in, so no relay numbered at or below the current `relay_seq` can reach the
- new connection.
-2. Core sends `relay_fence = hosted_launches.relay_seq` in `worker_attached`.
- Every relay after that takes a higher number and targets the new
- connection.
-3. Before its first idle report, the watcher resolves every number at or
- below the fence that has no `received` record as `not_delivered`, and
- every `received` one without `resolved` as `interrupted` if its process
- restarted.
-
-A hole therefore lasts at most until the next attach, and while it lasts the
-worker's connection is gone, so the report is absent and the launch is busy
-for that reason anyway. There are no Core-side voids to track: a queue
-refusal happens before a number is taken.
-
-### Decision (controller observation)
-
-`hosted_controller.observe` *(#538)* replaces `HostedLaunchStore.idle_busy`
-with `idle_evidence(launch)`. The launch is busy when any of these holds:
-
-1. there is no fresh report, the report says `busy`, or `relays_through <
- relay_seq`;
-2. a mutating relay of the launch is pending;
-3. a `hosted_operations` row is `queued`/`claimed` at the current revision;
-4. a `hosted_wake_mailbox` row for the agent is `pending`, `offered` or
- `accepted` (D3). `held` rows and `cancel_requested` tombstones do not
- count. A held input is waiting for a new address, not for time, and a
- tombstone is settled at the next attach. Whatever work either causes then
- shows in the report;
-5. an `approval_requests` row for the agent is `open` (main's
- `session_activity` tables).
-
-Busy sets `active_at = now` (as today). The launch stops only when all of the
-following hold, under the launch advisory lock, in one transaction:
-
-- not busy, `spec.auto_session` true, `hosted_idle_stop_minutes > 0`
- (`config.py` *(#538)*, default 0 = off);
-- `now - active_at >= hosted_idle_stop_minutes`;
-- the report's `received_at` is later than `active_at` (`note_addressed` and
- mutating relays bump `active_at` under the same lock);
-- `launch.revision` is the one the report's connection is bound to.
-
-Then `desired_state = stopped`, `state = stopping`, `sleeping = true` and
-`revision += 1`, as today. The bump evicts the worker (`launch_superseded`)
-and obsoletes its capability, so nothing it sends afterwards counts. An
-addressed event or relay racing the decision either commits first (bumps
-`active_at`, and the recheck fails) or after (the launch is `sleeping`: an
-addressed event wakes it and lands in the mailbox, and a relay is refused
-`worker_sleeping`). The EBS volume survives an EC2 stop.
-
-### Tests that prove D2
-
-- A pending approval, a running turn, a queued pump message and a claimed
- operation each keep the VM up across 3 grace windows.
-- No report, a stale report, a superseded generation, and a Core restart each
- count as busy.
-- `test_command_dispatch_races_autostop`: a Console prompt is dispatched after
- the watcher's last idle report and before the host posts `busy`, and
- observation runs in between → no stop (pending relay, then watermark).
-- `test_read_only_relay_does_not_renew_activity`: `list`, `snapshot`, `page`
- and an open relay stream for longer than the grace window → the VM stops.
-- A mention between the last idle report and the decision prevents the stop.
-- `test_permanent_startup_failure_holds_then_sleeps`: a host that always
- fails to start, one mention → one failure notice → busy for 15 minutes →
- rows `held` → the VM stops. A later mention wakes it, the held message is
- retried before the new one, and no second notice is posted for the held
- message.
-- `test_relay_crash_after_commit_before_enqueue`: Core crashes after
- committing `relay_seq = N` and before sending. After restart and reattach,
- `relay_fence = N`, the watcher resolves N `not_delivered`, `relays_through`
- reaches N, and the VM can stop.
-- `test_relay_queue_refusal_takes_no_sequence`: a full queue returns
- `worker_busy` and `relay_seq` is unchanged.
-- `test_relay_out_of_order_completion`: N+1 resolves before N;
- `relays_through` stays at N-1 until N resolves.
-- `test_delayed_handler_at_five_minutes_stays_busy`: Core's cached report
- says idle, and a command handler is blocked before dispatch for more than 5
- minutes. The launch stays busy the whole time (`relays_through` below
- `relay_seq`). At 5 minutes the watcher abandons the handler; unblocking it
- afterwards dispatches nothing; the watermark advances and the VM can stop.
- Variants: dispatched with a delayed host reply → resolved only after the
- barrier completes, with the barrier's busy state applied; a host that does
- not answer the barrier → `host_unknown`, busy, no resolution.
-- `test_gateway_cancel_after_commit_still_enqueues`: the Console request is
- cancelled right after the commit. The frame still reaches the worker on the
- same stream generation (no reconnect), the watermark advances, and
- `relay_seq` has no hole.
-- `test_gateway_cancel_with_teardown_and_immediate_reply`: through the real
- ASGI app and middleware, the client disconnects during the shielded step so
- the request's session is closed, and the fake worker replies the moment the
- frame arrives. The shielded task's commit and send succeed on its own
- session, the reply finds its registered relay (no 404), and no
- closed-session error is raised or logged.
-- `test_relay_worker_restart`: the watcher restarts with N received but not
- resolved. N resolves `interrupted` and the watermark advances; the host's
- recovered state decides `busy`.
-- Each mutating message kind (`command` accepted, `command` refused, `place`,
- `forget`, `attachment`, `attachmentCancel`) advances the watermark.
-- `sessionBusy` and parking agree (one predicate, table test).
-
-## D3 — Wake mailbox
-
-The mailbox **always writes** for hosted agents, whether or not a worker is
-attached. An "is it attached?" check followed by delivery can lose an event
-to a stream that dies, or a Core that restarts, before the watcher journals
-it.
-
-### Table `hosted_wake_mailbox`
-
-Tenant-scoped with RLS, like every table in main's `545f80e11f13`.
-
-| Column | Meaning |
-|---|---|
-| `tenant_id, agent_id, room_id, message_id` | Primary key. `message_id` is `roomInputId(event)` (`host/room-inbox.ts`): the message id for an addressed message, `type:sha256` for a listened `room_join` or task event. The dedupe key everywhere. |
-| `launch_id` | FK to `hosted_launches`, `ON DELETE CASCADE`. |
-| `thread_id` | The platform thread from the event payload (`payload.thread_id`, as `threadOf` in `shared-watcher.ts` reads it), so notices land in-thread without a lookup. |
-| `event` | JSONB `{type, payload, missed}` as the watcher's `onEvent` builds `Handoff.event`. |
-| `state` | See below. |
-| `ever_offered` | Set on the first offer and never cleared. |
-| `offered_to`, `offered_until` | `core_boot:connection_id:generation` and the lease, while `offered`. |
-| `origin` | `live` or `cutover` (Migration). |
-| `notice_owed` | The room notice a terminal move owes (`stopped`, `expired`, `expired_uncertain`, `started_before_*`), set in the same transaction as the move and cleared once the room has it. |
-| `addressed_at, updated_at, expires_at` | |
-
-Index `(tenant_id, agent_id, state, addressed_at)`.
-
-States:
-
-```
-pending ──offer──▶ offered ──ack journaled──▶ accepted ──ack admitted──▶ admitted
- ▲ │ │ ▲
- └──lease expiry / ─┘ ack held │ │ ack admitted (retry worked)
- foreign boot or generation ▼ │
- held
-explicit Stop: pending & !ever_offered ─▶ cancelled (definite)
- pending & ever_offered, offered, accepted, held ─▶ cancel_requested (stopped)
-24 h: accepted, held ─▶ cancel_requested (expired)
-cancel_requested ─watcher─▶ cancelled (not admitted) | admitted (already running)
-other terminal: refused, duplicate, expired (never offered), expired_uncertain
-```
-
-`held` is the durable form of the watcher's bounded failure hold (D2): the
-input is on the worker's disk, its failure notice has been posted, and it is
-waiting for the next address of its session, not for time. It does not keep
-the VM awake.
-
-### Write
-
-For a hosted agent every addressed event is written in the same transaction
-as `note_addressed` *(#538)*. That happens in `agent_client.py`
-`_note_hosted_addressed` and `on_task_delegate` *(#538)*.
-`ON CONFLICT DO NOTHING` makes a redelivered Matrix event a no-op. Traffic is
-chat-rate, so the cost is one insert per addressed event, for hosted agents
-only.
-
-The event is not written, and the room is told why through the existing
-unavailable reply, when `desired_state` is `stopped` with `sleeping` false
-(explicit Stop), `deleted`, or `state = error`. If the agent already has 500
-rows `pending` or `offered`, the insert is refused and the room is told:
-loud, not a silent drop.
-
-### Offer (lease, then send)
-
-Delivery to the watcher always follows **mark offered, then send**:
-
-1. `UPDATE … SET state = 'offered', ever_offered = true, offered_to = :me,
- offered_until = now() + 60 s WHERE state = 'pending'` (conditional; it can
- never overwrite an `accepted` or later row), commit;
-2. then enqueue: into the `EventBuffer` for a live event, or into a `wake`
- frame for a row sent on attach.
-
-- **Live, worker attached**: after the insert commits, the normal buffer path
- runs behind step 1. Without an attached worker the row stays `pending`.
-- **On attach**: every `pending` row, oldest first per room, as `wake` frames
- before the first buffered event.
-- **Reclaim** runs on attach (for that agent), at Core start, and every 30 s
- in the upkeep loop (`session_activity/maintenance.py`). It moves `offered →
- pending` when the lease has expired, when `offered_to` names another Core
- boot, or when it names a generation that is no longer attached. Reclaimed
- rows are offered again, and the watcher dedupes.
-
-A Core hard crash after step 1 leaves `offered` rows with a foreign boot;
-reclaim at start returns them. A crash between the insert and step 1 leaves
-`pending`, which is delivered on attach.
-
-### Ack (retried)
-
-`POST /agents/{agent_id}/connection/mailbox/ack`
-`{connection_id, generation, entries: [{room_id, message_id, outcome, reason?}]}`
-is accepted from the attached worker at the current revision. Every
-transition is a conditional update that only moves forward:
-
-| Outcome | Sent when | Row |
-|---|---|---|
-| `journaled` | the `parked` or assignment record is on disk (`host/journal.ts` syncs on append) | `pending`/`offered` → `accepted` |
-| `admitted` | the session host has taken it (host `inbox.jsonl` `accepted`) | `accepted` → `admitted` |
-| `duplicate` | the journal already has it | → `accepted` or `admitted` as the journal says |
-| `refused` | the watcher will not act on it (limit reached and auto-start off); the watcher posts the notice | → `refused` |
-| `held` | the session's host failed to start, the failure notice is posted (`room-notices`, reason `startup`), and the 15-minute hold has ended | `accepted` → `held` |
-| `admitted` after `held` | a later address of the session started its host, and the held input was admitted ahead of the new one | `held` → `admitted` |
-| `cancelled` / `admitted` | reply to a cancel (below) | `cancel_requested` → `cancelled` / `admitted` |
-
-**Held inputs.** The watcher keeps a held input's `parked` record and its
-`failure-notified` key. It retries the host start only on a new address of
-the same session (a live event or a mailbox row), or on a Console `start` /
-`restart` operation for it, and then admits held inputs in their original
-order before the new one. If the start fails again, the new message gets its
-own notice and goes `held` after its own hold, while the already-held inputs
-stay `held` and get no second notice (once per message and reason, Failure
-notices). Sleeping, waking and Core restarts do not touch held inputs; only
-the watcher's journal and the rows do.
-
-An ack for a row already at or past the target state is `ok`. An ack from a
-`pending` row (reclaimed while the ack was in flight) is applied, so a lease
-expiry never loses an ack. The watcher appends `mailbox-acked {room, message,
-outcome}` to `assignments.jsonl` after Core confirms. It re-sends every
-unconfirmed ack on attach and with every idle report, ≤ 200 per call. A lost
-ack without a reconnect is therefore retried within 30 s. Even if the retries
-fail, lease expiry re-offers the row and the watcher answers `duplicate`.
-
-The watcher treats `wake` and live events alike: a held delivery keyed by room
-and message, never by sequence. It goes through `SharedWatchAssignments.park`
-(the `parked` record stores the event and is keyed by `delivery(room,
-message)`), then the normal `admit` path. It skips a delivery when the
-journal already has an assignment, `parked` or `released` record for the same
-`(room, message)`, or the pump queue holds it.
-
-### Acceptance boundary and Stop
-
-**A row is accepted when the watcher's journal record is fsynced.** From then
-on only the watcher can say whether it ran. Core therefore announces an
-outcome as definite only when it knows it:
-
-- **Explicit Stop** (lifecycle route, in its transaction) moves rows that were
- never offered to `cancelled`, and posts one notice per room in-thread:
- "cancelled, not run". It moves `offered`, `accepted` and ever-offered
- `pending` rows and `held` rows to `cancel_requested` (reason `stopped`),
- with no notice yet. `admitted` rows are untouched. Explicit Stop never
- wakes.
-- If the worker is still attached, Core sends `mailbox_cancel` before the VM
- stops. In every case the `cancelled` list rides on the next
- `worker_attached`. `cancel_requested` rows are durable tombstones: kept until
- the watcher answers or the launch is deleted. They are exempt from the 24 h
- expiry and the 7-day prune below.
-- **Watcher reconcile, before admission.** On boot the watcher admits nothing
- from its journal (held rooms, parked deliveries, pump queue) until it has
- processed `worker_attached.cancelled`. For each entry:
- - journaled or held, and not yet admitted to a host: append `released`
- with the entry's reason (fsynced), then ack `cancelled`, and Core posts
- "cancelled, not run" (or "expired, not run" for reason `expired`);
- - already admitted: ack `admitted`, and Core posts "had already started
- before Stop";
- - unknown to the journal: ack `cancelled`, which is definite because
- nothing on the worker has it.
-- A launch deleted with `cancel_requested` rows posts nothing (the agent is
- removed); no outcome is invented.
-
-### Retention
-
-By state, 24 h after `addressed_at`:
-
-| State | Becomes | Notice |
-|---|---|---|
-| `pending`, never offered | `expired` | in-thread, one per room: "expired, not run" (definite: the worker never had it) |
-| `pending` ever offered, or `offered` | `expired_uncertain` | says delivery could not be confirmed; never claims the event did not run |
-| `accepted`, `held` | `cancel_requested` (reason `expired`) | none yet; the watcher settles it at the next attach, and the notice then says which way it went |
-| `cancel_requested` | unchanged | none. Tombstones are exempt from expiry and pruning and stay until the watcher answers or the launch is deleted. |
-
-Terminal rows (`admitted`, `cancelled`, `refused`, `duplicate`, `expired`,
-`expired_uncertain`) are pruned 7 days after `updated_at` by the upkeep loop.
-
-### Owed notices
-
-Every notice above is posted after its transaction commits, so a send can
-fail after the row has moved. The row keeps `notice_owed` until the room has
-the notice; the upkeep loop retries owed notices every 30 s, up to 100 rooms a
-pass, logging a warning for each failure. The per message and reason receipt
-(Failure notices) keeps a retry from posting twice. A row pruned with its
-notice still owed takes the obligation with it.
-
-### Tests that prove D3
-
-- Sleep → mention → wake → exactly one turn, with Core restarted before
- attach, after `wake` is sent but before ack, and after ack.
-- `test_core_crash_after_offer_reclaims`: rows `offered` under a dead boot go
- back to `pending` at start and are delivered once.
-- `test_lost_ack_retried_without_reconnect`: the ack POST fails once on a
- healthy stream → retried with the next idle report; the row is `accepted`.
-- `test_offer_ack_race_never_regresses`: an ack commits between reclaim and
- re-offer; the re-offer's conditional update does not touch the row.
-- `test_stop_between_fsync_and_ack`: the row is `cancel_requested`; next boot
- the watcher releases it and acks `cancelled`; notice "not run"; no turn.
-- `test_stop_between_ack_and_admission`: same, from `accepted`.
-- `test_stop_after_admission`: acked `admitted`; notice "had already started".
-- Mention while attached, stream killed before the watcher journals: one turn
- after reconnect.
-- The 501st `pending`/`offered` insert is refused with a room notice; expiry
- notices distinguish `expired` from `expired_uncertain`.
-- `test_tombstone_survives_expiry_and_prune`: a `cancel_requested` row 8 days
- old is still there and still in `worker_attached.cancelled`.
-- `test_held_row_expires_through_watcher`: a `held` row at 24 h becomes
- `cancel_requested (expired)`; on the next attach the watcher releases it and
- the notice says "expired, not run".
-- `test_permanent_startup_failure_holds_then_sleeps` (D2) covers the
- `accepted → held → admitted` path.
-
-## Operations: `hosted_operations` as the infra queue
-
-`HostedOperation` *(#538)* keeps `start` / `restart` of a cloud session:
-durable, idempotent by client-chosen `id`, with an outcome record. Console's
-other session actions go through the relay.
-
-- **Admission moves to the watcher.** `session_operation` *(#538)* stops
- reading `SdkSession`. Core checks only ownership, launch `ready`/`running`,
- id reuse, and one pending operation per session.
-- **Doorbell, not payload.** An insert sends an `operation {id}` frame to the
- attached worker. Core re-rings every 5 s while the row is `queued`, up to 6
- times. The durable catch-up is `worker_attached.queued_operations` and the
- idle-report response, so a doorbell lost on a healthy connection costs at
- most one report interval (30 s). `runHostedControl`'s 2 s poll *(#538)* goes.
-- **A doorbell only prompts a fenced claim; it never replays an operation.**
- `POST /hosted/operations/{id}/claim` `{connection_id, generation}` replaces
- `/operations/claim`. It moves `queued → claimed` only for the attached
- worker at `operation.launch_revision`, storing `claimed_by =
- core_boot:connection_id:generation` and `claimed_boot_id` (new columns).
- The host boot that claimed may claim again while the row is still `claimed`
- (`claimed_boot_id` matches): its claim reply may have been lost, and the
- watcher runs an operation only after journaling its claim, so a repeat claim
- never runs it twice. `worker_attached.queued_operations` and the idle-report
- response list those ids for that boot alongside the `queued` ones. Anything
- else is 409, including a claim from another boot. A doorbell for a claimed
- or finished id is ignored.
-- **Result**: `POST /hosted/operations/{id}/result` as today, plus
- `{connection_id, generation}`. The watcher writes the outcome to
- `operations.jsonl` before posting and re-posts every unconfirmed outcome on
- each attach. A lost result is recovered, not re-executed ("a lost result is
- never a reason to execute the operation again", `hosted-control.ts`
- *(#538)*, stays true). Core accepts a re-post from a later generation of the
- same host boot (`claimed_boot_id` matches).
-- **States** unchanged: a revision mismatch turns `queued → failed` and
- `claimed → unknown` (`fail_stale_operations`); `claimed` for 5 minutes with
- no result → `unknown`, which tells the owner to inspect the session via
- `snapshot`/`journal`.
-
-Tests: `test_operation_doorbell_lost_on_healthy_stream` (claimed within one
-report interval); a duplicate doorbell after claim causes no second
-execution; lost claim reply and lost result → one execution, outcome
-recovered.
-
-## Session existence and slot admission (watcher)
-
-One serial section in `runSharedWatcher` (the existing `pending` promise
-chain) decides for both auto-created sessions (`admit` →
-`assignments.assign`) and operations (`start` / `restart`):
-
-- `start` for a session id that exists under `sharedSessionsBase()` → result
- `failed` "already exists"; `restart` for one that does not, or that belongs
- to another agent → `failed` "not found" (as `executeHostedOperation`
- *(#538)* checks today).
-- Active means live or starting, not stopped, not retired; parked counts as
- active. `start`, `restart` of a stopped session, and a new auto-created
- session are refused when active ≥ `limits.sessions_per_agent`
- (`hosted_sessions_per_agent`, default 8). A refused operation is `failed`
- "session limit"; a refused auto-create posts a notice (reason `capacity`)
- and acks the mailbox row `refused`.
-- Local and SSH watchers get the same check with no limit configured.
-
-Test: two concurrent starts plus one auto-create at limit − 1 admit exactly
-one.
-
-## Credentials
-
-`POST /hosted/provider-credential` *(#538)* drops `sessions` (it lists
-`SdkSession`s; the watcher knows its own) and keeps `{status, kind, provider,
-revision, credential}` with `Cache-Control: no-store`.
-
-- **Doorbell**: a change to the owner's `provider_connections` row
- (`verified_at` moves, or the row is deleted) sends `credential {revision}`
- to the attached worker. The frame carries no secret. A delete sends
- `revision: null` and nothing else: the launch keeps its state and revision,
- so the worker is not failed or superseded — it stays attached, stops its
- hosts on the `revoked` fetch, and idle-stops like any quiet worker.
-- **Fetch**: on the doorbell, on each attach, and whenever
- `credential_revision` in an idle-report response differs from the one
- applied. The 2 s poll *(#538)* goes.
-- **Apply**: a new revision is materialized (`materializeHostedProvider`),
- then each live session restarts at its next idle point, never mid-turn.
- Codex keeps #538's `refreshCodexAuthentication` in `codex/home.ts` rather
- than main's copy-once, in hosted mode only. `revoked` stops every host
- (`supervision.stop`), reports idle, and makes addressed events answer
- "provider disconnected" instead of starting sessions. GitHub installation
- tokens (`/hosted/github-credential`, `github_issued_tokens`) are unchanged.
-- **Wake while revoked** does not start the VM: `note_addressed` refuses to
- wake a launch whose provider connection is missing, and the room is told to
- have the owner reconnect. #538's `note_addressed` has no such check.
-
-## Failure notices
-
-#538's `POST /agent-sessions/{id}/room-failure` depends on
-`SessionAuthority`, which is gone. It is replaced by an agent-authenticated
-`POST /agents/{agent_id}/room-notices`
-`{connection_id, generation, room_id, message_id, thread_id, reason}`, with
-reasons `startup | delivery | conversation | capacity | auto_start_off | stopped | expired |
-cancelled | revoked | upgrade`:
-
-- Authorization equals `post_message`'s for this agent (the agent's client is
- in the room, the room is not archived) plus: the caller is the attached
- worker, and `message_id` is a message in that room. Posting goes through
- `protocol.send_message` in `thread_id`.
-- **Once per message and reason**: the key `[agent, room, message, reason]`
- under an advisory lock, checked against `messages.content.switch_room_failure`
- as #538 does. Core-originated reasons (`stopped`, `expired`, `cancelled`,
- `upgrade`) use the same function, with the thread taken from the mailbox row.
-- Not an MCP tool, so the tool surface and the Switch skill are unchanged.
- Main's `announceStartFailure` (`watcher-tools.ts`) stays for local and SSH.
-
-The watcher keeps unsent notices in `notices.jsonl` and retries them on
-attach.
-
-## Room controls while sleeping
-
-`commands.py` `_dispatch_control_command`: compute `delivered = placed is not
-None and relay_session_command(...)`, where the relay goes only to the
-attached worker. When nothing was delivered and the agent is hosted:
-
-- `!reset`: if the launch is sleeping or waking, call `note_addressed` (which
- wakes it) and reply "The cloud worker is waking up. The reset was not
- queued … send `!reset` again" (#538 wording). This is two steps on purpose:
- a destructive command is never queued for later. Not written to the
- mailbox.
-- `!interrupt`, `!compact`: reply that the agent is asleep and nothing is
- running; do not wake.
-- Otherwise main's existing "controller is not connected" reply.
-
-Placements are in memory in Core and restated by the watcher on attach, so a
-sleeping agent has no placement. The check must not rely on `placed`.
-
-## Migration
-
-### Schema
-
-Our 11 revisions are applied on the shared pilot with parent `2d84b6f1c705`:
-`ab921ef034cd → c29f7018ea44 → e7240c165b92 → f138a612de04 →
-a472be90d1f6 → 48ab0298a34b → 51bc94a017d2 → 62cf05b128e3 → 73da16c239f4 →
-84eb27d340a5 → 95fc38e451b6`. None references the tables `b9e4d2a71c05`
-drops (checked by grep). Never edit or re-parent them.
-
-1. `a3c9e5f71d28_hosted_cutover_manifest`, child of `95fc38e451b6`, run by
- `hosted-cutover-upgrade prepare` (Pilot order). It adds
- `hosted_cutover_volumes (tenant_id, launch_id, preflight_state pending |
- blocked | complete, manifest_sha256, completed_at, blocked_reason,
- imports_queued_at)` and `hosted_cutover_items (tenant_id, agent_id,
- launch_id, session_id, kind, room_id, message_id, thread_id, evidence,
- disposition, payload JSONB NULL, notice_posted_at)`, with RLS; copies what
- the `sdk_*` tables and queued `hosted_operations` hold for hosted agents
- into items; and makes one `pending` volume per hosted agent's latest
- launch. It refuses a database whose launches outlived `sdk_sessions`.
-2. `33e037ee949f` merges it with `c4e9a1f7b203`. It runs after
- `b9e4d2a71c05`, in the same transaction, and raises unless every volume of
- a non-deleted launch is `complete`, every item of one is decided, and every
- `import` attachment still has its `media_blobs` row. The raise rolls the
- drop back. With no cutover volumes (every main database, every fresh one)
- it does nothing.
-3. `c4d8e2f1a9b7` (worker capability) and `d7e3a9c1f5b2`
- (`hosted_wake_mailbox`, RLS). The mailbox copy of `import` items is not a
- migration: `hosted-cutover-upgrade` queues them through
- `HostedMailboxStore.write` right after the upgrade (`pending`, `origin =
- cutover`, `expires_at = now() + 24 h`), once per volume, recorded in
- `imports_queued_at`. The worker's manifest upload queues any not yet
- queued, under the launch lock.
-
-An empty merge only proves the graph has one head. It does not prove the
-cutover is safe. The migration tests (`test_migration_hosted_merge.py`) are
-real upgrades with data, against PostgreSQL:
-
-- a pilot database at `95fc38e451b6` with launches, operations and `sdk_*`
- rows is captured at the manifest revision, its volume recorded, and
- upgraded to heads: launches and operations are intact, every item is
- decided, the `sdk_*` tables are gone;
-- the gate refuses, naming the launch, while a volume is `pending` or
- `blocked` (with the file and line the check named), an item is undecided,
- an import has no event, an import's blob would be dropped with its session,
- or the old tables changed after `prepare`; a bare `alembic upgrade heads`
- in each of those states rolls back with the drop undone;
-- once recorded, the upgrade keeps exactly the session blob an import needs,
- drops the others, and queues the import once;
-- a main database at `c4e9a1f7b203` upgrades with the hosted tables empty;
-- `alembic heads` is one head; `test_frozen_ddl_matches_create_all.py` still
- passes (no new NOTIFY DDL: mailbox delivery is offer-on-attach plus live
- buffer).
-
-### Preflight: pending work on retained volumes
-
-The #538 volumes hold work that main's code cannot pick up as-is (checked at
-`758bc5ae`):
-
-- `room-inbox.jsonl` `received` records carry `{sequence, roomId, messageId,
- missed, gap}`. There is **no event body**: #538 submitted the ids to Core
- (`shared-host.ts` → `/room-message`), and Core rebuilt the prompt from
- `messages` (`delivery/replay.py` `replay_room_event`, which refuses anything
- older than the 15-minute buffer window and multi-file groups). Main has
- neither the route nor `delivery/replay.py` / `transport/stored_event.py`.
-- `assignments.jsonl` holds `{sequence, roomId, messageId, config}` and
- `restarted` records only (no `handled`, `parked`, `released`). Main's schema
- is a superset, but `config` must parse under main's `sharedConfigSchema`.
-- Per-session `inbox.jsonl` uses the same record types as main
- (`accepted{command}`, `dispatched`, `finished`, `stopped`, `reset-started`,
- `reset-completed`, …). Commands carry `origin.{roomId, threadId,
- messageId}`.
-- #538 has no `handoff.ts`; main's `handoff.ts` is the in-memory `Handoff`
- type and `HostWaker`, not a file. So no handoff file needs importing.
-
-Main's host recovery (`session-host.ts`, on load) already settles what the
-host holds:
-
-- open requests become `request.settled` `interrupted`;
-- `accepted`/`dispatched` commands become `command.status` `unknown`
- (`HOST_RESTARTED`, not resent), with a transcript notice;
-- `reset-started` without `reset-completed` waits for a reset decision
- (`RESET_OUTCOME_UNKNOWN`).
-
-So reset state is preserved as-is, and request state is preserved as
-`interrupted`: a provider's pending permission call dies with its process on
-both branches. What remains is making room-originated work visible, or
-running it once.
-
-The **preflight** is `hosted-preflight` (agent-providers). The first boot of
-the new image runs it in place; before the drop it runs as
-`hosted-bootstrap.mjs --preflight-check <state-dir> <scratch-dir>` against a
-copy of each stopped volume, which it makes in the scratch directory and
-removes, so the volume itself is never written. The manifest it answers is
-the one the first boot will write, because the items depend only on the
-journals. For each session root and the watcher root it:
-
-1. parses every journal (`assignments.jsonl`, `room-inbox.jsonl`,
- `inbox.jsonl`, `events.jsonl`, `delivery-*.jsonl`, `shared-state.jsonl`)
- with **main's** schemas. Any failure blocks that volume and names the file
- and line;
-2. lists, without bodies, what it found per source:
- - `room_pending`: `received` records in a session's `room-inbox.jsonl`
- that are not acknowledged. The rule is exactly
- `SharedRoomInbox.readMessageState` *(#538)*: an `ack` matches by
- `identity` (`JSON.stringify([roomId, messageId])`) when it has one,
- otherwise by the `sequence` of the latest `received` for that message,
- and a `cursor` record with `reset: true` clears the sequence match.
- Legacy `received` records without `missed`/`gap` parse with the defaults
- (`storedReceivedSchema`). `failure-notified` identities are listed too;
- - host commands from each session's `inbox.jsonl`, with `origin.surface`,
- `origin.roomId` and `origin.messageId` and the furthest record seen
- (`accepted`, `dispatched`, `finished`);
- - `request_open` (from `events.jsonl`) and `reset_pending`, per session;
-3. prints `{"manifest": {manifest_sha256, items}}`, or `{"blocked": {step,
- file, line, error}}` with the file on the volume, and exits 2 when blocked.
- `hosted-cutover-upgrade record <launch-id> <file>` applies it: a manifest
- is merged with Core's capture and decided (below) and the volume becomes
- `complete` with its digest; a blocked check makes it `blocked` with the
- reason. After the upgrade the worker's first boot uploads the manifest to
- `POST /agents/{id}/connection/cutover-manifest`, which only confirms it:
- `409 cutover_manifest_unrecorded` for a volume with nothing recorded,
- `409 cutover_manifest_conflict` for a different digest (the volume changed
- after its check).
-
-**One disposition per logical room message.** The same message can appear in
-up to three sources at once: `room_pending` (its room-inbox ack was lost), a
-host command that got as far as `dispatched`, and a Core
-`sdk_session_commands` row whose status never became terminal. Core merges
-every record by `(agent_id, room_id, message_id)` before deciding. The Core
-receipt for a room message is found by its command id, which #538 derives as
-`uuid5(NAMESPACE_URL, "switch-room:{agent}:{room}:{message}")`
-(`sessions/service.py` `submit_room_message` *(#538)*). The strongest evidence
-wins, in this order:
-
-| Evidence, across all sources for the message | Disposition |
-|---|---|
-| a host `finished` record, or a receipt with status `applied` | `ran`: dropped, no notice |
-| a host `dispatched` record, or a receipt with status `dispatched` or `unknown` | `uncertain`: in-thread notice "may have been interrupted by the upgrade; re-send if needed". Never imported. |
-| a receipt with status `rejected` | `unrecoverable`: notice "was not run; please send it again" |
-| only `room_pending`, a host `accepted` record, or a receipt with status `accepted` | `import`: the payload is rebuilt with `to_inbound` and `message_payload` (replay's reconstruction without the 15-minute window) into `payload`. A multi-file group, a missing blob or a deleted message makes it `unrecoverable` instead. |
-
-Items that are not room messages are decided on their own:
-
-| Item | Disposition |
-|---|---|
-| Console-origin host command, `accepted` only | `settled_by_host`: main's host marks it `unknown` in the transcript Console shows; nothing is posted to a room, so no private prompt text leaks |
-| Console-origin Core receipt never delivered to the host | owner notice without the prompt text |
-| `request_open` | `interrupted`: in-thread notice "the approval was interrupted by an upgrade; ask the agent again" (reason `upgrade`) |
-| `reset_pending` | `preserved`: no action |
-
-A `failure-notified` message that is imported gets one more notice (reason
-`upgrade`): "will run now, after the upgrade", so the earlier failure notice
-is not the last word.
-
-An import into a session that also has a `reset_pending` item does not run
-now: the host holds room messages until the reset is decided. Every such
-import, failure-notified or not, gets the `conversation` notice instead,
-which says it waits and who acts: anyone in the room with `!reset @agent`,
-or the owner with Start a fresh conversation in Switch Console. It is keyed
-like the worker's own `conversation` notice, so a message #538 already told
-this is not told again.
-
-**Imported commands never meet the old host's dedupe record.** Main's host
-accepts a command id it has seen only if the command is identical, and
-otherwise returns the old status or `IDEMPOTENCY_CONFLICT`
-(`session-host.ts` `accept`). The old `accepted` record for an imported
-message is still in `inbox.jsonl`, and recovery marks it `unknown`. The ids
-already differ: main's `roomCommandId` hashes with SHA-256 (`room-prompt.ts`)
-where #538 used uuid5. The contract does not rely on that. A mailbox row with
-`origin = cutover` runs under `uuidFrom("switch-room-cutover:{agent}:{room}:{message}")` (the hash `roomCommandId` uses),
-a namespace no earlier build used. The transcript then shows the old command
-as interrupted and not sent, followed by the imported one running: both true.
-Only `import` produces a mailbox row, and one message has at most one row
-(primary key), so it runs at most once.
-
-Notices are posted by the new Core after the upgrade, once per key
-(`post_notice_once`), and each is recorded in `notice_posted_at`: at the
-manifest upload, and by the mailbox upkeep for every item still owed, so a
-volume whose worker has nothing to upload is not skipped and a failed post
-is retried. Recording a manifest keeps each import's attachment through the
-drop: its `media_blobs` row is detached from its #538 session, which
-`b9e4d2a71c05` would otherwise delete. Imported payloads live only in
-`hosted_cutover_items` until they are queued into the mailbox. Nothing is
-written to a file.
-
-Tests:
-
-- `test_cutover_merges_overlapping_records`. The fixture is a retained volume
- written in the exact 758bc5ae formats: a `room-inbox.jsonl` with a legacy
- `received` (no `missed`/`gap`), an `ack` without `identity`, a `cursor`
- reset, and a `received` whose ack was lost; an `inbox.jsonl` whose
- room-origin command for that same message reached `dispatched`; and an
- `sdk_session_commands` row for it (the uuid5 id) with status `accepted`.
- The result is one item, `uncertain`, with one notice and no mailbox row.
- Variants: host `accepted` only → one `import`; receipt `applied` → `ran`.
-- `test_imported_command_not_deduped_by_old_record`: a session whose
- `inbox.jsonl` holds the old `accepted` record for a message that is then
- imported. After recovery the old command is `unknown`, and the imported one
- is accepted under the cutover id and runs once.
-
-**Gate.** Before `b9e4d2a71c05` can run, every hosted agent's latest
-non-deleted launch needs a volume with `preflight_state = complete`, every
-item of one needs a disposition, every `import` needs its event, and every
-import attachment needs a `media_blobs` row no session owns.
-`hosted-cutover-upgrade` checks all of this, plus that every launch is
-stopped and that the `sdk_*` rows still match what `prepare` captured (the
-old Core did not run again), and refuses with one line per problem, naming
-the launch. The conditions are one set of plain-SQL queries in
-`switch_core/db/hosted_cutover_gate.py`, which the wrapper and `env.py` run;
-the merge revision carries a frozen copy of the checks that still apply after
-the drop.
-
-`alembic upgrade heads` orders `b9e4d2a71c05` freely between the branches (on
-a fresh database it runs the drop before the manifest revision), and Core
-runs it at boot, so the wrapper alone does not hold the gate. `migrations/env.py`
-therefore checks the whole gate on the migration connection, inside the
-migration transaction, before the first revision of any upgrade plan that
-includes an unapplied `b9e4d2a71c05`, and aborts the upgrade on any problem.
-The merge revision `33e037ee949f` checks it again after the drop and rolls the
-drop back, which also covers a runner that builds its own migration context
-rather than loading `env.py`; there the checks that read the old tables
-(stale capture, an import attachment still owned by a session) cannot run,
-and a lost attachment shows up as a missing blob instead. Both pass a database
-with no hosted launches, which is every fresh and every main-only database.
-
-**Revised: the guard lives in `env.py`.** An earlier revision of this design
-rejected an `env.py` guard and relied on the merge revision alone. That missed
-the stale-capture check: by the time the merge revision runs the old tables are
-gone, so a capture the old Core outran after `prepare` passed a bare
-`upgrade heads`. The only places that can see those tables are `b9e4d2a71c05`,
-a revision ordered before it, or `env.py`; the first two would mean editing or
-re-parenting revisions already on main, whose chain is frozen, so `env.py` it
-is. It inspects the resolved plan rather than the command line, so it runs
-exactly when the drop is about to be applied.
-
-**Revised from the approved design, and why.** The approved design had a
-transitional Core (#538 plus the manifest revision, a manifest route and the
-disposition job) receive each worker's upload and post every notice before
-the drop. That cannot be built as written, and the gate above is the closest
-lossless design:
-
-- *No Core can serve the upload before the drop.* The disposition code is
- main's (`to_inbound`, the attachment group rules, the mailbox), so a
- transitional Core would be a second release of #538 carrying a port of it;
- the new Core cannot run on a database still at `a3c9e5f71d28`, and the old
- Core must stay down from `prepare` on or its capture goes stale. A worker
- therefore has no Core to reach before the drop, and running the preflight
- needs no worker at all: the same code checks a copy of the stopped volume,
- and the one-shot `record` step (new code, directly against the database at
- `a3c9e5f71d28`) applies it with the tables it reads still present.
-- *Notices cannot be posted before the drop.* Posting needs a running Core's
- agent clients. Nothing a notice needs is dropped: the items, the rooms and
- the messages all survive, so the gate requires every notice to be decided,
- not posted, and the new Core posts each owed notice durably, once.
-- *Imports are queued after the drop, not copied by a revision,* so they go
- through the mailbox's one write path and are queued once per volume.
-
-### Pilot order
-
-The pilot sits on our head; main's `a7e1c4b90d23 … c4e9a1f7b203` are not
-applied. Among them `b9e4d2a71c05` drops `sdk_sessions`, `sdk_session_events`,
-`sdk_session_commands`, `sdk_room_admissions` and `session_request_posts`,
-and deletes blobs tied to them. Its downgrade raises.
-
-1. Stop every hosted launch through the controller with desired `stopped`
- and `sleeping = false` (explicit Stop, so nothing wakes), and wait for
- `stopped`. Volumes persist.
-2. Take the old Core down and keep it down until step 8; nothing may write
- to the `sdk_*` tables or a volume from here on.
-3. `pg_dump` the pilot database.
-4. `just hosted-cutover-upgrade prepare`: refuses unless every launch is
- stopped, then captures the old tables at `a3c9e5f71d28`.
-5. For each launch `just hosted-cutover-upgrade status` lists: snapshot its
- volume, attach the copy to a maintenance host with the new runtime build,
- and run `node hosted-bootstrap.mjs --preflight-check <state-dir>
- <scratch-dir> > <launch-id>.json` (`<state-dir>` is the volume's `state`
- directory, `<scratch-dir>` a directory that does not exist yet, outside
- it). Then `just hosted-cutover-upgrade record <launch-id>
- <launch-id>.json`, whether the check passed or blocked.
-6. For a blocked volume, repair the file and line the reason names on the
- real volume, snapshot it again and repeat step 5 for it.
-7. `just hosted-cutover-upgrade status` until it reports nothing blocking.
-8. `just hosted-cutover-upgrade` (the `upgrade` step): the gate, main's chain
- including the destructive revision, the merge and the mailbox, then the
- imports queued. Deploy the new Core; it posts the owed notices.
-9. Roll the worker image (first-boot migration below), then Console. Start
- the launches within 24 hours of step 8, or their queued imports expire
- (each with its notice).
-
-A step that fails can be run again: `record` of the same manifest changes
-nothing, and `upgrade` rolls back whole and, after the drop, only queues
-what is not yet queued. Should the gate report that the old tables changed
-after `prepare`, restore the dump from step 3 and start again at step 2.
-
-### Cutover gating (Core + worker + Console together)
-
-- Core refuses a hosted stream below protocol 7 (`upgrade_required`), so an
- old worker image cannot half-work against new Core. The controller launches
- the runtime build Core names in the spec.
-- Console detects `hosted_relay` on the version route (`version_routes.py`)
- and otherwise shows "update Switch Console" for cloud agents. An old
- Console against new Core fails on the removed `sharedList` path;
- compatibility is waived as in main's step 7.
-- An old Core never sees a new worker: images are rolled after Core.
-
-### Retained worker state (first boot of the new image)
-
-The bootstrap runs this before the daemon starts. It is idempotent, and each
-step is recorded in `<state root>/state-version.json` so a crash resumes it:
-
-- **Room inbox**: `room-inbox.jsonl` is renamed to
- `room-inbox.jsonl.pre-cutover` and **not** imported. Its pending entries
- reach the watcher through the mailbox (`origin = cutover`), so there is one
- source and dedupe by `(room, message)` covers any overlap.
-- **Assignments**: rewritten to main's schema if the preflight found `config`
- differences; the original is kept as `.pre-cutover`. Sequence numbers stay
- valid: #538 already used the per-boot floor and the new Core continues the
- same `agent_event_boot` sequence.
-- **Session journals** stay in place; main's host recovery settles them (see
- Preflight).
-- **Placements**: none to write. `SessionPlacements.open(root, () =>
- assignments.placements())` derives them from `assignments.jsonl` for a
- watcher with no `placements.json`; Console-started sessions use their
- config's `roomConnection.restoreRoomId`.
-- **Native, config and provider homes**: Codex homes are
- `<root>/<sha256(sessionId)>` on both branches (`codex/home.ts`). The
- preflight also checks that OpenCode data dirs and Claude project dirs
- resolve to the same paths under main's `provider-home.ts`; the first boot
- moves (renames on the same volume) where they do not. No credential is
- copied into a config file.
-- **Pending operations**: an operation applied before the cutover with an
- unposted result is recorded in `operations.jsonl` and re-posted. The Core
- row has become `failed`/`unknown` by revision bump, the re-post is refused
- 409, and that is logged, not retried.
-- **Activity**: with `activity-reported.jsonl` absent, main starts reporting at
- the journal's end, so no history is replayed to the platforms.
-
-## Data held where
-
-| What | Where | Durable? |
-|---|---|---|
-| Transcripts, provider state, native homes, placements, room journal (including held inputs), relay journal (`relays.jsonl`), operation outcomes, unsent notices, unconfirmed acks, pinned pages, staged attachments | Worker volume | Yes (survives EC2 stop); pages and partial transfers are cleared at start |
-| Launch, revision, sleeping, `active_at`, worker capability (hash, encrypted copy, revision), `relay_seq` | `hosted_launches` | Yes |
-| Bundle token required and persisted, per agent | controller agent store (SQLite) | Yes |
-| Worker bundle for the current revision | assignment secret, `AWSCURRENT` | Yes; one version per running revision |
-| Start/restart commands and outcomes | `hosted_operations` | Yes |
-| Addressed events until admitted, held inputs, and cancel tombstones | `hosted_wake_mailbox` | Yes: 24 h to expiry, 7 d terminal; tombstones until answered |
-| Pre-cutover pending work | `hosted_cutover_items` | Until the mailbox copy; drop after the pilot |
-| Approval requests, activity rows | main's `session_activity` tables | Yes (as main) |
-| Idle report, pending relays, live subscriptions, placements, frame queues | `ConnectionRegistry` | No: lost on Core restart, and loss means busy / retry |
-
-## What the contracts cannot say
-
-- A relay timeout leaves a command's outcome unknown until Console reads a
- snapshot. If the worker went to sleep in between, only a wake answers it.
-- Sleeping sessions cannot be inspected without waking: Core holds no
- transcript by design. A read-only cache of the last snapshot was rejected
- for that reason.
-- `console_recent` is a heuristic; ten minutes is arbitrary.
-- The mailbox guarantees one hand-off to the watcher journal and an honest
- outcome, not exactly-once turns. A host that crashes mid-turn after
- admission is main's recovery story: the turn is reported interrupted, not
- resent.
-- Pre-cutover room messages older than the platform's own history (a deleted
- message) can only be reported, not replayed.
-
-## Settled scope
-
-- **Relay is owner-only.** Tenant admins get no relay, read-only or
- otherwise, for now; a non-owner is 404 (D1). Support access would be a
- separate, audited feature.
-- **Mailbox limits are pilot defaults**: 500 rows `pending` or `offered` per
- agent (the bound D3 enforces; `accepted` and `held` rows are on the worker's
- disk and do not count), 24 h to
- expiry, 7 days to prune. Hitting a limit is always visible: a refused insert
- tells the room, and expiry posts its notice. The upkeep loop logs, per agent,
- pending-or-offered count, oldest such row and refusals, and the pilot sets the
- final values from those numbers.
-
-## Work packages
-
-- **WP1 base port.** Rebase #538's non-session parts onto main: controller,
- launches, provider and GitHub connections, bootstrap, the controller's
- per-revision bundle (`required_bundle_token` / `bundle_token`, start gated
- on it and on the version holding `AWSCURRENT`, `accept_launch` without the
- `instance_launch_issued` early return for the bundle step), `agent_event_boot`
- floor with the below-floor restart branch, `ac2ffa1d`. Remove every
- `SdkSession` / `SessionAuthority` use (`hosted_routes.py`,
- `hosted_launches.py`, `hosted_launch_store.py`, `session_routes.py`,
- `sessions/service.py`, `commands.py`). Merge revision. Tests: real upgrades
- from both heads; #538's controller and launch tests green;
- `test_stream_reports_restart_below_floor`,
- `test_sleep_wake_same_request_id_refreshes_bundle`,
- `test_controller_restart_between_issuance_and_persistence`.
-- **WP2 worker runtime on `--watch-worker`.** Main's `--watch-worker` branch
- of `shared-daemon.ts` with `SessionLinks` and in-process supervision; drop
- `runHostedControl`'s loop; factor `handleControlMessage` out of
- `serveControl`. Adds `list`, `journal`, paged answers, attachment transfers,
- capability file (from the bundle's `workerCapability`, accepted by
- `switch_hosted_worker.py`) and attach headers, `relays.jsonl` and the
- contiguous `relays_through` with the attach fence, handler cancellation and
- the `busyBarrier` host message, exit 75 on an obsolete capability with the
- worker's `obsolete-bundle` wait, idle report,
- `busy` IPC, the doorbell handlers (`operation`, `credential`) with claim and
- result via `operations.jsonl`, watcher-side admission, reconcile-before-admit
- for `cancelled`, held inputs and the `held` ack, cutover rows under the
- cutover command id, retried acks, `notices.jsonl`.
-- **WP3 server.** Worker capability: issued per running revision by
- `prepare`, idempotent at a revision (stored encrypted beside its hash),
- checked on stream open, one attached worker, `hosted_worker_only` gating of opens, reattach,
- subscribe and placements, `launch_superseded` eviction, and room-control
- relay narrowed to the worker. Protocol 7 frames (`worker_attached`, `relay`,
- `relay_cancel`, `wake`, `mailbox_cancel`, `operation`, `credential`) with
- bounded per-connection queues. Gateway relay route and relay stream
- (owner-only, 2 MiB requests, paged replies, resync reasons); agent relay
- reply and push routes. Mutating relays reserve a queue slot, then stamp
- `relay_seq` and `active_at` under the launch lock; register the pending
- relay, commit, then send, as one task shielded from request cancellation
- that owns its own database session (the request's session is closed by
- middleware at teardown, shield or not), tested by
- `test_gateway_cancel_with_teardown_and_immediate_reply`; `relay_fence` in
- `worker_attached`; pending-relay busy, idle report storage, `idle_evidence`
- with the contiguous watermark and without `held` rows or tombstones. Operation doorbell with re-ring, fenced claim, result re-post.
- Credential doorbell and idle-report catch-up.
- `/agents/{id}/room-notices`. `!reset` two-step.
-- **WP4 mailbox.** Table with states, leases and `ever_offered`; write in
- `note_addressed`'s transaction; mark-offered-then-send on the live path and
- on attach; reclaim on attach, at start and in upkeep; forward-only ack
- route; the `held` state; Stop's `cancelled` / `cancel_requested` split and
- tombstone delivery; expiry by state (`expired`, `expired_uncertain`,
- `cancel_requested (expired)`), tombstones exempt; pruning; the 500-row
- bound and its upkeep metrics.
-- **WP5 Console cloud.** A `CloudRelayClient` with `ControlClient`'s
- interface; `askHost`, `transcripts.ts` (subscribe-first, paged snapshot,
- resync), `host-sessions.ts`, the health monitor and stop/forget route cloud
- agents through it; chunked attachment upload; `sleeping` and `waking` shown
- as health states from the launch, with an explicit wake; the cloud sidebar
- reads `list` instead of polling `sharedList`.
-- **WP6 cutover and state migration.** The cutover-manifest revision,
- `hosted-preflight` and its `--preflight-check`, the per-message merge and
- dispositions, the `just hosted-cutover-upgrade` steps and gate with the
- merge-revision guard, the confirm-only manifest route, durable cutover
- notices, the pilot runbook above, the first-boot migration, and version
- gates on all three sides.
-- **WP7 acceptance** (`just bench` scenarios plus a pilot run):
- 1. retained Codex and OpenCode conversations resume after cutover; a
- pre-cutover pending room message runs exactly once or is reported;
- 2. two sessions on one worker, room takeover between them
- (`replace_placements`, `room_released`);
- 3. a pending approval, a running turn and a just-dispatched Console prompt
- never auto-stop; a lost relay and a permanently failing host do not keep
- the VM up forever;
- 4. sleep → mention → exactly one turn, across a Core restart at each step,
- with the woken VM on a fresh capability under the same `request_id`;
- 5. explicit Stop never wakes; nothing reported "not run" ever runs;
- 6. lost doorbell, lost claim and lost result: no double execution, outcome
- recovered;
- 7. a generation change mid-relay, mid-page and mid-operation is fenced;
- an old VM and a local Console cannot attach;
- 8. revoked provider credentials: hosts stop, no restart loop, no wake;
- 9. tenant isolation: foreign tenant and non-owner relay, ack, claim and
- notice calls all 404/403 and leak nothing;
- 10. load: many hosted agents with many parked sessions keep Core CPU and
- pool use flat (idle reports only; no per-session traffic).
diff --git a/docs/hosted-connections.md b/docs/hosted-connections.md
deleted file mode 100644
index be6100435..000000000
--- a/docs/hosted-connections.md
+++ /dev/null
@@ -1,170 +0,0 @@
-# Connections (v1)
-
-A connection gives a user's cloud agents access to an external service. Each
-connection comes with a skill: instructions, installed on the agent's VM, for
-using the service. v1 has a catalog and a grid in the Console. GitHub is the
-only connection that works; every other service in the catalog is shown as
-"Coming soon".
-
-## Decisions
-
-- **Owner.** A connection belongs to the user and can be granted to many of
- their agents.
-- **Catalog.** Built in only. It ships with Core; admins cannot add entries or
- upload skills.
-- **Accounts.** One account per service per user.
-- **Scope.** Cloud agents only. Local agents started by the Console get no
- connection skills.
-- **CLIs.** Baked into the VM image (`gh` today). Nothing is downloaded at
- boot.
-- **GitHub stays required** for a cloud launch, since the workspace is a
- GitHub repository. Every cloud launch is therefore granted the GitHub
- connection. Claude Code, Codex and OpenCode agents also get its skill.
- Cursor and Antigravity agents get no connection skills, because they have
- no skills directory.
-
-## Catalog
-
-One directory per service under `core/switch_core/connections/catalog/`:
-
-```
-catalog/<slug>/
- connection.yaml # required
- skill/ # required when enabled, forbidden when not
- SKILL.md
- ... # optional extra files, e.g. references or scripts
-```
-
-`connection.yaml` has exactly these fields:
-
-```yaml
-slug: github # must equal the directory name
-name: GitHub
-category: Source control
-description: Clone, branch, push and open pull requests in the repository you grant.
-enabled: true # false = "Coming soon" placeholder
-auth:
- type: oauth # oauth | api_key
-```
-
-The catalog has no logos. The Console bundles one logo per slug in
-`console/apps/switch-console-desktop/src/assets/images/connections/`, with
-sources in `NOTICE.md`, and draws a monogram from the name for a slug with no
-logo.
-
-`core/switch_core/connections/loader.py` validates the whole catalog when
-Core imports it, so a malformed entry stops Core from starting. The loader
-rejects:
-
-- unknown keys, a slug that does not match its directory, or an unknown auth
- type;
-- any file in an entry other than `connection.yaml` and `skill/`;
-- an enabled entry without a skill, or a placeholder that ships one;
-- a skill without `SKILL.md`, or whose `SKILL.md` frontmatter is missing
- `name: <slug>` or a `description`;
-- symlinks, unsafe paths, non-UTF-8 or NUL content, and skills larger than
- 32 KiB.
-
-v1 entries:
-
-| Category | Services |
-|---|---|
-| Source control | GitHub (enabled), GitLab, Bitbucket |
-| Project management | Jira, Asana, Linear |
-| Knowledge base | Notion |
-| Productivity | Google Workspace, Microsoft 365 |
-| File storage | Box |
-| Observability | Datadog, New Relic |
-| CRM | Salesforce |
-| Design | Canva |
-| Deployment | Vercel |
-
-## API
-
-`GET /provider-connections/catalog` (gateway, authenticated) returns the
-catalog along with the caller's status for each entry:
-
-```json
-{"connections": [{"slug": "github", "name": "GitHub", "category": "Source control",
- "description": "...", "enabled": true, "auth_type": "oauth",
- "status": "connected"}]}
-```
-
-`status` is `coming_soon` for a placeholder. For an enabled entry it is
-`connected` if the caller has a row in `provider_connections` for that slug,
-and `not_connected` otherwise. Connecting and disconnecting GitHub still use
-the existing GitHub endpoints.
-
-## Skill delivery
-
-The skills travel with each agent in the machine's agent list, next to the
-other agent data. The supervisor copies them into that agent's deployment
-document:
-
-```json
-"skills": [{"slug": "github", "files": {"SKILL.md": "..."}}]
-```
-
-1. **Core** (`GET /hosted/machines/{id}/agents`) adds `skills` to each agent
- for the connections granted to the launch. A provider that has no skills
- directory (Cursor, Antigravity) gets an empty list, and Core logs a warning.
-2. **The supervisor** (root) builds each agent's deployment. It copies
- `skills` into the deployment only when the list is not empty.
-3. **The supervisor** also validates `skills` strictly: at most 16 skills,
- unique and well-formed slugs, `SKILL.md` required, relative paths with no
- empty, `.` or `..` segment, no NUL, and 32 KiB in total. It writes nothing
- into the agent-owned state disk, because a root process writing into a
- directory the agent controls could be sent elsewhere through a symlink.
-4. **Bootstrap** (runs as the agent) applies the same limits. It refuses skills
- for a provider that has no skills directory. Before it starts the worker,
- on every start, it replaces each skill in the provider's skills directory,
- writing to a temporary directory first and then renaming it into place:
-
- | Provider | Directory |
- |---|---|
- | Claude Code | `$CLAUDE_CONFIG_DIR/skills/<slug>/` |
- | Codex | `$CODEX_HOME/skills/<slug>/`, linked into each session's home |
- | OpenCode | `$XDG_CONFIG_HOME/opencode/skills/<slug>/` |
-
-Each agent in the agent list must have `skills`, which can be empty. A
-deployment without `skills` keeps working.
-
-## Rollout
-
-The worker rejects a deployment key it does not know. A worker built before
-this change therefore rejects a deployment that carries `skills`, and the
-launch fails. That happens when a new Core sends skills to a worker on an
-older VM image.
-
-- Roll out the new VM image before, or together with, the new Core. A worker
- must accept `skills` before any launch carries them.
-- Existing workers pick up the new image only when they are replaced. Until
- then, do not deploy a Core that sends skills to them.
-- Images and a VM image built before this change cannot validate skill
- delivery. Validate it on a build that contains the whole change: Core,
- worker and bootstrap.
-
-## Console
-
-In the cloud onboarding flow, the step that used to connect GitHub directly
-is now a Connections grid:
-
-- a search box that matches name or category, ignoring case;
-- one card per entry, showing the logo, name, category and a status
- badge; enabled entries are listed first;
-- the GitHub card opens the existing GitHub connection step, and going back
- returns to the grid and refreshes it;
-- placeholder cards are disabled and marked "Coming soon";
-- "Continue to agent" appears once GitHub is connected.
-
-The renderer gets the catalog through `rpc.switchServers.getConnectionCatalog`,
-which calls the gateway from the main process.
-
-## Not in v1
-
-- Choosing, per launch, which connections an agent gets. Today GitHub is
- always granted.
-- Other services: their auth flows, credential storage and delivery
- (credential files, environment wrappers), and "Test connection" checks.
-- Custom catalog entries, and more than one account per service.
-- Connection skills for local agents.
diff --git a/justfile b/justfile
index 66d8d5592..c4d43ac3d 100644
--- a/justfile
+++ b/justfile
@@ -154,15 +154,6 @@ migrate:
uv run --project core alembic -c core/alembic.ini upgrade head
-# ── Upgrade a database that ran hosted agents on the old session tables ─────────
-# Steps: prepare, record <launch-id> <check.json>, status, upgrade (default).
-# Refuses the drop of the old session tables until every retained volume's
-# preflight check is recorded; see the cutover steps in
-# docs/hosted-activity-contracts.md.
-hosted-cutover-upgrade *args:
- uv run --project core python -m switch_core.hosted_cutover_upgrade {{args}}
-
-
# ── Generate a new alembic migration ──────────────────────────────────────────
migration msg:
uv run --project core alembic -c core/alembic.ini revision --autogenerate -m "{{ msg }}"