Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Use WebSocket on controller isolated agents (CHOO-3013) (#687)

Abel Dantas6ae882645cc1
diff --git a/artifacts.yaml b/artifacts.yaml
index a6090ab16..79f7ea373 100644
--- a/artifacts.yaml
+++ b/artifacts.yaml
@@ -195,7 +195,8 @@ contracts:
# window, so `accepts` stays at 1 on both sides. switch-core serves either
# transport, and agent-runtime prefers the socket and falls back to the
# stream and its beat against a server with no socket. Both go, and
- # `accepts` rises to 8, once no client still connects over the stream.
+ # `accepts` rises to 8, once `switch.agents.connected{transport:sse}` stays
+ # at zero.
artifacts:
switch-core: { speaks: 8, accepts: 1 }
agent-runtime: { speaks: 8, accepts: 1 }
diff --git a/console/packages/agent-controller/README.md b/console/packages/agent-controller/README.md
index ce93c1057..35ad52455 100644
--- a/console/packages/agent-controller/README.md
+++ b/console/packages/agent-controller/README.md
@@ -15,12 +15,12 @@ runs, and the agent hosts do the running.
An agent whose definition asks for `isolation: "isolated"` runs instead as an agent
host in a process of its own (the shared-host bundle with `--ensure-watch`, as Console
-runs an agent on an SSH host). It hears its events through the relay, which serves it
-the per-agent agent protocol (its own event stream, heartbeat and placements), and it
-is not stopped when the controller exits: it reconnects when the controller is back.
+runs an agent on an SSH host). It hears its events on the controller's **hub**, the
+same one in-process agent hosts use, over a WebSocket on the relay's port, and it is
+not stopped when the controller exits: it reconnects when the controller is back.
Changing an agent's isolation restarts it the other way.
-The controller holds **one** event stream to Switch for all of its agents and hands
+The controller holds **one** connection to Switch, a WebSocket, for all of its agents and hands
each shared agent's events to its agent host directly. Each agent host makes its calls to Switch
through a **local relay** on a loopback port, which adds the controller's
credentials. No agent holds a Switch credential: each is given a token that only the
@@ -158,24 +158,25 @@ session hosts it launches with `process.execPath` run as Node too.
### The controller stream
- Opens a connection (`POST /v1/controllers/{id}/connection`) with each agent's
- cursor (the last sequence its agent host confirmed, or `"head"`), then attaches the
- stream (`GET /v1/controllers/{id}/events`). The stream carries every bound agent's
+ cursor (the last sequence its agent host confirmed, or `"head"`), then attaches a
+ WebSocket to it (`/v1/controllers/{id}/connection/ws`). It carries every bound agent's
events (`agent.event`, `agent.gap`, `agent.session_command`,
`agent.approval_outcome`), its attachment and room membership (`agent.attached`,
`agent.detached`, `agent.rooms`), and the management nudges (`assignment.changed`,
`operation.pending`, `credential.revoked`).
-- Beats the connection (`POST .../connection/beat`) every `heartbeat_interval_s`
- (2 s) while the stream is attached, with each agent's confirmed cursor.
+- Answers every `ping` Switch sends on the socket (each `heartbeat_interval_s`, 2 s)
+ with a `pong` naming each agent's confirmed cursor: that pong is its heartbeat.
- The open and every beat also carry `placements`: for each agent whose running
agent host has a session placed in a room, those rooms. It is the
whole current map each time (Switch replaces what it held), and agents with no
placement are left out, so a placement the agent host makes reaches Switch on the
next beat.
-- A dropped stream is reattached to the same connection. A connection Switch no
+- A dropped socket is reattached to the same connection. A connection Switch no
longer knows (`unknown_connection`, `stale_generation`, or any `evicted` but
`taken_over`) is opened afresh, from the cursors as they stand. Reconnects back
- off with jitter, and a stream silent for 45 s (Switch writes a keepalive every
- 15 s) counts as dropped.
+ off with jitter, except after close code 1012 (Switch restarting), which is
+ retried within a second. A socket silent for 10 s (Switch pings every 2 s)
+ counts as dropped.
- `taken_over`, as a frame or a refusal, means another instance of this controller
opened the stream. This one stops its agents and exits with code `4`.
- Each agent's events are handed to its agent host one at a time, in order, and only
@@ -197,9 +198,18 @@ session hosts it launches with `process.execPath` run as Node too.
with a token minted for that agent (`swlr_…`) as `SWITCH_API_TOKEN`. A token the
relay did not mint is refused with `401`; while the controller is still starting,
with `503`.
-- It serves no event stream and no connection bookkeeping
- (`GET /agents/{id}/events`, `POST /agents/{id}/connection/…` answer `404`): the
+- It serves no event stream and no connection bookkeeping of the agent protocol
+ (`GET /agents/{id}/events`, `POST /agents/{id}/connection/…` answer `410`): the
controller holds each agent's connection to Switch itself.
+- **The hub**, at `ws://127.0.0.1:<port>/hub` with the agent's relay token, named in
+ the credentials file as `SWITCH_AGENT_HUB`. An agent host in a process of its own
+ hears its events there: the hub sends each event, gap, room control and approval
+ outcome as a request, and counts it handled when the agent host answers `done`, as
+ it does for an agent host in the controller's process. The agent host states its
+ sessions' rooms there too (`placements`). One agent host per agent: a newer one
+ takes the hub over (close `4409`, the older stands down). When the controller
+ stops, the hub closes with `1012` and the agent hosts reconnect when it is back,
+ resuming after the last event they handled.
- **Forwarded to Switch**, everything else under `/agents/{id}/…`,
`/agent-sessions/…`, `/sessions/…`, `/version` and `/health`: operations, media,
typing, history, session activity and approvals. The relay sends them with the
@@ -229,7 +239,8 @@ session hosts it launches with `process.execPath` run as Node too.
reboot, say): the agent host is launched again, once 15 s have passed since it
was last launched, so an agent host still coming up is not launched twice.
- **Desired `running`, relay credentials rewritten** (the relay came back on
- another port): a running agent host is restarted so it reads them.
+ another port, or they were written before the hub): a running agent host is
+ restarted so it reads them.
- **Agent host failed or was taken over:** it is left down, and reported as
`failed`. A new revision or an `agent.restart` brings it back. One that failed
because the relay refused its token is relaunched once it has a new one.
@@ -288,7 +299,7 @@ It holds:
|---|---|
| `controller.db` | SQLite: identity, cached assignment, per-agent applied revision and local failures, restart times, each agent's stream cursor, the relay's port, status seq. Everything except the identity can be rebuilt from the server. |
| `secrets/controller-credential` | The controller credential (see below). Absent when the credential is handed over with `--credential-stdin`. |
-| `agents/<id>/credentials.json` | Each agent's relay endpoint and relay token, in the layout the shared host reads. No Switch credential. |
+| `agents/<id>/credentials.json` | Each agent's relay endpoint, relay token and hub, in the layout the shared host reads. No Switch credential. |
| `agent hosts/<id>/` | Each agent's agent host state root: `watch.json`, `config.json`, `health.json` (what `status` reads), its journal, and `supervisor/failure.json` once it has failed for good. |
An agent whose definition names no directory works in
diff --git a/console/packages/agent-controller/package.json b/console/packages/agent-controller/package.json
index b793aed26..28147897a 100644
--- a/console/packages/agent-controller/package.json
+++ b/console/packages/agent-controller/package.json
@@ -26,11 +26,13 @@
"dependencies": {
"@switch-console/agent-providers": "workspace:*",
"@switch-console/plugins": "workspace:*",
+ "ws": "^8.21.3",
"zod": "^4.4.3"
},
"devDependencies": {
"@sandboxaq/switch-agent-runtime": "workspace:*",
"@types/node": "^24.13.3",
+ "@types/ws": "^8.18.2",
"@typescript/native-preview": "7.0.0-dev.20260609.1",
"tsdown": "^0.22.2",
"vitest": "^5.0.1"
diff --git a/console/packages/agent-controller/src/agent-hub.ts b/console/packages/agent-controller/src/agent-hub.ts
index 02ffb720d..0d3877880 100644
--- a/console/packages/agent-controller/src/agent-hub.ts
+++ b/console/packages/agent-controller/src/agent-hub.ts
@@ -140,6 +140,13 @@ export class AgentHub {
return cursors;
}
+ /** Where Core attached the agent on the controller stream, while it is attached. */
+ attachment(agentId: string): { fromSeq: number; rooms: string[] } | null {
+ const agent = this.agents.get(agentId);
+ if (!agent?.attached || agent.head === null) return null;
+ return { fromSeq: agent.head, rooms: [...(agent.rooms ?? [])] };
+ }
+
/** The agent's events flow on the controller stream and its agent host is taking them. */
attached(agentId: string): boolean {
return this.agents.get(agentId)?.host?.connected ?? false;
diff --git a/console/packages/agent-controller/src/api.test.ts b/console/packages/agent-controller/src/api.test.ts
index b34f3b95e..6c7e56cc8 100644
--- a/console/packages/agent-controller/src/api.test.ts
+++ b/console/packages/agent-controller/src/api.test.ts
@@ -9,11 +9,7 @@ import {
} from './api';
import { ConfigurationError } from './errors';
import { silentLogger } from './log';
-import {
- type Assignment,
- controllerBeatRequestSchema,
- controllerConnectionRequestSchema,
-} from './schemas';
+import { type Assignment, controllerConnectionRequestSchema } from './schemas';
import { FakeCore } from './testing/fake-core';
let core: FakeCore;
@@ -46,6 +42,7 @@ function client(credential = core.credential) {
controllerId: core.controllerId,
version: '0.1.0',
tokens,
+ openWebSocket: core.openWebSocket,
}),
};
}
@@ -131,7 +128,7 @@ describe('enroll', () => {
});
expect(result).toEqual({ controller_id: core.controllerId, credential: core.credential });
const request = core.requests.at(-1)!;
- expect(request.headers[PROTOCOL_HEADER.toLowerCase()]).toBe('1');
+ expect(request.headers[PROTOCOL_HEADER.toLowerCase()]).toBe('2');
expect(request.headers['idempotency-key']).toBeTruthy();
});
@@ -279,8 +276,8 @@ describe('errors', () => {
});
});
-describe('the controller stream routes', () => {
- it('opens a connection declaring itself and its cursors, beats it, and attaches the stream', async () => {
+describe('the controller connection', () => {
+ it('opens a connection declaring itself and its cursors, then attaches its socket', async () => {
core.setAssignment(assignment);
const { client: api } = client();
const signal = new AbortController().signal;
@@ -292,43 +289,36 @@ describe('the controller stream routes', () => {
client_version: '0.1.0',
cursors: { 'agent-1': 41, 'agent-2': 'head' },
});
- const connection = { connectionId: opened.connection_id, generation: opened.generation };
- const stream = new AbortController();
- const response = await api.openEvents(connection, stream.signal);
- expect(response.headers.get('content-type')).toContain('text/event-stream');
- await api.beat(connection, { 'agent-1': 42 }, signal);
- expect(controllerBeatRequestSchema.parse(core.requests.at(-1)!.body)).toEqual({
- connection_id: opened.connection_id,
+ const { socket } = await api.openSocket({
+ connectionId: opened.connection_id,
generation: opened.generation,
- cursors: { 'agent-1': 42 },
});
- stream.abort();
- });
-
- it('reads the refusals that decide between reopening and stopping', async () => {
- const { client: api } = client();
- const signal = new AbortController().signal;
- await expect(
- api.beat({ connectionId: 'nobody', generation: 1 }, {}, signal)
- ).rejects.toMatchObject({ status: 404, code: 'unknown_connection' });
- const first = await api.openConnection({}, signal);
- await api.openConnection({}, signal);
- await expect(
- api.beat({ connectionId: first.connection_id, generation: first.generation }, {}, signal)
- ).rejects.toMatchObject({ status: 409, code: 'taken_over' });
+ await new Promise((resolve) => socket.addEventListener('message', resolve, { once: true }));
+ const attach = core.requests.at(-1)!;
+ socket.close(1000);
+ expect(attach.method).toBe('WS');
+ expect(attach.path).toBe(`/v1/controllers/${core.controllerId}/connection/ws`);
+ expect(Object.fromEntries(new URLSearchParams(attach.search))).toEqual({
+ connection_id: opened.connection_id,
+ generation: String(opened.generation),
+ });
+ expect(attach.headers.authorization).toBe('Bearer access-token-1');
+ expect(attach.headers['switch-controller-protocol']).toBe('2');
});
it('reads a refusal in the agent bridge’s own envelope', async () => {
const { client: api } = client();
core.scripted.push({
method: 'POST',
- path: `/v1/controllers/${core.controllerId}/connection/beat`,
+ path: `/v1/controllers/${core.controllerId}/connection`,
+ status: 409,
+ body: { detail: { code: 'taken_over', message: 'someone else' } },
+ });
+ await expect(api.openConnection({}, new AbortController().signal)).rejects.toMatchObject({
status: 409,
- body: { detail: { code: 'no_stream', message: 'open the stream' } },
+ code: 'taken_over',
+ message: 'someone else',
});
- await expect(
- api.beat({ connectionId: 'c', generation: 1 }, {}, new AbortController().signal)
- ).rejects.toMatchObject({ status: 409, code: 'no_stream', message: 'open the stream' });
});
});
@@ -360,7 +350,7 @@ function statusReport() {
return {
seq: 1,
observed_at: new Date().toISOString(),
- controller: { version: '0.1.0', protocol: 1 as const, assignment_revision: 0 },
+ controller: { version: '0.1.0', protocol: 2 as const, assignment_revision: 0 },
machine: {
platform: { os: 'linux', arch: 'x64', os_version: '6.1' },
disk_free_bytes: 1,
diff --git a/console/packages/agent-controller/src/api.ts b/console/packages/agent-controller/src/api.ts
index fe3c74993..154bdb2c4 100644
--- a/console/packages/agent-controller/src/api.ts
+++ b/console/packages/agent-controller/src/api.ts
@@ -7,7 +7,6 @@ import {
type Assignment,
assignmentSchema,
type ControllerConnection,
- controllerBeatResponseSchema,
controllerConnectionResponseSchema,
credentialRotateResponseSchema,
type EnrollRequest,
@@ -32,6 +31,36 @@ export const CONTROLLER_CLIENT = 'switch-agent-controller';
export type Fetch = typeof fetch;
+/** What the controller needs of a WebSocket; Node's own satisfies it. */
+export type SocketLike = Pick<
+ WebSocket,
+ 'addEventListener' | 'removeEventListener' | 'send' | 'close' | 'readyState'
+>;
+
+/** Opens a WebSocket with request headers, as Node's does and a browser's does not. */
+export type OpenWebSocket = (url: string, headers: Record<string, string>) => SocketLike;
+
+/** Node's WebSocket (Node 22 and newer), which takes request headers. */
+export const nodeWebSocket: OpenWebSocket = (url, headers) => {
+ const Socket = (
+ globalThis as unknown as {
+ WebSocket?: new (url: string, init: { headers: Record<string, string> }) => WebSocket;
+ }
+ ).WebSocket;
+ if (!Socket)
+ throw new ConfigurationError(
+ `The agents controller needs Node 22 or newer to connect to Switch (this is ${process.version}, with no WebSocket).`
+ );
+ return new Socket(url, { headers });
+};
+
+/** The controller's socket, and how to say its token was refused. */
+export type OpenedSocket = {
+ socket: SocketLike;
+ /** The socket was refused with a 401 that is not a revocation: the token went stale. */
+ tokenRefused: () => void;
+};
+
/**
* A refusal from the server, read from its `{"error": {code, message,
* retryable}}` envelope. A response without one is still an error: it gets a
@@ -274,6 +303,7 @@ export class ControllerClient {
/** This controller's version, declared when the stream connection opens. */
version: string;
tokens: AccessTokens;
+ openWebSocket: OpenWebSocket;
}
) {}
@@ -407,48 +437,24 @@ export class ControllerClient {
}
/**
- * Proves the connection alive, and confirms how far each agent's host has
- * read.
+ * The socket that carries an open connection: its frames down, the
+ * controller's pongs (its beat) up. A refusal arrives on it as a `refused`
+ * frame, read by the caller.
*/
- async beat(
- connection: { connectionId: string; generation: number },
- cursors: Record<string, number>,
- signal: AbortSignal
- ): Promise<void> {
- const response = await this.request(`${this.streamPath}/connection/beat`, {
- method: 'POST',
- body: {
- connection_id: connection.connectionId,
- generation: connection.generation,
- cursors,
- },
- signal,
- });
- await parsed(response, controllerBeatResponseSchema);
- }
-
- /** Attaches the controller stream to an open connection; the caller reads the body. */
- async openEvents(
- connection: { connectionId: string; generation: number },
- signal: AbortSignal
- ): Promise<Response> {
+ async openSocket(connection: {
+ connectionId: string;
+ generation: number;
+ }): Promise<OpenedSocket> {
const query = new URLSearchParams({
connection_id: connection.connectionId,
generation: String(connection.generation),
});
- const response = await this.request(`${this.streamPath}/events?${query}`, {
- method: 'GET',
- headers: { Accept: 'text/event-stream' },
- signal,
+ const url = `${this.streamPath.replace(/^http/, 'ws')}/connection/ws?${query}`;
+ const token = await this.deps.tokens.get();
+ const socket = this.deps.openWebSocket(url, {
+ [PROTOCOL_HEADER]: String(PROTOCOL_VERSION),
+ Authorization: `Bearer ${token}`,
});
- if (!response.body)
- throw new ControllerApiError(
- response.status,
- 'invalid_response',
- 'The event stream opened with no body.',
- true,
- null
- );
- return response;
+ return { socket, tokenRefused: () => this.deps.tokens.invalidate(token) };
}
}
diff --git a/console/packages/agent-controller/src/controller.test.ts b/console/packages/agent-controller/src/controller.test.ts
index 9518d60b6..517e1e188 100644
--- a/console/packages/agent-controller/src/controller.test.ts
+++ b/console/packages/agent-controller/src/controller.test.ts
@@ -3,8 +3,9 @@ import { createServer, type Server } from 'node:net';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
-import { type AgentBridgeEvent, SwitchEventStream } from '@sandboxaq/switch-agent-runtime';
+import type { AgentBridgeEvent } from '@sandboxaq/switch-agent-runtime';
import { callOperation, SESSION_SELECTOR_HEADERS } from '@sandboxaq/switch-agent-runtime/hosted';
+import { openHubStream } from '@switch-console/agent-providers';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { type ControllerDeps, type ControllerExit, runController } from './controller';
import { ConfigurationError } from './errors';
@@ -69,6 +70,7 @@ function deps(server = core.url): ControllerDeps {
runtime: runtime.build,
locator: new FakeLocator(),
fetch,
+ openWebSocket: core.openWebSocket,
log: silentLogger,
dataDir: dir,
workspacesFor: () => join(dir, 'workspaces'),
@@ -83,7 +85,6 @@ function deps(server = core.url): ControllerDeps {
streamIdleMs: 2_000,
streamInitialBackoffMs: 10,
streamMaxBackoffMs: 50,
- relay: { heartbeatTtlMs: 6_000, heartbeatIntervalS: 2, sweepMs: 50, keepaliveMs: 15_000 },
eventBufferLimit: 100,
},
};
@@ -243,7 +244,7 @@ describe('runController', () => {
await expect(fetch(`${credentials.endpoint}/version`)).rejects.toThrow();
}, 20_000);
- it('serves an isolated agent its events as its own stream through the relay, and moves it in-process when that changes', async () => {
+ it('serves an isolated agent its events on the hub over the relay, and moves it in-process when that changes', async () => {
const isolatedAgent = agent(1);
isolatedAgent.definition.isolation = 'isolated';
core.setAssignment({ revision: 1, agents: [isolatedAgent] });
@@ -254,13 +255,12 @@ describe('runController', () => {
const events: AgentBridgeEvent[] = [];
const own = new AbortController();
watchers.push(own);
- new SwitchEventStream({
+ openHubStream(credentials.hub)({
creds: { agentId: 'agent-1', apiEndpoint: credentials.endpoint, token: credentials.token },
connectionId: 'isolated-host',
worker: null,
scope: 'all',
filter: 'addressed',
- spawnCapable: true,
rooms: [],
onEvent: (event) => void events.push(event),
onGap: () => {},
@@ -273,7 +273,7 @@ describe('runController', () => {
'the isolated host attached'
);
core.pushEvent('agent-1', 1, addressed(1));
- await waitFor(() => events.length === 1, 'the event on its own stream');
+ await waitFor(() => events.length === 1, 'the event on the hub');
await waitFor(() => store.cursors().get('agent-1') === 1, 'its cursor confirmed');
core.setAssignment({ revision: 2, agents: [agent(2)] });
@@ -328,6 +328,36 @@ describe('runController', () => {
expect(await running).toBe('stopped');
}, 20_000);
+ it('restarts an agent host whose credentials predate the hub, so it comes back on the hub', async () => {
+ const port = await freePort();
+ core.setAssignment({ revision: 1, agents: [agent(1)] });
+ store.saveRelayPort(port);
+ store.saveAssignment(core.assignment, '"1"', '2026-01-01T00:00:00Z');
+ store.recordApplied('agent-1', 1, '2026-01-01T00:00:00Z');
+ await runtime.launch('agent-1', runtimeTemplate(), {
+ isolation: 'isolated',
+ restart: false,
+ replaceIdentity: false,
+ clearTakenOver: false,
+ });
+ // As a controller from before the hub wrote them: the relay and a token, no hub.
+ await runtime.writeCredentials('agent-1', {
+ endpoint: `http://127.0.0.1:${port}`,
+ token: 'swlr_old',
+ hub: '',
+ });
+ runtime.calls.length = 0;
+ running = runController(deps(), stop.signal);
+ await waitFor(() => runtime.launches('agent-1').length === 1, 'a restart');
+ expect(runtime.launches('agent-1')[0]!.options.restart).toBe(true);
+ expect(runtime.credentials.get('agent-1')).toMatchObject({
+ endpoint: `http://127.0.0.1:${port}`,
+ hub: `ws://127.0.0.1:${port}/hub`,
+ });
+ stop.abort();
+ expect(await running).toBe('stopped');
+ });
+
it('points a watcher at the relay’s new port, and restarts it, when its old one is taken', async () => {
const port = await freePort();
const blocker = createServer();
@@ -346,6 +376,7 @@ describe('runController', () => {
await runtime.writeCredentials('agent-1', {
endpoint: `http://127.0.0.1:${port}`,
token: 'swlr_old',
+ hub: `ws://127.0.0.1:${port}/hub`,
});
runtime.calls.length = 0;
running = runController(deps(), stop.signal);
@@ -445,6 +476,7 @@ describe('runController', () => {
await runtime.writeCredentials('agent-1', {
endpoint: 'http://127.0.0.1:1',
token: 'swlr_k',
+ hub: 'ws://127.0.0.1:1/hub',
});
runtime.calls.length = 0;
await core.stop();
diff --git a/console/packages/agent-controller/src/controller.ts b/console/packages/agent-controller/src/controller.ts
index 75b9d788c..1c9e69bff 100644
--- a/console/packages/agent-controller/src/controller.ts
+++ b/console/packages/agent-controller/src/controller.ts
@@ -1,12 +1,13 @@
import type { OpenAgentStream } from '@switch-console/agent-providers';
import { AgentHub } from './agent-hub';
-import { AccessTokens, ControllerClient, type Fetch, isRevoked } from './api';
+import { AccessTokens, ControllerClient, type Fetch, isRevoked, type OpenWebSocket } from './api';
import { ConfigurationError } from './errors';
+import { HubSocket } from './hub-socket';
import { errorMessage, type Logger } from './log';
import { processPendingOperations } from './operations';
import { isSafeSegment } from './paths';
import { definitionProblem, reconcile, type ReconcileDeps, startAgent } from './reconcile';
-import { DEFAULT_RELAY_TIMING, LocalRelay, RELAY_TOKEN_PREFIX, type RelayTiming } from './relay';
+import { LocalRelay, RELAY_TOKEN_PREFIX } from './relay';
import { UpstreamForwarder } from './relay-forward';
import type { AgentRuntime } from './runtime';
import {
@@ -41,7 +42,6 @@ export type ControllerTiming = {
* every agent on the controller is offline while the stream is down.
*/
streamMaxBackoffMs: number;
- relay: RelayTiming;
/** The most events held per agent while its agent host is not taking them. */
eventBufferLimit: number;
};
@@ -51,11 +51,10 @@ export const DEFAULT_TIMING: ControllerTiming = {
statusPollMs: 5_000,
statusMinGapMs: 1_000,
defaultReportWithinS: 60,
- // Three missed 15 s keepalives.
- streamIdleMs: 45_000,
+ // Five missed pings, at the 2 s Switch sends them.
+ streamIdleMs: 10_000,
streamInitialBackoffMs: 1_000,
streamMaxBackoffMs: 8_000,
- relay: DEFAULT_RELAY_TIMING,
eventBufferLimit: 5_000,
};
@@ -69,6 +68,8 @@ export type ControllerDeps = {
runtime: (openStream: (agentId: string) => OpenAgentStream, workspaces: string) => AgentRuntime;
locator: ProviderLocator;
fetch: Fetch;
+ /** Opens the controller's connection to Switch. */
+ openWebSocket: OpenWebSocket;
log: Logger;
/** Where disk space is measured and provider checks run. */
dataDir: string;
@@ -88,10 +89,11 @@ export type ControllerDeps = {
export type ControllerExit = 'stopped' | 'revoked' | 'taken_over';
/**
- * Makes an agent's credentials file name the relay and a token it accepts:
- * the token already in the file when it names the relay as it listens now, a
- * new one otherwise. Resolves true when the file was (re)written, which a running
- * agent host only reads when it starts.
+ * Makes an agent's credentials file name the relay, its hub, and a token the
+ * relay accepts: the token already in the file when it names both as they
+ * listen now, a new one otherwise. Resolves true when the file was (re)written,
+ * which a running agent host only reads when it starts, so it is restarted:
+ * that is also what moves an agent host from before the hub onto it.
*/
export async function ensureRelayCredentials(
agentId: string,
@@ -99,16 +101,22 @@ export async function ensureRelayCredentials(
): Promise<boolean> {
const current = await deps.runtime.readCredentials(agentId);
const endpoint = deps.relay.endpoint;
- if (current && current.endpoint === endpoint && current.token.startsWith(RELAY_TOKEN_PREFIX)) {
+ const hub = deps.relay.hubUrl;
+ if (
+ current &&
+ current.endpoint === endpoint &&
+ current.hub === hub &&
+ current.token.startsWith(RELAY_TOKEN_PREFIX)
+ ) {
if (!deps.relay.isRegistered(agentId, current.token))
deps.relay.register(agentId, current.token);
return false;
}
const token = deps.relay.mint(agentId);
- await deps.runtime.writeCredentials(agentId, { endpoint, token });
+ await deps.runtime.writeCredentials(agentId, { endpoint, token, hub });
deps.log.info('Wrote the agent’s relay credentials', {
agentId,
- why: current ? 'they named another endpoint or a Switch key' : 'it had none',
+ why: current ? 'they named another endpoint, hub or a Switch key' : 'it had none',
});
return true;
}
@@ -195,6 +203,7 @@ export async function runController(
controllerId: identity.controllerId,
version: deps.version,
tokens,
+ openWebSocket: deps.openWebSocket,
});
const queue = new SerialQueue();
const cached = store.cachedAssignment();
@@ -224,14 +233,8 @@ export async function runController(
);
const relay = new LocalRelay({
log,
- version: deps.version,
- now: deps.now,
- timing: timing.relay,
- bufferLimit: timing.eventBufferLimit,
- onCursor: (agentId, cursor) =>
- store.saveCursor(agentId, cursor, new Date(deps.now()).toISOString()),
- onChange: () => reporter?.request(),
- sharedRoomFor: (agentId, sessionId) => hub.roomFor(agentId, sessionId),
+ roomFor: (agentId, sessionId) => hub.roomFor(agentId, sessionId),
+ hub: new HubSocket({ hub, log }),
forwarder: new UpstreamForwarder({
server: identity.server,
auth: {
@@ -242,31 +245,19 @@ export async function runController(
log,
}),
});
- /**
- * The agents whose host runs in a process of its own: their events go
- * through the relay, as their own stream. Every other agent's go to its host
- * in this process, through the hub.
- */
- const isolated = new Set<string>();
+ /** The agents whose saved cursor the hub has been given. */
const placed = new Set<string>();
- const delivery = (agentId: string) => (isolated.has(agentId) ? relay : hub);
- /** Sends each agent's events the way its definition asks, moving its cursor when that changes. */
+ /** Resumes each newly assigned agent where this controller last confirmed it. */
const placeAgents = (held: Assignment) => {
const cursors = store.cursors();
for (const entry of held.agents) {
const agentId = entry.agent_id;
- const wanted = entry.definition.isolation === 'isolated';
- if (placed.has(agentId) && wanted === isolated.has(agentId)) continue;
+ if (placed.has(agentId)) continue;
placed.add(agentId);
- if (wanted) {
- isolated.add(agentId);
- hub.forget(agentId);
- } else {
- isolated.delete(agentId);
- relay.forgetStreams(agentId);
- }
const cursor = cursors.get(agentId);
- if (cursor !== undefined) delivery(agentId).setCursor(agentId, cursor);
+ // Core may have attached the agent before the assignment that names it
+ // was read: where it attached it is then where the hub already stands.
+ if (cursor !== undefined && !hub.attachment(agentId)) hub.setCursor(agentId, cursor);
}
};
if (assignment) placeAgents(assignment);
@@ -326,7 +317,7 @@ export async function runController(
store,
runtime,
providers,
- attached: (agentId) => delivery(agentId).attached(agentId),
+ attached: (agentId) => hub.attached(agentId),
dataDir: deps.dataDir,
workspacesDir: workspaces,
version: deps.version,
@@ -455,33 +446,29 @@ export async function runController(
case 'evicted':
return;
case 'agent.event':
- delivery(frame.data.agent_id).ingest(frame.data);
+ hub.ingest(frame.data);
return;
case 'agent.gap':
log.warn('Switch reports events an agent missed', {
agentId: frame.data.agent_id,
reason: frame.data.reason,
});
- delivery(frame.data.agent_id).gap(frame.data);
+ hub.gap(frame.data);
return;
case 'agent.session_command':
- delivery(frame.data.agent_id).sessionCommand(frame.data);
+ hub.sessionCommand(frame.data);
return;
case 'agent.approval_outcome':
- delivery(frame.data.agent_id).approvalOutcome(frame.data);
+ hub.approvalOutcome(frame.data);
return;
case 'agent.attached':
- delivery(frame.data.agent_id).attach(
- frame.data.agent_id,
- frame.data.from_seq,
- frame.data.rooms
- );
+ hub.attach(frame.data.agent_id, frame.data.from_seq, frame.data.rooms);
return;
case 'agent.detached':
- delivery(frame.data.agent_id).detach(frame.data.agent_id, frame.data.reason);
+ hub.detach(frame.data.agent_id, frame.data.reason);
return;
case 'agent.rooms':
- delivery(frame.data.agent_id).setRooms(frame.data.agent_id, frame.data.rooms);
+ hub.setRooms(frame.data.agent_id, frame.data.rooms);
return;
case 'assignment.changed':
void sync(`assignment.changed to revision ${frame.data.revision}`);
@@ -522,9 +509,9 @@ export async function runController(
cursors: () => {
const cursors: Record<string, AgentCursor> = {};
for (const entry of assignment?.agents ?? []) cursors[entry.agent_id] = 'head';
- return { ...cursors, ...hub.cursors(), ...relay.cursors() };
+ return { ...cursors, ...hub.cursors() };
},
- confirmed: () => ({ ...hub.cursors(), ...relay.cursors() }),
+ confirmed: () => hub.cursors(),
onOpened: (connection) =>
log.info('Switch has these agents bound to this controller', {
agents: connection.agents,
@@ -532,13 +519,11 @@ export async function runController(
onConnected: () => {
log.info('Connected to Switch');
hub.streamAttached();
- relay.streamAttached();
void sync('connected');
void runOperations();
},
onDisconnected: () => {
hub.setUpstream(false);
- relay.setUpstream(false);
},
onFrame,
signal: stop.signal,
diff --git a/console/packages/agent-controller/src/detached-runtime.ts b/console/packages/agent-controller/src/detached-runtime.ts
index 4d2861109..c70b70954 100644
--- a/console/packages/agent-controller/src/detached-runtime.ts
+++ b/console/packages/agent-controller/src/detached-runtime.ts
@@ -34,11 +34,12 @@ const LAUNCH_TIMEOUT_MS = 60_000;
* runs an agent on an SSH host: a detached agent host from the agent-providers
* shared-host bundle, in a state root of its own, driven through the files it
* reads (`watch.json`, `config.json`) and observed through the files it
- * writes (`health.json`, `supervisor/failure.json`). It reaches Switch, its
- * event stream included, through the controller's relay.
+ * writes (`health.json`, `supervisor/failure.json`). It makes its calls to
+ * Switch through the controller's relay, and hears its events on the
+ * controller's hub, over a WebSocket on the relay's port.
*
* Being its own process, it is not stopped when the controller exits: it
- * keeps retrying the relay, and picks up again when the controller is back.
+ * keeps retrying the hub, and picks up again when the controller is back.
* A restart is a stop and a start: the agent host is turned off, waited out,
* and launched again from the new template.
*/
diff --git a/console/packages/agent-controller/src/hub-socket.ts b/console/packages/agent-controller/src/hub-socket.ts
new file mode 100644
index 000000000..113440703
--- /dev/null
+++ b/console/packages/agent-controller/src/hub-socket.ts
@@ -0,0 +1,194 @@
+import { randomUUID } from 'node:crypto';
+import type { IncomingMessage } from 'node:http';
+import type { Duplex } from 'node:stream';
+import type { SwitchEventStreamDeps } from '@sandboxaq/switch-agent-runtime';
+import {
+ type AgentEventStream,
+ HUB_CLOSE,
+ HUB_PROTOCOL,
+ type HubClientMessage,
+ hubClientMessageSchema,
+ type HubServerMessage,
+} from '@switch-console/agent-providers';
+import { type RawData, type WebSocket, WebSocketServer } from 'ws';
+import type { AgentHub } from './agent-hub';
+import { errorMessage, type Logger } from './log';
+
+/** How long an agent host has, once its socket is open, to say hello. */
+const HELLO_TIMEOUT_MS = 10_000;
+/** How long closing waits for the agent hosts to see the close before cutting them. */
+const CLOSE_GRACE_MS = 500;
+
+type Served = { socket: WebSocket; stream: AgentEventStream | null };
+
+/**
+ * The hub over a WebSocket, for agent hosts that run in a process of their
+ * own: each is an agent host on the hub exactly as one in this process is
+ * (`AgentHub.open`), with every callback the hub makes sent to it as a request
+ * and resolved when it answers `done`. So an event counts as taken once the
+ * agent host has handled it, wherever it runs.
+ *
+ * Served on the relay's loopback port at `/hub`; the relay authenticates the
+ * upgrade with the agent's relay token and hands it here with the agent named.
+ */
+export class HubSocket {
+ private readonly server = new WebSocketServer({ noServer: true });
+ /** The agent host connected for each agent; a newer one takes the agent over. */
+ private readonly served = new Map<string, Served>();
+
+ constructor(
+ private readonly deps: {
+ hub: Pick<AgentHub, 'open'>;
+ log: Logger;
+ }
+ ) {}
+
+ accept(agentId: string, req: IncomingMessage, socket: Duplex, head: Buffer): void {
+ this.server.handleUpgrade(req, socket, head, (ws) => this.serve(agentId, ws));
+ }
+
+ /**
+ * Closes every agent host's socket as a restart (1012), so each reconnects
+ * soon after the controller is back, and gives them a moment to read it.
+ */
+ async close(): Promise<void> {
+ const closing = [...this.server.clients].map(
+ (ws) =>
+ new Promise<void>((resolve) => {
+ if (ws.readyState === ws.CLOSED) return resolve();
+ ws.once('close', () => resolve());
+ ws.close(HUB_CLOSE.restarting, 'the agents controller is restarting');
+ })
+ );
+ await Promise.race([
+ Promise.all(closing),
+ new Promise((resolve) => setTimeout(resolve, CLOSE_GRACE_MS).unref()),
+ ]);
+ for (const ws of this.server.clients) ws.terminate();
+ this.served.clear();
+ }
+
+ private serve(agentId: string, ws: WebSocket): void {
+ const { log } = this.deps;
+ const lifetime = new AbortController();
+ const waiting = new Map<number, { resolve: () => void; reject: (error: Error) => void }>();
+ let nextId = 0;
+ const entry: Served = { socket: ws, stream: null };
+
+ const send = (message: HubServerMessage) => {
+ if (ws.readyState === ws.OPEN) ws.send(JSON.stringify(message));
+ };
+ const request = (message: (id: number) => HubServerMessage) =>
+ new Promise<void>((resolve, reject) => {
+ if (lifetime.signal.aborted)
+ return reject(new Error('The agent host is no longer connected.'));
+ const id = ++nextId;
+ waiting.set(id, { resolve, reject });
+ send(message(id));
+ });
+
+ const helloTimer = setTimeout(
+ () => ws.close(HUB_CLOSE.refused, 'no hello within 10 s'),
+ HELLO_TIMEOUT_MS
+ );
+ helloTimer.unref();
+
+ const hello = (message: HubClientMessage & { type: 'hello' }) => {
+ clearTimeout(helloTimer);
+ if (message.protocol !== HUB_PROTOCOL) {
+ ws.close(
+ HUB_CLOSE.refused,
+ `hub protocol ${message.protocol} is not ${HUB_PROTOCOL}; restart the agent host`
+ );
+ return;
+ }
+ const previous = this.served.get(agentId);
+ if (previous && previous.socket !== ws)
+ previous.socket.close(
+ HUB_CLOSE.takenOver,
+ 'another agent host for this agent connected to the hub'
+ );
+ this.served.set(agentId, entry);
+ const streamDeps: SwitchEventStreamDeps = {
+ creds: { agentId, apiEndpoint: '', token: '' },
+ connectionId: randomUUID(),
+ scope: 'all',
+ filter: 'addressed',
+ rooms: [],
+ ...(message.startCursor === null ? {} : { startCursor: message.startCursor }),
+ worker: null,
+ log,
+ signal: lifetime.signal,
+ onEvent: (event) => request((id) => ({ type: 'event', id, event })),
+ onGap: (gap) => request((id) => ({ type: 'gap', id, gap })),
+ onSessionCommand: (command) => request((id) => ({ type: 'session_command', id, command })),
+ onApprovalOutcome: (outcome) =>
+ request((id) => ({ type: 'approval_outcome', id, outcome })),
+ onConnected: () => send({ type: 'connected' }),
+ onDisconnected: ({ error }) => send({ type: 'disconnected', error }),
+ onEvicted: () => {},
+ };
+ const stream = this.deps.hub.open(agentId, streamDeps);
+ entry.stream = stream;
+ if (Object.keys(message.placements).length)
+ stream.replacePlacements(message.placements).catch((error: unknown) =>
+ log.warn('Refused the placements an agent host reconnected with', {
+ agentId,
+ error: errorMessage(error),
+ })
+ );
+ stream.start();
+ log.info('An isolated agent host connected to the hub', {
+ agentId,
+ startCursor: message.startCursor,
+ });
+ };
+
+ const receive = (raw: RawData) => {
+ let message: HubClientMessage;
+ try {
+ message = hubClientMessageSchema.parse(JSON.parse(raw.toString()));
+ } catch (error) {
+ log.warn('Closed a hub connection that sent something the hub does not read', {
+ agentId,
+ error: errorMessage(error),
+ });
+ ws.close(HUB_CLOSE.refused, 'unreadable message');
+ return;
+ }
+ if (message.type === 'hello') {
+ if (entry.stream) ws.close(HUB_CLOSE.refused, 'a second hello');
+ else hello(message);
+ return;
+ }
+ if (!entry.stream) {
+ ws.close(HUB_CLOSE.refused, 'hello first');
+ return;
+ }
+ if (message.type === 'done') {
+ const call = waiting.get(message.id);
+ waiting.delete(message.id);
+ if (message.error === undefined) call?.resolve();
+ else call?.reject(new Error(message.error));
+ return;
+ }
+ entry.stream.replacePlacements(message.placements).then(
+ () => send({ type: 'result', id: message.id }),
+ (error: unknown) => send({ type: 'result', id: message.id, error: errorMessage(error) })
+ );
+ };
+
+ ws.on('message', receive);
+ ws.on('close', () => {
+ clearTimeout(helloTimer);
+ lifetime.abort();
+ for (const call of waiting.values())
+ call.reject(new Error('The agent host disconnected from the hub.'));
+ waiting.clear();
+ if (this.served.get(agentId) === entry) this.served.delete(agentId);
+ });
+ ws.on('error', (error) =>
+ log.warn('A hub connection failed', { agentId, error: errorMessage(error) })
+ );
+ }
+}
diff --git a/console/packages/agent-controller/src/main.ts b/console/packages/agent-controller/src/main.ts
index 21d23915a..b41b0d9e0 100644
--- a/console/packages/agent-controller/src/main.ts
+++ b/console/packages/agent-controller/src/main.ts
@@ -1,7 +1,7 @@
import { hostname } from 'node:os';
import { parseArgs } from 'node:util';
import packageJson from '../package.json' with { type: 'json' };
-import { ControllerApiError, enroll, normalizeServerUrl } from './api';
+import { ControllerApiError, enroll, normalizeServerUrl, nodeWebSocket } from './api';
import { DEFAULT_TIMING, runController } from './controller';
import { DetachedRuntime } from './detached-runtime';
import { ConfigurationError, UsageError } from './errors';
@@ -212,6 +212,7 @@ async function runCommand(args: string[]): Promise<number> {
),
locator: new PathProviderLocator(process.env.PATH),
fetch,
+ openWebSocket: nodeWebSocket,
log,
dataDir,
workspacesFor: serverWorkspacesDir,
diff --git a/console/packages/agent-controller/src/reconcile.test.ts b/console/packages/agent-controller/src/reconcile.test.ts
index c4209830e..d6e33c138 100644
--- a/console/packages/agent-controller/src/reconcile.test.ts
+++ b/console/packages/agent-controller/src/reconcile.test.ts
@@ -55,7 +55,11 @@ function deps(): ReconcileDeps {
ensureCredentials: async (agentId) => {
if (runtime.credentials.get(agentId)?.endpoint === endpoint) return false;
minted.push(agentId);
- await runtime.writeCredentials(agentId, { endpoint, token: `swlr_${minted.length}` });
+ await runtime.writeCredentials(agentId, {
+ endpoint,
+ token: `swlr_${minted.length}`,
+ hub: `${endpoint.replace('http', 'ws')}/hub`,
+ });
return true;
},
forgetAgent: (agentId) => void forgotten.push(agentId),
@@ -85,7 +89,11 @@ describe('reconcile', () => {
it('starts a running agent: points it at the relay, launches its watcher', async () => {
await reconcile(assignment(agent()), deps());
expect(minted).toEqual(['agent-1']);
- expect(runtime.credentials.get('agent-1')).toEqual({ endpoint: RELAY, token: 'swlr_1' });
+ expect(runtime.credentials.get('agent-1')).toEqual({
+ endpoint: RELAY,
+ token: 'swlr_1',
+ hub: `${RELAY.replace('http', 'ws')}/hub`,
+ });
const [launch] = runtime.launches();
expect(launch!.options).toEqual({
isolation: 'shared',
diff --git a/console/packages/agent-controller/src/relay.test.ts b/console/packages/agent-controller/src/relay.test.ts
index 423c0bd5d..37d724fb3 100644
--- a/console/packages/agent-controller/src/relay.test.ts
+++ b/console/packages/agent-controller/src/relay.test.ts
@@ -3,15 +3,12 @@ import { request } from 'node:http';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
-import {
- type AgentBridgeEvent,
- type ApprovalOutcome,
- CONTRACTS,
- type Eviction,
- PlacementsRefusedError,
- type SessionCommand,
- SwitchEventStream,
- type SwitchEventStreamDeps,
+import type {
+ AgentBridgeEvent,
+ ApprovalOutcome,
+ Eviction,
+ SessionCommand,
+ SwitchEventStreamDeps,
} from '@sandboxaq/switch-agent-runtime';
import {
callOperation,
@@ -19,10 +16,14 @@ import {
fetchMediaToFile,
SESSION_SELECTOR_HEADERS,
} from '@sandboxaq/switch-agent-runtime/hosted';
+import { type AgentEventStream, openHubStream } from '@switch-console/agent-providers';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
+import WebSocket from 'ws';
+import { AgentHub } from './agent-hub';
import { AccessTokens } from './api';
+import { HubSocket } from './hub-socket';
import { silentLogger } from './log';
-import { LocalRelay, RELAY_AGENT_PROTOCOL } from './relay';
+import { LocalRelay } from './relay';
import { UpstreamForwarder } from './relay-forward';
import type { AgentAssignment } from './schemas';
import { FakeCore } from './testing/fake-core';
@@ -45,7 +46,7 @@ function assigned(agentId: string): AgentAssignment {
instructions: '',
auto_approve: false,
directory: null,
- isolation: 'shared',
+ isolation: 'isolated',
},
};
}
@@ -68,7 +69,6 @@ function message(sequence: number, addressed: boolean, room = 'room-a') {
timestamp: sequence,
},
sequence,
- ...(addressed ? { missed: { count: sequence, reason: null } } : {}),
},
};
}
@@ -82,16 +82,14 @@ async function waitFor(condition: () => boolean, what: string, timeoutMs = 15_00
}
let core: FakeCore;
+let hub: AgentHub;
let relay: LocalRelay;
let token: string;
let revokedCalls: number;
let cursorsSaved: [string, number][];
const stops: (() => void)[] = [];
-beforeEach(async () => {
- core = new FakeCore();
- await core.start();
- core.setAssignment({ revision: 1, agents: [assigned(AGENT), assigned('agent-2')] });
+function newRelay(): LocalRelay {
const tokens = new AccessTokens({
fetch,
server: core.url,
@@ -100,11 +98,10 @@ beforeEach(async () => {
now: Date.now,
log: silentLogger,
});
- revokedCalls = 0;
- cursorsSaved = [];
- relay = new LocalRelay({
+ return new LocalRelay({
log: silentLogger,
- version: '0.1.0',
+ roomFor: (agentId, sessionId) => hub.roomFor(agentId, sessionId),
+ hub: new HubSocket({ hub, log: silentLogger }),
forwarder: new UpstreamForwarder({
server: core.url,
auth: {
@@ -114,18 +111,26 @@ beforeEach(async () => {
},
log: silentLogger,
}),
+ });
+}
+
+beforeEach(async () => {
+ core = new FakeCore();
+ await core.start();
+ core.setAssignment({ revision: 1, agents: [assigned(AGENT), assigned('agent-2')] });
+ revokedCalls = 0;
+ cursorsSaved = [];
+ hub = new AgentHub({
+ log: silentLogger,
+ bufferLimit: 100,
onCursor: (agentId, cursor) => cursorsSaved.push([agentId, cursor]),
onChange: () => {},
- sharedRoomFor: (agentId, sessionId) =>
- agentId === 'agent-2' && sessionId === 'session-7' ? 'room-shared' : null,
- now: Date.now,
- timing: { heartbeatTtlMs: 6_000, heartbeatIntervalS: 2, sweepMs: 50, keepaliveMs: 15_000 },
- bufferLimit: 100,
});
+ relay = newRelay();
await relay.start(null);
token = relay.mint(AGENT);
- relay.streamAttached();
- relay.attach(AGENT, 0, ['room-a', 'room-b']);
+ hub.streamAttached();
+ hub.attach(AGENT, 0, ['room-a', 'room-b']);
relay.setReady();
});
@@ -135,37 +140,34 @@ afterEach(async () => {
await core.stop();
});
-type Watcher = ReturnType<typeof watch>;
+type Host = ReturnType<typeof host>;
-/** The runtime's own protocol client, pointed at the relay as a watcher points it. */
-function watch(overrides: Partial<SwitchEventStreamDeps> = {}) {
+/** An agent host in a process of its own, as the shared daemon opens its stream: on the hub. */
+function host(overrides: Partial<SwitchEventStreamDeps> = {}, as = token) {
const seen = {
events: [] as AgentBridgeEvent[],
gaps: [] as Parameters<SwitchEventStreamDeps['onGap']>[0][],
evictions: [] as Eviction[],
commands: [] as SessionCommand[],
outcomes: [] as ApprovalOutcome[],
- released: [] as { roomId: string; sessionId: string | null }[],
- rooms: [] as string[][],
connected: 0,
+ disconnected: [] as string[],
};
const controller = new AbortController();
- const stream = new SwitchEventStream({
- creds: { agentId: AGENT, apiEndpoint: relay.endpoint, token },
- connectionId: 'watcher-connection',
+ const stream: AgentEventStream = openHubStream(relay.hubUrl)({
+ creds: { agentId: AGENT, apiEndpoint: relay.endpoint, token: as },
+ connectionId: 'host-connection',
worker: null,
scope: 'all',
filter: 'addressed',
- spawnCapable: true,
rooms: [],
onEvent: (event) => void seen.events.push(event),
onGap: (gap) => void seen.gaps.push(gap),
onEvicted: (eviction) => void seen.evictions.push(eviction),
onSessionCommand: (command) => void seen.commands.push(command),
onApprovalOutcome: (outcome) => void seen.outcomes.push(outcome),
- onRoomReleased: (released) => void seen.released.push(released),
- onRooms: (rooms) => void seen.rooms.push(rooms),
onConnected: () => void seen.connected++,
+ onDisconnected: ({ error }) => void seen.disconnected.push(error),
log: quiet,
signal: controller.signal,
...overrides,
@@ -176,18 +178,14 @@ function watch(overrides: Partial<SwitchEventStreamDeps> = {}) {
return { stream, seen, stop };
}
-async function connected(watcher: Watcher, times = 1): Promise<void> {
- await waitFor(() => watcher.seen.connected >= times, 'the stream to connect');
+async function connected(agentHost: Host, times = 1): Promise<void> {
+ await waitFor(() => agentHost.seen.connected >= times, 'the hub to say connected');
}
-function caller(
- watcher: Watcher,
- overrides: Partial<CallerContext> = {},
- sessionId: string | null = 'session-1'
-): CallerContext {
+function caller(overrides: Partial<CallerContext> = {}, sessionId: string | null = 'session-1') {
return {
identity: { endpoint: relay.endpoint, agentId: AGENT, token },
- connectionId: 'watcher-connection',
+ connectionId: 'host-connection',
selector: sessionId
? {
[SESSION_SELECTOR_HEADERS.sessionId]: sessionId,
@@ -198,9 +196,9 @@ function caller(
room: null,
mediaDir: '/nonexistent',
cwd: '/',
- deadConnection: (operation) => `dead: ${operation}`,
+ deadConnection: (operation: string) => `dead: ${operation}`,
...overrides,
- };
+ } satisfies CallerContext;
}
async function post(path: string, body: unknown, bearer = token): Promise<Response> {
@@ -211,179 +209,86 @@ async function post(path: string, body: unknown, bearer = token): Promise<Respon
});
}
-describe('the relay as the agent protocol, read by the real SwitchEventStream', () => {
- it('serves switch-core’s revisions up to the socket, which the runtime still accepts', () => {
- const core = CONTRACTS['agent-protocol']['switch-core'];
- const runtime = CONTRACTS['agent-protocol']['agent-runtime'];
- // The relay serves the event stream, not the socket of agent-protocol 8.
- expect(RELAY_AGENT_PROTOCOL).toEqual({ speaks: 7, accepts: core.accepts });
- expect(runtime.accepts).toBeLessThanOrEqual(RELAY_AGENT_PROTOCOL.speaks);
- expect(RELAY_AGENT_PROTOCOL.accepts).toBeLessThanOrEqual(runtime.speaks);
- });
-
- it('connects, then delivers addressed events in order, with their ids and missed counts', async () => {
- const watcher = watch();
- await connected(watcher);
- // Switch keeps no record of its sessions, so the agent host says when it starts one.
- expect(watcher.stream.announcesSessionStarts).toBe(true);
- relay.ingest(message(1, true));
- relay.ingest(message(2, false));
- relay.ingest({
- agent_id: AGENT,
- seq: 3,
- event: {
- type: 'command',
- room_id: 'room-a',
- sequence: 3,
- payload: { command: 'reset', args: '', user_id: 'u', user_name: 'U' },
- },
+/** The status a WebSocket upgrade to the hub is answered with, as a raw client sees it. */
+function upgradeStatus(bearer: string, path = '/hub'): Promise<number> {
+ return new Promise((resolve, reject) => {
+ const socket = new WebSocket(`${relay.hubUrl.replace('/hub', path)}`, {
+ headers: { Authorization: `Bearer ${bearer}` },
});
- relay.ingest(message(4, true, 'room-b'));
- await waitFor(() => watcher.seen.events.length === 2, 'two addressed events');
- expect(watcher.seen.events.map((event) => event.sequence)).toEqual([1, 4]);
- expect(watcher.seen.events[0]).toMatchObject({
- type: 'message',
- room_id: 'room-a',
- bridge_id: null,
- missed: { count: 1, reason: null },
+ socket.on('unexpected-response', (_req, res) => {
+ resolve(res.statusCode ?? 0);
+ socket.terminate();
});
- expect(watcher.seen.events[0]).not.toHaveProperty('agent_id');
- expect(watcher.stream.position).toBe(4);
- });
-
- it('delivers every event to a connection filtering nothing', async () => {
- const watcher = watch({ filter: 'all', connectionId: 'everything' });
- await connected(watcher);
- relay.ingest(message(1, true));
- relay.ingest(message(2, false));
- await waitFor(() => watcher.seen.events.length === 2, 'both events');
- });
-
- it('resumes from Last-Event-ID without replaying what was read, and opens at head', async () => {
- const first = watch();
- await connected(first);
- relay.ingest(message(1, true));
- relay.ingest(message(2, true));
- await waitFor(() => first.seen.events.length === 2, 'the first two events');
- first.stop();
- relay.ingest(message(3, true));
- relay.ingest(message(4, true));
- const resumed = watch({ startCursor: 2 });
- await waitFor(() => resumed.seen.events.length === 2, 'the events missed while away');
- expect(resumed.seen.events.map((event) => event.sequence)).toEqual([3, 4]);
- const fresh = watch({ connectionId: 'fresh' });
- await connected(fresh);
- relay.ingest(message(5, true));
- await waitFor(() => fresh.seen.events.length === 1, 'only what came after opening');
- expect(fresh.seen.events[0]!.sequence).toBe(5);
+ socket.on('open', () => {
+ resolve(101);
+ socket.terminate();
+ });
+ socket.on('error', reject);
});
+}
- it('confirms the cursor the watcher beats, and only that', async () => {
- const watcher = watch();
- await connected(watcher);
- relay.ingest(message(1, true));
- relay.ingest(message(2, true));
- await waitFor(() => watcher.seen.events.length === 2, 'the events');
- await waitFor(() => relay.cursors()[AGENT] === 2, 'the beat confirming sequence 2', 5000);
- expect(cursorsSaved.at(-1)).toEqual([AGENT, 2]);
- expect(relay.attached(AGENT)).toBe(true);
+describe('the hub, over the relay’s WebSocket, read by openHubStream', () => {
+ it('serves the hub on the relay’s loopback port', () => {
+ expect(relay.hubUrl).toBe(`${relay.endpoint.replace('http', 'ws')}/hub`);
});
- it('answers the heartbeat as Switch does: 200, 404 unknown, 409 taken_over or no_stream', async () => {
- const watcher = watch();
- await connected(watcher);
- const unknown = await post(`/agents/${AGENT}/connection/beat`, {
- connection_id: 'nobody',
- cursor: 0,
- generation: 1,
- });
- expect(unknown.status).toBe(404);
- expect(await unknown.text()).toContain('nobody is not open');
- const stale = await post(`/agents/${AGENT}/connection/beat`, {
- connection_id: 'watcher-connection',
- cursor: 0,
- generation: 999_999,
- });
- expect(stale.status).toBe(409);
- expect((await stale.json()).detail.code).toBe('taken_over');
- const unfenced = await post(`/agents/${AGENT}/connection/beat`, {
- connection_id: 'watcher-connection',
- cursor: 0,
- });
- expect((await unfenced.json()).detail.code).toBe('unfenced');
- watcher.stop();
- await delay(50);
- const detached = await post(`/agents/${AGENT}/connection/beat`, {
- connection_id: 'watcher-connection',
- cursor: 0,
- generation: null,
+ it('connects, then delivers addressed events in order, each confirmed once handled', async () => {
+ let release: () => void = () => {};
+ const held = new Promise<void>((resolve) => (release = resolve));
+ const seen: number[] = [];
+ const agentHost = host({
+ onEvent: async (event) => {
+ seen.push(event.sequence!);
+ if (event.sequence === 4) await held;
+ },
});
- expect(detached.status).toBe(409);
- });
-
- it('evicts the stream another client takes over, and the runtime stands down', async () => {
- const first = watch();
- await connected(first);
- const second = watch();
- await connected(second);
- await waitFor(() => first.seen.evictions.length === 1, 'the first to be evicted');
- expect(first.seen.evictions[0]!.code).toBe('taken_over');
- relay.ingest(message(1, true));
- await waitFor(() => second.seen.events.length === 1, 'the event on the new holder');
- expect(first.seen.events).toEqual([]);
+ await connected(agentHost);
+ hub.ingest(message(1, true));
+ hub.ingest(message(2, false));
+ hub.ingest(message(4, true));
+ hub.ingest(message(5, true));
+ await waitFor(() => seen.length === 2, 'the events up to the held one');
+ // The event being handled is not confirmed, and the next is not sent.
+ expect(seen).toEqual([1, 4]);
+ expect(hub.cursors()[AGENT]).toBe(2);
+ release();
+ await waitFor(() => seen.length === 3, 'the event after the held one');
+ expect(seen).toEqual([1, 4, 5]);
+ await waitFor(() => hub.cursors()[AGENT] === 5, 'the cursor to move past it');
+ expect(cursorsSaved.at(-1)).toEqual([AGENT, 5]);
});
- it('closes a connection whose heartbeat lapses, with an evicted frame', async () => {
- await relay.close();
- relay = new LocalRelay({
- log: silentLogger,
- version: '0.1.0',
- forwarder: { forward: async () => {} },
- onCursor: () => {},
- onChange: () => {},
- sharedRoomFor: () => null,
- now: Date.now,
- timing: { heartbeatTtlMs: 300, heartbeatIntervalS: 2, sweepMs: 20, keepaliveMs: 15_000 },
- bufferLimit: 100,
- });
- await relay.start(null);
- token = relay.mint(AGENT);
- relay.setReady();
- const response = await fetch(
- `${relay.endpoint}/agents/${AGENT}/events?connection_id=raw&scope=all&protocol=6`,
- { headers: { Authorization: `Bearer ${token}`, Accept: 'text/event-stream' } }
- );
- const text = await response.text();
- expect(text).toContain('event: connection_state');
- expect(text).toContain('event: evicted');
- expect(text).toContain('"code":"heartbeat_lapsed"');
+ it('tells the agent host when the controller loses and regains Switch', async () => {
+ const agentHost = host();
+ await connected(agentHost);
+ hub.setUpstream(false);
+ await waitFor(() => agentHost.seen.disconnected.length === 1, 'disconnected');
+ expect(agentHost.seen.disconnected[0]).toMatch(/reconnecting to Switch/);
+ hub.streamAttached();
+ hub.attach(AGENT, 0, ['room-a', 'room-b']);
+ await connected(agentHost, 2);
});
it('tracks placements, names the session’s room on its calls, and routes room controls to it', async () => {
- const watcher = watch();
- await connected(watcher);
- await watcher.stream.replacePlacements({ 'session-1': 'room-a' });
- await waitFor(
- () => watcher.seen.rooms.some((rooms) => rooms.includes('room-a')),
- 'the subscription change naming room-a'
- );
-
- const placed = await callOperation(caller(watcher), 'post_message', { body: 'hi' });
+ const agentHost = host();
+ await connected(agentHost);
+ await agentHost.stream.replacePlacements({ 'session-1': 'room-a' });
+
+ const placed = await callOperation(caller(), 'post_message', { body: 'hi' });
expect(placed.isError).toBeFalsy();
expect(placed.structuredContent).toMatchObject({ room_id: 'room-a' });
const forwarded = core.requests.findLast((r) => r.path.endsWith('/ops/post_message'))!;
expect(forwarded.headers['x-switch-room-id']).toBe('room-a');
expect(forwarded.headers['x-switch-agent-id']).toBe(AGENT);
expect(forwarded.headers['x-switch-session-id']).toBe('session-1');
- expect(forwarded.headers['x-switch-connection-id']).toBeUndefined();
expect(forwarded.headers.authorization).toMatch(/^Bearer access-token-/);
- const unplaced = await callOperation(caller(watcher, {}, 'session-9'), 'post_message', {});
+ const unplaced = await callOperation(caller({}, 'session-9'), 'post_message', {});
expect(unplaced.structuredContent).toMatchObject({ room_id: null });
- const bySoleRoom = await callOperation(caller(watcher, {}, null), 'post_message', {});
+ const bySoleRoom = await callOperation(caller({}, null), 'post_message', {});
expect(bySoleRoom.structuredContent).toMatchObject({ room_id: 'room-a' });
- relay.sessionCommand({
+ hub.sessionCommand({
agent_id: AGENT,
room_id: 'room-a',
command: {
@@ -402,32 +307,22 @@ describe('the relay as the agent protocol, read by the real SwitchEventStream',
requesterName: 'Person',
},
});
- await waitFor(() => watcher.seen.commands.length === 1, 'the room control');
- expect(watcher.seen.commands[0]).toMatchObject({
- sessionId: 'session-1',
+ await waitFor(() => agentHost.seen.commands.length === 1, 'the room control');
+ expect(agentHost.seen.commands[0]).toMatchObject({
commandId: 'command-1',
+ roomId: 'room-a',
body: { type: 'session.reset' },
- requesterName: 'Person',
});
- expect(watcher.seen.commands[0]).not.toHaveProperty('room_id');
- expect(watcher.seen.commands[0]).not.toHaveProperty('agent_id');
- relay.sessionCommand({
- agent_id: AGENT,
- room_id: null,
- command: { commandId: 'command-2', origin: { roomId: 'room-b' } },
- });
await expect(
- watcher.stream.replacePlacements({ 'session-1': 'room-elsewhere' })
- ).rejects.toBeInstanceOf(PlacementsRefusedError);
- await delay(50);
- expect(watcher.seen.commands).toHaveLength(1);
+ agentHost.stream.replacePlacements({ 'session-1': 'room-elsewhere' })
+ ).rejects.toThrow(/not a member of room room-elsewhere/);
});
- it('passes approval outcomes to the watcher', async () => {
- const watcher = watch();
- await connected(watcher);
- relay.approvalOutcome({
+ it('passes approval outcomes to the agent host', async () => {
+ const agentHost = host();
+ await connected(agentHost);
+ hub.approvalOutcome({
agent_id: AGENT,
outcome: {
session_id: 'session-1',
@@ -438,27 +333,16 @@ describe('the relay as the agent protocol, read by the real SwitchEventStream',
answered_at: '2026-01-01T00:00:00Z',
},
});
- await waitFor(() => watcher.seen.outcomes.length === 1, 'the outcome');
- expect(watcher.seen.outcomes[0]).toMatchObject({ request_id: 'request-1', answer: 'yes' });
- });
-
- it('tells a connection its room was taken by a sibling placing a session there', async () => {
- const watcher = watch();
- await connected(watcher);
- await watcher.stream.replacePlacements({ 'session-1': 'room-a' });
- const other = watch({ connectionId: 'other-connection' });
- await connected(other);
- await other.stream.replacePlacements({ 'session-2': 'room-a' });
- await waitFor(() => watcher.seen.released.length === 1, 'room_released');
- expect(watcher.seen.released[0]).toEqual({ roomId: 'room-a', sessionId: 'session-1' });
+ await waitFor(() => agentHost.seen.outcomes.length === 1, 'the outcome');
+ expect(agentHost.seen.outcomes[0]).toMatchObject({ request_id: 'request-1', answer: 'yes' });
});
it('passes gaps through in order, and a reset as a cursor reset', async () => {
- const watcher = watch();
- await connected(watcher);
- relay.ingest(message(1, true));
- await waitFor(() => watcher.seen.events.length === 1, 'the first event');
- relay.gap({
+ const agentHost = host();
+ await connected(agentHost);
+ hub.ingest(message(1, true));
+ await waitFor(() => agentHost.seen.events.length === 1, 'the first event');
+ hub.gap({
agent_id: AGENT,
from_sequence: 1,
resumed_at: 10,
@@ -466,15 +350,10 @@ describe('the relay as the agent protocol, read by the real SwitchEventStream',
all_rooms: false,
reason: 'events older than the retention window were dropped',
});
- relay.ingest(message(11, true));
- await waitFor(() => watcher.seen.events.length === 2, 'the event after the gap');
- expect(watcher.seen.gaps[0]).toMatchObject({
- fromSequence: 1,
- resumedAt: 10,
- rooms: ['room-a'],
- cursorReset: false,
- });
- relay.gap({
+ hub.ingest(message(11, true));
+ await waitFor(() => agentHost.seen.events.length === 2, 'the event after the gap');
+ expect(agentHost.seen.gaps[0]).toMatchObject({ fromSequence: 1, resumedAt: 10 });
+ hub.gap({
agent_id: AGENT,
from_sequence: 11,
resumed_at: 2,
@@ -482,59 +361,68 @@ describe('the relay as the agent protocol, read by the real SwitchEventStream',
all_rooms: true,
reason: 'the server restarted',
});
- await waitFor(() => watcher.seen.gaps.length === 2, 'the reset');
- expect(watcher.seen.gaps[1]).toMatchObject({ resumedAt: 2, cursorReset: true });
- relay.ingest(message(3, true));
- await waitFor(() => watcher.seen.events.length === 3, 'numbering from the reset');
- expect(relay.cursors()[AGENT]).toBe(2);
+ await waitFor(() => agentHost.seen.gaps.length === 2, 'the reset');
+ expect(agentHost.seen.gaps[1]).toMatchObject({ resumedAt: 2, cursorReset: true });
+ hub.ingest(message(3, true));
+ await waitFor(() => agentHost.seen.events.length === 3, 'numbering from the reset');
+ await waitFor(() => hub.cursors()[AGENT] === 3, 'the cursor from the reset');
});
- it('drops what a reattached stream replays, and a gap behind what it holds', async () => {
- const watcher = watch();
- await connected(watcher);
- relay.ingest(message(1, true));
- relay.ingest(message(2, true));
- relay.streamAttached();
- relay.attach(AGENT, 0, ['room-a', 'room-b']);
- relay.gap({
- agent_id: AGENT,
- from_sequence: 0,
- resumed_at: 1,
- rooms: ['room-a'],
- all_rooms: false,
- reason: 'dropped',
+ it('a handler that fails leaves the event unconfirmed', async () => {
+ const agentHost = host({
+ onEvent: (event) => {
+ if (event.sequence === 2) throw new Error('the journal is full');
+ },
});
- relay.ingest(message(1, true));
- relay.ingest(message(2, true));
- relay.ingest(message(3, true));
- await waitFor(() => watcher.seen.events.length === 3, 'the new event');
- expect(watcher.seen.events.map((event) => event.sequence)).toEqual([1, 2, 3]);
- expect(watcher.seen.gaps).toEqual([]);
+ await connected(agentHost);
+ hub.ingest(message(1, true));
+ hub.ingest(message(2, true));
+ await waitFor(() => hub.cursors()[AGENT] === 1, 'the first event confirmed');
+ await delay(100);
+ expect(hub.cursors()[AGENT]).toBe(1);
});
- it('tells a watcher reconnecting from before what the relay still holds that it missed events', async () => {
- relay.setCursor('agent-2', 50);
- const tokenTwo = relay.mint('agent-2');
- const gaps: number[] = [];
- const controller = new AbortController();
- stops.push(() => controller.abort());
- const behind = new SwitchEventStream({
- creds: { agentId: 'agent-2', apiEndpoint: relay.endpoint, token: tokenTwo },
- connectionId: 'behind',
- worker: null,
- scope: 'all',
- filter: 'addressed',
- rooms: [],
- startCursor: 40,
- onEvent: () => {},
- onGap: (gap) => void gaps.push(gap.resumedAt ?? -1),
- onEvicted: () => {},
- log: quiet,
- signal: controller.signal,
+ it('reconnects when the controller restarts, resuming after what it handled', async () => {
+ const agentHost = host();
+ await connected(agentHost);
+ hub.ingest(message(1, true));
+ await waitFor(() => agentHost.seen.events.length === 1, 'the first event');
+ const port = Number(new URL(relay.endpoint).port);
+ await relay.close();
+ await waitFor(() => agentHost.seen.disconnected.length >= 1, 'the restart close');
+
+ // The controller comes back on the same port, its hub knowing nothing yet.
+ hub = new AgentHub({
+ log: silentLogger,
+ bufferLimit: 100,
+ onCursor: (agentId, cursor) => cursorsSaved.push([agentId, cursor]),
+ onChange: () => {},
});
- behind.start();
- await waitFor(() => gaps.length === 1, 'the gap');
- expect(gaps[0]).toBe(50);
+ relay = newRelay();
+ await relay.start(port);
+ relay.register(AGENT, token);
+ relay.setReady();
+ hub.streamAttached();
+ hub.attach(AGENT, 0, ['room-a', 'room-b']);
+ await connected(agentHost, 2);
+ // Switch replays from the opened cursor; the agent host said where it was.
+ hub.ingest(message(1, true));
+ hub.ingest(message(2, true));
+ await waitFor(() => agentHost.seen.events.length === 2, 'the new event');
+ expect(agentHost.seen.events.map((event) => event.sequence)).toEqual([1, 2]);
+ });
+
+ it('a second agent host for the agent takes the hub over, and the first stands down', async () => {
+ const first = host();
+ await connected(first);
+ const second = host();
+ await connected(second);
+ await waitFor(() => first.seen.evictions.length === 1, 'the first to be evicted');
+ expect(first.seen.evictions[0]).toMatchObject({ code: 'taken_over' });
+ hub.ingest(message(1, true));
+ await waitFor(() => second.seen.events.length === 1, 'the event at the second');
+ await delay(50);
+ expect(first.seen.events).toEqual([]);
});
});
@@ -544,19 +432,12 @@ describe('what the relay refuses', () => {
});
it('refuses a token it did not mint with 401, and asks to retry while starting', async () => {
- const refused = await post(`/agents/${AGENT}/connection/beat`, {}, 'swlr_not-a-token');
+ const refused = await post(`/agents/${AGENT}/ops/post_message`, {}, 'swlr_not-a-token');
expect(refused.status).toBe(401);
- const starting = new LocalRelay({
- log: silentLogger,
- version: '0.1.0',
- forwarder: { forward: async () => {} },
- onCursor: () => {},
- onChange: () => {},
- sharedRoomFor: () => null,
- now: Date.now,
- timing: { heartbeatTtlMs: 6_000, heartbeatIntervalS: 2, sweepMs: 50, keepaliveMs: 15_000 },
- bufferLimit: 100,
- });
+ expect(await upgradeStatus('swlr_not-a-token')).toBe(401);
+ expect(await upgradeStatus(token)).toBe(101);
+ expect(await upgradeStatus(token, '/elsewhere')).toBe(404);
+ const starting = newRelay();
await starting.start(null);
try {
const early = await fetch(`${starting.endpoint}/agents/${AGENT}/ops`, {
@@ -568,6 +449,18 @@ describe('what the relay refuses', () => {
}
});
+ it('answers the agent connection routes of an agent host from before the hub with 410', async () => {
+ for (const path of [`/agents/${AGENT}/connection/beat`, `/agents/${AGENT}/events`]) {
+ const response = await fetch(`${relay.endpoint}${path}`, {
+ method: path.endsWith('beat') ? 'POST' : 'GET',
+ headers: { Authorization: `Bearer ${token}` },
+ });
+ expect(response.status, path).toBe(410);
+ expect(JSON.stringify(await response.json())).toMatch(/\/hub/);
+ }
+ expect(core.requests.filter((r) => r.path.includes('/connection/'))).toEqual([]);
+ });
+
it('refuses another agent’s routes, and never relays the management routes', async () => {
const other = await fetch(`${relay.endpoint}/agents/agent-2/ops`, {
headers: { Authorization: `Bearer ${token}` },
@@ -612,11 +505,30 @@ describe('forwarding to Switch', () => {
expect(forwarded.headers.authorization).toMatch(/^Bearer access-token-/);
expect(forwarded.headers['x-switch-agent-id']).toBe(AGENT);
expect(forwarded.headers['x-switch-room-id']).toBeUndefined();
- expect(forwarded.headers['switch-controller-protocol']).toBe('1');
+ expect(forwarded.headers['switch-controller-protocol']).toBe('2');
expect(forwarded.body).toEqual({ row: 1 });
});
- it('names the room of a shared agent’s call from its host in the controller', async () => {
+ it('names the room of a call from its session’s placement on the hub', async () => {
+ // An agent host in the controller's own process states its placements to the hub directly.
+ hub.attach('agent-2', 0, ['room-shared']);
+ const inProcess = new AbortController();
+ stops.push(() => inProcess.abort());
+ await hub
+ .open('agent-2', {
+ creds: { agentId: 'agent-2', apiEndpoint: '', token: '' },
+ connectionId: 'in-process',
+ worker: null,
+ scope: 'all',
+ filter: 'addressed',
+ rooms: [],
+ onEvent: () => {},
+ onGap: () => {},
+ onEvicted: () => {},
+ log: quiet,
+ signal: inProcess.signal,
+ })
+ .replacePlacements({ 'session-7': 'room-shared' });
const shared = relay.mint('agent-2');
const response = await fetch(`${relay.endpoint}/agents/agent-2/ops/post_message`, {
method: 'POST',
diff --git a/console/packages/agent-controller/src/relay.ts b/console/packages/agent-controller/src/relay.ts
index f1adbeb0a..e9dc296a5 100644
--- a/console/packages/agent-controller/src/relay.ts
+++ b/console/packages/agent-controller/src/relay.ts
@@ -1,14 +1,9 @@
-import { createHash, randomBytes, randomInt } from 'node:crypto';
+import { createHash, randomBytes } from 'node:crypto';
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http';
import type { AddressInfo, Socket } from 'node:net';
-import { z } from 'zod';
+import type { Duplex } from 'node:stream';
+import { HUB_PATH } from '@switch-console/agent-providers';
import { errorMessage, type Logger } from './log';
-import type {
- AgentApprovalOutcomeFrame,
- AgentEventFrame,
- AgentGapFrame,
- AgentSessionCommandFrame,
-} from './schemas';
/**
* The controller's local relay: what each managed agent's host uses as
@@ -17,52 +12,18 @@ import type {
*
* Every call an agent host makes is forwarded to Switch with the controller's
* access token, the agent named in `X-Switch-Agent-Id`, and the calling
- * session's room in `X-Switch-Room-Id`.
+ * session's room in `X-Switch-Room-Id`. No agent holds a Switch credential.
*
- * An isolated agent's host runs in a process of its own, so it also hears its
- * events here: Switch delivers every agent's events on the one controller
- * stream, and the relay serves an isolated agent's to it as its own per-agent
- * stream, reproducing the agent protocol (`GET /agents/{id}/events` with its
- * frames, ids and resume, the heartbeat, placements and room claims). A shared
- * agent's host runs in the controller's process and hears its events from the
- * `AgentHub` instead; the room of its calls comes from there too.
+ * An agent host in a process of its own hears its events on the controller's
+ * hub, at `/hub` on this same port (`HubSocket`). The relay serves no event
+ * stream and no connection of its own: Switch holds one connection, the
+ * controller's, for every agent on this machine.
*/
-/**
- * The agent-protocol revisions the relay serves: switch-core's, up to the
- * socket. Revision 8 moved the connection onto a WebSocket, which the relay
- * does not serve; it serves the event stream and its beat, which the runtime
- * falls back to against a server that speaks 7 or older.
- */
-export const RELAY_AGENT_PROTOCOL = { speaks: 7, accepts: 1 } as const;
-/** From this revision a client names its connection incarnation on every beat and room request. */
-const FENCED_PROTOCOL_REVISION = 2;
-const APPROVAL_OUTCOME_PROTOCOL_REVISION = 4;
-const SESSION_COMMAND_PROTOCOL_REVISION = 5;
-const ROOM_RELEASED_PROTOCOL_REVISION = 6;
-const MAX_CONNECTIONS_PER_AGENT = 32;
-const MAX_JSON_BODY_BYTES = 256 * 1024;
-
/** Every token the relay mints starts with this, which tells it apart from a Switch credential. */
export const RELAY_TOKEN_PREFIX = 'swlr_';
-export type RelayTiming = {
- /** A connection with no beat for this long is closed, as Switch closes one. */
- heartbeatTtlMs: number;
- /** What `connection_state` tells the client to beat on. */
- heartbeatIntervalS: number;
- sweepMs: number;
- keepaliveMs: number;
-};
-
-export const DEFAULT_RELAY_TIMING: RelayTiming = {
- heartbeatTtlMs: 6_000,
- heartbeatIntervalS: 2,
- sweepMs: 1_000,
- keepaliveMs: 15_000,
-};
-
-/** Where a request that is not connection bookkeeping goes. */
+/** Where a request that is not the hub goes. */
export interface Forwarder {
forward(
req: IncomingMessage,
@@ -71,77 +32,18 @@ export interface Forwarder {
): Promise<void>;
}
+/** Where the hub's upgrades go, with the agent the token named. */
+export interface HubEndpoint {
+ accept(agentId: string, req: IncomingMessage, socket: Duplex, head: Buffer): void;
+ close(): Promise<void>;
+}
+
export type RelayDeps = {
log: Logger;
- /** The room of a call for a shared agent (one whose host runs in the controller), from the calling session. */
- sharedRoomFor: (agentId: string, sessionId: string | null) => string | null;
- version: string;
+ /** The room of a call for an agent, from the calling session's placement. */
+ roomFor: (agentId: string, sessionId: string | null) => string | null;
forwarder: Forwarder;
- /** An agent's confirmed cursor moved; the controller persists it and beats it upstream. */
- onCursor: (agentId: string, cursor: number) => void;
- /** Something `attached()` reads changed. */
- onChange: () => void;
- now: () => number;
- timing: RelayTiming;
- /** The most events held per agent for local resume; past it the oldest go, and say so. */
- bufferLimit: number;
-};
-
-type Declaration = {
- speaks: number | null;
- accepts: number | null;
- artifact: string | null;
- version: string | null;
-};
-
-type LocalStream = {
- res: ServerResponse;
- generation: number;
- /** The rooms the client was last told it covers. */
- told: Set<string>;
-};
-
-type LocalConnection = {
- id: string;
- agentId: string;
- scope: 'single' | 'all';
- filter: 'all' | 'addressed';
- spawnCapable: boolean;
- /** The last sequence written out (or skipped); null until the agent's head is known. */
- cursor: number | null;
- /** The last sequence the client confirmed on a beat, never moving back. */
- confirmed: number | null;
- generation: number;
- lastBeat: number;
- stream: LocalStream | null;
- rooms: Set<string>;
- released: Map<string, string | null>;
- declaration: Declaration;
-};
-
-type Buffered =
- | { kind: 'event'; seq: number; type: string; roomId: string; notifiable: boolean; data: object }
- | { kind: 'gap'; seq: number; data: Record<string, unknown> };
-
-type AgentState = {
- agentId: string;
- /** Core attached the agent to the controller stream. */
- attached: boolean;
- /** The rooms the agent belongs to, as Core last said; null before it has. */
- rooms: Set<string> | null;
- /** The highest sequence known for this agent. */
- head: number | null;
- /** Below or at this, the relay holds nothing: a cursor here gets a gap. */
- droppedThrough: number;
- /** The cursor the controller resumes the agent from upstream. */
- confirmed: number | null;
- buffer: Buffered[];
- /** The last time Core reset the agent's numbering (a Core restart). */
- reset: { reason: string; rooms: string[] } | null;
- connections: Map<string, LocalConnection>;
- /** Each session's room, and the connection that placed it there. */
- placements: Map<string, { roomId: string; owner: string }>;
- overflowing: boolean;
+ hub: HubEndpoint;
};
class HttpRefusal extends Error {
@@ -153,47 +55,16 @@ class HttpRefusal extends Error {
}
}
-function coded(status: number, code: string, message: string): HttpRefusal {
- return new HttpRefusal(status, { code, message });
-}
-
-function notOpen(connectionId: string): HttpRefusal {
- return new HttpRefusal(
- 404,
- `connection ${connectionId} is not open; reconnect and resume from your cursor`
- );
-}
-
function hashToken(token: string): string {
return createHash('sha256').update(token).digest('hex');
}
-/** Whether a client filtering to what addresses the agent is sent this event, as Switch decides. */
-export function isNotifiable(type: string, payload: Record<string, unknown>): boolean {
- if (type === 'message') return payload.addressed === true;
- if (type === 'room_join') return payload.listening === true;
- if (type === 'command') return false;
- return type.startsWith('task_');
-}
-
-const beatSchema = z.object({
- connection_id: z.string().min(1),
- cursor: z.number().int().nonnegative().default(0),
- generation: z.number().int().nullish(),
-});
-const roomRequestSchema = z.object({
- connection_id: z.string().min(1),
- room_id: z.string().min(1),
- takeover: z.boolean().default(false),
- generation: z.number().int().nullish(),
-});
-const placementsSchema = z.object({
- connection_id: z.string().min(1),
- placements: z.record(z.string().min(1), z.string().min(1)),
- generation: z.number().int().nullish(),
-});
-
-const LOCAL_ROUTE = /^\/agents\/([^/]+)\/(events|connection\/[^/]+)$/;
+/**
+ * The agent protocol's own connection routes, which an agent host from before
+ * the hub still calls. Refused rather than forwarded: Switch refuses them to a
+ * controller, and the agent host is restarted onto the hub.
+ */
+const CONNECTION_ROUTE = /^\/agents\/[^/]+\/(events|connection\/[^/]+)$/;
/** The prefixes forwarded to Switch. Anything else, the management routes above all, stays here. */
const FORWARDED = /^\/(agents\/[^/]+\/.+|agent-sessions\/.+|sessions\/.+|version|health)$/;
/** `/agents/<segment>/...` routes whose segment is not an agent. */
@@ -205,20 +76,11 @@ export class LocalRelay {
private readonly sockets = new Set<Socket>();
private readonly tokens = new Map<string, string>();
private readonly tokenOf = new Map<string, string>();
- private readonly agents = new Map<string, AgentState>();
- private readonly connections = new Map<string, LocalConnection>();
- /**
- * Starts anywhere, as Switch's does, so a client still holding an
- * incarnation from before a restart cannot name one this relay issues.
- */
- private incarnation = randomInt(2 ** 31);
private ready = false;
- private upstream = false;
- private timers: ReturnType<typeof setInterval>[] = [];
constructor(private readonly deps: RelayDeps) {}
- /** Listens on 127.0.0.1: on `preferredPort` when it is free, so running watchers find it again. */
+ /** Listens on 127.0.0.1: on `preferredPort` when it is free, so running agent hosts find it again. */
async start(preferredPort: number | null): Promise<number> {
const server = createServer((req, res) => {
void this.handle(req, res).catch((error: unknown) => {
@@ -230,6 +92,7 @@ export class LocalRelay {
else res.destroy();
});
});
+ server.on('upgrade', (req, socket, head) => this.upgrade(req, socket, head));
server.on('connection', (socket) => {
this.sockets.add(socket);
socket.on('close', () => this.sockets.delete(socket));
@@ -255,11 +118,6 @@ export class LocalRelay {
}
this.server = server;
this.port = (server.address() as AddressInfo).port;
- this.timers = [
- setInterval(() => this.sweep(), this.deps.timing.sweepMs),
- setInterval(() => this.keepalive(), this.deps.timing.keepaliveMs),
- ];
- for (const timer of this.timers) timer.unref();
return this.port;
}
@@ -268,14 +126,18 @@ export class LocalRelay {
return `http://127.0.0.1:${this.port}`;
}
+ /** Where an agent host in a process of its own hears its events. */
+ get hubUrl(): string {
+ if (!this.server) throw new Error('The relay is not listening.');
+ return `ws://127.0.0.1:${this.port}${HUB_PATH}`;
+ }
+
/**
- * Stops serving. Streams are cut without an `evicted` frame: the watchers
- * outlive the controller, and a dropped socket is what tells them to keep
- * retrying until it is back.
+ * Stops serving. The hub's agent hosts are told the controller is
+ * restarting, and keep retrying until it is back: they outlive it.
*/
async close(): Promise<void> {
- for (const timer of this.timers) clearInterval(timer);
- this.timers = [];
+ await this.deps.hub.close();
for (const socket of this.sockets) socket.destroy();
const server = this.server;
this.server = null;
@@ -284,7 +146,7 @@ export class LocalRelay {
/**
* Unknown tokens are told to retry (503) until this is called: right after
- * a restart a watcher can reach the relay before the controller has read
+ * a restart an agent host can reach the relay before the controller has read
* back its token, and a 401 would stop it for good.
*/
setReady(): void {
@@ -300,7 +162,6 @@ export class LocalRelay {
const hash = hashToken(token);
this.tokens.set(hash, agentId);
this.tokenOf.set(agentId, hash);
- this.agent(agentId);
}
/** A new token for the agent, replacing any it had. */
@@ -314,263 +175,62 @@ export class LocalRelay {
return this.tokenOf.get(agentId) === hashToken(token);
}
- /**
- * Forgets what is held for the agent's own stream, keeping its token: its
- * host now runs in the controller's process.
- */
- forgetStreams(agentId: string): void {
- const agent = this.agents.get(agentId);
- if (!agent) return;
- for (const conn of agent.connections.values()) {
- conn.stream?.res.destroy();
- this.connections.delete(conn.id);
- }
- this.agents.delete(agentId);
- this.deps.onChange();
- }
-
- /** Forgets the agent: its token stops working and its streams are cut. */
+ /** Forgets the agent: its token stops working. */
unregister(agentId: string): void {
const hash = this.tokenOf.get(agentId);
if (hash) this.tokens.delete(hash);
this.tokenOf.delete(agentId);
- const agent = this.agents.get(agentId);
- if (!agent) return;
- for (const conn of agent.connections.values()) {
- conn.stream?.res.destroy();
- this.connections.delete(conn.id);
- }
- this.agents.delete(agentId);
- this.deps.onChange();
- }
-
- /** Where the controller resumes an agent from, as persisted. */
- setCursor(agentId: string, cursor: number): void {
- const agent = this.agent(agentId);
- agent.confirmed = cursor;
- agent.head = cursor;
- agent.droppedThrough = cursor;
- }
-
- /** Each agent's confirmed cursor, for the upstream beat and reopen. */
- cursors(): Record<string, number> {
- const cursors: Record<string, number> = {};
- for (const agent of this.agents.values())
- if (agent.confirmed !== null) cursors[agent.agentId] = agent.confirmed;
- return cursors;
- }
-
- /** The agent's events flow on the controller stream and its watcher is connected here. */
- attached(agentId: string): boolean {
- const agent = this.agents.get(agentId);
- if (!agent || !this.upstream || !agent.attached) return false;
- return [...agent.connections.values()].some((conn) => conn.stream && this.alive(conn));
- }
-
- // -- What arrives on the controller stream ----------------------------------
-
- setUpstream(connected: boolean): void {
- if (this.upstream === connected) return;
- this.upstream = connected;
- this.deps.onChange();
- }
-
- /**
- * The controller stream (re)attached. Switch attaches every bound agent
- * afresh on each stream, with `agent.attached`, so none counts as attached
- * until it says so again.
- */
- streamAttached(): void {
- for (const agent of this.agents.values()) agent.attached = false;
- this.upstream = true;
- this.deps.onChange();
- }
-
- attach(agentId: string, fromSeq: number, rooms: string[]): void {
- const agent = this.agent(agentId);
- agent.attached = true;
- this.setRooms(agentId, rooms);
- if (agent.head === null) agent.head = fromSeq;
- else if (fromSeq > agent.head) {
- // Switch starts the agent past what the relay holds: whatever lay
- // between is not coming, and a watcher behind it is told so.
- agent.head = fromSeq;
- agent.droppedThrough = Math.max(agent.droppedThrough, fromSeq);
- }
- for (const conn of agent.connections.values())
- if (conn.cursor === null) {
- conn.cursor = agent.head;
- conn.confirmed ??= agent.head;
- }
- this.deps.log.info('Agent attached to the controller stream', { agentId, fromSeq });
- this.pumpAll(agent);
- this.deps.onChange();
- }
-
- detach(agentId: string, reason: string): void {
- const agent = this.agents.get(agentId);
- if (!agent) return;
- agent.attached = false;
- this.deps.log.warn('Agent detached from the controller stream', { agentId, reason });
- this.deps.onChange();
}
- /** The agent's membership changed: rooms it left stop being covered here. */
- setRooms(agentId: string, rooms: string[]): void {
- const agent = this.agent(agentId);
- agent.rooms = new Set(rooms);
- for (const conn of agent.connections.values()) {
- for (const room of conn.rooms) if (!agent.rooms.has(room)) conn.rooms.delete(room);
- this.pump(conn);
- }
- }
-
- /**
- * A domain event: buffered for local resume, and written to every stream
- * that covers it. One the relay already has (Switch replays from where the
- * connection opened each time the stream reattaches) is dropped.
- */
- ingest(frame: AgentEventFrame): void {
- const { agent_id: agentId, seq } = frame;
- const data = { ...frame.event, sequence: seq };
- const agent = this.agent(agentId);
- if (agent.head !== null && seq <= agent.head) return;
- agent.buffer.push({
- kind: 'event',
- seq,
- type: data.type,
- roomId: data.room_id,
- notifiable: isNotifiable(data.type, data.payload),
- data,
- });
- for (const conn of agent.connections.values())
- if (conn.cursor === null) {
- conn.cursor = seq - 1;
- conn.confirmed ??= seq - 1;
- }
- agent.head = seq;
- this.pumpAll(agent);
- this.trim(agent);
- }
-
- /**
- * Switch could not serve the agent's cursor. A reset (Switch restarted, so
- * its numbering went back: `all_rooms`, resuming below what the relay
- * holds) clears what is held and goes to every stream now. A gap resuming
- * past what the relay holds takes its place in the buffer, so each stream
- * meets it in order. One resuming at or below it describes events the relay
- * already has — a reattached stream starting from where the connection
- * opened — and is not passed on.
- */
- gap(frame: AgentGapFrame): void {
- const { agent_id: agentId, ...data } = frame;
- const agent = this.agent(agentId);
- const resumedAt = data.resumed_at;
- if (resumedAt === undefined) {
- for (const conn of agent.connections.values()) this.write(conn, 'gap', data);
- return;
- }
- if (agent.head !== null && resumedAt <= agent.head && data.all_rooms !== true) {
- this.deps.log.debug('Dropped a gap behind what the relay already holds', {
- agentId,
- resumedAt,
- head: agent.head,
- });
- return;
- }
- if (agent.head !== null && resumedAt < agent.head) {
- agent.buffer = [];
- agent.head = resumedAt;
- agent.droppedThrough = resumedAt;
- agent.reset = { reason: data.reason, rooms: data.rooms ?? [] };
- agent.confirmed = resumedAt;
- this.deps.onCursor(agentId, resumedAt);
- for (const conn of agent.connections.values()) {
- if (conn.confirmed !== null) conn.confirmed = Math.min(conn.confirmed, resumedAt);
- if (!conn.stream) continue;
- this.write(conn, 'gap', data);
- conn.cursor = resumedAt;
- }
- return;
- }
- agent.buffer.push({ kind: 'gap', seq: resumedAt, data });
- agent.head = resumedAt;
- this.pumpAll(agent);
- this.trim(agent);
- }
+ // -- HTTP -------------------------------------------------------------------
- /** A room control, for the session placed in its room. */
- sessionCommand(frame: AgentSessionCommandFrame): void {
- const { agent_id: agentId, command } = frame;
- const origin = command.origin as { roomId?: unknown } | undefined;
- const roomId = frame.room_id ?? (typeof origin?.roomId === 'string' ? origin.roomId : null);
- const agent = this.agents.get(agentId);
- const sessionId = agent && roomId ? this.sessionIn(agent, roomId) : null;
- if (!agent || !sessionId) {
- this.deps.log.warn(
- 'Dropped a room control: no session of the agent is placed in its room here',
- { agentId, roomId, commandId: command.commandId }
- );
- return;
- }
- const watchers = this.watchers(agent, SESSION_COMMAND_PROTOCOL_REVISION);
- if (!watchers.length) {
- this.deps.log.warn('Dropped a room control: the agent has no watcher connected', {
- agentId,
- roomId,
- commandId: command.commandId,
- });
- return;
- }
- for (const conn of watchers) this.write(conn, 'session_command', { ...command, sessionId });
+ /** The agent a request's bearer token names, or the refusal for one it does not. */
+ private authenticate(req: IncomingMessage): string | HttpRefusal {
+ const auth = req.headers.authorization ?? '';
+ const token = auth.startsWith('Bearer ') ? auth.slice(7).trim() : '';
+ const agentId = token ? this.tokens.get(hashToken(token)) : undefined;
+ if (agentId) return agentId;
+ if (!this.ready)
+ return new HttpRefusal(503, 'The agents controller is starting; retry in a moment.');
+ return new HttpRefusal(
+ 401,
+ 'This token is not one the agents controller issued, or its agent is no longer assigned to this machine.'
+ );
}
- approvalOutcome(frame: AgentApprovalOutcomeFrame): void {
- const { agent_id: agentId, outcome } = frame;
- const agent = this.agents.get(agentId);
- const watchers = agent ? this.watchers(agent, APPROVAL_OUTCOME_PROTOCOL_REVISION) : [];
- if (!watchers.length) {
- this.deps.log.warn(
- 'An approval outcome arrived while the agent has no watcher connected; Switch sends it again until it is delivered',
- { agentId, requestId: outcome.request_id }
+ private upgrade(req: IncomingMessage, socket: Duplex, head: Buffer): void {
+ const path = new URL(req.url ?? '/', 'http://relay.invalid').pathname;
+ const refuse = (status: number, reason: string) => {
+ socket.end(`HTTP/1.1 ${status} ${reason}\r\nConnection: close\r\nContent-Length: 0\r\n\r\n`);
+ };
+ if (path !== HUB_PATH) return refuse(404, 'Not Found');
+ const agentId = this.authenticate(req);
+ if (agentId instanceof HttpRefusal)
+ return refuse(
+ agentId.status,
+ agentId.status === 503 ? 'Service Unavailable' : 'Unauthorized'
);
- return;
- }
- for (const conn of watchers) this.write(conn, 'approval_outcome', outcome);
+ this.deps.hub.accept(agentId, req, socket, head);
}
- // -- HTTP -------------------------------------------------------------------
-
private async handle(req: IncomingMessage, res: ServerResponse): Promise<void> {
const raw = req.url ?? '';
if (!raw.startsWith('/') || raw.startsWith('//'))
return this.send(res, 400, { detail: 'The relay serves origin-form paths only.' });
const url = new URL(raw, 'http://relay.invalid');
- const auth = req.headers.authorization ?? '';
- const token = auth.startsWith('Bearer ') ? auth.slice(7).trim() : '';
- const agentId = token ? this.tokens.get(hashToken(token)) : undefined;
- if (!agentId) {
- if (!this.ready)
- return this.send(res, 503, {
- detail: 'The agents controller is starting; retry in a moment.',
- });
- return this.send(res, 401, {
- detail:
- 'This token is not one the agents controller issued, or its agent is no longer assigned to this machine.',
- });
- }
+ const agentId = this.authenticate(req);
+ if (agentId instanceof HttpRefusal)
+ return this.send(res, agentId.status, { detail: agentId.detail });
try {
- const local = LOCAL_ROUTE.exec(url.pathname);
- if (local) {
- const named = decodeURIComponent(local[1]!);
- if (named !== agentId)
- throw new HttpRefusal(403, `authenticated as agent ${agentId}, not ${named}`);
- return await this.handleLocal(req, res, agentId, local[2]!, url);
- }
+ if (CONNECTION_ROUTE.test(url.pathname))
+ throw new HttpRefusal(
+ 410,
+ `The agents controller no longer serves the agent connection; its agent hosts hear their events on ${HUB_PATH}. Restart the agent host.`
+ );
this.checkForwardable(url, raw, agentId);
await this.deps.forwarder.forward(req, res, {
agentId,
- roomId: this.roomFor(agentId, req),
+ roomId: this.deps.roomFor(agentId, header(req, 'x-switch-session-id')),
path: `${url.pathname}${url.search}`,
});
} catch (error) {
@@ -590,558 +250,6 @@ export class LocalRelay {
throw new HttpRefusal(403, `authenticated as agent ${agentId}, not ${named}`);
}
- private async handleLocal(
- req: IncomingMessage,
- res: ServerResponse,
- agentId: string,
- route: string,
- url: URL
- ): Promise<void> {
- if (route === 'events') {
- if (req.method !== 'GET') throw new HttpRefusal(405, 'Method Not Allowed');
- if (!(req.headers.accept ?? '').includes('text/event-stream'))
- throw new HttpRefusal(
- 406,
- 'The agents controller relays the event stream only; ask for text/event-stream.'
- );
- return this.openStream(req, res, agentId, url);
- }
- if (req.method !== 'POST') throw new HttpRefusal(405, 'Method Not Allowed');
- const body = await readJson(req);
- switch (route) {
- case 'connection/beat':
- return this.send(res, 200, this.beat(agentId, parse(beatSchema, body)));
- case 'connection/placements':
- return this.send(res, 200, this.placements(agentId, parse(placementsSchema, body)));
- case 'connection/subscribe':
- return this.send(res, 200, this.subscribe(agentId, parse(roomRequestSchema, body)));
- case 'connection/unsubscribe':
- return this.send(res, 200, this.unsubscribe(agentId, parse(roomRequestSchema, body)));
- default:
- throw new HttpRefusal(404, 'Not Found');
- }
- }
-
- private openStream(req: IncomingMessage, res: ServerResponse, agentId: string, url: URL): void {
- const query = url.searchParams;
- const connectionId = query.get('connection_id');
- if (!connectionId)
- throw new HttpRefusal(
- 400,
- 'connection_id is required to open an event stream; generate a UUID and reuse it when reconnecting so the connection survives the drop'
- );
- const scope = query.get('scope') ?? 'single';
- if (scope !== 'single' && scope !== 'all')
- throw new HttpRefusal(400, `scope must be 'single' or 'all', got '${scope}'`);
- const filter = query.get('filter') ?? 'all';
- if (filter !== 'all' && filter !== 'addressed')
- throw new HttpRefusal(400, `filter must be 'all' or 'addressed', got '${filter}'`);
- const agent = this.agent(agentId);
- const lastEventId = req.headers['last-event-id'];
- const rawCursor =
- (typeof lastEventId === 'string' && lastEventId) || query.get('start_from') || 'head';
- let cursor: number | null;
- if (rawCursor === 'head') cursor = agent.head;
- else {
- cursor = Number(rawCursor);
- if (!Number.isSafeInteger(cursor))
- throw new HttpRefusal(
- 400,
- `start_from must be 'head' or a sequence number, got '${rawCursor}'`
- );
- cursor = Math.max(cursor, 0);
- }
- const declaration = declarationOf(query);
- if (
- declaration.speaks !== null &&
- declaration.accepts !== null &&
- !(
- declaration.accepts <= RELAY_AGENT_PROTOCOL.speaks &&
- RELAY_AGENT_PROTOCOL.accepts <= declaration.speaks
- )
- )
- throw new HttpRefusal(409, {
- message: `agent-protocol ${declaration.accepts}-${declaration.speaks} does not overlap the ${RELAY_AGENT_PROTOCOL.accepts}-${RELAY_AGENT_PROTOCOL.speaks} the agents controller serves`,
- contract: 'agent-protocol',
- server: { version: null, ...RELAY_AGENT_PROTOCOL },
- client: { speaks: declaration.speaks, accepts: declaration.accepts },
- remedy: 'Update the agents controller, or the agent runtime, so their ranges overlap.',
- });
- const expectedRaw = query.get('expected_generation');
- const expected = expectedRaw === null ? null : Number(expectedRaw);
-
- let conn = this.connections.get(connectionId);
- if (conn && conn.agentId !== agentId) throw new HttpRefusal(409, notOpen(connectionId).detail);
- if (conn && !this.alive(conn)) {
- this.closeConnection(conn, null);
- conn = undefined;
- }
- if (conn) {
- if (expected !== null && expected !== conn.generation)
- throw coded(
- 409,
- 'taken_over',
- `connection ${connectionId} has been reopened since incarnation ${expected} and is now at ${conn.generation}; another client holds it, so this reattach was refused and the connection was left untouched`
- );
- if (conn.stream)
- this.evict(
- conn,
- 'taken_over',
- 'another stream attached to this connection and took it over'
- );
- } else {
- if (agent.connections.size >= MAX_CONNECTIONS_PER_AGENT)
- throw new HttpRefusal(
- 409,
- `agent ${agentId} already holds ${MAX_CONNECTIONS_PER_AGENT} connections; close one before opening another`
- );
- conn = {
- id: connectionId,
- agentId,
- scope,
- filter,
- spawnCapable: false,
- cursor: null,
- confirmed: null,
- generation: 0,
- lastBeat: 0,
- stream: null,
- rooms: new Set(),
- released: new Map(),
- declaration,
- };
- this.connections.set(connectionId, conn);
- agent.connections.set(connectionId, conn);
- }
- conn.scope = scope;
- conn.filter = filter;
- conn.spawnCapable = query.get('spawn_capable') === 'true';
- conn.declaration = declaration;
- conn.cursor = cursor;
- conn.confirmed = cursor;
- conn.lastBeat = this.deps.now();
- conn.generation = ++this.incarnation;
-
- for (const roomId of (query.get('rooms') ?? '').split(',').filter(Boolean)) {
- if (agent.rooms && !agent.rooms.has(roomId)) {
- this.closeConnection(conn, null);
- throw new HttpRefusal(403, `agent ${agentId} is not a member of room ${roomId}`);
- }
- this.claim(conn, roomId, true);
- }
-
- res.writeHead(200, {
- 'Content-Type': 'text/event-stream',
- 'Cache-Control': 'no-cache',
- Connection: 'keep-alive',
- 'X-Accel-Buffering': 'no',
- });
- const stream: LocalStream = { res, generation: conn.generation, told: new Set(conn.rooms) };
- conn.stream = stream;
- const owner = conn;
- res.on('close', () => {
- if (owner.stream === stream) {
- owner.stream = null;
- this.deps.onChange();
- }
- });
- this.write(conn, 'connection_state', {
- connection_id: conn.id,
- agent_id: agentId,
- generation: conn.generation,
- scope: conn.scope,
- filter: conn.filter,
- spawn_capable: conn.spawnCapable,
- rooms: [...conn.rooms].sort(),
- cursor: conn.cursor ?? 0,
- protocol: RELAY_AGENT_PROTOCOL.speaks,
- heartbeat_interval_seconds: this.deps.timing.heartbeatIntervalS,
- // Switch keeps no record of an agent's sessions: the agent host says when it starts one.
- announce_session_starts: true,
- server: {
- version: null,
- contracts: { 'agent-protocol': { ...RELAY_AGENT_PROTOCOL } },
- relay: { artifact: 'switch-agent-controller', version: this.deps.version },
- },
- client: declaration,
- });
- if (agent.reset && conn.cursor !== null && agent.head !== null && conn.cursor > agent.head) {
- this.write(conn, 'gap', {
- from_sequence: agent.head,
- resumed_at: agent.head,
- rooms: [...conn.rooms].sort(),
- all_rooms: true,
- reason: agent.reset.reason,
- });
- conn.cursor = agent.head;
- conn.confirmed = agent.head;
- }
- this.pump(conn);
- this.deps.onChange();
- }
-
- private beat(
- agentId: string,
- body: z.infer<typeof beatSchema>
- ): { ok: true; rooms: string[]; cursor: number } {
- const conn = this.live(agentId, body.connection_id);
- this.fence(conn, body.generation ?? null, 'heartbeat');
- if (!conn.stream)
- throw coded(
- 409,
- 'no_stream',
- `connection ${conn.id} has no stream attached; reopen the event stream`
- );
- conn.lastBeat = this.deps.now();
- const agent = this.agent(agentId);
- const cursor = agent.head === null ? body.cursor : Math.min(body.cursor, agent.head);
- if (conn.cursor !== null && cursor > conn.cursor) conn.cursor = cursor;
- if (conn.confirmed !== null && cursor > conn.confirmed) conn.confirmed = cursor;
- this.confirm(agent);
- this.trim(agent);
- return { ok: true, rooms: [...conn.rooms].sort(), cursor: conn.cursor ?? cursor };
- }
-
- private placements(agentId: string, body: z.infer<typeof placementsSchema>) {
- const conn = this.current(agentId, body.connection_id, body.generation ?? null);
- const agent = this.agent(agentId);
- const rooms = Object.values(body.placements);
- for (const roomId of new Set(rooms))
- if (agent.rooms && !agent.rooms.has(roomId))
- throw new HttpRefusal(403, `agent ${agentId} is not a member of room ${roomId}`);
- const twice = [...new Set(rooms.filter((room, index) => rooms.indexOf(room) !== index))];
- if (twice.length)
- throw new HttpRefusal(
- 400,
- `placements name room(s) ${twice.sort().join(', ')} for more than one session; one session of an agent acts in a room`
- );
- const released: { connection_id: string; room_id: string; session_id: string | null }[] = [];
- const previousRooms = new Set<string>();
- for (const [sessionId, placed] of agent.placements)
- if (placed.owner === conn.id) {
- previousRooms.add(placed.roomId);
- agent.placements.delete(sessionId);
- }
- for (const [sessionId, roomId] of Object.entries(body.placements)) {
- agent.placements.delete(sessionId);
- for (const [other, placed] of agent.placements) {
- if (placed.roomId !== roomId) continue;
- agent.placements.delete(other);
- const loser = this.connections.get(placed.owner);
- if (loser && loser !== conn && loser.agentId === agentId) {
- this.notifyReleased(loser, roomId, other);
- released.push({ connection_id: loser.id, room_id: roomId, session_id: other });
- }
- }
- agent.placements.set(sessionId, { roomId, owner: conn.id });
- }
- for (const roomId of new Set(rooms)) {
- const evicted = this.claim(conn, roomId, true);
- if (evicted && !released.some((r) => r.connection_id === evicted.id && r.room_id === roomId))
- released.push({ connection_id: evicted.id, room_id: roomId, session_id: null });
- }
- for (const roomId of previousRooms) if (!rooms.includes(roomId)) conn.rooms.delete(roomId);
- for (const other of agent.connections.values()) this.pump(other);
- return {
- ok: true,
- placements: this.connectionPlacements(conn),
- rooms: [...conn.rooms].sort(),
- released,
- };
- }
-
- private subscribe(agentId: string, body: z.infer<typeof roomRequestSchema>) {
- const conn = this.current(agentId, body.connection_id, body.generation ?? null);
- const agent = this.agent(agentId);
- if (agent.rooms && !agent.rooms.has(body.room_id))
- throw new HttpRefusal(403, `agent ${agentId} is not a member of room ${body.room_id}`);
- const departing =
- conn.scope === 'single' ? [...conn.rooms].filter((r) => r !== body.room_id) : [];
- const evicted = this.claim(conn, body.room_id, body.takeover);
- for (const roomId of departing) conn.rooms.delete(roomId);
- for (const other of agent.connections.values()) this.pump(other);
- return {
- ok: true,
- rooms: [...conn.rooms].sort(),
- evicted_connection_id: evicted?.id ?? null,
- warning: evicted
- ? `Room ${body.room_id} was being attended by connection ${evicted.id} of this agent. That connection no longer receives the room's messages; this one does.`
- : null,
- };
- }
-
- private unsubscribe(agentId: string, body: z.infer<typeof roomRequestSchema>) {
- const conn = this.current(agentId, body.connection_id, body.generation ?? null);
- conn.rooms.delete(body.room_id);
- for (const other of this.agent(agentId).connections.values()) this.pump(other);
- return { ok: true, rooms: [...conn.rooms].sort() };
- }
-
- // -- Connections ------------------------------------------------------------
-
- private agent(agentId: string): AgentState {
- let agent = this.agents.get(agentId);
- if (!agent) {
- agent = {
- agentId,
- attached: false,
- rooms: null,
- head: null,
- droppedThrough: 0,
- confirmed: null,
- buffer: [],
- reset: null,
- connections: new Map(),
- placements: new Map(),
- overflowing: false,
- };
- this.agents.set(agentId, agent);
- }
- return agent;
- }
-
- private alive(conn: LocalConnection): boolean {
- return this.deps.now() - conn.lastBeat < this.deps.timing.heartbeatTtlMs;
- }
-
- /** The agent's open connection by id, or the 404 Switch answers for one that is not. */
- private live(agentId: string, connectionId: string): LocalConnection {
- const conn = this.connections.get(connectionId);
- if (!conn || conn.agentId !== agentId) throw notOpen(connectionId);
- if (!this.alive(conn)) {
- this.closeConnection(conn, 'heartbeat_lapsed');
- throw notOpen(connectionId);
- }
- return conn;
- }
-
- /** Refuses a request from a client that is not, or cannot show it is, the connection's holder. */
- private fence(conn: LocalConnection, generation: number | null, what: string): void {
- if (generation === null) {
- if ((conn.declaration.speaks ?? 0) >= FENCED_PROTOCOL_REVISION)
- throw coded(
- 409,
- 'unfenced',
- `connection ${conn.id} is held by a client speaking agent-protocol ${conn.declaration.speaks}, which names the connection incarnation on every ${what}; this one named none`
- );
- return;
- }
- if (generation !== conn.generation)
- throw coded(
- 409,
- 'taken_over',
- `connection ${conn.id} was reopened since incarnation ${generation} and is now at ${conn.generation}; another client holds it, so this ${what} was refused`
- );
- }
-
- private current(agentId: string, connectionId: string, generation: number | null) {
- const conn = this.live(agentId, connectionId);
- this.fence(conn, generation, 'room request');
- return conn;
- }
-
- /** Takes a room for `conn`, from whichever sibling held it. Returns that sibling. */
- private claim(conn: LocalConnection, roomId: string, takeover: boolean): LocalConnection | null {
- const agent = this.agent(conn.agentId);
- let evicted: LocalConnection | null = null;
- for (const sibling of agent.connections.values()) {
- if (sibling === conn || !sibling.rooms.has(roomId) || !this.alive(sibling)) continue;
- if (!takeover)
- throw new HttpRefusal(
- 409,
- `room ${roomId} is already held by connection ${sibling.id} for this agent; close it or pass takeover`
- );
- sibling.rooms.delete(roomId);
- this.notifyReleased(sibling, roomId, this.placedBy(agent, sibling, roomId));
- evicted = sibling;
- }
- conn.released.delete(roomId);
- conn.rooms.add(roomId);
- return evicted;
- }
-
- private notifyReleased(conn: LocalConnection, roomId: string, sessionId: string | null): void {
- if ((conn.declaration.speaks ?? 0) < ROOM_RELEASED_PROTOCOL_REVISION) return;
- if (sessionId !== null || !conn.released.has(roomId)) conn.released.set(roomId, sessionId);
- }
-
- private placedBy(agent: AgentState, conn: LocalConnection, roomId: string): string | null {
- for (const [sessionId, placed] of agent.placements)
- if (placed.roomId === roomId && placed.owner === conn.id) return sessionId;
- return null;
- }
-
- private connectionPlacements(conn: LocalConnection): Record<string, string> {
- const placements: Record<string, string> = {};
- for (const [sessionId, placed] of this.agent(conn.agentId).placements)
- if (placed.owner === conn.id) placements[sessionId] = placed.roomId;
- return placements;
- }
-
- private sessionIn(agent: AgentState, roomId: string): string | null {
- for (const [sessionId, placed] of agent.placements)
- if (placed.roomId === roomId) return sessionId;
- return null;
- }
-
- /**
- * The room a forwarded call is made in: the calling session's placement when
- * the call names a session, otherwise the single room of the connection it
- * names. Null when neither says, and Switch answers as it does for a caller
- * bound to no room.
- */
- roomFor(agentId: string, req: Pick<IncomingMessage, 'headers'>): string | null {
- const agent = this.agents.get(agentId);
- const sessionId = header(req, 'x-switch-session-id');
- if (!agent || agent.connections.size === 0) return this.deps.sharedRoomFor(agentId, sessionId);
- if (sessionId) return agent.placements.get(sessionId)?.roomId ?? null;
- const connectionId = header(req, 'x-switch-connection-id');
- const conn = connectionId ? agent.connections.get(connectionId) : undefined;
- if (conn && conn.rooms.size === 1) return [...conn.rooms][0]!;
- return null;
- }
-
- private covers(conn: LocalConnection, roomId: string): boolean {
- if (conn.rooms.has(roomId)) return true;
- if (conn.scope === 'single') return false;
- for (const sibling of this.agent(conn.agentId).connections.values())
- if (sibling !== conn && sibling.rooms.has(roomId) && this.alive(sibling)) return false;
- return true;
- }
-
- /** The scope-`all` connections with a stream that take frames of this revision. */
- private watchers(agent: AgentState, revision: number): LocalConnection[] {
- return [...agent.connections.values()].filter(
- (conn) =>
- conn.scope === 'all' &&
- conn.stream !== null &&
- this.alive(conn) &&
- (conn.declaration.speaks ?? 0) >= revision
- );
- }
-
- /** The lowest cursor any live connection has confirmed, never moving back. */
- private confirm(agent: AgentState): void {
- const confirmed = [...agent.connections.values()]
- .filter((conn) => conn.confirmed !== null && this.alive(conn))
- .map((conn) => conn.confirmed!);
- if (!confirmed.length) return;
- const lowest = Math.min(...confirmed);
- if (agent.confirmed !== null && lowest <= agent.confirmed) return;
- agent.confirmed = lowest;
- this.deps.onCursor(agent.agentId, lowest);
- }
-
- /** Drops what every live connection has confirmed, and the oldest past the limit. */
- private trim(agent: AgentState): void {
- const confirmed = [...agent.connections.values()]
- .filter((conn) => this.alive(conn))
- .map((conn) => conn.confirmed ?? -1);
- const floor = confirmed.length ? Math.min(...confirmed) : -1;
- let drop = 0;
- while (drop < agent.buffer.length && agent.buffer[drop]!.seq <= floor) drop++;
- let overflow = false;
- if (agent.buffer.length - drop > this.deps.bufferLimit) {
- drop = agent.buffer.length - this.deps.bufferLimit;
- overflow = true;
- }
- if (drop > 0) {
- agent.droppedThrough = Math.max(agent.droppedThrough, agent.buffer[drop - 1]!.seq);
- agent.buffer.splice(0, drop);
- }
- if (overflow && !agent.overflowing)
- this.deps.log.warn(
- 'The relay is holding more events than it keeps for an agent whose watcher is not reading; the oldest are dropped, and the watcher is told when it reconnects',
- { agentId: agent.agentId, limit: this.deps.bufferLimit }
- );
- agent.overflowing = overflow;
- }
-
- // -- Writing ----------------------------------------------------------------
-
- private write(conn: LocalConnection, event: string, data: unknown, id?: number): void {
- const stream = conn.stream;
- if (!stream) return;
- const lines = id === undefined ? '' : `id: ${id}\n`;
- stream.res.write(`${lines}event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);
- }
-
- private pumpAll(agent: AgentState): void {
- for (const conn of agent.connections.values()) this.pump(conn);
- }
-
- /** Writes what the connection is owed: room changes, then every event past its cursor. */
- private pump(conn: LocalConnection): void {
- const stream = conn.stream;
- if (!stream) return;
- for (const [roomId, sessionId] of conn.released)
- this.write(conn, 'room_released', { room_id: roomId, session_id: sessionId });
- conn.released.clear();
- if (conn.rooms.size !== stream.told.size || [...conn.rooms].some((r) => !stream.told.has(r))) {
- stream.told = new Set(conn.rooms);
- this.write(conn, 'subscription_changed', {
- rooms: [...conn.rooms].sort(),
- reason: 'subscription updated',
- });
- }
- // A session connection with no room yet must not read: it would skip,
- // and so lose, every event before the room it is about to claim.
- if (conn.cursor === null || (conn.scope === 'single' && conn.rooms.size === 0)) return;
- const agent = this.agent(conn.agentId);
- if (conn.cursor < agent.droppedThrough) {
- this.write(conn, 'gap', {
- from_sequence: conn.cursor,
- resumed_at: agent.droppedThrough,
- rooms: [...conn.rooms].sort(),
- all_rooms: true,
- reason: 'the agents controller no longer holds events this far back; re-read room context',
- });
- conn.cursor = agent.droppedThrough;
- }
- for (let index = firstAfter(agent.buffer, conn.cursor); index < agent.buffer.length; index++) {
- const item = agent.buffer[index]!;
- if (item.kind === 'gap')
- this.write(conn, 'gap', { ...item.data, from_sequence: conn.cursor });
- else if (this.covers(conn, item.roomId) && (conn.filter === 'all' || item.notifiable))
- this.write(conn, item.type, item.data, item.seq);
- conn.cursor = item.seq;
- }
- }
-
- private evict(conn: LocalConnection, code: string, reason: string): void {
- const stream = conn.stream;
- if (!stream) return;
- this.write(conn, 'evicted', { code, reason, room_id: null });
- conn.stream = null;
- stream.res.end();
- }
-
- /** Ends a connection; with a code, its open stream is told why first. */
- private closeConnection(conn: LocalConnection, code: 'heartbeat_lapsed' | null): void {
- if (code)
- this.evict(conn, code, 'heartbeat lapsed; reopen the stream and resume from your cursor');
- else conn.stream?.res.end();
- conn.stream = null;
- this.connections.delete(conn.id);
- this.agents.get(conn.agentId)?.connections.delete(conn.id);
- this.deps.onChange();
- }
-
- private sweep(): void {
- for (const conn of [...this.connections.values()])
- if (!this.alive(conn)) {
- this.deps.log.debug('Closing a local connection whose heartbeat lapsed', {
- agentId: conn.agentId,
- connectionId: conn.id,
- });
- this.closeConnection(conn, 'heartbeat_lapsed');
- }
- }
-
- private keepalive(): void {
- for (const conn of this.connections.values()) conn.stream?.res.write(': keepalive\n\n');
- }
-
private send(res: ServerResponse, status: number, body: unknown): void {
const text = JSON.stringify(body);
res.writeHead(status, {
@@ -1157,59 +265,3 @@ function header(req: Pick<IncomingMessage, 'headers'>, name: string): string | n
const text = Array.isArray(value) ? value[0] : value;
return text ? text : null;
}
-
-function declarationOf(query: URLSearchParams): Declaration {
- const integer = (name: string) => {
- const raw = query.get(name);
- if (raw === null) return null;
- const value = Number(raw);
- return Number.isSafeInteger(value) ? value : null;
- };
- const speaks = integer('protocol');
- return {
- speaks,
- accepts: integer('protocol_accepts') ?? speaks,
- artifact: query.get('client'),
- version: query.get('client_version'),
- };
-}
-
-/** The index of the first buffered item past `cursor`. */
-function firstAfter(buffer: Buffered[], cursor: number): number {
- let low = 0;
- let high = buffer.length;
- while (low < high) {
- const middle = (low + high) >> 1;
- if (buffer[middle]!.seq <= cursor) low = middle + 1;
- else high = middle;
- }
- return low;
-}
-
-async function readJson(req: IncomingMessage): Promise<unknown> {
- const chunks: Buffer[] = [];
- let size = 0;
- for await (const chunk of req) {
- size += (chunk as Buffer).length;
- if (size > MAX_JSON_BODY_BYTES) throw new HttpRefusal(413, 'Request body too large.');
- chunks.push(chunk as Buffer);
- }
- try {
- return JSON.parse(Buffer.concat(chunks).toString('utf8'));
- } catch {
- throw new HttpRefusal(400, 'The request body is not JSON.');
- }
-}
-
-function parse<S extends z.ZodType>(schema: S, body: unknown): z.infer<S> {
- const result = schema.safeParse(body);
- if (!result.success)
- throw new HttpRefusal(
- 422,
- result.error.issues.map((issue) => ({
- loc: ['body', ...issue.path.map(String)],
- msg: issue.message,
- }))
- );
- return result.data;
-}
diff --git a/console/packages/agent-controller/src/runtime.test.ts b/console/packages/agent-controller/src/runtime.test.ts
index 808b39d89..dc14be23f 100644
--- a/console/packages/agent-controller/src/runtime.test.ts
+++ b/console/packages/agent-controller/src/runtime.test.ts
@@ -28,7 +28,11 @@ if (args[0] === '--probe') {
}
`;
-const RELAY = { endpoint: 'http://127.0.0.1:43210', token: 'swlr_relay-token-placeholder' };
+const RELAY = {
+ endpoint: 'http://127.0.0.1:43210',
+ token: 'swlr_relay-token-placeholder',
+ hub: 'ws://127.0.0.1:43210/hub',
+};
const LAUNCH = {
isolation: 'shared' as const,
restart: false,
@@ -128,10 +132,7 @@ async function fakeWatcher(root: string): Promise<ChildProcess> {
describe('InProcessRuntime', () => {
it('writes credentials the shared host reads, owner-only, outside the working directory', async () => {
- await runtime.writeCredentials('agent-1', {
- endpoint: 'http://127.0.0.1:43210',
- token: 'swlr_relay-token-placeholder',
- });
+ await runtime.writeCredentials('agent-1', RELAY);
const path = runtime.credentialsPath('agent-1');
expect(path).toBe(join(dir, 'data', 'agents', 'agent-1', 'credentials.json'));
expect(statSync(path).mode & 0o777).toBe(0o600);
@@ -140,11 +141,9 @@ describe('InProcessRuntime', () => {
SWITCH_API_ENDPOINT: 'http://127.0.0.1:43210',
SWITCH_API_TOKEN: 'swlr_relay-token-placeholder',
SWITCH_AGENT_ID: 'agent-1',
+ SWITCH_AGENT_HUB: 'ws://127.0.0.1:43210/hub',
});
- expect(await runtime.readCredentials('agent-1')).toEqual({
- endpoint: 'http://127.0.0.1:43210',
- token: 'swlr_relay-token-placeholder',
- });
+ expect(await runtime.readCredentials('agent-1')).toEqual(RELAY);
await runtime.deleteCredentials('agent-1');
await runtime.deleteCredentials('agent-1');
expect(await runtime.readCredentials('agent-1')).toBeNull();
diff --git a/console/packages/agent-controller/src/runtime.ts b/console/packages/agent-controller/src/runtime.ts
index c815c4b73..f903cfb96 100644
--- a/console/packages/agent-controller/src/runtime.ts
+++ b/console/packages/agent-controller/src/runtime.ts
@@ -76,8 +76,11 @@ export type LaunchOptions = {
clearTakenOver: boolean;
};
-/** What an agent host reads to reach Switch: the controller's relay, and a token for it. */
-export type RelayCredentials = { endpoint: string; token: string };
+/**
+ * What an agent host reads to reach Switch: the controller's relay, a token
+ * for it, and the hub an agent host in a process of its own hears its events on.
+ */
+export type RelayCredentials = { endpoint: string; token: string; hub: string };
/** Runs agent hosts one way: in this controller's process, or each in a process of its own. */
export interface AgentRunner {
@@ -286,8 +289,12 @@ export class InProcessRuntime implements AgentRuntime {
const env = (JSON.parse(text) as { env?: Record<string, unknown> }).env ?? {};
const endpoint = env.SWITCH_API_ENDPOINT;
const token = env.SWITCH_API_TOKEN;
+ // Written before the hub: an empty one, which no relay names, so the file is written again.
+ const hub = typeof env.SWITCH_AGENT_HUB === 'string' ? env.SWITCH_AGENT_HUB : '';
if (env.SWITCH_AGENT_ID !== agentId) return null;
- return typeof endpoint === 'string' && typeof token === 'string' ? { endpoint, token } : null;
+ return typeof endpoint === 'string' && typeof token === 'string'
+ ? { endpoint, token, hub }
+ : null;
} catch {
return null;
}
@@ -302,6 +309,7 @@ export class InProcessRuntime implements AgentRuntime {
SWITCH_API_ENDPOINT: credentials.endpoint,
SWITCH_API_TOKEN: credentials.token,
SWITCH_AGENT_ID: agentId,
+ SWITCH_AGENT_HUB: credentials.hub,
},
})
);
diff --git a/console/packages/agent-controller/src/schemas.test.ts b/console/packages/agent-controller/src/schemas.test.ts
index 08c967515..0b1b20b56 100644
--- a/console/packages/agent-controller/src/schemas.test.ts
+++ b/console/packages/agent-controller/src/schemas.test.ts
@@ -4,9 +4,8 @@ import { describe, expect, it } from 'vitest';
import { z } from 'zod';
import {
assignmentSchema,
- controllerBeatRequestSchema,
- controllerBeatResponseSchema,
controllerConnectionRequestSchema,
+ controllerPongSchema,
controllerConnectionResponseSchema,
credentialRotateResponseSchema,
enrollRequestSchema,
@@ -46,8 +45,7 @@ const FIXTURES = join(
*/
const SCHEMA_FOR: Record<string, z.ZodType> = {
'assignment_response.json': assignmentSchema,
- 'controller_beat_request.json': controllerBeatRequestSchema,
- 'controller_beat_response.json': controllerBeatResponseSchema,
+ 'controller_pong.json': controllerPongSchema,
'controller_connection_request.json': controllerConnectionRequestSchema,
'controller_connection_response.json': controllerConnectionResponseSchema,
'credential_rotate_response.json': credentialRotateResponseSchema,
@@ -178,7 +176,7 @@ describe('status report', () => {
const report = {
seq: 1,
observed_at: '2026-01-01T00:00:00Z',
- controller: { version: '0.1.0', protocol: 1, assignment_revision: 0 },
+ controller: { version: '0.1.0', protocol: 2, assignment_revision: 0 },
machine: {
platform: { os: 'linux', arch: 'x64', os_version: '6.1' },
disk_free_bytes: 1,
diff --git a/console/packages/agent-controller/src/schemas.ts b/console/packages/agent-controller/src/schemas.ts
index 9254fa2cb..a4f4571b4 100644
--- a/console/packages/agent-controller/src/schemas.ts
+++ b/console/packages/agent-controller/src/schemas.ts
@@ -11,7 +11,7 @@ import { z } from 'zod';
* known enum is read as `unknown` rather than failing the whole message.
*/
-export const PROTOCOL_VERSION = 1;
+export const PROTOCOL_VERSION = 2;
export const PROVIDERS = ['claude', 'codex', 'opencode', 'antigravity', 'cursor'] as const;
export type Provider = (typeof PROVIDERS)[number];
@@ -295,14 +295,12 @@ export const controllerConnectionResponseSchema = z.object({
});
export type ControllerConnection = z.infer<typeof controllerConnectionResponseSchema>;
-export const controllerBeatRequestSchema = z.object({
- connection_id: id,
- generation: z.number().int(),
+/** The controller's answer to each `ping`: its beat, with how far each agent's host has read. */
+export const controllerPongSchema = z.object({
+ type: z.literal('pong'),
cursors: z.record(z.string().min(1), sequence),
});
-export type ControllerBeatRequest = z.infer<typeof controllerBeatRequestSchema>;
-
-export const controllerBeatResponseSchema = z.object({ agents: z.array(id) });
+export type ControllerPong = z.infer<typeof controllerPongSchema>;
/** The stream's first frame. */
export const connectionStateSchema = z.object({
diff --git a/console/packages/agent-controller/src/status.test.ts b/console/packages/agent-controller/src/status.test.ts
index 4a98b69f9..79a04f548 100644
--- a/console/packages/agent-controller/src/status.test.ts
+++ b/console/packages/agent-controller/src/status.test.ts
@@ -315,7 +315,7 @@ describe('StatusCollector', () => {
const assignment = { revision: 7, agents: [entry()] };
const first = await collector.collect(assignment);
expect(statusReportSchema.safeParse({ ...first, seq: 1 }).success).toBe(true);
- expect(first.controller).toEqual({ version: '0.1.0', protocol: 1, assignment_revision: 7 });
+ expect(first.controller).toEqual({ version: '0.1.0', protocol: 2, assignment_revision: 7 });
expect(first.providers.map((p) => p.provider)).toEqual(['claude']);
expect(first.agents[0]).toMatchObject({
agent_id: 'agent-1',
diff --git a/console/packages/agent-controller/src/stream.test.ts b/console/packages/agent-controller/src/stream.test.ts
index c3b973c36..05441d721 100644
--- a/console/packages/agent-controller/src/stream.test.ts
+++ b/console/packages/agent-controller/src/stream.test.ts
@@ -2,61 +2,72 @@ import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
-import { AccessTokens, ControllerApiError, ControllerClient } from './api';
+import {
+ AccessTokens,
+ ControllerApiError,
+ ControllerClient,
+ type OpenedSocket,
+ type SocketLike,
+} from './api';
import { DEFAULT_TIMING } from './controller';
import { silentLogger } from './log';
import type { AgentCursor, ControllerConnection } from './schemas';
-import {
- type ControllerFrame,
- type ControllerStreamOptions,
- parseSse,
- runControllerStream,
- type SseItem,
-} from './stream';
+import { type ControllerFrame, type ControllerStreamOptions, runControllerStream } from './stream';
import { FakeCore } from './testing/fake-core';
-function streamOf(chunks: string[], keepOpen = false): ReadableStream<Uint8Array> {
- const encoder = new TextEncoder();
- return new ReadableStream({
- start(controller) {
- for (const chunk of chunks) controller.enqueue(encoder.encode(chunk));
- if (!keepOpen) controller.close();
- },
- });
-}
+type Message = { event: string; data: unknown };
+
+/**
+ * A socket that opens, delivers `messages` one per turn of the event loop,
+ * then closes with `end` (or stays open). What the controller sends is kept
+ * in `sent`, and `deliver` adds more as the test goes.
+ */
+class ScriptedSocket extends EventTarget {
+ readyState = 0;
+ readonly sent: unknown[] = [];
+
+ constructor(messages: (Message | string)[], end: number | 'open') {
+ super();
+ setImmediate(() => {
+ this.readyState = 1;
+ this.dispatchEvent(new Event('open'));
+ for (const message of messages) this.deliver(message);
+ if (end !== 'open') this.end(end);
+ });
+ }
-async function collect(body: ReadableStream<Uint8Array>): Promise<SseItem[]> {
- const items: SseItem[] = [];
- for await (const item of parseSse(body)) items.push(item);
- return items;
-}
+ deliver(message: Message | string): void {
+ const data = typeof message === 'string' ? message : JSON.stringify(message);
+ setImmediate(() => {
+ if (this.readyState !== 3) this.dispatchEvent(Object.assign(new Event('message'), { data }));
+ });
+ }
-describe('parseSse', () => {
- it('reads events, comments and multi-line data', async () => {
- expect(
- await collect(
- streamOf([': keepalive\n\n', 'event: a\ndata: {"x":1}\n\n', 'data: line1\ndata: line2\n\n'])
- )
- ).toEqual([
- { kind: 'comment', text: 'keepalive' },
- { kind: 'event', event: 'a', data: '{"x":1}', id: null },
- { kind: 'event', event: 'message', data: 'line1\nline2', id: null },
- ]);
- });
+ end(code: number): void {
+ setImmediate(() => {
+ if (this.readyState === 3) return;
+ this.readyState = 3;
+ this.dispatchEvent(Object.assign(new Event('close'), { code }));
+ });
+ }
- it('handles CRLF, fields without a space, and frames split across chunks', async () => {
- expect(
- await collect(streamOf(['event:b\r', '\ndata:{"y"', ':2}\r\nid: 7\r\n', '\r\n']))
- ).toEqual([{ kind: 'event', event: 'b', data: '{"y":2}', id: '7' }]);
- });
+ send(data: string): void {
+ this.sent.push(JSON.parse(data));
+ }
- it('drops a trailing frame the stream never finished', async () => {
- expect(await collect(streamOf(['event: c\ndata: {}\n']))).toEqual([]);
- });
-});
+ close(code = 1000): void {
+ this.end(code);
+ }
+}
+
+function scriptedSocket(messages: (Message | string)[], end: number | 'open'): ScriptedSocket {
+ return new ScriptedSocket(messages, end);
+}
+
+const state: Message = { event: 'connection_state', data: { connection_id: 'connection-1' } };
-function sse(event: string, data: unknown): string {
- return `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`;
+function refused(status: number, code: string): Message {
+ return { event: 'refused', data: { status, detail: { code, message: code } } };
}
async function waitFor(condition: () => boolean, what: string, timeoutMs = 15_000): Promise<void> {
@@ -78,23 +89,24 @@ const opened: ControllerConnection = {
class ScriptedClient {
opens: Record<string, AgentCursor>[] = [];
attaches: { connectionId: string; generation: number }[] = [];
- beats: Record<string, number>[] = [];
+ opened: ScriptedSocket[] = [];
+ staleTokens = 0;
open: () => Promise<ControllerConnection> = async () => opened;
- events: () => Promise<Response> = async () => new Response(streamOf([], true));
- beat: () => Promise<void> = async () => {};
+ sockets: () => ScriptedSocket = () => scriptedSocket([], 'open');
readonly client: ControllerStreamOptions['client'] = {
openConnection: async (cursors) => {
this.opens.push(cursors);
return this.open();
},
- openEvents: async (connection) => {
+ openSocket: async (connection): Promise<OpenedSocket> => {
this.attaches.push(connection);
- return this.events();
- },
- beat: async (_connection, cursors) => {
- this.beats.push(cursors);
- return this.beat();
+ const socket = this.sockets();
+ this.opened.push(socket);
+ return {
+ socket: socket as unknown as SocketLike,
+ tokenRefused: () => void this.staleTokens++,
+ };
},
};
}
@@ -128,15 +140,20 @@ function run(
describe('runControllerStream', () => {
it('opens with the cursors, attaches, and hands over typed frames in order, skipping bad ones', async () => {
const scripted = new ScriptedClient();
- scripted.events = async () =>
- new Response(
- streamOf(
- [
- ': ping\n\n',
- sse('agent.attached', { agent_id: 'agent-1', from_seq: 7, rooms: ['room-a'] }),
- sse('agent.event', { agent_id: 'agent-1' }),
- sse('agent.unknown', { agent_id: 'agent-1' }),
- sse('agent.event', {
+ scripted.sockets = () =>
+ scriptedSocket(
+ [
+ { event: 'ping', data: {} },
+ 'not json',
+ {
+ event: 'agent.attached',
+ data: { agent_id: 'agent-1', from_seq: 7, rooms: ['room-a'] },
+ },
+ { event: 'agent.event', data: { agent_id: 'agent-1' } },
+ { event: 'agent.unknown', data: { agent_id: 'agent-1' } },
+ {
+ event: 'agent.event',
+ data: {
agent_id: 'agent-1',
seq: 8,
event: {
@@ -146,16 +163,15 @@ describe('runControllerStream', () => {
payload: { addressed: true },
future: 1,
},
- }),
- sse('assignment.changed', { revision: 2 }),
- sse('operation.pending', {
- operation_id: 'op-1',
- kind: 'agent.restart',
- agent_id: null,
- }),
- ],
- true
- )
+ },
+ },
+ { event: 'assignment.changed', data: { revision: 2 } },
+ {
+ event: 'operation.pending',
+ data: { operation_id: 'op-1', kind: 'agent.restart', agent_id: null },
+ },
+ ],
+ 'open'
);
const stop = new AbortController();
const { frames, states, ending } = run(scripted, stop.signal);
@@ -174,19 +190,21 @@ describe('runControllerStream', () => {
expect(frames[1]!.data).toMatchObject({ seq: 8, event: { future: 1 } });
});
- it('beats the connection with the confirmed cursors while attached', async () => {
+ it('answers every ping with a pong naming the confirmed cursors: its beat', async () => {
const scripted = new ScriptedClient();
+ const ping = { event: 'ping', data: {} };
+ scripted.sockets = () => scriptedSocket([state, ping, ping], 'open');
const stop = new AbortController();
const { ending } = run(scripted, stop.signal);
- await waitFor(() => scripted.beats.length >= 2, 'two beats');
+ await waitFor(() => (scripted.opened[0]?.sent.length ?? 0) >= 2, 'two pongs');
stop.abort();
await ending;
- expect(scripted.beats[0]).toEqual({ 'agent-1': 7 });
+ expect(scripted.opened[0]!.sent[0]).toEqual({ type: 'pong', cursors: { 'agent-1': 7 } });
});
- it('reattaches to the same connection after the stream ends', async () => {
+ it('reattaches to the same connection after the socket closes', async () => {
const scripted = new ScriptedClient();
- scripted.events = async () => new Response(streamOf([': ping\n\n']));
+ scripted.sockets = () => scriptedSocket([state], 1000);
const stop = new AbortController();
const { states, ending } = run(scripted, stop.signal);
await waitFor(() => scripted.attaches.length >= 3, 'three attaches');
@@ -199,6 +217,18 @@ describe('runControllerStream', () => {
expect(states).toContain('disconnected');
});
+ it('comes back within a second, without backing off, when the server is restarting', async () => {
+ const scripted = new ScriptedClient();
+ scripted.sockets = () => scriptedSocket([state], 1012);
+ const stop = new AbortController();
+ const started = Date.now();
+ const { ending } = run(scripted, stop.signal, { initialBackoffMs: 60_000 });
+ await waitFor(() => scripted.attaches.length >= 3, 'three attaches', 3_000);
+ stop.abort();
+ await ending;
+ expect(Date.now() - started).toBeGreaterThanOrEqual(490);
+ });
+
it('reconnects when nothing arrives within the idle timeout', async () => {
const scripted = new ScriptedClient();
const started = Date.now();
@@ -212,13 +242,11 @@ describe('runControllerStream', () => {
it('opens a new connection, from the cursors as they stand, when Switch no longer knows it', async () => {
const scripted = new ScriptedClient();
- let attaches = 0;
- scripted.events = async () => {
- attaches++;
- if (attaches === 2)
- throw new ControllerApiError(404, 'unknown_connection', 'gone', false, null);
- return new Response(streamOf([]));
- };
+ scripted.sockets = () =>
+ scriptedSocket(
+ scripted.attaches.length === 2 ? [refused(404, 'unknown_connection')] : [state],
+ scripted.attaches.length === 2 ? 4404 : 1000
+ );
let cursor = 7;
const stop = new AbortController();
const { ending } = run(scripted, stop.signal, {
@@ -230,13 +258,9 @@ describe('runControllerStream', () => {
expect(scripted.opens).toEqual([{ 'agent-1': 7 }, { 'agent-1': 8 }]);
});
- it('opens a new connection when the heartbeat is refused as stale', async () => {
+ it('opens a new connection when an attach is refused as stale', async () => {
const scripted = new ScriptedClient();
- let beats = 0;
- scripted.beat = async () => {
- beats++;
- if (beats === 1) throw new ControllerApiError(409, 'stale_generation', 'old', false, null);
- };
+ scripted.sockets = () => scriptedSocket([refused(409, 'stale_generation')], 4409);
const stop = new AbortController();
const { ending } = run(scripted, stop.signal);
await waitFor(() => scripted.opens.length === 2, 'a second open');
@@ -244,11 +268,12 @@ describe('runControllerStream', () => {
await ending;
});
- it('opens a new connection when Switch ends the stream for any reason but a takeover', async () => {
+ it('opens a new connection when Switch evicts it for any reason but a takeover', async () => {
const scripted = new ScriptedClient();
- scripted.events = async () =>
- new Response(
- streamOf([sse('evicted', { code: 'heartbeat_lapsed', reason: 'lapsed' })], true)
+ scripted.sockets = () =>
+ scriptedSocket(
+ [state, { event: 'evicted', data: { code: 'no_stream', reason: 'beat refused' } }],
+ 1000
);
const stop = new AbortController();
const { ending } = run(scripted, stop.signal);
@@ -259,39 +284,50 @@ describe('runControllerStream', () => {
it('stops for good when another instance took the connection over', async () => {
const evicted = new ScriptedClient();
- evicted.events = async () =>
- new Response(streamOf([sse('evicted', { code: 'taken_over', reason: 'x' })], true));
+ evicted.sockets = () =>
+ scriptedSocket(
+ [state, { event: 'evicted', data: { code: 'taken_over', reason: 'x' } }],
+ 1000
+ );
expect(await run(evicted, new AbortController().signal).ending).toBe('taken_over');
- const scripted = new ScriptedClient();
- scripted.beat = async () => {
- throw new ControllerApiError(409, 'taken_over', 'someone else', false, null);
- };
- expect(await run(scripted, new AbortController().signal).ending).toBe('taken_over');
- const refused = new ScriptedClient();
- refused.events = async () => {
- throw new ControllerApiError(409, 'taken_over', 'someone else', false, null);
- };
- expect(await run(refused, new AbortController().signal).ending).toBe('taken_over');
+ const refusedAttach = new ScriptedClient();
+ refusedAttach.sockets = () => scriptedSocket([refused(409, 'taken_over')], 4409);
+ expect(await run(refusedAttach, new AbortController().signal).ending).toBe('taken_over');
});
- it('ends as revoked on the frame, on a refused open, and on a refused beat', async () => {
+ it('ends as revoked on the frame, on a refused open, and on a refused attach', async () => {
const framed = new ScriptedClient();
- framed.events = async () => new Response(streamOf([sse('credential.revoked', {})], true));
+ framed.sockets = () => scriptedSocket([{ event: 'credential.revoked', data: {} }], 'open');
const first = run(framed, new AbortController().signal);
expect(await first.ending).toBe('revoked');
expect(first.frames.map((frame) => frame.type)).toEqual(['credential.revoked']);
- const refused = new ScriptedClient();
- refused.open = async () => {
+ const refusedOpen = new ScriptedClient();
+ refusedOpen.open = async () => {
throw new ControllerApiError(401, 'controller_revoked', 'revoked', false, null);
};
- expect(await run(refused, new AbortController().signal).ending).toBe('revoked');
+ expect(await run(refusedOpen, new AbortController().signal).ending).toBe('revoked');
- const beaten = new ScriptedClient();
- beaten.beat = async () => {
- throw new ControllerApiError(401, 'controller_revoked', 'revoked', false, null);
- };
- expect(await run(beaten, new AbortController().signal).ending).toBe('revoked');
+ const refusedAttach = new ScriptedClient();
+ refusedAttach.sockets = () => scriptedSocket([refused(401, 'controller_revoked')], 4401);
+ const third = run(refusedAttach, new AbortController().signal);
+ expect(await third.ending).toBe('revoked');
+ expect(refusedAttach.staleTokens).toBe(0);
+ });
+
+ it('drops a token the socket refused as stale, and attaches again', async () => {
+ const scripted = new ScriptedClient();
+ scripted.sockets = () =>
+ scripted.attaches.length === 1
+ ? scriptedSocket([refused(401, 'token_expired')], 4401)
+ : scriptedSocket([state], 'open');
+ const stop = new AbortController();
+ const { states, ending } = run(scripted, stop.signal);
+ await waitFor(() => states.includes('connected'), 'a connection');
+ stop.abort();
+ await ending;
+ expect(scripted.staleTokens).toBe(1);
+ expect(scripted.opens).toHaveLength(1);
});
it('keeps retrying an open that fails', async () => {
@@ -302,6 +338,7 @@ describe('runControllerStream', () => {
if (opens < 3) throw new TypeError('fetch failed');
return opened;
};
+ scripted.sockets = () => scriptedSocket([state], 'open');
const stop = new AbortController();
const { states, ending } = run(scripted, stop.signal);
await waitFor(() => states.includes('connected'), 'a connection');
@@ -310,7 +347,7 @@ describe('runControllerStream', () => {
expect(opens).toBe(3);
});
- it('waits less each time a stream attached, however briefly, and never long', async () => {
+ it('waits less each time a socket attached, however briefly, and never long', async () => {
const scripted = new ScriptedClient();
let opens = 0;
scripted.open = async () => {
@@ -318,12 +355,8 @@ describe('runControllerStream', () => {
if (opens <= 3) throw new TypeError('fetch failed');
return opened;
};
- let attaches = 0;
- scripted.events = async () => {
- attaches++;
- if (attaches === 1) return new Response(streamOf([]));
- throw new TypeError('fetch failed');
- };
+ scripted.sockets = () =>
+ scripted.attaches.length === 1 ? scriptedSocket([state], 1000) : scriptedSocket([], 1006);
const waits: number[] = [];
const log = {
...silentLogger,
@@ -340,7 +373,7 @@ describe('runControllerStream', () => {
await waitFor(() => waits.length >= 5, 'five reconnect waits');
stop.abort();
await ending;
- // Doubling from the start after the stream that attached, capped, with
+ // Doubling from the start after the socket that attached, capped, with
// jitter taking half off (random() is 0 here).
expect(waits.slice(0, 5)).toEqual([4, 8, 16, 8, 16]);
});
@@ -370,13 +403,7 @@ describe('runControllerStream', () => {
data: unknown;
}[];
const scripted = new ScriptedClient();
- scripted.events = async () =>
- new Response(
- streamOf(
- fixtureFrames.map((frame) => sse(frame.event, frame.data)),
- true
- )
- );
+ scripted.sockets = () => scriptedSocket(fixtureFrames, 'open');
const stop = new AbortController();
const { frames, ending } = run(scripted, stop.signal);
await waitFor(
@@ -396,7 +423,7 @@ describe('runControllerStream', () => {
});
});
-describe('runControllerStream against the controller stream routes', () => {
+describe('runControllerStream against the controller routes', () => {
let core: FakeCore;
beforeEach(async () => {
@@ -414,6 +441,7 @@ describe('runControllerStream against the controller stream routes', () => {
server: core.url,
controllerId: core.controllerId,
version: '0.1.0',
+ openWebSocket: core.openWebSocket,
tokens: new AccessTokens({
fetch,
server: core.url,
@@ -425,7 +453,7 @@ describe('runControllerStream against the controller stream routes', () => {
});
}
- it('opens, reads, beats, reattaches, and reopens with cursors after the connection lapses', async () => {
+ it('opens, reads, pongs, reattaches, and reopens with cursors after the connection lapses', async () => {
let confirmed = 3;
const frames: ControllerFrame[] = [];
const stop = new AbortController();
@@ -439,20 +467,23 @@ describe('runControllerStream against the controller stream routes', () => {
onFrame: async (frame) => void frames.push(frame),
signal: stop.signal,
log: silentLogger,
- // Long enough that a loaded machine does not lapse the stream before the test does.
+ // Long enough that a loaded machine does not lapse the socket before the test does.
idleTimeoutMs: 30_000,
initialBackoffMs: 5,
maxBackoffMs: 20,
random: () => 0,
});
- await waitFor(() => core.streamCount === 1, 'the stream');
+ await waitFor(() => core.streamCount === 1, 'the socket');
core.push('assignment.changed', { revision: 4 });
- await waitFor(() => frames.length === 1, 'the frame');
- await waitFor(() => core.beats.length > 0, 'a beat');
+ await waitFor(
+ () => frames.some((frame) => frame.type === 'assignment.changed'),
+ 'the pushed frame'
+ );
+ await waitFor(() => core.beats.length > 0, 'a pong');
expect(core.beats[0]).toEqual({ 'agent-1': 3 });
core.closeStreams();
- await waitFor(() => core.streamCount === 1, 'the stream reattached');
+ await waitFor(() => core.streamCount === 1, 'the socket reattached');
expect(core.opens).toHaveLength(1);
confirmed = 9;
diff --git a/console/packages/agent-controller/src/stream.ts b/console/packages/agent-controller/src/stream.ts
index c845fc660..0285e15ab 100644
--- a/console/packages/agent-controller/src/stream.ts
+++ b/console/packages/agent-controller/src/stream.ts
@@ -1,6 +1,12 @@
import { setTimeout as delay } from 'node:timers/promises';
import type { z } from 'zod';
-import { ControllerApiError, type ControllerClient, isRevoked, isTakenOver } from './api';
+import {
+ ControllerApiError,
+ type ControllerClient,
+ isRevoked,
+ isTakenOver,
+ type OpenedSocket,
+} from './api';
import { errorMessage, type Logger } from './log';
import {
type AgentApprovalOutcomeFrame,
@@ -29,60 +35,7 @@ import {
operationPendingSchema,
} from './schemas';
-export type SseItem =
- | { kind: 'comment'; text: string }
- | { kind: 'event'; event: string; data: string; id: string | null };
-
-/**
- * Splits an SSE byte stream into events and comments. Comments are surfaced
- * rather than skipped because the server's `: keepalive` is how a silent but
- * healthy stream is told apart from a dead one.
- */
-export async function* parseSse(body: ReadableStream<Uint8Array>): AsyncGenerator<SseItem> {
- const reader = body.getReader();
- const decoder = new TextDecoder();
- let buffered = '';
- let event = '';
- let id: string | null = null;
- let data: string[] = [];
- try {
- for (;;) {
- const { done, value } = await reader.read();
- if (done) return;
- buffered += decoder.decode(value, { stream: true });
- let newline: number;
- while ((newline = buffered.search(/\r\n|\r|\n/)) !== -1) {
- const line = buffered.slice(0, newline);
- const width = buffered.startsWith('\r\n', newline) ? 2 : 1;
- // A lone CR at the end of the buffer may be the first half of a CRLF.
- if (width === 1 && buffered[newline] === '\r' && newline === buffered.length - 1) break;
- buffered = buffered.slice(newline + width);
- if (line === '') {
- if (data.length)
- yield { kind: 'event', event: event || 'message', data: data.join('\n'), id };
- event = '';
- data = [];
- continue;
- }
- if (line.startsWith(':')) {
- yield { kind: 'comment', text: line.slice(1).trimStart() };
- continue;
- }
- const colon = line.indexOf(':');
- const field = colon === -1 ? line : line.slice(0, colon);
- let value = colon === -1 ? '' : line.slice(colon + 1);
- if (value.startsWith(' ')) value = value.slice(1);
- if (field === 'event') event = value;
- else if (field === 'data') data.push(value);
- else if (field === 'id') id = value;
- }
- }
- } finally {
- void reader.cancel().catch(() => {});
- }
-}
-
-/** The schema for each SSE event type the controller stream carries, by `event:` name. */
+/** The schema for each frame the controller's socket carries, by its `event`. */
export const STREAM_FRAME_SCHEMAS = {
connection_state: connectionStateSchema,
evicted: evictedSchema,
@@ -116,10 +69,10 @@ export type ControllerFrame =
export type StreamEnding = 'stopped' | 'revoked' | 'taken_over';
export type ControllerStreamOptions = {
- client: Pick<ControllerClient, 'openConnection' | 'beat' | 'openEvents'>;
+ client: Pick<ControllerClient, 'openConnection' | 'openSocket'>;
/** Where each agent resumes when a connection is opened. */
cursors: () => Record<string, AgentCursor>;
- /** How far each agent's watcher has confirmed reading, sent on every beat. */
+ /** How far each agent's watcher has confirmed reading, sent with every pong. */
confirmed: () => Record<string, number>;
/** A connection was opened: Core attached these agents to it. */
onOpened: (connection: ControllerConnection) => Promise<void> | void;
@@ -131,7 +84,7 @@ export type ControllerStreamOptions = {
onFrame: (frame: ControllerFrame) => Promise<void>;
signal: AbortSignal;
log: Logger;
- /** No byte for this long, keepalives included, and the stream is presumed dead. */
+ /** Nothing on the socket for this long, pings included, and it is presumed dead. */
idleTimeoutMs: number;
/** The first reconnect wait, and where the wait returns to once a stream attaches. */
initialBackoffMs: number;
@@ -141,9 +94,18 @@ export type ControllerStreamOptions = {
random: () => number;
};
+/**
+ * The close code a server sends as it begins shutting down (uvicorn sends it
+ * to every socket). The server is about to be back, so the reconnect waits
+ * are short and do not grow.
+ */
+const SERVICE_RESTART = 1012;
+const RESTART_RETRY_MS = 250;
+const RESTART_RETRY_JITTER_MS = 500;
+
class IdleTimeout extends Error {
constructor(ms: number) {
- super(`No data on the event stream for ${Math.round(ms / 1000)} s.`);
+ super(`Nothing on the controller socket for ${Math.round(ms / 1000)} s.`);
}
}
@@ -153,35 +115,40 @@ function connectionGone(error: unknown): boolean {
error instanceof ControllerApiError &&
(error.status === 404 ||
error.code === 'unknown_connection' ||
- error.code === 'stale_generation')
+ error.code === 'stale_generation' ||
+ error.code === 'no_stream')
);
}
-type Held = { connectionId: string; generation: number; heartbeatMs: number };
+type Held = { connectionId: string; generation: number };
-/** One attempt at attaching the stream: its socket, and what it learned. */
+/** One attempt at attaching the socket, and what it learned. */
type Attempt = {
- socket: AbortController;
+ abort: AbortController;
/** Set when the stream stopped for good. */
ending: StreamEnding | null;
/** When the stream attached, or 0 if it never did. */
attachedAt: number;
- beating: Promise<void> | null;
+ /** The server closed the socket because it is restarting. */
+ restarting: boolean;
};
+type Message = { event: string; data: unknown };
+
/**
- * The controller stream, held open for as long as `signal` lives.
+ * The controller's connection to Switch, held open for as long as `signal`
+ * lives.
*
* A connection is opened once (`POST .../connection`, with every agent's
- * cursor) and the stream attached to it (`GET .../events`). A dropped socket
- * reattaches to the same connection and generation, so Core resumes where it
- * was; a connection Core no longer knows (its 6 s heartbeat lapsed, or it was
- * superseded) is opened afresh, from the cursors as they stand. While the
- * stream is attached the connection is beaten every `heartbeat_interval_s`
- * with each agent's confirmed cursor and the rooms its sessions work in, so a
- * placement made locally reaches Switch on the next beat.
+ * cursor) and a WebSocket attached to it (`.../connection/ws`). The socket
+ * carries the stream's frames down, and the server's `ping` every heartbeat
+ * interval, which is answered with a `pong` naming each agent's confirmed
+ * cursor: that pong is the controller's beat. A dropped socket reattaches to
+ * the same connection and generation, so Core resumes where it was; a
+ * connection Core no longer knows (its heartbeat lapsed, or it was
+ * superseded) is opened afresh, from the cursors as they stand.
*
- * Ends with `'revoked'` on `credential.revoked` or a request refused as
+ * Ends with `'revoked'` on `credential.revoked` or a refusal saying
* `controller_revoked`, with `'taken_over'` when another instance of this
* controller took the connection over (an `evicted` frame or a refusal saying
* so: reopening would take it straight back, so this one stops), and with
@@ -203,12 +170,12 @@ class ControllerStream {
const { signal, log } = this.options;
while (!signal.aborted) {
const attempt: Attempt = {
- socket: new AbortController(),
+ abort: new AbortController(),
ending: null,
attachedAt: 0,
- beating: null,
+ restarting: false,
};
- const stop = () => attempt.socket.abort();
+ const stop = () => attempt.abort.abort();
signal.addEventListener('abort', stop, { once: true });
let failure: string | null = null;
try {
@@ -218,31 +185,32 @@ class ControllerStream {
if (isRevoked(error)) attempt.ending = 'revoked';
else if (isTakenOver(error)) attempt.ending = 'taken_over';
else {
- const reason = attempt.socket.signal.reason;
+ const reason = attempt.abort.signal.reason;
failure = errorMessage(reason instanceof IdleTimeout ? reason : error);
}
}
} finally {
signal.removeEventListener('abort', stop);
- attempt.socket.abort();
- await attempt.beating;
+ attempt.abort.abort();
}
if (attempt.ending) return attempt.ending;
if (signal.aborted) break;
if (attempt.attachedAt > 0) {
- this.options.onDisconnected(failure ?? 'the stream ended');
- // Every agent on this controller is offline until the stream is back,
- // so a stream that attached at all starts the waits over.
+ this.options.onDisconnected(failure ?? 'the socket closed');
+ // Every agent on this controller is offline until the socket is back,
+ // so a socket that attached at all starts the waits over.
this.backoff = this.options.initialBackoffMs;
}
- const wait = Math.round(this.backoff * (0.5 + this.options.random() * 0.5));
+ const wait = attempt.restarting
+ ? RESTART_RETRY_MS + Math.round(this.options.random() * RESTART_RETRY_JITTER_MS)
+ : Math.round(this.backoff * (0.5 + this.options.random() * 0.5));
if (failure !== null)
- log.warn('The controller stream is down; reconnecting.', {
+ log.warn('The controller socket is down; reconnecting.', {
error: failure,
retryInMs: wait,
});
await delay(wait, undefined, { signal }).catch(() => {});
- this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs);
+ if (!attempt.restarting) this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs);
}
return 'stopped';
}
@@ -250,13 +218,9 @@ class ControllerStream {
private async attach(attempt: Attempt): Promise<void> {
const { client, log } = this.options;
if (!this.held) {
- const opened = await client.openConnection(this.options.cursors(), attempt.socket.signal);
- this.held = {
- connectionId: opened.connection_id,
- generation: opened.generation,
- heartbeatMs: opened.heartbeat_interval_s * 1000,
- };
- log.info('Opened the controller stream connection', {
+ const opened = await client.openConnection(this.options.cursors(), attempt.abort.signal);
+ this.held = { connectionId: opened.connection_id, generation: opened.generation };
+ log.info('Opened the controller connection', {
connectionId: opened.connection_id,
generation: opened.generation,
agents: opened.agents.length,
@@ -264,135 +228,160 @@ class ControllerStream {
await this.options.onOpened(opened);
}
const current = this.held;
- let response: Response;
try {
- response = await client.openEvents(current, attempt.socket.signal);
+ await this.read(await client.openSocket(current), attempt);
} catch (error) {
if (!connectionGone(error)) throw error;
- log.warn('Switch no longer knows the stream connection; opening a new one.', {
+ log.warn('Switch no longer knows the controller connection; opening a new one.', {
error: errorMessage(error),
});
- this.held = null;
- return;
+ if (this.held === current) this.held = null;
}
- attempt.attachedAt = Date.now();
- this.options.onConnected();
- attempt.beating = this.beat(current, attempt);
- await this.read(response.body!, attempt);
}
- private async read(body: ReadableStream<Uint8Array>, attempt: Attempt): Promise<void> {
- const { socket } = attempt;
+ /**
+ * Reads the socket until it closes: answers each ping with the confirmed
+ * cursors and hands every other frame on, in order. A refusal is thrown as
+ * the error an HTTP request would have raised.
+ */
+ private async read(opened: OpenedSocket, attempt: Attempt): Promise<void> {
+ const { socket } = opened;
+ const abort = attempt.abort;
+ const pending: Message[] = [];
+ let refused: { status: number; code: string; message: string } | null = null;
+ let closedWith: number | null = null;
+ let wake: (() => void) | null = null;
+ const notify = () => {
+ const resolve = wake;
+ wake = null;
+ resolve?.();
+ };
let idle: ReturnType<typeof setTimeout> | null = null;
const touch = () => {
if (idle) clearTimeout(idle);
idle = setTimeout(
- () => socket.abort(new IdleTimeout(this.options.idleTimeoutMs)),
+ () => abort.abort(new IdleTimeout(this.options.idleTimeoutMs)),
this.options.idleTimeoutMs
);
};
- const aborted = new Promise<never>((_resolve, reject) => {
- const fail = () => reject(socket.signal.reason ?? new Error('aborted'));
- if (socket.signal.aborted) fail();
- socket.signal.addEventListener('abort', fail, { once: true });
- });
- aborted.catch(() => {});
+ const onMessage = (message: MessageEvent) => {
+ touch();
+ let parsed: { event?: unknown; data?: unknown };
+ try {
+ parsed = JSON.parse(String(message.data)) as typeof parsed;
+ } catch {
+ this.options.log.error('Dropped a controller socket message that is not JSON');
+ return;
+ }
+ if (parsed.event === 'ping') {
+ socket.send(JSON.stringify({ type: 'pong', cursors: this.options.confirmed() }));
+ return;
+ }
+ if (parsed.event === 'refused') {
+ refused = refusalOf(parsed.data);
+ return;
+ }
+ if (typeof parsed.event !== 'string') return;
+ pending.push({ event: parsed.event, data: parsed.data });
+ notify();
+ };
+ const onClose = (event: CloseEvent) => {
+ closedWith = event.code;
+ if (event.code === SERVICE_RESTART) attempt.restarting = true;
+ notify();
+ };
+ const onAbort = () => notify();
+ socket.addEventListener('message', onMessage);
+ socket.addEventListener('close', onClose);
+ socket.addEventListener('error', notify);
+ abort.signal.addEventListener('abort', onAbort, { once: true });
try {
touch();
- const items = parseSse(body);
for (;;) {
- const next = await Promise.race([items.next(), aborted]);
- if (next.done) break;
- touch();
- const item = next.value;
- if (item.kind === 'comment') continue;
- const frame = parseFrame(item, this.options.log);
- if (!frame) continue;
- await this.options.onFrame(frame);
- if (frame.type === 'credential.revoked') {
- attempt.ending = 'revoked';
- return;
- }
- if (frame.type === 'evicted') {
- if (frame.data.code === 'taken_over') {
- this.options.log.error(
- 'Another instance of this controller took its stream over; this one stops. Run one controller per data directory.'
- );
- attempt.ending = 'taken_over';
- return;
+ while (pending.length > 0) {
+ const message = pending.shift() as Message;
+ if (attempt.attachedAt === 0) {
+ attempt.attachedAt = Date.now();
+ this.options.onConnected();
}
- this.options.log.warn('Switch ended the controller stream; opening a new connection.', {
- code: frame.data.code,
- reason: frame.data.reason,
- });
- this.held = null;
- return;
+ const frame = parseFrame(message, this.options.log);
+ if (!frame) continue;
+ await this.options.onFrame(frame);
+ if (this.ended(frame, attempt)) return;
}
+ if (closedWith !== null || abort.signal.aborted) break;
+ await new Promise<void>((resolve) => {
+ wake = resolve;
+ });
+ }
+ const refusal = refused as { status: number; code: string; message: string } | null;
+ if (refusal !== null) {
+ if (refusal.status === 401 && refusal.code !== 'controller_revoked') opened.tokenRefused();
+ throw new ControllerApiError(refusal.status, refusal.code, refusal.message, false, null);
}
+ if (abort.signal.aborted && abort.signal.reason instanceof IdleTimeout)
+ throw abort.signal.reason;
+ if (!this.options.signal.aborted && attempt.attachedAt === 0)
+ throw new Error(`The controller socket closed before it attached (code ${closedWith}).`);
if (!this.options.signal.aborted)
- this.options.log.warn('The controller stream ended; reconnecting.');
+ this.options.log.warn('The controller socket closed; reconnecting.', { code: closedWith });
} finally {
if (idle) clearTimeout(idle);
+ socket.removeEventListener('message', onMessage);
+ socket.removeEventListener('close', onClose);
+ socket.removeEventListener('error', notify);
+ abort.signal.removeEventListener('abort', onAbort);
+ if (socket.readyState === 0 || socket.readyState === 1) socket.close(1000);
}
}
- /** Beats the connection while this attempt's stream is attached. */
- private async beat(connection: Held, attempt: Attempt): Promise<void> {
- const { client, log } = this.options;
- const signal = attempt.socket.signal;
- let failures = 0;
- while (!signal.aborted) {
- await delay(connection.heartbeatMs, undefined, { signal }).catch(() => {});
- if (signal.aborted) return;
- try {
- await client.beat(connection, this.options.confirmed(), signal);
- if (failures > 0)
- log.info('The controller stream heartbeat recovered', { afterFailures: failures });
- failures = 0;
- } catch (error) {
- if (signal.aborted) return;
- if (isRevoked(error) || isTakenOver(error)) {
- if (isTakenOver(error))
- log.error(
- 'Another instance of this controller took its stream connection over; this one stops. Run one controller per data directory.'
- );
- attempt.ending = isRevoked(error) ? 'revoked' : 'taken_over';
- attempt.socket.abort();
- return;
- }
- if (connectionGone(error)) {
- log.warn('Switch refused the heartbeat for a connection it no longer holds; reopening.', {
- error: errorMessage(error),
- });
- if (this.held === connection) this.held = null;
- attempt.socket.abort();
- return;
- }
- failures++;
- // The first failure, then powers of two: an outage costs a few lines.
- if ((failures & (failures - 1)) === 0)
- log.warn('The controller stream heartbeat failed', {
- failures,
- error: errorMessage(error),
- });
- }
+ /** Whether this frame ends the attempt: a revocation, or an eviction. */
+ private ended(frame: ControllerFrame, attempt: Attempt): boolean {
+ if (frame.type === 'credential.revoked') {
+ attempt.ending = 'revoked';
+ return true;
}
+ if (frame.type !== 'evicted') return false;
+ if (frame.data.code === 'taken_over') {
+ this.options.log.error(
+ 'Another instance of this controller took its connection over; this one stops. Run one controller per data directory.'
+ );
+ attempt.ending = 'taken_over';
+ return true;
+ }
+ this.options.log.warn('Switch ended the controller connection; opening a new one.', {
+ code: frame.data.code,
+ reason: frame.data.reason,
+ });
+ this.held = null;
+ return true;
}
}
-function parseFrame(item: SseItem & { kind: 'event' }, log: Logger): ControllerFrame | null {
- const schema = (STREAM_FRAME_SCHEMAS as Record<string, z.ZodType>)[item.event];
+function refusalOf(data: unknown): { status: number; code: string; message: string } {
+ const body = (data ?? {}) as { status?: unknown; detail?: unknown };
+ const detail = (body.detail ?? {}) as { code?: unknown; message?: unknown };
+ return {
+ status: typeof body.status === 'number' ? body.status : 0,
+ code: typeof detail.code === 'string' ? detail.code : 'refused',
+ message:
+ typeof detail.message === 'string' ? detail.message : 'Switch refused the controller socket.',
+ };
+}
+
+function parseFrame(message: Message, log: Logger): ControllerFrame | null {
+ const schema = (STREAM_FRAME_SCHEMAS as Record<string, z.ZodType>)[message.event];
if (!schema) {
- log.debug('Ignoring an event type this controller does not know', { event: item.event });
+ log.debug('Ignoring a frame this controller does not know', { event: message.event });
return null;
}
- try {
- return { type: item.event, data: schema.parse(JSON.parse(item.data)) } as ControllerFrame;
- } catch (error) {
- log.error('Dropped a controller stream frame that does not match the protocol', {
- event: item.event,
- error: errorMessage(error),
+ const result = schema.safeParse(message.data);
+ if (!result.success) {
+ log.error('Dropped a controller frame that does not match the protocol', {
+ event: message.event,
+ error: result.error.message,
});
return null;
}
+ return { type: message.event, data: result.data } as ControllerFrame;
}
diff --git a/console/packages/agent-controller/src/testing/fake-core.ts b/console/packages/agent-controller/src/testing/fake-core.ts
index 58b69ac33..a10238e0e 100644
--- a/console/packages/agent-controller/src/testing/fake-core.ts
+++ b/console/packages/agent-controller/src/testing/fake-core.ts
@@ -1,5 +1,6 @@
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http';
import type { AddressInfo } from 'node:net';
+import type { OpenWebSocket, SocketLike } from '../api';
import type {
AgentCursor,
Assignment,
@@ -19,6 +20,56 @@ type Recorded = {
bytes: number;
};
+/**
+ * A WebSocket held in memory: what the controller sends lands in `received`,
+ * and what the fake core sends arrives as `message` events, one per turn of
+ * the event loop, in order.
+ */
+class FakeSocket extends EventTarget {
+ readyState = 0;
+ readonly url: string;
+ readonly headers: Record<string, string>;
+ onSend: (data: string) => void = () => {};
+
+ constructor(url: string, headers: Record<string, string>) {
+ super();
+ this.url = url;
+ this.headers = headers;
+ }
+
+ send(data: string): void {
+ if (this.readyState === 1) this.onSend(String(data));
+ }
+
+ close(code = 1000): void {
+ this.end(code);
+ }
+
+ open(): void {
+ this.readyState = 1;
+ this.dispatchEvent(new Event('open'));
+ }
+
+ deliver(event: string, data: unknown): void {
+ if (this.readyState !== 1) return;
+ const text = JSON.stringify({ event, data });
+ setImmediate(() => {
+ if (this.readyState !== 3)
+ this.dispatchEvent(Object.assign(new Event('message'), { data: text }));
+ });
+ }
+
+ /** Closes after whatever was sent before it has arrived. */
+ end(code: number): void {
+ if (this.readyState >= 2) return;
+ this.readyState = 2;
+ setImmediate(() => {
+ this.readyState = 3;
+ this.dispatchEvent(Object.assign(new Event('close'), { code }));
+ });
+ }
+}
+
type StreamConnection = {
id: string;
generation: number;
@@ -63,7 +114,8 @@ export class FakeCore {
private generations = 0;
private current: StreamConnection | null = null;
private readonly connections = new Map<string, StreamConnection>();
- private readonly streams = new Map<ServerResponse, StreamConnection>();
+ private readonly streams = new Map<FakeSocket, StreamConnection>();
+ private readonly pings = new Set<ReturnType<typeof setInterval>>();
private server: Server | null = null;
url = '';
@@ -82,7 +134,8 @@ export class FakeCore {
}
async stop(): Promise<void> {
- for (const stream of this.streams.keys()) stream.destroy();
+ for (const ping of this.pings) clearInterval(ping);
+ for (const stream of this.streams.keys()) stream.end(1001);
this.server?.closeAllConnections();
await new Promise<void>((resolve) => this.server?.close(() => resolve()) ?? resolve());
}
@@ -109,8 +162,7 @@ export class FakeCore {
}
push(event: string, data: unknown): void {
- for (const stream of this.streams.keys())
- stream.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);
+ for (const stream of this.streams.keys()) stream.deliver(event, data);
}
/** An agent's domain event on the controller stream, as Switch wraps it. */
@@ -119,12 +171,87 @@ export class FakeCore {
this.push('agent.event', { agent_id: agentId, seq, event: { ...event, sequence: seq } });
}
- /** Ends the open streams; the connection itself lives on. */
- closeStreams(): void {
- for (const stream of this.streams.keys()) stream.end();
+ /** Ends the open sockets; the connection itself lives on. */
+ closeStreams(code = 1000): void {
+ for (const stream of this.streams.keys()) stream.end(code);
this.streams.clear();
}
+ /** The controller's socket, held by this fake core rather than a server. */
+ readonly openWebSocket: OpenWebSocket = (url, headers) => {
+ const socket = new FakeSocket(url, headers);
+ setImmediate(() => this.attach(socket));
+ return socket as unknown as SocketLike;
+ };
+
+ private attach(socket: FakeSocket): void {
+ const url = new URL(socket.url.replace(/^ws/, 'http'));
+ this.requests.push({
+ method: 'WS',
+ path: url.pathname,
+ search: url.search,
+ headers: Object.fromEntries(
+ Object.entries(socket.headers).map(([name, value]) => [name.toLowerCase(), value])
+ ),
+ body: null,
+ bytes: 0,
+ });
+ socket.open();
+ const refuse = (status: number, code: string) => {
+ socket.deliver('refused', { status, detail: { code, message: `refused: ${code}` } });
+ socket.end(4000 + status);
+ };
+ if (url.pathname !== `/v1/controllers/${this.controllerId}/connection/ws`)
+ return refuse(404, 'not_found');
+ const bearer = socket.headers.Authorization?.replace(/^Bearer /, '') ?? '';
+ if (this.revoked) return refuse(401, 'controller_revoked');
+ if (!this.tokens.has(bearer)) return refuse(401, 'token_expired');
+ const connection = this.connections.get(url.searchParams.get('connection_id') ?? '');
+ if (!connection) return refuse(404, 'unknown_connection');
+ if (connection.superseded) return refuse(409, 'taken_over');
+ if (Number(url.searchParams.get('generation')) !== connection.generation)
+ return refuse(409, 'stale_generation');
+ socket.deliver('connection_state', {
+ controller_id: this.controllerId,
+ assignment_revision: this.assignment.revision,
+ report_within_s: this.reportWithinS,
+ connection_id: connection.id,
+ generation: connection.generation,
+ heartbeat_interval_s: this.heartbeatIntervalS,
+ });
+ // Every bound agent is attached afresh on each socket, from where the
+ // connection was opened.
+ for (const agentId of this.bound()) {
+ const cursor = connection.cursors[agentId];
+ socket.deliver('agent.attached', {
+ agent_id: agentId,
+ from_seq: typeof cursor === 'number' ? cursor : (this.heads.get(agentId) ?? 0),
+ rooms: this.rooms.get(agentId) ?? [],
+ });
+ }
+ this.streams.set(socket, connection);
+ const ping = setInterval(() => socket.deliver('ping', {}), this.heartbeatIntervalS * 1000);
+ this.pings.add(ping);
+ socket.onSend = (data) => {
+ const message = JSON.parse(data) as { type?: string; cursors?: Record<string, number> };
+ if (message.type !== 'pong') return;
+ if (connection.superseded || !this.connections.has(connection.id)) {
+ socket.deliver('evicted', {
+ code: connection.superseded ? 'taken_over' : 'unknown_connection',
+ reason: 'this connection is no longer current',
+ });
+ socket.end(1000);
+ return;
+ }
+ this.beats.push(message.cursors ?? {});
+ };
+ socket.addEventListener('close', () => {
+ clearInterval(ping);
+ this.pings.delete(ping);
+ this.streams.delete(socket);
+ });
+ }
+
/** Forgets the connection, as a lapsed heartbeat would: the next attach or beat is a 404. */
forgetConnection(): void {
if (this.current) this.connections.delete(this.current.id);
@@ -249,10 +376,11 @@ export class FakeCore {
this.current.superseded = true;
for (const [stream, connection] of this.streams)
if (connection === this.current) {
- stream.write(
- `event: evicted\ndata: ${JSON.stringify({ code: 'taken_over', reason: 'another connection of this controller took over' })}\n\n`
- );
- stream.end();
+ stream.deliver('evicted', {
+ code: 'taken_over',
+ reason: 'another connection of this controller took over',
+ });
+ stream.end(1000);
}
}
const connection: StreamConnection = {
@@ -270,52 +398,6 @@ export class FakeCore {
agents: this.bound(),
});
}
- if (method === 'GET' && url.pathname === `${streamBase}/events`) {
- const connection = this.connections.get(url.searchParams.get('connection_id') ?? '');
- if (!connection) return this.refuse(res, 404, 'unknown_connection');
- if (connection.superseded) return this.refuse(res, 409, 'taken_over');
- if (Number(url.searchParams.get('generation')) !== connection.generation)
- return this.refuse(res, 409, 'stale_generation');
- res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache' });
- const frame = (event: string, data: unknown) =>
- res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);
- frame('connection_state', {
- controller_id: this.controllerId,
- assignment_revision: this.assignment.revision,
- report_within_s: this.reportWithinS,
- connection_id: connection.id,
- generation: connection.generation,
- heartbeat_interval_s: this.heartbeatIntervalS,
- });
- // Every bound agent is attached afresh on each stream, from where the
- // connection was opened.
- for (const agentId of this.bound()) {
- const cursor = connection.cursors[agentId];
- frame('agent.attached', {
- agent_id: agentId,
- from_seq: typeof cursor === 'number' ? cursor : (this.heads.get(agentId) ?? 0),
- rooms: this.rooms.get(agentId) ?? [],
- });
- }
- this.streams.set(res, connection);
- res.on('close', () => this.streams.delete(res));
- return;
- }
- if (method === 'POST' && url.pathname === `${streamBase}/connection/beat`) {
- const beat = body as {
- connection_id: string;
- generation: number;
- cursors: Record<string, number>;
- };
- const connection = this.connections.get(beat.connection_id);
- if (!connection) return this.refuse(res, 404, 'unknown_connection');
- if (connection.superseded) return this.refuse(res, 409, 'taken_over');
- if (beat.generation !== connection.generation)
- return this.refuse(res, 409, 'stale_generation');
- if ('placements' in (body as object)) return this.refuse(res, 422, 'validation_error');
- this.beats.push(beat.cursors);
- return this.json(res, 200, { agents: this.bound() });
- }
if (method === 'GET' && url.pathname === `${base}/assignment`) {
const etag = `"${this.assignment.revision}"`;
if (req.headers['if-none-match'] === etag) {
diff --git a/console/packages/agent-providers/src/host/hub-stream.ts b/console/packages/agent-providers/src/host/hub-stream.ts
new file mode 100644
index 000000000..d66d6dce5
--- /dev/null
+++ b/console/packages/agent-providers/src/host/hub-stream.ts
@@ -0,0 +1,252 @@
+import {
+ type AgentBridgeEvent,
+ type ApprovalOutcome,
+ EVICTION_TAKEN_OVER,
+ type SessionCommand,
+ type SwitchEventStreamDeps,
+} from '@sandboxaq/switch-agent-runtime';
+import { z } from 'zod';
+import type { AgentEventStream, OpenAgentStream } from './agent-host';
+
+/**
+ * The agents controller's hub, over a WebSocket on the controller's loopback
+ * relay: how an agent host in a process of its own (an `isolated` agent) hears
+ * its agent's events. The controller holds the agent's connection to Switch
+ * and hands each event to the hub; the hub hands it here, one at a time, and
+ * counts it taken once this side's handler has resolved, exactly as it does
+ * for an agent host in the controller's own process.
+ *
+ * Every request the hub makes carries an `id` and is answered with `done`;
+ * `placements` from this side is answered with `result`. The agent is the one
+ * the bearer token names.
+ */
+export const HUB_PATH = '/hub';
+export const HUB_PROTOCOL = 1;
+
+/** Closes the hub sends. */
+export const HUB_CLOSE = {
+ /** The controller is stopping or restarting: reconnect soon. */
+ restarting: 1012,
+ /** A newer agent host for the same agent took the hub over: stand down. */
+ takenOver: 4409,
+ /** The hello was missing, malformed, or named a protocol the hub does not speak. */
+ refused: 4400,
+} as const;
+
+const placementsSchema = z.record(z.string().min(1), z.string().min(1));
+
+export const hubClientMessageSchema = z.discriminatedUnion('type', [
+ z.object({
+ type: z.literal('hello'),
+ protocol: z.number().int(),
+ startCursor: z.number().int().nonnegative().nullable(),
+ placements: placementsSchema,
+ }),
+ z.object({ type: z.literal('done'), id: z.number().int(), error: z.string().optional() }),
+ z.object({ type: z.literal('placements'), id: z.number().int(), placements: placementsSchema }),
+]);
+export type HubClientMessage = z.infer<typeof hubClientMessageSchema>;
+
+export type HubGap = Parameters<SwitchEventStreamDeps['onGap']>[0];
+
+export type HubServerMessage =
+ | { type: 'connected' }
+ | { type: 'disconnected'; error: string }
+ | { type: 'event'; id: number; event: AgentBridgeEvent }
+ | { type: 'gap'; id: number; gap: HubGap }
+ | { type: 'session_command'; id: number; command: SessionCommand }
+ | { type: 'approval_outcome'; id: number; outcome: ApprovalOutcome }
+ | { type: 'result'; id: number; error?: string };
+
+/** Node's WebSocket, which, unlike a browser's, takes request headers. */
+type HeaderedWebSocket = new (url: string, init: { headers: Record<string, string> }) => WebSocket;
+
+const INITIAL_BACKOFF_MS = 250;
+const MAX_BACKOFF_MS = 5_000;
+
+/**
+ * Opens the agent's stream on the controller's hub at `url`
+ * (`ws://127.0.0.1:<relay port>/hub`), with the relay token the agent host
+ * was given.
+ */
+export function openHubStream(url: string): OpenAgentStream {
+ return (deps) => new HubEventStream(url, deps);
+}
+
+class HubEventStream implements AgentEventStream {
+ /** The agent host, not Switch, tells a room when it starts a session for it. */
+ readonly announcesSessionStarts = true;
+ private started = false;
+ private socket: WebSocket | null = null;
+ /** The last sequence this side has handled; sent on every hello. */
+ private cursor: number | null;
+ private placements: Record<string, string> = {};
+ private nextId = 0;
+ private readonly calls = new Map<number, (error: string | undefined) => void>();
+ /** Requests are handled one at a time, in the order they came. */
+ private handling: Promise<void> = Promise.resolve();
+ private standingDown = false;
+
+ constructor(
+ private readonly url: string,
+ private readonly deps: SwitchEventStreamDeps
+ ) {
+ this.cursor = deps.startCursor ?? null;
+ }
+
+ start(): void {
+ if (this.started) return;
+ this.started = true;
+ void this.run();
+ }
+
+ // Switch decides who may start a session from the agent's binding.
+ setSpawnCapable(): void {}
+
+ async replacePlacements(placements: Record<string, string>): Promise<void> {
+ this.placements = { ...placements };
+ const socket = this.socket;
+ // Not connected: the next hello carries them.
+ if (!socket || socket.readyState !== WebSocket.OPEN) return;
+ const id = ++this.nextId;
+ const error = await new Promise<string | undefined>((resolve) => {
+ this.calls.set(id, resolve);
+ socket.send(JSON.stringify({ type: 'placements', id, placements }));
+ });
+ if (error !== undefined) throw new Error(error);
+ }
+
+ workerCall(): Promise<unknown> {
+ return Promise.reject(
+ new Error('An agent run by an agents controller is not a hosted worker.')
+ );
+ }
+
+ private async run(): Promise<void> {
+ const { signal, log } = this.deps;
+ let backoff = INITIAL_BACKOFF_MS;
+ let failures = 0;
+ while (!signal.aborted && !this.standingDown) {
+ const { code, reason, helloed } = await this.connectOnce();
+ if (signal.aborted) return;
+ if (code === HUB_CLOSE.takenOver) {
+ this.standingDown = true;
+ this.deps.onEvicted({
+ code: EVICTION_TAKEN_OVER,
+ reason: reason || 'another agent host for this agent took the controller hub over',
+ roomId: null,
+ });
+ return;
+ }
+ const error = reason || `the agents controller hub closed (${code})`;
+ this.deps.onDisconnected?.({ error });
+ if (helloed) {
+ backoff = INITIAL_BACKOFF_MS;
+ failures = 0;
+ }
+ failures++;
+ // The first failure, then powers of two: a long outage costs a few lines.
+ if ((failures & (failures - 1)) === 0)
+ log.warn(`The agents controller hub is unreachable (${error}); retrying.`);
+ const wait = code === HUB_CLOSE.restarting ? INITIAL_BACKOFF_MS : backoff;
+ await sleep(Math.round(wait * (0.5 + Math.random() * 0.5)), signal);
+ backoff = Math.min(backoff * 2, MAX_BACKOFF_MS);
+ }
+ }
+
+ /** One socket, from open to close. */
+ private connectOnce(): Promise<{ code: number; reason: string; helloed: boolean }> {
+ return new Promise((resolve) => {
+ const Socket = (globalThis as { WebSocket?: HeaderedWebSocket }).WebSocket;
+ if (!Socket) throw new Error('The agents controller hub needs Node 22 or newer.');
+ const socket = new Socket(this.url, {
+ headers: { Authorization: `Bearer ${this.deps.creds.token}` },
+ });
+ this.socket = socket;
+ let helloed = false;
+ const abort = () => socket.close(1000, 'the agent host is stopping');
+ this.deps.signal.addEventListener('abort', abort, { once: true });
+ socket.addEventListener('open', () => {
+ helloed = true;
+ socket.send(
+ JSON.stringify({
+ type: 'hello',
+ protocol: HUB_PROTOCOL,
+ startCursor: this.cursor,
+ placements: this.placements,
+ })
+ );
+ });
+ socket.addEventListener('message', (message) => {
+ let parsed: HubServerMessage;
+ try {
+ parsed = JSON.parse(String(message.data)) as HubServerMessage;
+ } catch {
+ this.deps.log.error('Dropped a message from the agents controller hub that is not JSON');
+ return;
+ }
+ if (parsed.type === 'result') {
+ const call = this.calls.get(parsed.id);
+ this.calls.delete(parsed.id);
+ call?.(parsed.error);
+ return;
+ }
+ this.handling = this.handling.then(() => this.handle(socket, parsed));
+ });
+ socket.addEventListener('close', (event) => {
+ this.deps.signal.removeEventListener('abort', abort);
+ if (this.socket === socket) this.socket = null;
+ // The placements a call was sending go again with the next hello.
+ for (const call of this.calls.values()) call(undefined);
+ this.calls.clear();
+ resolve({ code: event.code, reason: event.reason, helloed });
+ });
+ // A failed handshake also closes; the close says what happened.
+ socket.addEventListener('error', () => {});
+ });
+ }
+
+ private async handle(socket: WebSocket, message: HubServerMessage): Promise<void> {
+ const { deps } = this;
+ if (message.type === 'connected') return deps.onConnected?.();
+ if (message.type === 'disconnected') return deps.onDisconnected?.({ error: message.error });
+ if (message.type === 'result') return;
+ let error: string | undefined;
+ try {
+ switch (message.type) {
+ case 'event':
+ await deps.onEvent(message.event);
+ if (typeof message.event.sequence === 'number')
+ this.cursor = Math.max(this.cursor ?? 0, message.event.sequence);
+ break;
+ case 'gap':
+ await deps.onGap(message.gap);
+ if (message.gap.resumedAt !== undefined) this.cursor = message.gap.resumedAt;
+ break;
+ case 'session_command':
+ await deps.onSessionCommand?.(message.command);
+ break;
+ case 'approval_outcome':
+ await deps.onApprovalOutcome?.(message.outcome);
+ break;
+ }
+ } catch (failure) {
+ error = failure instanceof Error ? failure.message : String(failure);
+ }
+ if (socket.readyState === WebSocket.OPEN)
+ socket.send(JSON.stringify({ type: 'done', id: message.id, ...(error ? { error } : {}) }));
+ }
+}
+
+function sleep(ms: number, signal: AbortSignal): Promise<void> {
+ return new Promise((resolve) => {
+ if (signal.aborted) return resolve();
+ const timer = setTimeout(done, ms);
+ function done() {
+ clearTimeout(timer);
+ signal.removeEventListener('abort', done);
+ resolve();
+ }
+ signal.addEventListener('abort', done, { once: true });
+ });
+}
diff --git a/console/packages/agent-providers/src/host/shared-config.ts b/console/packages/agent-providers/src/host/shared-config.ts
index 8da979f0e..b34d5cc4a 100644
--- a/console/packages/agent-providers/src/host/shared-config.ts
+++ b/console/packages/agent-providers/src/host/shared-config.ts
@@ -117,6 +117,9 @@ export async function readSharedCredentials(config: SharedHostConfig) {
SWITCH_API_ENDPOINT: z.string().min(1),
SWITCH_API_TOKEN: z.string().min(1),
SWITCH_AGENT_ID: z.string().min(1),
+ // Set when an agents controller runs the agent host in a process of
+ // its own: the hub it hears its events on instead of Switch.
+ SWITCH_AGENT_HUB: z.string().min(1).optional(),
}),
})
.parse(JSON.parse(await readFile(config.execution.credentialsPath, 'utf8'))).env;
diff --git a/console/packages/agent-providers/src/host/shared-daemon.ts b/console/packages/agent-providers/src/host/shared-daemon.ts
index 50978f147..190e66ab7 100644
--- a/console/packages/agent-providers/src/host/shared-daemon.ts
+++ b/console/packages/agent-providers/src/host/shared-daemon.ts
@@ -7,13 +7,14 @@ import { AttachmentTransfers } from './attachment-transfers';
import { type ControlContext, ensureSessions, serveControl } from './control';
import { OBSOLETE_BUNDLE_EXIT_CODE, WorkerObsoleteError } from './exit-codes';
import { hostedWorker } from './hosted-watcher';
+import { openHubStream } from './hub-stream';
import { detachedSupervision, ensureSharedProcess, inProcessSupervision } from './launch';
import { replaceOwner } from './ownership-lock';
import { ownProcessGroup } from './process-fence';
import { checkProviderReadiness } from './provider-readiness';
import { adapterFor } from './server';
import { HOST_EXIT_GRACE_MS, SessionLinks } from './session-channel';
-import { sharedConfigSchema } from './shared-config';
+import { readSharedCredentials, sharedConfigSchema } from './shared-config';
import { hostSessionProcess } from './shared-host';
import { superviseSharedHost } from './supervisor';
import { recordWatcherHealth } from './watcher-health-file';
@@ -135,6 +136,11 @@ async function main(): Promise<void> {
transfers,
};
const hosted = await hostedWorker(config, resolve(root), context);
+ // An agents controller running this agent host in a process of its own
+ // names its hub: the agent's events come from there, not from Switch.
+ const hub = config.execution
+ ? (await readSharedCredentials(config)).SWITCH_AGENT_HUB
+ : undefined;
// Console reads the watcher's connection state from this file, with the
// rest of the host's watcher state, rather than from the control port.
const stopRecording = recordWatcherHealth(resolve(root), control);
@@ -151,7 +157,7 @@ async function main(): Promise<void> {
supervision,
control,
hosted,
- openSwitchStream
+ hub ? openHubStream(hub) : openSwitchStream
).finally(() => stop.abort()),
serveControl(resolve(root), context, stop.signal),
]);
diff --git a/console/packages/agent-providers/src/index.ts b/console/packages/agent-providers/src/index.ts
index 7478443d7..0e9cfb689 100644
--- a/console/packages/agent-providers/src/index.ts
+++ b/console/packages/agent-providers/src/index.ts
@@ -65,6 +65,15 @@ export {
openSwitchStream,
runAgentHost,
} from './host/agent-host';
+export {
+ HUB_CLOSE,
+ HUB_PATH,
+ HUB_PROTOCOL,
+ type HubClientMessage,
+ hubClientMessageSchema,
+ type HubServerMessage,
+ openHubStream,
+} from './host/hub-stream';
export { clearTakenOver, readTakenOver, type TakenOver } from './host/taken-over';
export {
recordWatcherHealth,
diff --git a/console/pnpm-lock.yaml b/console/pnpm-lock.yaml
index 91ff45164..67e5f6428 100644
--- a/console/pnpm-lock.yaml
+++ b/console/pnpm-lock.yaml
@@ -305,6 +305,9 @@ importers:
'@switch-console/plugins':
specifier: workspace:*
version: link:../plugins
+ ws:
+ specifier: ^8.21.3
+ version: 8.21.3
zod:
specifier: ^4.4.3
version: 4.4.3
@@ -315,6 +318,9 @@ importers:
'@types/node':
specifier: ^24.13.3
version: 24.13.3
+ '@types/ws':
+ specifier: ^8.18.2
+ version: 8.18.2
'@typescript/native-preview':
specifier: 7.0.0-dev.20260609.1
version: 7.0.0-dev.20260609.1
@@ -2565,6 +2571,9 @@ packages:
resolution: {integrity: sha512-ko/gIFJRv177XgZsZcBwnqJN5x/Gien8qNOn0D5bQU/zAzVf9Zt3BlcUiLqhV9y4ARk0GbT3tnUiPNgnTXzc/Q==}
+ '@types/[email protected]':
+ resolution: {integrity: sha512-67MQl+fpWKVTT1NYdnmo3U4sc/xPo/zQBncVnI74qmQa0z/b+1g6iYqNmGCPbxO+zz2aklb08a0oHfegiVd0/w==}
+
'@typescript/[email protected]':
resolution: {integrity: sha512-Yf/zHEadP/yUiWUdM/mZVfEVFJuGMf6nhRSFif0vp+FwtfGU4jmlpNF7BTJJdOHrrcWkwEJKzAoMCtEtyxhuyQ==}
engines: {node: '>=16.20.0'}
@@ -7423,6 +7432,10 @@ snapshots:
'@types/[email protected]': {}
+ '@types/[email protected]':
+ dependencies:
+ '@types/node': 24.13.3
+
'@typescript/[email protected]':
optional: true
diff --git a/core/switch_core/bridges/agent/api/handlers.py b/core/switch_core/bridges/agent/api/handlers.py
index 1b29b2049..4ac585df7 100644
--- a/core/switch_core/bridges/agent/api/handlers.py
+++ b/core/switch_core/bridges/agent/api/handlers.py
@@ -133,6 +133,8 @@ from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.db.stores.feature_flag_store import FeatureFlagStore
from switch_core.feature_flags import is_known_flag
from switch_core.gateway.known_agents import KNOWN_AGENTS
+from switch_core.observability.catalogue import AGENT_CONNECTIONS_REFUSED
+from switch_core.observability.metrics import metrics
from switch_core.version import switch_core_version
logger = logging.getLogger(__name__)
@@ -193,12 +195,17 @@ async def _resolve_registration_user_id(
"bootstrap" if key.type == BOOTSTRAP_KEY_TYPE else "personal_key"
)
try:
- return await resolve_registration_owner_id(session, protocol.user_store, key)
+ owner_id = await resolve_registration_owner_id(
+ session, protocol.user_store, key
+ )
except RuntimeError as exc:
logger.error("Agent-registration bootstrap owner resolution failed: %s", exc)
raise HTTPException(
status_code=503, detail="Agent registration is temporarily unavailable"
) from exc
+ # Not held across registration, which opens its own session and calls every bridge.
+ await session.commit()
+ return owner_id
# How the current registration authenticated. A contextvar rather than a
@@ -397,6 +404,8 @@ async def register_known_agents_bulk_endpoint(
),
)
+ # Not held across registration, which opens its own session and calls every bridge.
+ await session.commit()
results: list[BulkRegisterResult] = []
for subagent_name, name, description in derived:
# Inherited parent settings are the base; explicit request options
@@ -764,8 +773,8 @@ async def poll_events(
stream, with its heartbeat on `POST /connection/beat`: that is how an agent
runtime built before the WebSocket connects (agent-protocol revision 7 and
older), and it is kept for those clients for a compatibility window. It
- goes once no client still connects over it. Anything else falls back to
- the long poll, served from the same buffer.
+ goes once `switch.agents.connected{transport:sse}` stays at zero. Anything
+ else falls back to the long poll, served from the same buffer.
The declaration parameters are all optional and all default to None,
meaning *unknown* (CHOO-1865): a client that says nothing still connects.
@@ -1038,6 +1047,11 @@ async def _open_connection(
return conn, frames
+def _refusal_reason(exc: HTTPException) -> str:
+ """The `reason` a refused open is counted under."""
+ return "protocol" if isinstance(exc.__cause__, ProtocolVersionError) else "other"
+
+
async def _open_event_stream(
*,
agent: Agent,
@@ -1063,25 +1077,29 @@ async def _open_event_stream(
loop, so the two cannot drift while both exist. A refusal is the HTTP
error the socket would have sent as its `refused` frame.
"""
- _conn, frames = await _open_connection(
- agent=agent,
- protocol=protocol,
- config=config,
- connection_id=connection_id,
- scope=scope,
- event_filter=event_filter,
- start_from=start_from,
- spawn_capable=spawn_capable,
- declaration=declaration,
- rooms=rooms,
- expected_generation=expected_generation,
- worker_capability=worker_capability,
- host_boot_id=host_boot_id,
- host_instance_id=host_instance_id,
- worker_state_version=worker_state_version,
- last_event_id=last_event_id,
- transport="sse",
- )
+ try:
+ _conn, frames = await _open_connection(
+ agent=agent,
+ protocol=protocol,
+ config=config,
+ connection_id=connection_id,
+ scope=scope,
+ event_filter=event_filter,
+ start_from=start_from,
+ spawn_capable=spawn_capable,
+ declaration=declaration,
+ rooms=rooms,
+ expected_generation=expected_generation,
+ worker_capability=worker_capability,
+ host_boot_id=host_boot_id,
+ host_instance_id=host_instance_id,
+ worker_state_version=worker_state_version,
+ last_event_id=last_event_id,
+ transport="sse",
+ )
+ except HTTPException as exc:
+ metrics().increment(AGENT_CONNECTIONS_REFUSED, {"reason": _refusal_reason(exc)})
+ raise
return StreamingResponse(
sse_stream(frames),
media_type="text/event-stream",
@@ -1236,6 +1254,7 @@ async def connection_socket(
worker_state_version=worker_state_version,
)
except HTTPException as exc:
+ metrics().increment(AGENT_CONNECTIONS_REFUSED, {"reason": _refusal_reason(exc)})
await websocket.send_json(
{
"event": "refused",
diff --git a/core/switch_core/bridges/agent/auth.py b/core/switch_core/bridges/agent/auth.py
index edf161bbe..b0f64b458 100644
--- a/core/switch_core/bridges/agent/auth.py
+++ b/core/switch_core/bridges/agent/auth.py
@@ -31,6 +31,8 @@ from switch_core.db.tenant_lookup import (
tenant_of_api_key,
)
from switch_core.logging_context import log_context
+from switch_core.observability.catalogue import AGENT_CONNECTIONS_REFUSED
+from switch_core.observability.metrics import metrics
from switch_core.tenant_context import tenant_scope
logger = logging.getLogger(__name__)
@@ -78,7 +80,7 @@ _NOT_AN_AGENT_SEGMENT = frozenset({"rooms", "feature-flags"})
# Registration: a controller registers nothing, so its token is refused here.
_REGISTRATION_SEGMENTS = frozenset({"register-known", "register-known-bulk"})
# The connection surface a controller serves its agents itself, from its own
-# stream (`GET /v1/controllers/{id}/events`). A controller-backed agent has no
+# connection (`/v1/controllers/{id}/connection/ws`). A controller-backed agent has no
# connection of its own, and the legacy heartbeats would make it look live from
# a second source.
_SERVED_ON_THE_CONTROLLER_STREAM = re.compile(
@@ -86,6 +88,11 @@ _SERVED_ON_THE_CONTROLLER_STREAM = re.compile(
)
+# The agent connection's socket, whose refusals are counted as connections
+# refused. Any other path's 401 is an ordinary failed request.
+_AGENT_CONNECTION_PATH = re.compile(r"^/agents/[^/]+/connection/ws$")
+
+
class OIDCTokenValidator:
def __init__(
self,
@@ -425,7 +432,7 @@ class BearerAuthMiddleware:
await _controller_refusal(
MANAGED_BY_CONTROLLER,
"A controller receives its agents' events on its own stream, "
- "GET /v1/controllers/{id}/events; this agent route is not "
+ "/v1/controllers/{id}/connection/ws; this agent route is not "
"served to it.",
409,
)(scope, receive, send)
@@ -589,6 +596,8 @@ async def _unauthorized(
connection uses for every refusal.
"""
if scope["type"] == "websocket":
+ if _AGENT_CONNECTION_PATH.match(scope.get("path", "")):
+ metrics().increment(AGENT_CONNECTIONS_REFUSED, {"reason": "unauthorized"})
await receive() # the client's websocket.connect
await send({"type": "websocket.accept"})
await send({"type": "websocket.close", "code": 4401, "reason": reason})
diff --git a/core/switch_core/bridges/agent/protocol/agent_connections.py b/core/switch_core/bridges/agent/protocol/agent_connections.py
index 182535d25..784299116 100644
--- a/core/switch_core/bridges/agent/protocol/agent_connections.py
+++ b/core/switch_core/bridges/agent/protocol/agent_connections.py
@@ -31,7 +31,7 @@ from contextlib import asynccontextmanager
from dataclasses import dataclass, field
from typing import Any, Literal
-from switch_core.artifacts import contract_range
+from switch_core.artifacts import ARTIFACT_VERSIONS, contract_range
from switch_core.bridges.agent.protocol.controller_presence import ControllerPresence
from switch_core.bridges.agent.protocol.hosted_workers import (
IDLE_FRESH_FOR_SECONDS,
@@ -55,7 +55,10 @@ from switch_core.bridges.agent.protocol.liveness import TAKEN_OVER as TAKEN_OVER
from switch_core.bridges.agent.protocol.liveness import CloseCode as CloseCode
from switch_core.bridges.agent.protocol.liveness import Closure as Closure
from switch_core.logging_context import log_context
-from switch_core.observability.catalogue import AGENT_CONNECTIONS_EXPIRED
+from switch_core.observability.catalogue import (
+ AGENT_CONNECTIONS_EXPIRED,
+ AGENT_CONNECTIONS_OPENED,
+)
from switch_core.observability.metrics import metrics
logger = logging.getLogger(__name__)
@@ -64,7 +67,7 @@ Scope = Literal["single", "all"]
DeliveryFilter = Literal["all", "addressed"]
# How a connection's stream reaches its client. `sse` is the event stream an
# agent runtime built before the WebSocket still asks for, served for a
-# compatibility window and told apart so we can see those clients drain.
+# compatibility window and counted apart so we can see those clients drain.
Transport = Literal["websocket", "sse"]
# Refuse a client that cannot meet this server's protocol rather than degrading
@@ -273,6 +276,11 @@ class WorkerAlreadyAttachedError(ConnectionError_):
self.agent_id = agent_id
+# How many closed connection ids are remembered for the reconnect counter.
+# Far more than the live fleet, so a lapse storm is still recognised when its
+# agents come back.
+_RECENTLY_CLOSED_LIMIT = 10_000
+
#: The hosted launch moved to a newer revision than the one this worker was
#: attached for. Terminal: the worker's capability is obsolete, and only a
#: restart onto the current bundle can attach again.
@@ -359,6 +367,20 @@ class ClientDeclaration:
}
+def client_label(declaration: ClientDeclaration) -> str:
+ """The declared artifact as a metric value: a registered name, or a bucket.
+
+ The client says what it is, so only names in the artifact registry pass
+ through. The declared version is never a label: it is the client's own
+ string and unbounded, and stays in the `[CONN] opened` log line instead.
+ """
+ if not declaration.artifact:
+ return "unknown"
+ if declaration.artifact in ARTIFACT_VERSIONS:
+ return declaration.artifact
+ return "other"
+
+
class ProtocolVersionError(ConnectionError_):
"""A client's declared agent-protocol range cannot meet this server's.
@@ -447,6 +469,18 @@ class AgentConnection:
return self.closure is None and (now - self.last_beat) < HEARTBEAT_TTL_SECONDS
+# The `(transport, client)` an agent run by an agents controller counts under:
+# it travels on its controller's connection, whatever that connection uses.
+CONTROLLER_LABEL = ("controller", "agents-controller")
+
+
+def connection_transport(conn: AgentConnection) -> str:
+ """`websocket` or `sse` while a stream is attached, by what it travels over;
+ `detached` while it has dropped and the connection waits out its heartbeat
+ window for a reconnect."""
+ return conn.stream_transport if conn.stream_attached else "detached"
+
+
class AgentConnectionRegistry:
"""The live set of agent connections. Authoritative, in memory."""
@@ -484,6 +518,10 @@ class AgentConnectionRegistry:
# presence is their controller's. Every presence question below asks
# it first for those agents, and this registry for the rest.
self.controllers = ControllerPresence(on_bound=self._close_for_controller)
+ # Why each recently closed connection id closed, so its next open can
+ # be counted as a return after a lapse rather than a first connect.
+ # Insertion-ordered and capped: only the counter reads it.
+ self._recently_closed: dict[str, str] = {}
def _new_incarnation(self) -> int:
"""The next never-before-used incarnation number.
@@ -580,6 +618,7 @@ class AgentConnectionRegistry:
# what is on the other end of it need not.
existing.declaration = declaration
self._reset_worker(existing)
+ metrics().increment(AGENT_CONNECTIONS_OPENED, {"kind": "reattach"})
logger.info(
"[CONN] reattached agent=%s connection=%s scope=%s generation=%s "
"transport=%s",
@@ -612,6 +651,10 @@ class AgentConnectionRegistry:
)
self._by_id[connection_id] = conn
owned.add(connection_id)
+ metrics().increment(
+ AGENT_CONNECTIONS_OPENED,
+ {"kind": self._recently_closed.pop(connection_id, "fresh")},
+ )
logger.info(
"[CONN] opened agent=%s connection=%s scope=%s filter=%s spawn=%s "
"client=%s version=%s protocol=%s transport=%s",
@@ -674,6 +717,14 @@ class AgentConnectionRegistry:
self._drop_placements(conn)
self.relays.fail_connection(conn.id, None)
conn.wake.set()
+ lapsed = closure is HEARTBEAT_LAPSED
+ self._remember_closed(connection_id, "after_lapse" if lapsed else "after_close")
+ if lapsed:
+ # Counted here rather than in the sweep: a lapse is also noticed
+ # when the dead connection is next used (`require`) and by its own
+ # event stream, and counting only the sweep under-reported lapses
+ # against the reconnects that follow them.
+ metrics().increment(AGENT_CONNECTIONS_EXPIRED, {})
# `beats` and the age separate the two ways a connection dies, which
# otherwise look identical in the log: a client that never beat at all
# (beats=0 — it is not running the heartbeat, or cannot reach us) versus
@@ -724,6 +775,12 @@ class AgentConnectionRegistry:
),
)
+ def _remember_closed(self, connection_id: str, kind: str) -> None:
+ self._recently_closed.pop(connection_id, None)
+ self._recently_closed[connection_id] = kind
+ while len(self._recently_closed) > _RECENTLY_CLOSED_LIMIT:
+ del self._recently_closed[next(iter(self._recently_closed))]
+
def sweep(self) -> list[AgentConnection]:
"""Close connections whose heartbeat has lapsed. Returns those closed."""
now = time.monotonic()
@@ -737,10 +794,6 @@ class AgentConnectionRegistry:
gone = self.close(conn.id, HEARTBEAT_LAPSED)
if gone is not None:
closed.append(gone)
- if closed:
- # A gauge of live connections cannot show churn: agents
- # reconnecting as fast as they expire hold it perfectly flat.
- metrics().increment(AGENT_CONNECTIONS_EXPIRED, {}, float(len(closed)))
return closed
# ------------------------------------------------------------------
@@ -1376,6 +1429,26 @@ class AgentConnectionRegistry:
conn.agent_id for conn in self._by_id.values() if conn.is_alive(now)
} | self.controllers.live_agent_ids()
+ def live_agents_by_transport(self) -> dict[tuple[str, str], int]:
+ """Agents with a live connection, per `(transport, client)` label.
+
+ Counted like `live_agent_ids`: an agent is one, however many
+ connections it holds, but an agent connected two different ways counts
+ once under each, so the labels can sum to more than the agent count.
+ An agent run by an agents controller has no connection of its own and
+ counts under `CONTROLLER_LABEL` while its controller is live.
+ """
+ now = time.monotonic()
+ agents: dict[tuple[str, str], set[str]] = {}
+ for conn in self._by_id.values():
+ if conn.is_alive(now):
+ key = (connection_transport(conn), client_label(conn.declaration))
+ agents.setdefault(key, set()).add(conn.agent_id)
+ controlled = self.controllers.live_agent_ids()
+ if controlled:
+ agents.setdefault(CONTROLLER_LABEL, set()).update(controlled)
+ return {key: len(ids) for key, ids in agents.items()}
+
def live_connection_ids(self) -> set[str]:
"""Every connection currently alive, by id.
diff --git a/core/switch_core/bridges/agent/protocol/controller_stream.py b/core/switch_core/bridges/agent/protocol/controller_stream.py
index eee014858..195e350d2 100644
--- a/core/switch_core/bridges/agent/protocol/controller_stream.py
+++ b/core/switch_core/bridges/agent/protocol/controller_stream.py
@@ -1,4 +1,5 @@
-"""One event stream per agents controller (`GET /v1/controllers/{id}/events`).
+"""One stream per agents controller, sent on its socket
+(`/v1/controllers/{id}/connection/ws`).
A read-side merge over the per-agent `EventBuffer`: every agent bound to the
controller is read from its own cursor, with nothing filtered (the controller
@@ -28,17 +29,17 @@ of its agents' watchers exactly the stream it would have had from Switch:
- `evicted {code, reason}`: the stream ends. `taken_over` is terminal for the
client that receives it; the rest are recovered by opening again.
-A keepalive comment is written whenever nothing else has been for the
-keepalive interval, only so idle proxies keep the connection.
+Frames are `{"event", "data"}` dicts, which the socket sends as JSON. When
+nothing has happened for the idle interval the stream yields `IDLE`, which is
+not sent: it only lets the loop look at the connection again.
"""
from __future__ import annotations
import asyncio
-import json
import logging
import time
-from collections.abc import AsyncIterator, Awaitable, Callable
+from collections.abc import AsyncGenerator, Awaitable, Callable
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Protocol
@@ -59,15 +60,12 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
-KEEPALIVE_INTERVAL_SECONDS = 15.0
-KEEPALIVE = b": keepalive\n\n"
+IDLE_INTERVAL_SECONDS = 15.0
CATCH_UP_BATCH = 200
-STREAM_HEADERS = {
- "Cache-Control": "no-cache",
- "Connection": "keep-alive",
- "X-Accel-Buffering": "no",
-}
+ControllerFrame = dict[str, Any]
+# Yielded when nothing happened for the idle interval; never sent.
+IDLE: ControllerFrame = {"event": "idle", "data": {}}
# The nudge after which nothing more is written: the controller stops every
# agent on it and exits.
@@ -85,10 +83,8 @@ class ControllerNudges(Protocol):
def close(self) -> None: ...
-def frame(event: str, data: dict[str, Any]) -> bytes:
- return (
- f"event: {event}\ndata: {json.dumps(data, separators=(',', ':'))}\n\n"
- ).encode()
+def frame(event: str, data: dict[str, Any]) -> ControllerFrame:
+ return {"event": event, "data": data}
@dataclass
@@ -117,8 +113,8 @@ def controller_event_stream(
nudges: ControllerNudges,
opening: dict[str, Any],
rooms_of: Callable[[str], Awaitable[set[str]]],
- keepalive_seconds: float,
-) -> AsyncIterator[bytes]:
+ idle_seconds: float,
+) -> AsyncGenerator[ControllerFrame]:
"""The stream for an open controller connection whose stream was just
attached as `stream_token`. Takes ownership of `nudges` and closes it."""
return _ControllerStream(
@@ -129,7 +125,7 @@ def controller_event_stream(
approvals=approvals,
nudges=nudges,
rooms_of=rooms_of,
- keepalive_seconds=keepalive_seconds,
+ idle_seconds=idle_seconds,
).run(opening)
@@ -144,7 +140,7 @@ class _ControllerStream:
approvals: ApprovalOutcomes | None,
nudges: ControllerNudges,
rooms_of: Callable[[str], Awaitable[set[str]]],
- keepalive_seconds: float,
+ idle_seconds: float,
) -> None:
self._conn = conn
self._token = stream_token
@@ -153,11 +149,11 @@ class _ControllerStream:
self._approvals = approvals
self._nudges = nudges
self._rooms_of = rooms_of
- self._keepalive = keepalive_seconds
+ self._idle = idle_seconds
self._tenant_id = current_tenant_id()
self._attached: dict[str, _Attached] = {}
- async def run(self, opening: dict[str, Any]) -> AsyncIterator[bytes]:
+ async def run(self, opening: dict[str, Any]) -> AsyncGenerator[ControllerFrame]:
conn = self._conn
try:
yield frame("connection_state", opening)
@@ -213,7 +209,7 @@ class _ControllerStream:
continue
if not await self._wait():
- yield KEEPALIVE
+ yield IDLE
finally:
for attached in self._attached.values():
if attached.unsubscribe is not None:
@@ -239,11 +235,11 @@ class _ControllerStream:
return conn.closure
return None
- def _detach_departed(self) -> list[bytes]:
+ def _detach_departed(self) -> list[ControllerFrame]:
conn = self._conn
owed = self._presence.take_detached(conn)
bound = self._presence.agents_of(conn.controller_id)
- frames: list[bytes] = []
+ frames: list[ControllerFrame] = []
for agent_id in sorted(self._attached):
if agent_id in bound and agent_id not in owed:
continue
@@ -264,9 +260,9 @@ class _ControllerStream:
)
return frames
- async def _attach_arrivals(self) -> list[bytes]:
+ async def _attach_arrivals(self) -> list[ControllerFrame]:
conn = self._conn
- frames: list[bytes] = []
+ frames: list[ControllerFrame] = []
for agent_id in sorted(
self._presence.agents_of(conn.controller_id) - set(self._attached)
):
@@ -357,8 +353,8 @@ class _ControllerStream:
)
attached.resync = True
- async def _owed_outcomes(self) -> list[bytes]:
- frames: list[bytes] = []
+ async def _owed_outcomes(self) -> list[ControllerFrame]:
+ frames: list[ControllerFrame] = []
for attached in list(self._attached.values()):
if attached.resync and self._approvals is not None:
attached.resync = False
@@ -377,7 +373,7 @@ class _ControllerStream:
)
return frames
- def _deliver(self, attached: _Attached) -> list[bytes]:
+ def _deliver(self, attached: _Attached) -> list[ControllerFrame]:
agent_id = attached.agent_id
for room_id in self._presence.rooms(agent_id):
self._buffer.ensure_counting(
@@ -409,7 +405,7 @@ class _ControllerStream:
# would find work it can never read and spin.
attached.cursor = max(attached.cursor, self._buffer.head(agent_id))
return []
- frames: list[bytes] = []
+ frames: list[ControllerFrame] = []
for item in pending:
data = item.event.model_dump(mode="json")
data["sequence"] = item.seq
@@ -441,13 +437,13 @@ class _ControllerStream:
async def _wait(self) -> bool:
"""Wait for an event, a nudge or a change to the connection. False when
- the keepalive interval passed with none of them."""
+ the idle interval passed with none of them."""
bells = [self._buffer.doorbell(agent_id) for agent_id in self._attached]
for bell in bells:
bell.clear()
self._conn.wake.clear()
# Re-checked after clearing: anything that arrived between the last
- # read and the clear would otherwise wait for the keepalive timeout.
+ # read and the clear would otherwise wait for the idle timeout.
if self._has_work() or self._nudges.wake.is_set():
return True
waiters = [
@@ -456,7 +452,7 @@ class _ControllerStream:
]
try:
done, _ = await asyncio.wait(
- waiters, timeout=self._keepalive, return_when=asyncio.FIRST_COMPLETED
+ waiters, timeout=self._idle, return_when=asyncio.FIRST_COMPLETED
)
return bool(done)
finally:
diff --git a/core/switch_core/bridges/agent/protocol/event_buffer.py b/core/switch_core/bridges/agent/protocol/event_buffer.py
index 482dd7dce..66bcd3b73 100644
--- a/core/switch_core/bridges/agent/protocol/event_buffer.py
+++ b/core/switch_core/bridges/agent/protocol/event_buffer.py
@@ -37,7 +37,11 @@ from dataclasses import dataclass
from switch_core.bridges.agent.protocol.types import AgentEvent
from switch_core.logging_context import log_context
-from switch_core.observability.catalogue import AGENT_EVENTS_DROPPED
+from switch_core.observability.catalogue import (
+ AGENT_BUFFER_SCAN_DURATION,
+ AGENT_BUFFER_SCANNED,
+ AGENT_EVENTS_DROPPED,
+)
from switch_core.observability.metrics import metrics
logger = logging.getLogger(__name__)
@@ -52,6 +56,13 @@ DEFAULT_MAX_EVENTS_PER_AGENT = 2000
# later is told it has a gap rather than handed a partial stream.
DEFAULT_RETENTION_SECONDS = 15 * 60
+# An agent past the cap drops an event on every enqueue, and a warning per drop
+# is a log line per delivered message. One per agent per interval says the same.
+OVERFLOW_WARNING_INTERVAL_SECONDS = 60.0
+
+_READ = {"operation": "read"}
+_UNREAD = {"operation": "unread"}
+
# Why an unread count is absent or incomplete. Never a bare zero: a reader that
# is told nothing went by must be able to trust it.
NO_BASELINE = "nothing recorded what you had already seen in this room"
@@ -204,6 +215,10 @@ class EventBuffer:
# restart is gone from the buffer, so a count taken from what is left
# would be a floor presented as a total.
self._restarted: set[str] = set()
+ # agent -> when its last overflow warning was logged, and how many
+ # overflow drops have gone unlogged since.
+ self._overflow_warned_at: dict[str, float] = {}
+ self._overflow_unlogged: dict[str, int] = {}
# ------------------------------------------------------------------
# Producing
@@ -264,11 +279,14 @@ class EventBuffer:
readable when its window ends is relied on to decide how long a record
about it has to be kept.
"""
+ started = time.perf_counter()
self._trim(agent_id)
self._check_cursor(agent_id, after_seq)
out: list[BufferedEvent] = []
- for item in self._events.get(agent_id, ()):
+ events = self._events.get(agent_id, ())
+ scanned = len(events)
+ for index, item in enumerate(events):
if item.seq <= after_seq:
continue
if rooms is not None and item.room_id not in rooms:
@@ -277,7 +295,9 @@ class EventBuffer:
continue
out.append(item)
if limit is not None and len(out) >= limit:
+ scanned = index + 1
break
+ self._record_scan(_READ, scanned, started)
return out
async def wait(self, agent_id: str, timeout: float) -> None:
@@ -469,14 +489,17 @@ class EventBuffer:
if baseline is None:
return Unread(count=None, reason=RESTARTED)
+ started = time.perf_counter()
+ events = self._events.get(agent_id, ())
count = sum(
1
- for item in self._events.get(agent_id, ())
+ for item in events
if item.room_id == room_id
and baseline < item.seq <= through_seq
and item.event.type == "message"
and not item.notifiable
)
+ self._record_scan(_UNREAD, len(events), started)
dropped = self._dropped_through.get(agent_id, {}).get(room_id, 0)
if dropped > baseline:
return Unread(count=count, reason=COUNTED_FROM_A_HOLE)
@@ -512,6 +535,8 @@ class EventBuffer:
self._dropped_through.pop(agent_id, None)
self._counting.pop(agent_id, None)
self._restarted.discard(agent_id)
+ self._overflow_warned_at.pop(agent_id, None)
+ self._overflow_unlogged.pop(agent_id, None)
def _begin(
self, agent_id: str, room_id: str, occupant: Reader, baseline: int
@@ -725,6 +750,16 @@ class EventBuffer:
# Internals
# ------------------------------------------------------------------
+ @staticmethod
+ def _record_scan(operation: dict[str, str], scanned: int, started: float) -> None:
+ registry = metrics()
+ registry.observe(
+ AGENT_BUFFER_SCAN_DURATION,
+ operation,
+ (time.perf_counter() - started) * 1000.0,
+ )
+ registry.observe(AGENT_BUFFER_SCANNED, operation, scanned)
+
def _wake(self, agent_id: str) -> None:
notify = self._notify.get(agent_id)
if notify is not None:
@@ -766,23 +801,44 @@ class EventBuffer:
item = events.popleft()
dropped[item.room_id] = item.seq
metrics().increment(AGENT_EVENTS_DROPPED, {"reason": "overflow"}, overflow)
- # The agent is a field as well as being in the message: this fires
- # from the buffer's own bookkeeping rather than from anything the
- # agent called, so there is no request context to inherit it from,
- # and "which agent is falling behind" is the only question this
- # line is ever read to answer.
- with log_context(agent_id=agent_id):
- logger.warning(
- "[EVENT-BUF] agent=%s exceeded %s buffered events; dropped "
- "through seq=%s in rooms %s — readers resuming from before "
- "this will be told they missed events",
- agent_id,
- self._max_events,
- max(dropped.values()),
- ", ".join(sorted(dropped)),
- )
+ self._warn_overflow(agent_id, overflow, dropped)
if dropped:
markers = self._dropped_through.setdefault(agent_id, {})
for room_id, seq in dropped.items():
markers[room_id] = max(seq, markers.get(room_id, 0))
+
+ def _warn_overflow(
+ self, agent_id: str, overflow: int, dropped: dict[str, int]
+ ) -> None:
+ """Log that the cap forced events out, at most once per interval per agent.
+
+ The drops in between are not lost: the metric still counts each one, and
+ the next line says how many went unlogged.
+ """
+ now = time.monotonic()
+ last = self._overflow_warned_at.get(agent_id)
+ if last is not None and now - last < OVERFLOW_WARNING_INTERVAL_SECONDS:
+ self._overflow_unlogged[agent_id] = (
+ self._overflow_unlogged.get(agent_id, 0) + overflow
+ )
+ return
+ unlogged = self._overflow_unlogged.pop(agent_id, 0)
+ self._overflow_warned_at[agent_id] = now
+ # The agent is a field as well as being in the message: this fires
+ # from the buffer's own bookkeeping rather than from anything the
+ # agent called, so there is no request context to inherit it from,
+ # and "which agent is falling behind" is the only question this
+ # line is ever read to answer.
+ with log_context(agent_id=agent_id):
+ logger.warning(
+ "[EVENT-BUF] agent=%s exceeded %s buffered events; dropped "
+ "through seq=%s in rooms %s, plus %s dropped since the last "
+ "warning; readers resuming from before this will be told they "
+ "missed events",
+ agent_id,
+ self._max_events,
+ max(dropped.values()),
+ ", ".join(sorted(dropped)),
+ unlogged,
+ )
diff --git a/core/switch_core/bridges/agent/protocol/stream.py b/core/switch_core/bridges/agent/protocol/stream.py
index a872e768c..fe101188d 100644
--- a/core/switch_core/bridges/agent/protocol/stream.py
+++ b/core/switch_core/bridges/agent/protocol/stream.py
@@ -103,8 +103,8 @@ async def sse_stream(frames: AsyncGenerator[Frame]) -> AsyncIterator[bytes]:
"""The connection's frames as a `text/event-stream` body.
Kept for clients built before the WebSocket (agent-protocol revision 7 and
- older), for a compatibility window: it goes once no client still connects
- over it.
+ older), for a compatibility window: it goes once
+ `switch.agents.connected{transport:sse}` stays at zero.
"""
try:
async for frame in frames:
diff --git a/core/switch_core/clients/agent_consumer.py b/core/switch_core/clients/agent_consumer.py
index 69f9684f1..54dee2e16 100644
--- a/core/switch_core/clients/agent_consumer.py
+++ b/core/switch_core/clients/agent_consumer.py
@@ -708,7 +708,9 @@ class AgentConsumer(Consumer[AgentActor]):
if hosted is not None and not hosted.deliver:
return
- logger.debug("Enqueuing event %s", agent_event.model_dump_json(indent=2))
+ # Serialising runs once per agent per message, so only when it will be seen.
+ if logger.isEnabledFor(logging.DEBUG):
+ logger.debug("Enqueuing event %s", agent_event.model_dump_json(indent=2))
self._event_buffer.enqueue(
self.agent.id,
meta.room_id,
@@ -932,7 +934,10 @@ class AgentConsumer(Consumer[AgentActor]):
if hosted is not None and not hosted.deliver:
return
- logger.debug("Enqueuing media event %s", agent_event.model_dump_json(indent=2))
+ if logger.isEnabledFor(logging.DEBUG):
+ logger.debug(
+ "Enqueuing media event %s", agent_event.model_dump_json(indent=2)
+ )
self._event_buffer.enqueue(
self.agent.id,
meta.room_id,
diff --git a/core/switch_core/clients/client_factory.py b/core/switch_core/clients/client_factory.py
index 5fd33da4a..8bc782f0f 100644
--- a/core/switch_core/clients/client_factory.py
+++ b/core/switch_core/clients/client_factory.py
@@ -19,6 +19,7 @@ from switch_core.transport import MessageTransport
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
from switch_core.transport.postgres import PostgresTransport
+from switch_core.transport.room_cache import RoomDeliveryCache
logger = logging.getLogger(__name__)
@@ -37,6 +38,7 @@ class ClientFactory:
listener: MessageListener,
invites: InviteBus,
ephemeral: EphemeralBus,
+ room_cache: RoomDeliveryCache,
) -> None:
self._client_store = client_store
self._session_factory = session_factory
@@ -48,6 +50,8 @@ class ClientFactory:
self._listener = listener
self._invites = invites
self._ephemeral = ephemeral
+ # One for the process, so every member of a room shares it.
+ self._room_cache = room_cache
self._registry: dict[
str,
tuple[
@@ -122,4 +126,5 @@ class ClientFactory:
listener=self._listener,
invites=self._invites,
ephemeral=self._ephemeral,
+ room_cache=self._room_cache,
)
diff --git a/core/switch_core/config.py b/core/switch_core/config.py
index 3bd15fe50..0b4516dde 100644
--- a/core/switch_core/config.py
+++ b/core/switch_core/config.py
@@ -506,6 +506,16 @@ class SwitchConfig(BaseSettings):
# `report_within_s`; a controller that has not reported for three of these
# is shown as unknown and refused new placements.
controller_status_interval_seconds: int = 60
+ # Share one read of a room's new messages between every client in it
+ # (`transport/room_cache.py`). Without it, a room of N agents reads each
+ # new page N times, which is what exhausted the pool in a restart burst.
+ # The limits bound memory, not correctness: anything outside them is read
+ # from the database, as every client did before the cache.
+ room_delivery_cache_max_bytes: int = 64 * 1024 * 1024
+ room_delivery_cache_max_rooms: int = 5000
+ # At least one delivery page (200), or a fill could not be held.
+ room_delivery_cache_max_rows_per_room: int = 1000
+ room_delivery_cache_max_age_seconds: float = 300.0
# Postgres terminates a connection that sits inside an open transaction
# without executing anything for longer than this (a Postgres interval such
diff --git a/core/switch_core/db/models.py b/core/switch_core/db/models.py
index 51547f0b7..063502810 100644
--- a/core/switch_core/db/models.py
+++ b/core/switch_core/db/models.py
@@ -2469,12 +2469,25 @@ class Message(TenantScoped, Base):
)
id: Mapped[str] = mapped_column(Text, primary_key=True, default=_uuid)
- # Position within the room, from 1, and the cursor the read path pages on.
- # Assigned by MessageStore.create under a per-room lock rather than by a
- # sequence: a sequence hands out numbers when a statement runs, not when it
- # commits, so a row can commit after one with a higher number and a reader
- # paging on `seq > n` would step straight over it. See the store for the
- # argument in full.
+ # The message's position in its room's log: the room's offset, in Kafka
+ # terms (room = partition, seq = offset, a reader's cursor = its consumer
+ # offset). It means something only within its room: unique per room, and
+ # two rooms can each have a message 5.
+ #
+ # Live messages (MessageStore.create) count up from 1 with no gaps, in
+ # commit order, and a number is never renumbered or reused. Reconstructed
+ # history (MessageStore.create_historical) counts down from -1, below the
+ # room's oldest, so seq can be negative and ordering by it still walks the
+ # room in the order things happened.
+ #
+ # A reader's cursor is the last seq it has, and it reads `seq > cursor`.
+ # Commit order is what makes that safe: nothing can later commit behind a
+ # cursor. A Postgres sequence would not give it, because it hands out
+ # numbers when a statement runs rather than when it commits. See
+ # MessageStore._next_seq for the argument in full.
+ #
+ # Not the `sequence` an agent sees on its event stream: that one is the
+ # event buffer's, numbered per agent across all its rooms.
seq: Mapped[int] = mapped_column(BigInteger, nullable=False)
room_id: Mapped[str] = mapped_column(Text, nullable=False)
# Global on purpose: a random, globally-unique identifier — scoping it
diff --git a/core/switch_core/db/stores/delivery_cursor_store.py b/core/switch_core/db/stores/delivery_cursor_store.py
index 5506bb06e..5d3bd45b1 100644
--- a/core/switch_core/db/stores/delivery_cursor_store.py
+++ b/core/switch_core/db/stores/delivery_cursor_store.py
@@ -19,9 +19,10 @@ class DeliveryCursorStore:
async def position(self, session: AsyncSession, agent_id: str, room_id: str) -> int:
"""How far this agent has been delivered, 0 if it never has.
- Zero rather than None because `seq` starts at 1: an agent with no
- cursor is behind the room's first message, which is the same statement
- the number already makes.
+ Zero rather than None because live messages are numbered from 1: an
+ agent with no cursor is behind the room's first live message, which is
+ the same statement the number already makes. Reconstructed history sits
+ below zero and is not delivered.
"""
result = await session.execute(
select(DeliveryCursor.last_seq).where(
diff --git a/core/switch_core/db/stores/message_store.py b/core/switch_core/db/stores/message_store.py
index 42638fa45..90fa3c7a8 100644
--- a/core/switch_core/db/stores/message_store.py
+++ b/core/switch_core/db/stores/message_store.py
@@ -92,6 +92,10 @@ class MessageStore:
async def _next_seq(self, session: AsyncSession, room_id: str) -> int:
"""The next position in this room, allocated in commit order.
+ One above the room's highest, so live positions run 1, 2, 3 with no
+ gaps: a transaction that rolls back takes its number with it, and the
+ next writer gets the same one.
+
A database sequence — `Identity`, `SERIAL`, `nextval` — is the obvious
way to number rows and the wrong one here. It hands out a number when
the INSERT runs, not when the transaction commits, and it does so
diff --git a/core/switch_core/gateway/agents.py b/core/switch_core/gateway/agents.py
index 3dcceaace..4e83fe3ae 100644
--- a/core/switch_core/gateway/agents.py
+++ b/core/switch_core/gateway/agents.py
@@ -110,6 +110,8 @@ async def delete_agent_by_name(
)
if hosted_launch_of(agent.metadata_) is not None:
raise HTTPException(status_code=409, detail=CLOUD_AGENT_DELETE_REFUSED)
+ # Not held across the delete, which opens its own sessions and calls every bridge.
+ await session.commit()
try:
await protocol.delete_agent(agent_name=agent_name)
except ValueError as exc:
@@ -143,6 +145,8 @@ async def delete_agent(
)
if hosted_launch_of(agent.metadata_) is not None:
raise HTTPException(status_code=409, detail=CLOUD_AGENT_DELETE_REFUSED)
+ # Not held across the delete, which opens its own sessions and calls every bridge.
+ await session.commit()
try:
await protocol.delete_agent(agent_id=agent_id)
except ValueError as exc:
@@ -299,6 +303,8 @@ async def register_known_subagents(
),
)
+ # Not held across registration, which opens its own session and calls every bridge.
+ await session.commit()
results: list[BulkRegisterResult] = []
for d in derived:
try:
diff --git a/core/switch_core/gateway/auth.py b/core/switch_core/gateway/auth.py
index df0a63e7b..08532287e 100644
--- a/core/switch_core/gateway/auth.py
+++ b/core/switch_core/gateway/auth.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+import asyncio
import datetime
import logging
from collections.abc import AsyncIterator
@@ -68,6 +69,23 @@ def verify_password(password: str, password_hash: str | None) -> bool:
return result
+async def hash_password_off_loop(password: str) -> str:
+ """`hash_password` on a worker thread.
+
+ bcrypt is slow on purpose, a few hundred milliseconds of CPU per call, and
+ it holds the event loop for all of it: on the loop, every other request in
+ the process waits behind one login. Call it with no transaction open, too,
+ or the connection waits with them.
+ """
+ return await asyncio.to_thread(hash_password, password)
+
+
+async def verify_password_off_loop(password: str, password_hash: str | None) -> bool:
+ """`verify_password` on a worker thread, for the same reason as
+ `hash_password_off_loop`."""
+ return await asyncio.to_thread(verify_password, password, password_hash)
+
+
def create_jwt(
user_id: str, email: str, role: str, keyring: Keyring, tenant_id: str | None
) -> str:
@@ -716,11 +734,15 @@ async def get_tenant_is_admin(
should be built from — see ``UserStore.administers`` for what it actually
checks.
- Also callable directly (not just as a FastAPI dependency) from any
- handler or service function that already holds a bound `session`, the
- caller's `User`, and a `UserStore`.
+ Ends the read's transaction before returning, the way `get_current_user`
+ does, so the connection is back in the pool before the route body runs. A
+ route that went on to open a second session, or to call a messaging
+ platform, would otherwise do it holding this one. So it must not be called
+ on a session with writes pending.
"""
- return await user_store.administers(session, user)
+ is_admin = await user_store.administers(session, user)
+ await session.commit()
+ return is_admin
async def get_tenant_is_owner(
@@ -732,9 +754,12 @@ async def get_tenant_is_owner(
Strictly narrower than ``get_tenant_is_admin`` — see
``authz.owns_tenant``. A route that changes the ownership set needs both:
- ``require_tenant_admin`` to get in, this to make the call.
+ ``require_tenant_admin`` to get in, this to make the call. Ends its read's
+ transaction for the same reason ``get_tenant_is_admin`` does.
"""
- return await user_store.owns(session, user)
+ is_owner = await user_store.owns(session, user)
+ await session.commit()
+ return is_owner
async def require_tenant_admin(
diff --git a/core/switch_core/gateway/auth_routes.py b/core/switch_core/gateway/auth_routes.py
index cd8f88bce..1fc547ff2 100644
--- a/core/switch_core/gateway/auth_routes.py
+++ b/core/switch_core/gateway/auth_routes.py
@@ -24,12 +24,12 @@ from switch_core.gateway.auth import (
describe_session_state,
get_authenticated_session,
get_current_user,
- hash_password,
+ hash_password_off_loop,
initial_tenant_claim,
list_tenant_memberships,
require_admin,
set_session_cookie,
- verify_password,
+ verify_password_off_loop,
)
from switch_core.gateway.dependencies import (
get_bridge_store,
@@ -125,11 +125,12 @@ async def login(
raise HTTPException(status_code=403, detail="Password login is disabled")
user = await user_store.get_by_email(session, req.email)
- if user is None or not verify_password(req.password, user.password_hash):
- raise HTTPException(status_code=401, detail="Invalid email or password")
- # Ends the read so its connection goes back to the pool before the
- # membership lookup takes one: a request holds one connection at a time.
+ # One connection at a time, and none held across bcrypt.
await session.commit()
+ if user is None or not await verify_password_off_loop(
+ req.password, user.password_hash
+ ):
+ raise HTTPException(status_code=401, detail="Invalid email or password")
set_session_cookie(
response,
@@ -163,6 +164,30 @@ async def _prewarm(
return SignupMachine(status="starting", reason=None)
+async def _refuse_signup(
+ session: AsyncSession, user_store: UserStore, email: str, config: SwitchConfig
+) -> None:
+ """Raise if this sign-up must be refused: the hourly cap is reached, or the
+ email is taken. Rolls back before raising, so a refusal never leaves the
+ transaction, or an advisory lock taken in it, open."""
+ created, retry_after = await user_store.created_in_last_hour(session)
+ if created >= config.gateway_signup_max_per_hour:
+ logger.warning(
+ "Refused sign-up: %d users created in the last hour (cap %d)",
+ created,
+ config.gateway_signup_max_per_hour,
+ )
+ await session.rollback()
+ raise HTTPException(
+ status_code=429,
+ detail="Too many sign-ups on this server in the last hour. Try again later.",
+ headers={"Retry-After": str(retry_after)},
+ )
+ if await user_store.get_by_email(session, email) is not None:
+ await session.rollback()
+ raise HTTPException(status_code=409, detail="Email already registered")
+
+
@router.post("/auth/signup", status_code=201)
async def signup(
req: SignupRequest,
@@ -185,31 +210,22 @@ async def signup(
)
with tenant_scope(TENANT_ZERO_ID):
+ # Checked before the hash so a refusal stays cheap, and again under the
+ # lock after it; neither the lock nor a connection is held across bcrypt.
+ await _refuse_signup(session, user_store, req.email, config)
+ await session.rollback()
+ password_hash = await hash_password_off_loop(req.password)
+
await session.execute(
text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
{"key": "gateway-signup"},
)
- created, retry_after = await user_store.created_in_last_hour(session)
- if created >= config.gateway_signup_max_per_hour:
- logger.warning(
- "Refused sign-up: %d users created in the last hour (cap %d)",
- created,
- config.gateway_signup_max_per_hour,
- )
- await session.rollback()
- raise HTTPException(
- status_code=429,
- detail="Too many sign-ups on this server in the last hour. Try again later.",
- headers={"Retry-After": str(retry_after)},
- )
- if await user_store.get_by_email(session, req.email) is not None:
- await session.rollback()
- raise HTTPException(status_code=409, detail="Email already registered")
+ await _refuse_signup(session, user_store, req.email, config)
user = User(
name=req.display_name or req.email.split("@")[0],
email=req.email,
role="user",
- password_hash=hash_password(req.password),
+ password_hash=password_hash,
)
try:
async with session.begin_nested():
@@ -355,10 +371,10 @@ async def change_password(
session: Annotated[AsyncSession, Depends(get_session)],
user: Annotated[User, Depends(get_current_user)],
) -> dict[str, bool]:
- if not verify_password(req.current_password, user.password_hash):
+ if not await verify_password_off_loop(req.current_password, user.password_hash):
raise HTTPException(status_code=403, detail="Current password is incorrect")
- user.password_hash = hash_password(req.new_password)
+ user.password_hash = await hash_password_off_loop(req.new_password)
await session.commit()
return {"ok": True}
@@ -392,12 +408,15 @@ async def create_user(
existing = await user_store.get_by_email(session, req.email)
if existing is not None:
raise HTTPException(status_code=409, detail="Email already registered")
+ # Not held across bcrypt.
+ await session.commit()
+ password_hash = await hash_password_off_loop(req.password)
user = User(
name=req.name,
email=req.email,
role=req.role,
- password_hash=hash_password(req.password),
+ password_hash=password_hash,
)
await user_store.create(session, user)
await session.commit()
diff --git a/core/switch_core/gateway/collaborations.py b/core/switch_core/gateway/collaborations.py
index f4cf2cc5d..842f0406e 100644
--- a/core/switch_core/gateway/collaborations.py
+++ b/core/switch_core/gateway/collaborations.py
@@ -324,18 +324,6 @@ async def update_bridge(
),
)
- if payload.agent_greetings_enabled is not None:
- bridge = await bridge_store.set_agent_greetings_enabled(
- session, bridge_id, payload.agent_greetings_enabled
- )
- if payload.channel_creation_enabled is not None:
- bridge = await bridge_store.set_channel_creation_enabled(
- session, bridge_id, payload.channel_creation_enabled
- )
- if payload.preconfigured is not None:
- bridge = await bridge_store.set_preconfigured(
- session, bridge_id, payload.preconfigured
- )
if payload.connection_config is not None:
current = bridge.connection_config or {}
merged = {**current, **payload.connection_config}
@@ -347,6 +335,9 @@ async def update_bridge(
status_code=422,
detail="event_delivery cannot be changed on an existing connection.",
)
+ # Not held across the platform's credential check; nothing is written
+ # until it passes, so a refused edit still changes nothing.
+ await session.commit()
try:
await collab_lifecycle.check_edited_connection_config(
bridge_id=bridge_id, bridge_type=bridge.type, connection_config=merged
@@ -366,6 +357,20 @@ async def update_bridge(
raise HTTPException(status_code=422, detail=str(exc)) from exc
except (BridgeClaimConflict, BridgeCredentialError) as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+ if payload.agent_greetings_enabled is not None:
+ bridge = await bridge_store.set_agent_greetings_enabled(
+ session, bridge_id, payload.agent_greetings_enabled
+ )
+ if payload.channel_creation_enabled is not None:
+ bridge = await bridge_store.set_channel_creation_enabled(
+ session, bridge_id, payload.channel_creation_enabled
+ )
+ if payload.preconfigured is not None:
+ bridge = await bridge_store.set_preconfigured(
+ session, bridge_id, payload.preconfigured
+ )
+ if payload.connection_config is not None:
bridge = await bridge_store.merge_connection_config(
session, bridge_id, dict(payload.connection_config)
)
@@ -497,6 +502,8 @@ async def search_bridge_directory(
u.external_user_id: u
for u in await external_user_store.get_by_bridge(session, bridge_id)
}
+ # Not held across the platform search below.
+ await session.commit()
source: Literal["directory", "known"] = "directory"
note: str | None = None
@@ -682,6 +689,8 @@ async def claim_bridge_identity(
# the account exists. Provisioning mints a human actor, so taking the
# request's word for it would let any signed-in user conjure accounts
# for people who do not exist.
+ # Not held across the directory search or provisioning's own session.
+ await session.commit()
await _require_directory_account(
collab_lifecycle,
bridge_id=bridge_id,
@@ -844,6 +853,8 @@ async def delete_bridge(
"display_name": bridge.display_name,
"rooms_deleted": len(rooms),
}
+ # Not held across the deletes, which open their own sessions and call Matrix.
+ await session.commit()
for room in rooms:
await room_service.delete_room(room.id)
diff --git a/core/switch_core/gateway/connectors.py b/core/switch_core/gateway/connectors.py
index 8dd425657..a32b6b587 100644
--- a/core/switch_core/gateway/connectors.py
+++ b/core/switch_core/gateway/connectors.py
@@ -143,7 +143,8 @@ async def delete_connector(
raise HTTPException(
status_code=403, detail="Not authorized to delete this connector"
)
-
+ # Not held across removal, which deletes the agents on sessions of its own.
+ await session.commit()
try:
await connector_lifecycle.remove(connector_id)
except ValueError as exc:
diff --git a/core/switch_core/gateway/provider_verifications.py b/core/switch_core/gateway/provider_verifications.py
index 2dead89c8..518bf467a 100644
--- a/core/switch_core/gateway/provider_verifications.py
+++ b/core/switch_core/gateway/provider_verifications.py
@@ -72,6 +72,8 @@ async def worker_session(
raise HTTPException(
403, "Connection check owner is no longer a member."
)
+ # Not held while the route reads the worker's body, at the worker's pace.
+ await session.commit()
yield session
diff --git a/core/switch_core/gateway/rooms.py b/core/switch_core/gateway/rooms.py
index c8b13f180..773cbf240 100644
--- a/core/switch_core/gateway/rooms.py
+++ b/core/switch_core/gateway/rooms.py
@@ -91,6 +91,28 @@ async def _require_room(
return room
+async def _require_room_released(
+ session: AsyncSession,
+ room_store: RoomStore,
+ room_id: str,
+ user: User,
+ action: Action,
+ is_admin: bool,
+) -> Room:
+ """`_require_room`, then end the read's transaction.
+
+ For a route that goes on to call `RoomService` or `RoomYamlService`, which
+ open sessions of their own and may call Matrix or a messaging platform.
+ Holding this read's connection while waiting for another from the same
+ pool is how the pool runs dry under load: every connection held by a
+ request that is waiting for one more. The room stays readable after the
+ commit (`expire_on_commit=False`).
+ """
+ room = await _require_room(session, room_store, room_id, user, action, is_admin)
+ await session.commit()
+ return room
+
+
async def _external_channel_url(room: Room) -> str | None:
"""Native deeplink to open the room's external channel in the messaging
app, built by the live collaboration adapter. None when not bridged, the
@@ -149,6 +171,9 @@ async def _build_room_detail(
roles = await _list_room_role_details(session, room.id, protocol)
join_event_listeners = await room_store.get_join_event_listeners(session, room.id)
+ # Not held across the deeplink, which can ask the platform on a cache miss.
+ await session.commit()
+ external_channel_url = await _external_channel_url(room)
return RoomDetail(
id=room.id,
@@ -162,7 +187,7 @@ async def _build_room_detail(
bridge_id=room.bridge_id,
bridge_display_name=bridge_display_name,
bridge_type=bridge_type,
- external_channel_url=await _external_channel_url(room),
+ external_channel_url=external_channel_url,
group_id=room.group_id,
group_name=group_name,
read_visibility=room.read_visibility,
@@ -478,7 +503,7 @@ async def export_room_yaml(
) -> Response:
"""Export a room to YAML in the same surface ``/rooms/from-yaml`` accepts.
Each section can be dropped via its boolean toggle (default included)."""
- await _require_room(session, room_store, room_id, user, "read", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "read", is_admin)
yaml_text = await rooms_yaml.export(
room_id,
agents=agents,
@@ -520,7 +545,9 @@ async def patch_room(
user: Annotated[User, Depends(get_current_user)],
is_admin: Annotated[bool, Depends(get_tenant_is_admin)],
) -> RoomDetail:
- target = await _require_room(session, room_store, room_id, user, "write", is_admin)
+ target = await _require_room_released(
+ session, room_store, room_id, user, "write", is_admin
+ )
# Changing a room's access permission (read/write visibility) is reserved
# for the room's owner and tenant admins — plain write access (which a
# publicly-writable room grants to everyone) is not enough. Otherwise any
@@ -677,7 +704,7 @@ async def put_protection(
user: Annotated[User, Depends(get_current_user)],
is_admin: Annotated[bool, Depends(get_tenant_is_admin)],
) -> RoomDetail:
- await _require_room(session, room_store, room_id, user, "write", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)
try:
await room_service.update_protection_config(room_id, req.protection_config)
except ValueError:
@@ -704,7 +731,7 @@ async def put_observe(
user: Annotated[User, Depends(get_current_user)],
is_admin: Annotated[bool, Depends(get_tenant_is_admin)],
) -> RoomDetail:
- await _require_room(session, room_store, room_id, user, "write", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)
try:
await room_service.update_observe_config(room_id, req.observe_config)
except ValueError:
@@ -731,7 +758,7 @@ async def post_room_agents(
user: Annotated[User, Depends(get_current_user)],
is_admin: Annotated[bool, Depends(get_tenant_is_admin)],
) -> RoomDetail:
- await _require_room(session, room_store, room_id, user, "write", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)
try:
await room_service.add_agents_to_room(
room_id,
@@ -764,7 +791,7 @@ async def patch_room_agent(
user: Annotated[User, Depends(get_current_user)],
is_admin: Annotated[bool, Depends(get_tenant_is_admin)],
) -> RoomDetail:
- await _require_room(session, room_store, room_id, user, "write", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)
try:
await room_service.set_join_event_listeners(
room_id, {agent_id: req.receives_join_events}
@@ -792,7 +819,7 @@ async def delete_room_agent(
user: Annotated[User, Depends(get_current_user)],
is_admin: Annotated[bool, Depends(get_tenant_is_admin)],
) -> RoomDetail:
- await _require_room(session, room_store, room_id, user, "write", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)
try:
await room_service.remove_agents_from_room(room_id, [agent_id])
except ValueError as e:
@@ -819,7 +846,7 @@ async def post_room_users(
user: Annotated[User, Depends(get_current_user)],
is_admin: Annotated[bool, Depends(get_tenant_is_admin)],
) -> RoomDetail:
- await _require_room(session, room_store, room_id, user, "write", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)
try:
await room_service.add_users_to_room(room_id, req.user_names)
except ValueError as e:
@@ -845,7 +872,7 @@ async def _set_archived(
user: User,
is_admin: bool,
) -> RoomDetail:
- await _require_room(session, room_store, room_id, user, "write", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)
try:
await room_service.set_room_archived(room_id, archived)
except ValueError:
@@ -922,7 +949,7 @@ async def delete_room(
user: Annotated[User, Depends(get_current_user)],
is_admin: Annotated[bool, Depends(get_tenant_is_admin)],
) -> dict[str, bool]:
- await _require_room(session, room_store, room_id, user, "delete", is_admin)
+ await _require_room_released(session, room_store, room_id, user, "delete", is_admin)
try:
await room_service.delete_room(room_id)
except ValueError:
@@ -951,6 +978,8 @@ async def bulk_delete_rooms(
"Skipping room %s during bulk delete: not authorized", room_id
)
continue
+ # See `_require_room_released`.
+ await session.commit()
await room_service.delete_room(room_id)
deleted += 1
return BulkDeleteResponse(deleted=deleted)
@@ -980,6 +1009,8 @@ async def bulk_archive_rooms(
"Skipping room %s during bulk archive: not authorized", room_id
)
continue
+ # See `_require_room_released`.
+ await session.commit()
await room_service.set_room_archived(room_id, req.archived)
updated += 1
return BulkArchiveResponse(updated=updated)
diff --git a/core/switch_core/gateway/template_runs.py b/core/switch_core/gateway/template_runs.py
index c46f4e19a..6f14d05c8 100644
--- a/core/switch_core/gateway/template_runs.py
+++ b/core/switch_core/gateway/template_runs.py
@@ -199,6 +199,8 @@ async def _control(
"template, or an admin can do that."
),
)
+ # Not held across set_state, which opens its own session and posts in every room.
+ await session.commit()
try:
await service.set_state(
root_id,
diff --git a/core/switch_core/gateway/templates.py b/core/switch_core/gateway/templates.py
index 5af5e995e..6baa5ca93 100644
--- a/core/switch_core/gateway/templates.py
+++ b/core/switch_core/gateway/templates.py
@@ -417,6 +417,8 @@ async def patch_template(
require_manage(principal, locked.owner_id)
if req.content is not None:
+ # Not held across linting (up to 1 MiB of YAML); the update re-checks under lock.
+ await session.commit()
_require_within_size_limit(req.content, config)
_require_storable(req.content)
try:
diff --git a/core/switch_core/gateway/tenants.py b/core/switch_core/gateway/tenants.py
index cca9d8cf4..c1491eb9a 100644
--- a/core/switch_core/gateway/tenants.py
+++ b/core/switch_core/gateway/tenants.py
@@ -460,6 +460,8 @@ async def create_tenant(
user = await user_store.get(session, caller.id)
if user is None:
raise HTTPException(status_code=401, detail="User not found")
+ # Not held across provisioning, which opens a session per tenant.
+ await session.commit()
tenant = await _provision_workspace(
session_factory, user_store, client_lifecycle, caller, req.name
)
diff --git a/core/switch_core/main.py b/core/switch_core/main.py
index bc0595b8f..8c59e09c0 100644
--- a/core/switch_core/main.py
+++ b/core/switch_core/main.py
@@ -151,7 +151,12 @@ from switch_core.observability.bootstrap import (
RuntimeProbes,
start_observability,
)
-from switch_core.observability.pool import install_pool_watermark, pool_stats
+from switch_core.observability.pool import (
+ WaitTimedQueuePool,
+ install_hold_timer,
+ install_pool_watermark,
+ pool_stats,
+)
from switch_core.observability.query import instrument_queries
from switch_core.observability.runtime import EventLoopLag
from switch_core.provisioning import Provisioning
@@ -169,6 +174,7 @@ from switch_core.telemetry.setup import build_telemetry
from switch_core.tenant_context import no_tenant
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
from switch_core.version import switch_core_version
logger = logging.getLogger(__name__)
@@ -396,7 +402,7 @@ async def run(config: SwitchConfig) -> None:
# the schema owner's connection where one is configured, and neither
# belongs on the pooled application engine built below.
encrypted_json.configure(config.keyring)
- engine = create_engine_from_config(config)
+ engine = create_engine_from_config(config, poolclass=WaitTimedQueuePool)
tenants_isolated = await _check_tenant_isolation(config, engine)
session_factory = create_session_factory(engine)
# Wired here rather than inside the engine factory, so the database layer
@@ -408,6 +414,7 @@ async def run(config: SwitchConfig) -> None:
# not change how queries execute.
instrument_queries(engine)
pool_watermark = install_pool_watermark(engine)
+ install_hold_timer(engine)
# Its connection is held rather than borrowed, so it builds its own outside
# the pool. Nothing subscribes yet; it starts with the server so that the
@@ -531,6 +538,18 @@ async def run(config: SwitchConfig) -> None:
# fresh heartbeat row (CHOO-1857 stage B).
connections = AgentConnectionRegistry()
+ # One shared read per room for every client in this process.
+ room_cache = RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=message_store,
+ limits=RoomCacheLimits(
+ max_bytes=config.room_delivery_cache_max_bytes,
+ max_rooms=config.room_delivery_cache_max_rooms,
+ max_rows_per_room=config.room_delivery_cache_max_rows_per_room,
+ max_age_seconds=config.room_delivery_cache_max_age_seconds,
+ ),
+ )
+
# ── Client factory ───────────────────────────────────────────────────────
client_factory = ClientFactory(
client_store=client_store,
@@ -543,6 +562,7 @@ async def run(config: SwitchConfig) -> None:
listener=message_listener,
invites=invites,
ephemeral=ephemeral,
+ room_cache=room_cache,
)
client_factory.register(
"agent",
@@ -608,6 +628,7 @@ async def run(config: SwitchConfig) -> None:
resource_service=resource_service,
session_factory=session_factory,
telemetry=telemetry,
+ room_cache=room_cache,
)
collab_lifecycle._room_service = room_service
@@ -817,8 +838,9 @@ async def run(config: SwitchConfig) -> None:
consumers_running=client_lifecycle.running_count,
connectors_running=connector_lifecycle.running_count,
connectors_configured=connector_lifecycle.expected_count,
- agents_connected=lambda: len(connections.live_agent_ids()),
+ agents_connected=connections.live_agents_by_transport,
pool_stats=lambda: pool_stats(engine, pool_watermark),
+ room_cache_stats=room_cache.stats,
)
# ── Lifespan: start server-side connectors once HTTP is serving ────────
@@ -841,6 +863,7 @@ async def run(config: SwitchConfig) -> None:
version=switch_core_version(),
session_factory=session_factory,
probes=probes,
+ db_server_engine=lambda: create_unpooled_engine(config),
)
asyncio.create_task(connector_lifecycle.start_all())
sweep_task = asyncio.create_task(_runtime_state_sweep_loop(protocol))
diff --git a/core/switch_core/management/controller_routes.py b/core/switch_core/management/controller_routes.py
index e28424ca2..66dfb2094 100644
--- a/core/switch_core/management/controller_routes.py
+++ b/core/switch_core/management/controller_routes.py
@@ -6,26 +6,36 @@ names a controller then requires it to be the token's own. Enrollment and
token exchange carry their secret in the body and resolve their tenant from
it here.
-The controller's stream (`/v1/controllers/{id}/...`) is Core's: these routes
-authenticate and parse, and Core's `ControllerPresence` and controller stream
-do the rest. Management only adds its nudges and the first frame's
+The controller's connection (`/v1/controllers/{id}/connection` and its
+socket) is Core's: these routes authenticate and parse, and Core's
+`ControllerPresence` and controller stream do the rest. Management only adds its nudges and the first frame's
assignment revision.
Every failure is answered in the contract's error envelope (`errors.py`).
`Switch-Controller-Protocol` is checked when sent, and every response says
-which protocol versions this server accepts.
+which protocol versions this server accepts. Protocol 2 moved the stream and
+the beat onto one WebSocket; a protocol 1 controller is refused with 426.
"""
from __future__ import annotations
+import asyncio
import hashlib
import json
import logging
-from collections.abc import Awaitable, Callable, Coroutine
+from collections.abc import AsyncGenerator, Awaitable, Callable, Coroutine
from typing import Annotated, Any
-from fastapi import APIRouter, Depends, Header, Request, Response
-from fastapi.responses import JSONResponse, StreamingResponse
+from fastapi import (
+ APIRouter,
+ Depends,
+ Header,
+ Request,
+ Response,
+ WebSocket,
+ WebSocketDisconnect,
+)
+from fastapi.responses import JSONResponse
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.bridges.agent.auth import ControllerPrincipal
@@ -35,8 +45,9 @@ from switch_core.bridges.agent.protocol.controller_presence import (
ControllerConnectionError,
)
from switch_core.bridges.agent.protocol.controller_stream import (
- KEEPALIVE_INTERVAL_SECONDS,
- STREAM_HEADERS,
+ IDLE,
+ IDLE_INTERVAL_SECONDS,
+ ControllerFrame,
controller_event_stream,
)
from switch_core.bridges.agent.protocol.liveness import HEARTBEAT_INTERVAL_SECONDS
@@ -53,7 +64,6 @@ from switch_core.management.dependencies import (
from switch_core.management.errors import ManagementError, ManagementRoute, error_body
from switch_core.management.schemas import (
MAX_STATUS_BYTES,
- ControllerBeatRequest,
ControllerConnectionRequest,
DefinitionV1,
EnrollRequest,
@@ -69,8 +79,8 @@ logger = logging.getLogger(__name__)
PROTOCOL_HEADER = "Switch-Controller-Protocol"
PROTOCOL_ACCEPTS_HEADER = "Switch-Controller-Protocol-Accepts"
-SUPPORTED_PROTOCOLS = "1-1"
-_SUPPORTED_PROTOCOL_VERSIONS = frozenset({"1"})
+SUPPORTED_PROTOCOLS = "2-2"
+_SUPPORTED_PROTOCOL_VERSIONS = frozenset({"2"})
class ControllerRoute(ManagementRoute):
@@ -315,7 +325,7 @@ async def open_connection(
The agents bound to it are attached on the stream, each from its cursor
here; the response names them. Opening again is a takeover: the earlier
- connection's stream ends with `taken_over`, and its beats are refused.
+ connection's socket ends with `taken_over`, and its beats are refused.
"""
presence = protocol.connections.controllers
try:
@@ -341,21 +351,21 @@ async def open_connection(
}
[email protected]("/v1/controllers/{controller_id}/events")
-async def controller_events(
- principal: PathController,
- management: Management,
- protocol: Protocol,
- session_factory: Annotated[
- async_sessionmaker[AsyncSession], Depends(get_management_session_factory)
- ],
+async def attach_stream(
+ *,
+ principal: ControllerPrincipal,
+ management: ManagementService,
+ protocol: AgentCore,
+ session_factory: async_sessionmaker[AsyncSession],
connection_id: str,
generation: int,
-) -> StreamingResponse:
- """The controller's one stream: its agents' events and the management nudges.
+) -> AsyncGenerator[ControllerFrame]:
+ """Attach the stream to an open connection: its agents' events and the
+ management nudges, starting with `connection_state`.
The nudge subscription is taken before the first frame is built, so a
change that commits while the stream is opening is still delivered.
+ Raises the refusal when the connection cannot take a stream.
"""
presence = protocol.connections.controllers
nudges = management.notifier.subscribe(principal.controller_id)
@@ -376,48 +386,40 @@ async def controller_events(
"heartbeat_interval_s": HEARTBEAT_INTERVAL_SECONDS,
}
token = presence.attach_stream(conn)
- return StreamingResponse(
- controller_event_stream(
- conn=conn,
- stream_token=token,
- presence=presence,
- buffer=protocol.event_buffer,
- approvals=protocol.approval_outcomes,
- nudges=nudges,
- opening=opening,
- rooms_of=_rooms_reader(protocol),
- keepalive_seconds=KEEPALIVE_INTERVAL_SECONDS,
- ),
- media_type="text/event-stream",
- headers=STREAM_HEADERS,
+ return controller_event_stream(
+ conn=conn,
+ stream_token=token,
+ presence=presence,
+ buffer=protocol.event_buffer,
+ approvals=protocol.approval_outcomes,
+ nudges=nudges,
+ opening=opening,
+ rooms_of=_rooms_reader(protocol),
+ idle_seconds=IDLE_INTERVAL_SECONDS,
)
[email protected]("/v1/controllers/{controller_id}/connection/beat")
-async def connection_beat(
- body: ControllerBeatRequest,
- principal: PathController,
- protocol: Protocol,
-) -> dict[str, Any]:
+def record_beat(
+ *,
+ principal: ControllerPrincipal,
+ protocol: AgentCore,
+ connection_id: str,
+ generation: int,
+ cursors: dict[str, int],
+) -> list[str]:
"""Keep the controller, and so every agent on it, live; confirm cursors.
- The cursors confirmed here are where a stream reattached to this
- connection resumes each agent.
-
- Every two seconds; six without one and its agents are not live. Refused
- with `taken_over` once another connection has replaced this one, which is
- terminal for the client, and with `no_stream` or `unknown_connection`
- when the stream has to be opened again.
+ A pong on the socket, every heartbeat interval; six seconds without one
+ and its agents are not live. The cursors confirmed here are where a
+ stream reattached to this connection resumes each agent. Raises the
+ presence's refusal: `taken_over` once another connection has replaced
+ this one, terminal for the client, and `no_stream` or
+ `unknown_connection` when the connection has to be opened again.
"""
presence = protocol.connections.controllers
- try:
- conn = presence.beat(
- principal.controller_id, body.connection_id, body.generation
- )
- except ControllerConnectionError as exc:
- raise _connection_refusal(exc) from exc
+ conn = presence.beat(principal.controller_id, connection_id, generation)
agents = presence.agents_of(principal.controller_id)
- for agent_id, cursor in body.cursors.items():
+ for agent_id, cursor in cursors.items():
binding = presence.binding(agent_id)
if agent_id not in agents or binding is None:
continue
@@ -426,4 +428,180 @@ async def connection_beat(
confirmed = min(cursor, protocol.event_buffer.head(agent_id))
protocol.event_buffer.confirm(agent_id, presence.holder_id(binding), confirmed)
presence.resume_from(conn, agent_id, confirmed)
- return {"agents": sorted(agents)}
+ return sorted(agents)
+
+
+def _refused(error: ManagementError) -> dict[str, Any]:
+ """The frame a refused socket gets before it closes with 4000 + status,
+ as the bearer middleware sends its own refusals."""
+ return {
+ "event": "refused",
+ "data": {
+ "status": error.status_code,
+ "detail": {"code": error.code, "message": error.message},
+ },
+ }
+
+
+def _cursors_of(message: dict[str, Any]) -> dict[str, int]:
+ raw = message.get("cursors")
+ if not isinstance(raw, dict):
+ return {}
+ return {
+ agent_id: cursor
+ for agent_id, cursor in raw.items()
+ if isinstance(agent_id, str)
+ and isinstance(cursor, int)
+ and not isinstance(cursor, bool)
+ and cursor >= 0
+ }
+
+
[email protected]("/v1/controllers/{controller_id}/connection/ws")
+async def connection_socket(
+ websocket: WebSocket,
+ controller_id: str,
+ management: Management,
+ protocol: Protocol,
+ session_factory: Annotated[
+ async_sessionmaker[AsyncSession], Depends(get_management_session_factory)
+ ],
+ connection_id: str,
+ generation: int,
+) -> None:
+ """The controller's one connection: its agents' events and the management
+ nudges down, its heartbeat up, on one WebSocket.
+
+ Attaches to a connection opened with `POST .../connection`. Frames are
+ JSON `{"event", "data"}`, the events the stream carries. The server sends
+ `{"event": "ping"}` every heartbeat interval and the controller answers
+ `{"type": "pong", "cursors": {agent_id: n}}`, which is its beat. A refused
+ attach is a `refused` frame, then a close with 4000 plus the status; a
+ refused beat is an `evicted` frame naming why, then a close.
+ """
+ await websocket.accept()
+ try:
+ principal = _socket_principal(websocket, controller_id)
+ frames = await attach_stream(
+ principal=principal,
+ management=management,
+ protocol=protocol,
+ session_factory=session_factory,
+ connection_id=connection_id,
+ generation=generation,
+ )
+ except ManagementError as error:
+ await websocket.send_json(_refused(error))
+ await websocket.close(code=4000 + error.status_code)
+ return
+
+ outbox: asyncio.Queue[ControllerFrame] = asyncio.Queue(maxsize=_SOCKET_OUTBOX)
+ pump = asyncio.create_task(_pump(frames, outbox))
+ pongs = asyncio.create_task(
+ _receive_pongs(websocket, principal, protocol, connection_id, generation)
+ )
+ loop = asyncio.get_running_loop()
+ next_ping = loop.time() + HEARTBEAT_INTERVAL_SECONDS
+ try:
+ # One writer: frames and pings both go out from this loop.
+ while not pump.done() or not outbox.empty():
+ if pongs.done():
+ refusal = pongs.result()
+ if refusal is not None:
+ await websocket.send_json(
+ {
+ "event": "evicted",
+ "data": {"code": refusal.code, "reason": str(refusal)},
+ }
+ )
+ return
+ try:
+ frame = await asyncio.wait_for(
+ outbox.get(), timeout=max(0.0, next_ping - loop.time())
+ )
+ except TimeoutError:
+ frame = None
+ if frame is not None and frame is not IDLE:
+ await websocket.send_json(frame)
+ if loop.time() >= next_ping:
+ await websocket.send_json({"event": "ping", "data": {}})
+ next_ping = loop.time() + HEARTBEAT_INTERVAL_SECONDS
+ except WebSocketDisconnect:
+ return
+ except RuntimeError:
+ # The server closed the socket under us (uvicorn does on shutdown).
+ return
+ finally:
+ pump.cancel()
+ pongs.cancel()
+ failure = pump.exception() if pump.done() and not pump.cancelled() else None
+ if failure is not None:
+ logger.error(
+ "Controller %s connection %s: delivery failed, closing its socket",
+ principal.controller_id,
+ connection_id,
+ exc_info=failure,
+ )
+ try:
+ await websocket.close(code=1011 if failure is not None else 1000)
+ except RuntimeError:
+ pass # the client closed it first
+
+
+# Frames buffered between the stream and the socket.
+_SOCKET_OUTBOX = 64
+
+
+def _socket_principal(websocket: WebSocket, controller_id: str) -> ControllerPrincipal:
+ """The controller the bearer middleware authenticated, checked against the
+ path and the protocol it declares, as the HTTP routes check theirs."""
+ requested = websocket.headers.get(PROTOCOL_HEADER)
+ if requested is not None and requested.strip() not in _SUPPORTED_PROTOCOL_VERSIONS:
+ raise ManagementError(
+ 426,
+ reason_codes.PROTOCOL_UNSUPPORTED,
+ f"Controller protocol {requested!r} is not supported; "
+ f"this server accepts {SUPPORTED_PROTOCOLS}.",
+ )
+ return require_controller(controller_id, get_controller_principal(websocket))
+
+
+async def _pump(
+ frames: AsyncGenerator[ControllerFrame], outbox: asyncio.Queue[ControllerFrame]
+) -> None:
+ # Closed here: cancelled while waiting on the outbox, the stream would
+ # otherwise detach only when the generator is garbage collected.
+ try:
+ async for frame in frames:
+ await outbox.put(frame)
+ finally:
+ await frames.aclose()
+
+
+async def _receive_pongs(
+ websocket: WebSocket,
+ principal: ControllerPrincipal,
+ protocol: AgentCore,
+ connection_id: str,
+ generation: int,
+) -> ControllerConnectionError | None:
+ """Turn each pong into a beat, until the controller goes or a beat is
+ refused. Returns the refusal, so the socket can say which before it
+ closes; None when the controller simply went away."""
+ while True:
+ try:
+ message = await websocket.receive_json()
+ except (WebSocketDisconnect, ValueError, RuntimeError):
+ return None
+ if not isinstance(message, dict) or message.get("type") != "pong":
+ continue
+ try:
+ record_beat(
+ principal=principal,
+ protocol=protocol,
+ connection_id=connection_id,
+ generation=generation,
+ cursors=_cursors_of(message),
+ )
+ except ControllerConnectionError as refusal:
+ return refusal
diff --git a/core/switch_core/management/dependencies.py b/core/switch_core/management/dependencies.py
index d9f14b50c..593f83967 100644
--- a/core/switch_core/management/dependencies.py
+++ b/core/switch_core/management/dependencies.py
@@ -10,8 +10,8 @@ from __future__ import annotations
from typing import Any
-from fastapi import Request
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+from starlette.requests import HTTPConnection
from switch_core.bridges.agent.auth import ControllerPrincipal
from switch_core.management import reason_codes
@@ -45,7 +45,7 @@ def get_management_session_factory() -> async_sessionmaker[AsyncSession]:
return _state["session_factory"] # type: ignore[no-any-return]
-def get_controller_principal(request: Request) -> ControllerPrincipal:
+def get_controller_principal(request: HTTPConnection) -> ControllerPrincipal:
"""The controller the bearer middleware authenticated."""
principal = request.scope.get("controller")
if not isinstance(principal, ControllerPrincipal):
diff --git a/core/switch_core/management/schemas.py b/core/switch_core/management/schemas.py
index 2373d0c19..f815994a8 100644
--- a/core/switch_core/management/schemas.py
+++ b/core/switch_core/management/schemas.py
@@ -320,14 +320,6 @@ class ControllerConnectionRequest(_ControllerStreamBody):
}
-class ControllerBeatRequest(_ControllerStreamBody):
- """A beat: the connection is still there, and these cursors are confirmed."""
-
- connection_id: str
- generation: int
- cursors: dict[str, int]
-
-
# ── Gateway requests ──────────────────────────────────────────────────────────
diff --git a/core/switch_core/observability/bootstrap.py b/core/switch_core/observability/bootstrap.py
index dfd4966a2..31439dcd9 100644
--- a/core/switch_core/observability/bootstrap.py
+++ b/core/switch_core/observability/bootstrap.py
@@ -14,7 +14,7 @@ from collections.abc import Callable, Iterator, Mapping
from dataclasses import dataclass
import httpx
-from sqlalchemy.ext.asyncio import async_sessionmaker
+from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker
from switch_core.config import SwitchConfig
from switch_core.logging_context import LogContextFilter
@@ -26,7 +26,11 @@ from switch_core.observability.catalogue import (
DB_POOL_IN_USE,
DB_POOL_OVERFLOW,
DB_POOL_SIZE,
+ DELIVERY_CACHE_BYTES,
+ DELIVERY_CACHE_ROOMS,
+ RUNTIME_STARTS,
)
+from switch_core.observability.db_server import DbServerSampler
from switch_core.observability.exporter import MetricsExporter
from switch_core.observability.health import (
HealthMonitor,
@@ -55,6 +59,7 @@ from switch_core.observability.runtime import (
RuntimeMetrics,
log_unreadable_sources,
)
+from switch_core.transport.room_cache import RoomCacheStats
logger = logging.getLogger(__name__)
@@ -83,10 +88,13 @@ class RuntimeProbes:
consumers_running: Callable[[], int]
connectors_running: Callable[[], int]
connectors_configured: Callable[[], int]
- agents_connected: Callable[[], int]
+ # Agents per (transport, client); see
+ # `AgentConnectionRegistry.live_agents_by_transport`.
+ agents_connected: Callable[[], Mapping[tuple[str, str], int]]
# None when the engine's pool does not keep these — see
# :mod:`switch_core.observability.pool`.
pool_stats: Callable[[], PoolStats | None]
+ room_cache_stats: Callable[[], RoomCacheStats]
@dataclass
@@ -119,8 +127,19 @@ class Observability:
def _state_readings(probes: RuntimeProbes) -> Callable[[], Iterator[GaugeReading]]:
+ # Every (transport, client) pair reported so far: one that empties out is
+ # reported as 0, or the dashboard would keep showing its last count.
+ seen: set[tuple[str, str]] = {("websocket", "unknown")}
+
def readings() -> Iterator[GaugeReading]:
- yield GaugeReading(AGENTS_CONNECTED, float(probes.agents_connected()), {})
+ connected = probes.agents_connected()
+ seen.update(connected)
+ for transport, client in sorted(seen):
+ yield GaugeReading(
+ AGENTS_CONNECTED,
+ float(connected.get((transport, client), 0)),
+ {"transport": transport, "client": client},
+ )
yield GaugeReading(CONSUMERS_RUNNING, float(probes.consumers_running()), {})
for platform, running in probes.bridges_running_by_platform().items():
yield GaugeReading(
@@ -136,6 +155,10 @@ def _state_readings(probes: RuntimeProbes) -> Callable[[], Iterator[GaugeReading
yield GaugeReading(DB_POOL_SIZE, float(stats.size), {})
yield GaugeReading(DB_POOL_OVERFLOW, float(stats.overflow), {})
+ cache = probes.room_cache_stats()
+ yield GaugeReading(DELIVERY_CACHE_BYTES, float(cache.bytes), {})
+ yield GaugeReading(DELIVERY_CACHE_ROOMS, float(cache.rooms), {})
+
return readings
@@ -144,10 +167,13 @@ def start_observability(
version: str | None,
session_factory: async_sessionmaker,
probes: RuntimeProbes,
+ db_server_engine: Callable[[], AsyncEngine] | None = None,
) -> Observability:
"""Install the registry, start the loops, and hand back the handle.
- Called once, from the server's lifespan.
+ Called once, from the server's lifespan. `db_server_engine` builds the
+ engine the database sampler takes its one connection from; it must not be
+ the application's pooled engine.
"""
monitor = HealthMonitor(
checks=[
@@ -179,6 +205,7 @@ def start_observability(
registry = MetricsRegistry()
install(registry)
+ registry.increment(RUNTIME_STARTS, {})
monitor.install(registry)
RuntimeMetrics(lag).install(registry)
@@ -209,6 +236,12 @@ def start_observability(
tasks.append(
asyncio.create_task(exporter.run_forever(), name="metrics-exporter")
)
+ if db_server_engine is not None:
+ sampler = DbServerSampler(db_server_engine)
+ registry.register_observer(sampler.readings)
+ tasks.append(
+ asyncio.create_task(sampler.run_forever(), name="db-server-sampler")
+ )
logger.info(
"Reporting metrics to %s every %.0fs as service %r.",
client.url_for("metrics"),
diff --git a/core/switch_core/observability/catalogue.py b/core/switch_core/observability/catalogue.py
index 16502642b..48941ec03 100644
--- a/core/switch_core/observability/catalogue.py
+++ b/core/switch_core/observability/catalogue.py
@@ -124,6 +124,33 @@ DB_POOL_TIMEOUTS = _spec(
"at its ceiling is only suggestive: this is a request that got no connection "
"at all within the timeout.",
)
+# How long a connection stayed borrowed, by who borrowed it. The peak above
+# says the pool filled; this says which code was holding it. `caller` is the
+# first switch_core function on the stack when the connection was handed out,
+# as `module:function`, so the series count is bounded by the code, not the
+# traffic. A hold far longer than the queries inside it is code awaiting
+# something else with a connection in hand.
+DB_POOL_HOLD_DURATION = _spec(
+ "switch.db.pool.hold.duration",
+ "histogram",
+ "ms",
+ "How long a pooled connection stayed checked out, by the code that "
+ "borrowed it. Compare with query duration: a hold much longer than its "
+ "queries is code waiting on something else while holding a connection.",
+ "caller",
+)
+# The timeouts above count requests that waited the full `db_pool_timeout`;
+# this is every wait, so a pool that makes requests queue for 200 ms shows up
+# long before one makes them give up.
+DB_POOL_WAIT_DURATION = _spec(
+ "switch.db.pool.wait.duration",
+ "histogram",
+ "ms",
+ "Time from asking the pool for a connection to getting one, including "
+ "opening a new one when the pool grows. Requests that timed out are not "
+ "here; they are counted in switch.db.pool.timeouts.",
+ bounds=SUB_MILLISECOND_BOUNDS_MS,
+)
# The pool gauges say whether connections are scarce; this says whether the
# database is slow, which is the other half and the one a slow request is
# usually about. `operation` is the statement's leading keyword mapped through
@@ -141,6 +168,30 @@ DB_QUERY_DURATION = _spec(
bounds=SUB_MILLISECOND_BOUNDS_MS,
)
+# What the database itself reports, sampled by this server on a connection of
+# its own rather than one from the pool above: the case worth seeing is the pool
+# exhausted, when a pooled connection is the one thing a sampler cannot get.
+# Only the server role's own sessions in its own database are counted, which is
+# also all `pg_stat_activity` shows a role that is not a superuser.
+DB_SERVER_CONNECTIONS = _spec(
+ "switch.db.server.connections",
+ "gauge",
+ "{connection}",
+ "Sessions the database holds for this server's role, by state, as of the "
+ "last sample. `idle_in_transaction` is a connection checked out and held "
+ "across work that is not a query, the usual way a pool runs dry. Counts "
+ "every replica's sessions, not only this one's.",
+ "state",
+)
+DB_SERVER_TRANSACTIONS = _spec(
+ "switch.db.server.transactions",
+ "sum",
+ "{transaction}",
+ "Transactions committed or rolled back in this server's database, from the "
+ "database's own counters. Counts every client of the database, every "
+ "replica included.",
+)
+
# ── Message transport ────────────────────────────────────────────────────────
MESSAGES_SENT = _spec(
"switch.messages.sent",
@@ -191,6 +242,59 @@ DELIVERY_LAG = _spec(
"actor",
)
+# ── Shared room reads ────────────────────────────────────────────────────────
+# The per-room cache of recent rows (`transport/room_cache.py`). Every value
+# below comes from a fixed set in that module.
+DELIVERY_CACHE_READS = _spec(
+ "switch.delivery_cache.reads",
+ "sum",
+ "{read}",
+ "A transport asking the room cache for the rows after its cursor. "
+ "`outcome` is hit (served from memory), filled (served after waiting for "
+ "the room's one read), or why it fell back to its own database read: "
+ "behind (cursor below what is held), evicted (dropped while it waited, "
+ "and the read it waited for could not answer it), "
+ "unwatched, or gave_up.",
+ "outcome",
+)
+DELIVERY_CACHE_FILLS = _spec(
+ "switch.delivery_cache.fills",
+ "sum",
+ "{fill}",
+ "Database reads made by the room cache on behalf of every member of a "
+ "room. `outcome` is ok, failed, or discarded (the room was dropped while "
+ "it read).",
+ "outcome",
+)
+DELIVERY_CACHE_ROWS_READ = _spec(
+ "switch.delivery_cache.rows_read",
+ "sum",
+ "{row}",
+ "Rows the room cache read from the database. Against "
+ "switch.messages.delivered, the number of deliveries each read served.",
+)
+DELIVERY_CACHE_EVICTIONS = _spec(
+ "switch.delivery_cache.evictions",
+ "sum",
+ "{eviction}",
+ "Rows or rooms the room cache let go of, by reason: bytes or rooms (the "
+ "process-wide limits), rows or age (one room's), unwatched (its last "
+ "member left), invalidated.",
+ "reason",
+)
+DELIVERY_CACHE_BYTES = _spec(
+ "switch.delivery_cache.bytes",
+ "gauge",
+ "By",
+ "Estimated memory the room cache holds, against room_delivery_cache_max_bytes.",
+)
+DELIVERY_CACHE_ROOMS = _spec(
+ "switch.delivery_cache.rooms",
+ "gauge",
+ "{room}",
+ "Rooms the room cache holds rows for.",
+)
+
# ── Bridges ──────────────────────────────────────────────────────────────────
# Both bridges report the same metrics. `bridge` is "collaboration" or
# "agent". For a collaboration bridge `platform` is one of the registered
@@ -264,20 +368,82 @@ AGENT_EVENTS_DROPPED = _spec(
"(retention).",
"reason",
)
+# The buffer answers a read and an unread count by walking the agent's retained
+# events from the oldest. These say what that walk costs, so whether to index
+# the buffer is decided from data. `operation` is "read" or "unread".
+AGENT_BUFFER_SCANNED = _spec(
+ "switch.agent.buffer.scanned",
+ "histogram",
+ "{event}",
+ "Retained events walked to answer one buffer read or unread count. Bounded "
+ "by the per-agent cap, so a p95 near it means every read walks a full buffer.",
+ "operation",
+ bounds=(0.0, 10.0, 50.0, 100.0, 250.0, 500.0, 1000.0, 1500.0, 2000.0),
+)
+AGENT_BUFFER_SCAN_DURATION = _spec(
+ "switch.agent.buffer.scan.duration",
+ "histogram",
+ "ms",
+ "Wall time of one buffer read or unread count. A read runs on every pass "
+ "of a connection's delivery loop, an unread count on every addressed event "
+ "it delivers.",
+ "operation",
+ bounds=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 25.0, 100.0),
+)
+# `reason` is from a fixed set the refusal sites choose. `transport_removed` is
+# a client still asking for the retired event stream, the one way to see old
+# clients that never manage to connect at all. Nothing sends it while the event
+# stream is kept for its compatibility window; it is kept here for the release
+# that removes the stream, when it starts firing again.
+AGENT_CONNECTIONS_REFUSED = _spec(
+ "switch.agent.connections_refused",
+ "sum",
+ "{connection}",
+ "Agent connections refused on opening, by reason: transport_removed (an "
+ "old client asking for the event stream), protocol (no agent-protocol "
+ "revision in common), unauthorized (no valid credential), other.",
+ "reason",
+)
AGENT_CONNECTIONS_EXPIRED = _spec(
"switch.agent.connections_expired",
"sum",
"{connection}",
- "Agent connections closed on a lapsed heartbeat. A steady rate against a "
- "flat connection count is churn, which a gauge alone cannot show.",
+ "Agent connections closed on a lapsed heartbeat, whichever path noticed: "
+ "the sweep, the next use of the dead connection, or its own event stream. "
+ "A steady rate against a flat connection count is churn, which a gauge "
+ "alone cannot show.",
+)
+
+# Every stream open, by what it was. `reattach` is a live connection whose
+# socket came back; `after_lapse` and `after_close` are a connection id this
+# process closed recently, coming back; `fresh` is one this process has never
+# seen, which is a first connect or anything after a server restart. A burst
+# of `after_lapse` is the reconnect storm the expiry counter only implies.
+AGENT_CONNECTIONS_OPENED = _spec(
+ "switch.agent.connections_opened",
+ "sum",
+ "{connection}",
+ "Agent event streams opened, by kind: reattach, after_lapse, after_close or fresh.",
+ "kind",
)
# ── Agents and clients ───────────────────────────────────────────────────────
+# `client` is the declared artifact when the registry knows the name, else
+# `other` or `unknown`. Never the declared version: the client chooses it.
+# `transport` is `websocket`, `sse` (an old client on the event stream, kept
+# for a compatibility window that ends when this stays at zero), `detached`
+# (the stream dropped, and the connection waits out its heartbeat window), or
+# `controller` (run by an agents controller, on that controller's connection).
AGENTS_CONNECTED = _spec(
"switch.agents.connected",
"gauge",
"{agent}",
- "Agents holding a live protocol connection.",
+ "Agents holding a live protocol connection, by transport (websocket, sse, "
+ "detached, controller) and client. An agent connected more than one way counts once "
+ "under each, so summing across the attributes can exceed the number of "
+ "agents.",
+ "transport",
+ "client",
)
CONSUMERS_RUNNING = _spec(
"switch.consumers.running",
@@ -326,6 +492,15 @@ RUNTIME_EVENT_LOOP_LAG = _spec(
"single-threaded, so this is the one number that says whether anything is "
"being starved.",
)
+# One per process start, so a restart lines up against everything else on a
+# dashboard. The version is already on every series' resource.
+RUNTIME_STARTS = _spec(
+ "switch.runtime.starts",
+ "sum",
+ "{start}",
+ "Process starts. A marker: deploys and restarts, to line up against "
+ "reconnect bursts and pool peaks.",
+)
RUNTIME_GC_COLLECTIONS = _spec(
"switch.runtime.gc_collections",
"sum",
@@ -353,19 +528,33 @@ CATALOGUE: dict[str, MetricSpec] = {
DB_POOL_SIZE,
DB_POOL_OVERFLOW,
DB_POOL_TIMEOUTS,
+ DB_POOL_HOLD_DURATION,
+ DB_POOL_WAIT_DURATION,
DB_QUERY_DURATION,
+ DB_SERVER_CONNECTIONS,
+ DB_SERVER_TRANSACTIONS,
MESSAGES_SENT,
MESSAGES_DELIVERED,
SEND_FAILURES,
DELIVERY_FAILURES,
DELIVERY_LAG,
+ DELIVERY_CACHE_READS,
+ DELIVERY_CACHE_FILLS,
+ DELIVERY_CACHE_ROWS_READ,
+ DELIVERY_CACHE_EVICTIONS,
+ DELIVERY_CACHE_BYTES,
+ DELIVERY_CACHE_ROOMS,
BRIDGE_EVENTS_IN,
BRIDGE_EVENTS_OUT,
BRIDGE_ERRORS,
BRIDGES_RUNNING,
BRIDGE_CALL_DURATION,
AGENT_EVENTS_DROPPED,
+ AGENT_BUFFER_SCANNED,
+ AGENT_BUFFER_SCAN_DURATION,
+ AGENT_CONNECTIONS_REFUSED,
AGENT_CONNECTIONS_EXPIRED,
+ AGENT_CONNECTIONS_OPENED,
AGENTS_CONNECTED,
CONSUMERS_RUNNING,
CONNECTORS_RUNNING,
@@ -373,6 +562,7 @@ CATALOGUE: dict[str, MetricSpec] = {
RUNTIME_CPU_SECONDS,
RUNTIME_OPEN_FDS,
RUNTIME_EVENT_LOOP_LAG,
+ RUNTIME_STARTS,
RUNTIME_GC_COLLECTIONS,
HEALTH_CHECK,
)
diff --git a/core/switch_core/observability/db_server.py b/core/switch_core/observability/db_server.py
new file mode 100644
index 000000000..635211ce9
--- /dev/null
+++ b/core/switch_core/observability/db_server.py
@@ -0,0 +1,208 @@
+"""The database's own view of this server: its sessions and its transactions.
+
+The pool gauges are this process's side of the connection. What they cannot
+show is what the database is holding, and in particular how many sessions sit
+`idle in transaction`, a connection checked out and held across work that is
+not a query, which is the usual way a pool runs dry. A managed database with no
+monitoring integration has nowhere else to read that from.
+
+Sampled on a connection of its own, never one from the application pool: the
+moment worth seeing is the pool exhausted, which is exactly when a pooled
+connection cannot be had. One connection, held between samples and reopened
+when it fails, so the sampler costs the database one idle session.
+
+Only the server role's own sessions in its own database are counted. That is
+also all `pg_stat_activity` shows in full to a role that is not a superuser,
+so the numbers mean the same under the restricted runtime role as under the
+owner.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+from collections.abc import Callable, Iterator
+from dataclasses import dataclass
+
+from sqlalchemy import text
+from sqlalchemy.exc import ProgrammingError
+from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine
+
+from switch_core.observability.catalogue import (
+ DB_SERVER_CONNECTIONS,
+ DB_SERVER_TRANSACTIONS,
+)
+from switch_core.observability.metrics import GaugeReading, metrics
+
+logger = logging.getLogger(__name__)
+
+SAMPLE_INTERVAL_SECONDS = 10.0
+SAMPLE_TIMEOUT_SECONDS = 2.0
+# A database that stays unreachable is reported once a minute, not every sample.
+FAILURE_LOG_INTERVAL_SECONDS = 60.0
+
+STATES = ("active", "idle_in_transaction", "idle", "other")
+
+# `pg_stat_activity.state` mapped onto STATES; anything else (fastpath, an
+# aborted transaction, a null state) is `other`.
+_STATE_OF = {
+ "active": "active",
+ "idle in transaction": "idle_in_transaction",
+ "idle": "idle",
+}
+
+_SESSIONS = text(
+ "SELECT state, count(*) FROM pg_stat_activity "
+ "WHERE usename = current_user AND datname = current_database() "
+ "AND pid <> pg_backend_pid() GROUP BY state"
+)
+_TRANSACTIONS = text(
+ "SELECT xact_commit + xact_rollback FROM pg_stat_database "
+ "WHERE datname = current_database()"
+)
+
+
+@dataclass(frozen=True)
+class DbServerSample:
+ connections: dict[str, int]
+ transactions: int
+
+
+class DbServerSampler:
+ """Samples the database's session and transaction counters on a timer.
+
+ Connection counts are a gauge read from the last sample, and absent once a
+ sample has failed: a stale count would draw a healthy database through an
+ outage. Transactions are a counter, recorded as the rise between samples.
+ """
+
+ def __init__(
+ self,
+ engine_factory: Callable[[], AsyncEngine],
+ *,
+ interval_seconds: float = SAMPLE_INTERVAL_SECONDS,
+ timeout_seconds: float = SAMPLE_TIMEOUT_SECONDS,
+ ) -> None:
+ self._engine_factory = engine_factory
+ self._interval = interval_seconds
+ self._timeout = timeout_seconds
+ self._engine: AsyncEngine | None = None
+ self._conn: AsyncConnection | None = None
+ self._latest: DbServerSample | None = None
+ self._last_transactions: int | None = None
+ self._last_failure_log = float("-inf")
+ self.disabled = False
+
+ def readings(self) -> Iterator[GaugeReading]:
+ latest = self._latest
+ if latest is None:
+ return
+ for state in STATES:
+ yield GaugeReading(
+ DB_SERVER_CONNECTIONS,
+ float(latest.connections.get(state, 0)),
+ {"state": state},
+ )
+
+ async def run_forever(self) -> None:
+ try:
+ while not self.disabled:
+ await self.sample_once()
+ await asyncio.sleep(self._interval)
+ finally:
+ await self.aclose()
+
+ async def sample_once(self) -> DbServerSample | None:
+ """Take one sample and record it. None when it could not be taken."""
+ if self.disabled or not self._postgres():
+ return None
+ try:
+ sample = await asyncio.wait_for(self._read(), self._timeout)
+ except ProgrammingError:
+ # The views are missing or unreadable, which no retry will change.
+ logger.warning(
+ "The database's activity views cannot be read, so "
+ "switch.db.server.* will not be reported by this server.",
+ exc_info=True,
+ )
+ self.disabled = True
+ self._latest = None
+ await self.aclose()
+ return None
+ except Exception as exc:
+ self._latest = None
+ await self._drop_connection()
+ self._log_failure(exc)
+ return None
+ self._record(sample)
+ return sample
+
+ def _postgres(self) -> bool:
+ if self._engine is None:
+ self._engine = self._engine_factory()
+ if self._engine.dialect.name == "postgresql":
+ return True
+ logger.info(
+ "The database is not Postgres, so switch.db.server.* is not reported."
+ )
+ self.disabled = True
+ return False
+
+ async def _read(self) -> DbServerSample:
+ conn = await self._connection()
+ sessions = await conn.execute(_SESSIONS)
+ connections = dict.fromkeys(STATES, 0)
+ for state, count in sessions.all():
+ connections[_STATE_OF.get(state, "other")] += int(count)
+ transactions = (await conn.execute(_TRANSACTIONS)).scalar_one_or_none()
+ return DbServerSample(
+ connections=connections, transactions=int(transactions or 0)
+ )
+
+ async def _connection(self) -> AsyncConnection:
+ if self._conn is None:
+ assert self._engine is not None
+ conn = await self._engine.connect()
+ # Autocommit, so the sampler never sits idle in a transaction itself.
+ self._conn = await conn.execution_options(isolation_level="AUTOCOMMIT")
+ return self._conn
+
+ def _record(self, sample: DbServerSample) -> None:
+ self._latest = sample
+ previous = self._last_transactions
+ self._last_transactions = sample.transactions
+ # The first sample is a baseline, and a fall is a statistics reset.
+ if previous is not None and sample.transactions >= previous:
+ metrics().increment(
+ DB_SERVER_TRANSACTIONS, {}, float(sample.transactions - previous)
+ )
+
+ def _log_failure(self, exc: BaseException) -> None:
+ now = time.monotonic()
+ if now - self._last_failure_log < FAILURE_LOG_INTERVAL_SECONDS:
+ return
+ self._last_failure_log = now
+ logger.warning(
+ "Could not sample the database's own connection and transaction "
+ "counts (%s: %s); retrying every %.0fs, and saying so at most once a "
+ "minute.",
+ type(exc).__name__,
+ exc,
+ self._interval,
+ )
+
+ async def _drop_connection(self) -> None:
+ conn, self._conn = self._conn, None
+ if conn is None:
+ return
+ try:
+ await asyncio.wait_for(conn.close(), self._timeout)
+ except Exception:
+ logger.debug("Closing the database sampler's connection failed.")
+
+ async def aclose(self) -> None:
+ await self._drop_connection()
+ engine, self._engine = self._engine, None
+ if engine is not None:
+ await engine.dispose()
diff --git a/core/switch_core/observability/pool.py b/core/switch_core/observability/pool.py
index 537c075ce..b4177b415 100644
--- a/core/switch_core/observability/pool.py
+++ b/core/switch_core/observability/pool.py
@@ -12,11 +12,27 @@ absent one.
from __future__ import annotations
+import logging
+import os
+import sys
import threading
+import time
from dataclasses import dataclass
+from types import FrameType
+from typing import Any
+import greenlet
from sqlalchemy import event
from sqlalchemy.ext.asyncio import AsyncEngine
+from sqlalchemy.pool import AsyncAdaptedQueuePool, ConnectionPoolEntry
+
+from switch_core.observability.catalogue import (
+ DB_POOL_HOLD_DURATION,
+ DB_POOL_WAIT_DURATION,
+)
+from switch_core.observability.metrics import metrics
+
+logger = logging.getLogger(__name__)
class PoolInUseWatermark:
@@ -99,3 +115,166 @@ def _overflow_beyond_nominal(raw: int) -> int:
approaching `size`.
"""
return max(0, raw)
+
+
+# ── Who holds a connection, and for how long ─────────────────────────────────
+
+# A hold this long is code that awaited something else with a connection in
+# hand: the slowest query on this schema is a few hundred milliseconds.
+LONG_HOLD_SECONDS = 1.0
+# One warning per caller per window, so a storm of long holds is a line per
+# culprit rather than one per request.
+_LONG_HOLD_LOG_INTERVAL_SECONDS = 60.0
+# Frames shown with a long-hold warning, innermost first.
+_LONG_HOLD_STACK_DEPTH = 6
+
+_PACKAGE_MARKER = f"{os.sep}switch_core{os.sep}"
+# Frames that are the plumbing every borrow passes through, not the borrower.
+_PLUMBING = (
+ f"{_PACKAGE_MARKER}observability{os.sep}",
+ f"{_PACKAGE_MARKER}db{os.sep}engine.py",
+ f"{_PACKAGE_MARKER}db{os.sep}session_scope.py",
+ f"{_PACKAGE_MARKER}db{os.sep}tenant_session.py",
+)
+# A test tree mirrors the package path, so its frames would otherwise pass as ours.
+_TESTS_MARKER = f"{os.sep}tests{os.sep}"
+_HOLD_KEY = "_switch_hold"
+UNKNOWN_CALLER = "unknown"
+
+
+@dataclass(frozen=True)
+class _Hold:
+ started: float
+ caller: str
+ stack: tuple[str, ...]
+
+
+def _frames_of_borrower() -> list[FrameType]:
+ """The call stack of the code borrowing a connection, innermost first.
+
+ A checkout on the async engine runs inside SQLAlchemy's greenlet, whose
+ own stack starts at the greenlet and knows nothing of the coroutine that
+ awaited it. That coroutine is suspended in the parent greenlet, so the
+ walk continues from the parent's frame when the local one runs out.
+ """
+ frames: list[FrameType] = []
+ frame: FrameType | None = sys._getframe(1)
+ while frame is not None:
+ frames.append(frame)
+ frame = frame.f_back
+ current = greenlet.getcurrent()
+ parent = current.parent
+ if parent is not None and parent.gr_frame is not None:
+ frame = parent.gr_frame
+ while frame is not None:
+ frames.append(frame)
+ frame = frame.f_back
+ return frames
+
+
+def _ours(frame: FrameType) -> bool:
+ filename = frame.f_code.co_filename
+ return (
+ _PACKAGE_MARKER in filename
+ and _TESTS_MARKER not in filename
+ and not any(p in filename for p in _PLUMBING)
+ )
+
+
+def _module(frame: FrameType) -> str:
+ filename = frame.f_code.co_filename
+ tail = filename.rsplit(_PACKAGE_MARKER, 1)[-1]
+ return "switch_core." + tail.removesuffix(".py").replace(os.sep, ".")
+
+
+def describe_borrower() -> tuple[str, tuple[str, ...]]:
+ """`module:function` of the innermost switch_core frame, and a short stack.
+
+ The label is bounded by the code rather than the traffic, so it can be a
+ metric attribute. Asked on every checkout: walking a few dozen frames is
+ microseconds against a round trip of a hundred or more.
+ """
+ ours = [f for f in _frames_of_borrower() if _ours(f)]
+ if not ours:
+ return UNKNOWN_CALLER, ()
+ caller = f"{_module(ours[0])}:{ours[0].f_code.co_name}"
+ stack = tuple(
+ f"{_module(f)}:{f.f_code.co_name}:{f.f_lineno}"
+ for f in ours[:_LONG_HOLD_STACK_DEPTH]
+ )
+ return caller, stack
+
+
+class _LongHoldLog:
+ """Rate-limits the long-hold warning per caller."""
+
+ def __init__(self) -> None:
+ self._lock = threading.Lock()
+ self._last: dict[str, float] = {}
+ self._suppressed: dict[str, int] = {}
+
+ def report(self, hold: _Hold, held_seconds: float) -> None:
+ now = time.monotonic()
+ with self._lock:
+ last = self._last.get(hold.caller)
+ if last is not None and now - last < _LONG_HOLD_LOG_INTERVAL_SECONDS:
+ self._suppressed[hold.caller] = self._suppressed.get(hold.caller, 0) + 1
+ return
+ self._last[hold.caller] = now
+ suppressed = self._suppressed.pop(hold.caller, 0)
+ logger.warning(
+ "A database connection was held for %.2f s by %s (%d more long holds "
+ "by it in the last %d s). Stack: %s",
+ held_seconds,
+ hold.caller,
+ suppressed,
+ int(_LONG_HOLD_LOG_INTERVAL_SECONDS),
+ " < ".join(hold.stack) or "unknown",
+ )
+
+
+def install_hold_timer(engine: AsyncEngine) -> None:
+ """Time every checkout to its checkin, by the code that borrowed it.
+
+ The checkout event fires in the borrower's call, which is the only moment
+ its stack is there to read; the checkin fires wherever the connection
+ happens to be returned, so the borrower is stored on the connection
+ record in between.
+ """
+ long_holds = _LongHoldLog()
+
+ @event.listens_for(engine.sync_engine, "checkout")
+ def _on_checkout(_dbapi: object, record: Any, _proxy: object) -> None:
+ caller, stack = describe_borrower()
+ record.info[_HOLD_KEY] = _Hold(time.perf_counter(), caller, stack)
+
+ @event.listens_for(engine.sync_engine, "checkin")
+ def _on_checkin(_dbapi: object, record: Any) -> None:
+ hold = record.info.pop(_HOLD_KEY, None) if record is not None else None
+ if hold is None:
+ return
+ held = time.perf_counter() - hold.started
+ metrics().observe(DB_POOL_HOLD_DURATION, {"caller": hold.caller}, held * 1000.0)
+ if held >= LONG_HOLD_SECONDS:
+ long_holds.report(hold, held)
+
+
+class WaitTimedQueuePool(AsyncAdaptedQueuePool):
+ """The default async pool, timing how long each request waits for a
+ connection.
+
+ A subclass rather than an event because the pool has no event for the
+ request, only for the handover. Passed in by `main.py` as the engine's
+ `poolclass`, so the database layer stays unaware of any of this; a pool
+ rebuilt by `dispose()` keeps the class, and with it the timing.
+ """
+
+ def _do_get(self) -> ConnectionPoolEntry:
+ started = time.perf_counter()
+ try:
+ return super()._do_get()
+ finally:
+ # A wait that ends in a pool timeout is the one that matters most.
+ metrics().observe(
+ DB_POOL_WAIT_DURATION, {}, (time.perf_counter() - started) * 1000.0
+ )
diff --git a/core/switch_core/room_service.py b/core/switch_core/room_service.py
index b7bdaf06e..cdb385892 100644
--- a/core/switch_core/room_service.py
+++ b/core/switch_core/room_service.py
@@ -52,6 +52,7 @@ if TYPE_CHECKING:
from switch_core.bridges.collaboration.lifecycle_service import (
CollaborationBridgeLifecycleService,
)
+ from switch_core.transport.room_cache import RoomDeliveryCache
logger = logging.getLogger(__name__)
@@ -167,6 +168,7 @@ class RoomService:
collab_bridge_store: CollaborationBridgeStore,
resource_service: ResourceService,
session_factory: async_sessionmaker[AsyncSession],
+ room_cache: RoomDeliveryCache,
telemetry: TelemetryService | None = None,
) -> None:
self._provisioning = provisioning
@@ -180,6 +182,8 @@ class RoomService:
# Optional because several tests and tooling build a RoomService
# without one; `emit_safely` treats None as "report nothing".
self._telemetry = telemetry
+ # The shared delivery read; told when a room goes.
+ self._room_cache = room_cache
async def _resolve_agent_ids(self, config: RoomCreateConfig) -> list[str]:
if config.agent_ids is not None:
@@ -806,6 +810,10 @@ class RoomService:
await self._room_store.delete(session, room_id)
await session.commit()
+ # Kicking the members above already emptied it if they were all
+ # running here; this covers any that were not.
+ self._room_cache.invalidate(room.tenant_id, room_id)
+
if bridge_id:
collaboration_core = self._collab_lifecycle.get(bridge_id)
if collaboration_core:
diff --git a/core/switch_core/transport/postgres.py b/core/switch_core/transport/postgres.py
index 9ab82f0c5..0ca83ce15 100644
--- a/core/switch_core/transport/postgres.py
+++ b/core/switch_core/transport/postgres.py
@@ -6,7 +6,7 @@ of that. **The write is the send.** A send is an INSERT, numbered per room
under an advisory lock (`MessageStore.create`); the `messages_notify` trigger
announces it on commit (`db/notify_ddl.py`); one `MessageListener` wakes the
transports watching that room (`messages/notify.py`); and each transport reads
-the rows after its own cursor, `_DELIVERY_PAGE` at a time. There is no second
+the rows after its own cursor, `DELIVERY_PAGE` at a time. There is no second
store to agree with. Switch used to run on a Matrix homeserver; the
`matrix_*` column names are what is left of it, kept as stable ids.
@@ -40,7 +40,6 @@ from switch_core.db.models import (
ClientRoom,
MediaBlob,
Message,
- MessageAttachment,
UsageMetric,
)
from switch_core.db.session_scope import tenant_session
@@ -60,6 +59,7 @@ from switch_core.transport.content import media_content, message_content
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
from switch_core.transport.port import Handler, TransportHandlers
+from switch_core.transport.room_cache import DELIVERY_PAGE
from switch_core.transport.stored import to_inbound
from switch_core.transport.types import (
DownloadResult,
@@ -83,13 +83,12 @@ if TYPE_CHECKING:
from switch_core.db.stores.room_store import RoomStore
from switch_core.db.stores.usage_store import UsageStore
from switch_core.messages.notify import MessageListener
+ from switch_core.transport.room_cache import RoomDeliveryCache
+ from switch_core.transport.stored import DeliveryAttachments, DeliveryRow
-logger = logging.getLogger(__name__)
+ DeliveryPage = tuple[list[tuple[DeliveryRow, DeliveryAttachments]], bool]
-# How many rows one wake-up reads at a time. A page rather than everything
-# outstanding, so a room that moved a long way while a handler was busy is
-# delivered in bounded steps instead of one unbounded read.
-_DELIVERY_PAGE = 200
+logger = logging.getLogger(__name__)
# What makes an `m.room.message` a file rather than text.
_MEDIA_MSGTYPES = frozenset({"m.image", "m.file", "m.video", "m.audio"})
@@ -175,6 +174,7 @@ class PostgresTransport:
listener: MessageListener,
invites: InviteBus,
ephemeral: EphemeralBus,
+ room_cache: RoomDeliveryCache,
) -> None:
self.user_id = user_id
self.client_id = client_id
@@ -204,6 +204,11 @@ class PostgresTransport:
self._listener = listener
self._invites = invites
self._ephemeral = ephemeral
+ # Shared with every other transport in the process.
+ self._room_cache = room_cache
+ # When this client was last woken for each room, on the cache's clock.
+ # A shared read only counts for this client if it started after this.
+ self._woken: dict[str, int] = {}
self._handlers = TransportHandlers()
# Per-room delivery position, and the transport-side id to hand back
# to a handler. Both are keyed by the Switch room id, which is what
@@ -435,13 +440,13 @@ class PostgresTransport:
# already undone the claim and there is nothing subscribed to undo.
return
self._cursors[room_id] = seq
+ self._room_cache.attach(self.tenant_id, room_id, self)
self._listener.subscribe(room_id, self._on_room_advanced)
self._ephemeral.subscribe(transport_room_id, self._on_ephemeral)
if from_seq is not None:
# The announcement for anything already written went out before
# this subscription existed, so nothing will wake the loop for it.
- self._pending.add(room_id)
- self._wake.set()
+ self._mark_pending(room_id)
def _unwatch(self, transport_room_id: str) -> None:
"""Stop delivering one room. Not watching it is success.
@@ -469,6 +474,9 @@ class PostgresTransport:
# A wake-up already queued for this room would otherwise be drained
# after the subscription was dropped.
self._pending.discard(room_id)
+ self._woken.pop(room_id, None)
+ # The last member out takes the room's cached rows with it.
+ self._room_cache.detach(self.tenant_id, room_id, self)
def _unwatch_all(self) -> None:
"""Drop every room, through the same path a single removal takes."""
@@ -496,6 +504,15 @@ class PostgresTransport:
"""
if room_id not in self._watching:
return
+ self._mark_pending(room_id)
+
+ def _mark_pending(self, room_id: str) -> None:
+ """Queue a room for this client's loop, remembering when.
+
+ The tick is taken here, after the commit that caused the wake, so a
+ shared read that started later is known to have seen it.
+ """
+ self._woken[room_id] = self._room_cache.tick()
self._pending.add(room_id)
self._wake.set()
@@ -516,6 +533,11 @@ class PostgresTransport:
Every delivery is a suspension point, so the subscription is re-read
per row. A removal landing mid-page has to stop the rows it has not
reached, and must not be undone by a cursor write after it.
+
+ A page comes from the room's one shared read when the cache can vouch
+ for it from this cursor, and from this client's own read when it
+ cannot. Only the read is shared: the rows are still handed over here,
+ one at a time, under the same checks.
"""
transport_room_id = self._watching.get(room_id)
if transport_room_id is None:
@@ -527,6 +549,10 @@ class PostgresTransport:
# row; the map only ever held this value, and two sources for one fact
# is how they come to disagree.
tenant_id = self.tenant_id
+ # Taken once per pass: a wake landing during it queues another pass,
+ # which takes the newer one. A room queued without a wake time (no
+ # path does that today) gets "now", which only ever costs a read.
+ woken_at = self._woken.get(room_id) or self._room_cache.tick()
# Bound for the read *and* the delivery below: a handler (posting to a
# bridge, gating a command) opens its own sessions rather than reusing
# this one, and those still need the room's tenant. The contextvar
@@ -537,33 +563,72 @@ class PostgresTransport:
cursor = self._cursors.get(room_id)
if cursor is None or room_id not in self._watching:
return
- async with tenant_session(self._session_factory, tenant_id) as session:
- rows = await self._message_store.list_for_room(
- session,
- room_id,
- after_seq=cursor,
- limit=_DELIVERY_PAGE,
- )
- attachments = await self._message_store.attachments_for(
- session, [row.id for row in rows]
- )
+ rows, done = await self._read_shared_page(
+ room_id, tenant_id, cursor, woken_at
+ )
+ if self._cursors.get(room_id) != cursor:
+ # Removed and put back while waiting on the room's
+ # shared read: these rows are from the old
+ # subscription's position, not the new one's.
+ continue
if not rows:
return
- for row in rows:
+ for row, attachments in rows:
if room_id not in self._watching:
return
self._cursors[room_id] = row.seq
- await self._deliver(
- transport_room_id, row, attachments.get(row.id, [])
- )
- if len(rows) < _DELIVERY_PAGE:
+ await self._deliver(transport_room_id, row, attachments)
+ if done:
return
+ async def _read_page(
+ self, room_id: str, tenant_id: str, cursor: int
+ ) -> DeliveryPage:
+ """This client's own read of the next page, for when the room's
+ shared read cannot serve it. `done` when the page came back short."""
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ rows = await self._message_store.list_for_room(
+ session,
+ room_id,
+ after_seq=cursor,
+ limit=DELIVERY_PAGE,
+ )
+ attachments = await self._message_store.attachments_for(
+ session, [row.id for row in rows]
+ )
+ page: list[tuple[DeliveryRow, DeliveryAttachments]] = [
+ (row, attachments.get(row.id, [])) for row in rows
+ ]
+ return page, len(rows) < DELIVERY_PAGE
+
+ async def _read_shared_page(
+ self, room_id: str, tenant_id: str, cursor: int, woken_at: int
+ ) -> DeliveryPage:
+ """The next page from the room's shared read, or this client's own
+ when the cache cannot vouch for it (cursor behind what it holds,
+ entry dropped while waiting).
+
+ The key is this client's tenant and a room it resolved under that
+ tenant, which is what makes a hit as good as the read it replaces.
+ """
+ page = await self._room_cache.read(
+ tenant_id,
+ room_id,
+ after_seq=cursor,
+ woken_at=woken_at,
+ limit=DELIVERY_PAGE,
+ )
+ if page is None:
+ if room_id not in self._watching:
+ return [], True
+ return await self._read_page(room_id, tenant_id, cursor)
+ return [(row, row.attachments) for row in page.rows], page.done
+
async def _deliver(
self,
transport_room_id: str,
- row: Message,
- attachments: list[MessageAttachment],
+ row: DeliveryRow,
+ attachments: DeliveryAttachments,
) -> None:
room = RoomRef(room_id=transport_room_id)
event = to_inbound(row, attachments, transport_room_id=transport_room_id)
diff --git a/core/switch_core/transport/room_cache.py b/core/switch_core/transport/room_cache.py
new file mode 100644
index 000000000..7f1ee0c4d
--- /dev/null
+++ b/core/switch_core/transport/room_cache.py
@@ -0,0 +1,587 @@
+"""One read of a room's new messages, shared by every client watching it.
+
+Each client has its own `PostgresTransport`, and each transport reads the rows
+after its own cursor when the listener says a room moved. A room with six
+agents in it therefore reads every new message six times, through six pool
+checkouts, at the same moment. Under a restart burst that is what exhausted
+the pool. This keeps one bounded copy of a room's recent rows in the process,
+so the first transport to need them reads them and the rest are handed the
+same rows.
+
+**What an entry claims.** An entry for a room holds every row whose `seq` is
+in `(floor, ceiling]`, exactly, and nothing else. A fill reads `seq >
+ceiling` and appends, so the range only ever grows at the top and stays
+contiguous. Trimming raises `floor`. A reader whose cursor is in
+`[floor, ceiling)` is served from memory; one below `floor` reads the
+database itself, and comes back once its cursor reaches the range. No read is
+ever served across a hole, because the entry never has one.
+
+**When an entry is current enough.** A row is visible to a read only if it
+committed before the read's snapshot, and a reader woken for a row must not
+be told "nothing new" by a read that started before that row committed. So
+time here is a counter, `tick()`: a transport takes one when it is woken
+(after the NOTIFY, so after the commit), and a fill takes one before it
+queries. A fill that started after a reader's wake has seen what the reader
+was woken for. `complete_as_of` is the start of the latest fill that reached
+the end of the room: every row committed before it is at or below `ceiling`.
+A reader may be told it has caught up only when that tick is later than its
+own wake. This is what keeps the missed-notification window closed on the
+first read after subscribing, without a database read per transport.
+
+**One fill per room at a time.** It runs as a task of its own, not in any
+transport's loop, so a client that is cancelled mid-read does not cancel the
+read every other member is waiting on. Never from the listener's fan-out
+either: a waker only notes the room, as it always has.
+
+**Rows are immutable snapshots.** A row and its attachments are published
+together, in one step with no await, after both reads finish. `content` is
+kept as JSON text and parsed afresh for every delivery, so no two clients ever
+hold the same dict. Attachment metadata only: the bytes live in `media_blobs`
+and are never read here.
+
+**Messages are append-only.** The NOTIFY trigger fires on INSERT only
+(`db/notify_ddl.py`), nothing updates a delivered row, and deleting a room
+cascades its messages away. A cached row is therefore never stale. A future
+edit or redaction path must call `invalidate` for the room, or members would
+keep being handed the old text from memory.
+
+**What it does not do.** It checks nothing about who may read. The key is
+`(tenant_id, room_id)`, taken from a transport that resolved the room under
+its own tenant, and every fill reads under that tenant's session, so row-level
+security applies to the fill exactly as it did to each transport's own read.
+Whether *this* client is still in the room, and whether it wants the message,
+stays with the transport and its consumer, checked after every await.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import bisect
+import json
+import logging
+import time
+from collections import OrderedDict
+from dataclasses import dataclass
+from datetime import datetime
+from typing import TYPE_CHECKING, Any, cast
+
+from switch_core.db.session_scope import tenant_session
+from switch_core.observability.catalogue import (
+ DELIVERY_CACHE_EVICTIONS,
+ DELIVERY_CACHE_FILLS,
+ DELIVERY_CACHE_READS,
+ DELIVERY_CACHE_ROWS_READ,
+)
+from switch_core.observability.metrics import metrics
+from switch_core.tenant_context import no_tenant
+
+if TYPE_CHECKING:
+ from collections.abc import Iterable
+
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+ from switch_core.db.models import Message, MessageAttachment
+ from switch_core.db.stores.message_store import MessageStore
+
+logger = logging.getLogger(__name__)
+
+# How many rows one wake-up reads at a time, by a transport or by a fill. A
+# page rather than everything outstanding, so a room that moved a long way
+# while a handler was busy is delivered in bounded steps instead of one
+# unbounded read. Here rather than beside the transport so the process can
+# size the cache without naming a transport implementation.
+DELIVERY_PAGE = 200
+
+# What a row costs beyond its strings: the object, its tuple slot, the
+# datetime. An estimate, so the byte limit is a budget rather than a measure.
+_ROW_OVERHEAD_BYTES = 400
+
+# How many fills one read may wait for before it gives up and lets the
+# transport read the database. Each fill started after the reader's wake
+# either reaches the end of the room or adds a page, so one or two is the
+# norm; this bounds the odd case of a cursor far ahead of an old entry.
+_MAX_ROUNDS = 4
+
+CacheKey = tuple[str, str]
+
+
+@dataclass(frozen=True, slots=True)
+class CachedAttachment:
+ """A file's metadata, as `to_inbound` reads it off `MessageAttachment`."""
+
+ uri: str
+ filename: str | None
+ mimetype: str | None
+ size: int | None
+
+
+@dataclass(frozen=True, slots=True)
+class CachedRow:
+ """A message row frozen at the moment it was read.
+
+ Carries the attributes `to_inbound` and the delivery metrics read off a
+ `Message`, under the same names, so both paths build an event the same
+ way.
+ """
+
+ id: str
+ seq: int
+ transport_event_id: str
+ sender_id: str
+ sender_name: str | None
+ event_type: str
+ msgtype: str | None
+ body: str | None
+ formatted_body: str | None
+ thread_root_event_id: str | None
+ sent_at: datetime
+ content_json: str
+ attachments: tuple[CachedAttachment, ...]
+ size: int
+
+ @property
+ def content(self) -> dict[str, Any]:
+ """A fresh dict on every access, so one client's edit is its own."""
+ return cast("dict[str, Any]", json.loads(self.content_json))
+
+ @classmethod
+ def of(cls, row: Message, attachments: Iterable[MessageAttachment]) -> CachedRow:
+ files = tuple(
+ CachedAttachment(
+ uri=file.uri,
+ filename=file.filename,
+ mimetype=file.mimetype,
+ size=file.size,
+ )
+ for file in attachments
+ )
+ content_json = json.dumps(row.content, separators=(",", ":"))
+ texts = (
+ row.transport_event_id,
+ row.sender_id,
+ row.sender_name,
+ row.event_type,
+ row.msgtype,
+ row.body,
+ row.formatted_body,
+ row.thread_root_event_id,
+ content_json,
+ )
+ size = _ROW_OVERHEAD_BYTES + sum(len(text) for text in texts if text)
+ size += sum(
+ len(file.uri) + len(file.filename or "") + len(file.mimetype or "") + 64
+ for file in files
+ )
+ return cls(
+ id=row.id,
+ seq=row.seq,
+ transport_event_id=row.transport_event_id,
+ sender_id=row.sender_id,
+ sender_name=row.sender_name,
+ event_type=row.event_type,
+ msgtype=row.msgtype,
+ body=row.body,
+ formatted_body=row.formatted_body,
+ thread_root_event_id=row.thread_root_event_id,
+ sent_at=cast("datetime", row.sent_at),
+ content_json=content_json,
+ attachments=files,
+ size=size,
+ )
+
+
+@dataclass(frozen=True)
+class RoomCacheLimits:
+ """Process-wide bounds. Whatever falls outside them is read from the
+ database, so a limit set too low costs reads, never messages."""
+
+ max_bytes: int
+ max_rooms: int
+ max_rows_per_room: int
+ max_age_seconds: float
+
+
+@dataclass(frozen=True)
+class RoomCacheStats:
+ bytes: int
+ rooms: int
+ rows: int
+
+
+@dataclass(frozen=True)
+class CachedPage:
+ """Rows after a reader's cursor, in `seq` order with no gaps.
+
+ `done` means nothing else was committed before the reader was woken, so
+ it can stop without asking again.
+ """
+
+ rows: tuple[CachedRow, ...]
+ done: bool
+
+
+class _Fill:
+ __slots__ = ("after", "rows", "started", "task")
+
+ def __init__(self, started: int, after: int) -> None:
+ self.started = started
+ # It reads the rows after this, so what it holds is (after, last].
+ self.after = after
+ # What it read, once checked; kept so its waiters can still use it if
+ # the entry is dropped before it lands.
+ self.rows: tuple[CachedRow, ...] | None = None
+ self.task: asyncio.Task[None] | None = None
+
+
+class _Entry:
+ __slots__ = (
+ "born",
+ "bytes",
+ "ceiling",
+ "complete_as_of",
+ "dropped",
+ "filling",
+ "floor",
+ "rows",
+ )
+
+ def __init__(self, floor: int) -> None:
+ self.floor = floor
+ self.ceiling = floor
+ self.rows: tuple[CachedRow, ...] = ()
+ # When each row was cached, for the age limit. Parallel to `rows`.
+ self.born: tuple[float, ...] = ()
+ self.bytes = 0
+ # No fill has reached the end yet, so no wake is covered.
+ self.complete_as_of = -1
+ self.filling: _Fill | None = None
+ # Why it was dropped, once it has been.
+ self.dropped: str | None = None
+
+
+class RoomDeliveryCache:
+ """The process's shared read of recent rows, one entry per watched room."""
+
+ def __init__(
+ self,
+ *,
+ session_factory: async_sessionmaker[AsyncSession],
+ message_store: MessageStore,
+ limits: RoomCacheLimits,
+ page: int = DELIVERY_PAGE,
+ ) -> None:
+ if limits.max_rooms < 1 or limits.max_bytes < 1:
+ raise ValueError("the room delivery cache needs room for one room")
+ if limits.max_rows_per_room < page:
+ # A fill reads a page; a room that cannot hold one would trim what
+ # it just read before anyone was served from it.
+ raise ValueError(
+ f"max_rows_per_room ({limits.max_rows_per_room}) must be at "
+ f"least the delivery page ({page})"
+ )
+ self._session_factory = session_factory
+ self._message_store = message_store
+ self._limits = limits
+ self._page = page
+ # Least recently read first, so eviction pops from the front.
+ self._entries: OrderedDict[CacheKey, _Entry] = OrderedDict()
+ # Who is watching each room. An entry exists only while someone is,
+ # which is what purges a room every member has left or that was
+ # deleted (deleting a room removes its members first).
+ self._watchers: dict[CacheKey, set[object]] = {}
+ self._bytes = 0
+ self._clock = 0
+
+ # ── Time ──────────────────────────────────────────────────────────────────
+
+ def tick(self) -> int:
+ """A moment, ordered against every other moment in this process.
+
+ Synchronous and strictly increasing, which is all the handoff needs:
+ whether a fill started after a reader was woken.
+ """
+ self._clock += 1
+ return self._clock
+
+ # ── Who is watching ───────────────────────────────────────────────────────
+
+ def attach(self, tenant_id: str, room_id: str, watcher: object) -> None:
+ self._watchers.setdefault((tenant_id, room_id), set()).add(watcher)
+
+ def detach(self, tenant_id: str, room_id: str, watcher: object) -> None:
+ """Forget a watcher; drop the room's rows when it was the last.
+
+ Safe for a watcher that never attached, like the buses' unsubscribe,
+ because `_release_claim` undoes a claim that may never have
+ subscribed.
+ """
+ key = (tenant_id, room_id)
+ watchers = self._watchers.get(key)
+ if watchers is None:
+ return
+ watchers.discard(watcher)
+ if watchers:
+ return
+ del self._watchers[key]
+ self._drop(key, "unwatched")
+
+ def invalidate(self, tenant_id: str, room_id: str) -> None:
+ """Drop everything held for a room.
+
+ The hook for anything that changes a row after it was written (an
+ edit, a redaction) or removes the room. Nothing does the former today;
+ see the module docstring. A fill in flight for the room is discarded
+ when it lands, and its waiters read the database: it may have read
+ rows from before the change.
+ """
+ self._drop((tenant_id, room_id), "invalidated")
+
+ # ── Reading ───────────────────────────────────────────────────────────────
+
+ async def read(
+ self,
+ tenant_id: str,
+ room_id: str,
+ *,
+ after_seq: int,
+ woken_at: int,
+ limit: int,
+ ) -> CachedPage | None:
+ """Rows after `after_seq`, or None for "read the database yourself".
+
+ `woken_at` is the reader's tick from when it was last woken for this
+ room. An empty page is only ever answered once a fill that started
+ after it has reached the end of the room.
+ """
+ key = (tenant_id, room_id)
+ waited = False
+ for round_ in range(_MAX_ROUNDS + 1):
+ if key not in self._watchers:
+ self._count("unwatched")
+ return None
+ entry = self._entries.get(key)
+ if entry is None:
+ entry = self._create(key, floor=after_seq)
+ else:
+ self._entries.move_to_end(key)
+ self._expire(key, entry)
+ if after_seq < entry.floor:
+ self._count("behind")
+ return None
+ if after_seq < entry.ceiling:
+ start = bisect.bisect_right(entry.rows, after_seq, key=_seq_of)
+ rows = entry.rows[start : start + limit]
+ done = rows[-1].seq >= entry.ceiling and entry.complete_as_of > woken_at
+ self._count("filled" if waited else "hit")
+ return CachedPage(rows=rows, done=done)
+ if entry.complete_as_of > woken_at:
+ self._count("filled" if waited else "hit")
+ return CachedPage(rows=(), done=True)
+ if round_ == _MAX_ROUNDS:
+ break
+ fill = entry.filling
+ if fill is None:
+ fill = self._start_fill(key, entry)
+ # A fill that started before this reader was woken may not have
+ # seen what woke it. It is still waited for rather than run
+ # beside, so a room has one read in flight; the next round starts
+ # a fresh one.
+ waited = True
+ assert fill.task is not None
+ await asyncio.shield(fill.task)
+ if self._entries.get(key) is not entry:
+ return self._from_dropped(
+ key,
+ entry,
+ fill,
+ after_seq=after_seq,
+ woken_at=woken_at,
+ limit=limit,
+ )
+ self._count("gave_up")
+ return None
+
+ def _from_dropped(
+ self,
+ key: CacheKey,
+ entry: _Entry,
+ fill: _Fill,
+ *,
+ after_seq: int,
+ woken_at: int,
+ limit: int,
+ ) -> CachedPage | None:
+ """Serve a waiter from a fill whose entry was dropped while it read.
+
+ Evicted for room or bytes, the rows are as good as ever; they are just
+ not kept. Sending every waiter to the database instead would put a
+ read per member on top of the one already made, which under churn is
+ the load this cache exists to remove. Invalidated means a row may have
+ changed under the read, so its waiters still read the database.
+ """
+ if key not in self._watchers:
+ self._count("unwatched")
+ return None
+ if (
+ entry.dropped == "invalidated"
+ or fill.rows is None
+ or after_seq < fill.after
+ ):
+ self._count("evicted")
+ return None
+ rows = tuple(row for row in fill.rows if row.seq > after_seq)
+ reached_end = len(fill.rows) < self._page and fill.started > woken_at
+ if not rows and not reached_end:
+ self._count("evicted")
+ return None
+ served = rows[:limit]
+ self._count("filled")
+ return CachedPage(rows=served, done=reached_end and len(served) == len(rows))
+
+ def stats(self) -> RoomCacheStats:
+ return RoomCacheStats(
+ bytes=self._bytes,
+ rooms=len(self._entries),
+ rows=sum(len(entry.rows) for entry in self._entries.values()),
+ )
+
+ # ── Filling ───────────────────────────────────────────────────────────────
+
+ def _create(self, key: CacheKey, *, floor: int) -> _Entry:
+ entry = _Entry(floor)
+ self._entries[key] = entry
+ while len(self._entries) > self._limits.max_rooms:
+ oldest = next(iter(self._entries))
+ self._drop(oldest, "rooms")
+ return entry
+
+ def _start_fill(self, key: CacheKey, entry: _Entry) -> _Fill:
+ # Ticked here, before the task can query, so a reader woken after
+ # this moment never mistakes this fill for one that saw its row.
+ fill = _Fill(started=self.tick(), after=entry.ceiling)
+ task = asyncio.create_task(
+ self._fill(key, entry, fill), name="room-delivery-cache-fill"
+ )
+ # Every waiter may have been cancelled; the outcome is still
+ # retrieved, so a failed fill does not warn "never retrieved".
+ task.add_done_callback(_retrieve)
+ fill.task = task
+ entry.filling = fill
+ return fill
+
+ async def _fill(self, key: CacheKey, entry: _Entry, fill: _Fill) -> None:
+ tenant_id, room_id = key
+ after = fill.after
+ try:
+ # The task inherits its creator's context; the session below binds
+ # the tenant explicitly, and nothing else here should inherit one.
+ with no_tenant():
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ rows = await self._message_store.list_for_room(
+ session, room_id, after_seq=after, limit=self._page
+ )
+ attachments = await self._message_store.attachments_for(
+ session, [row.id for row in rows]
+ )
+ snapshots = tuple(
+ CachedRow.of(row, attachments.get(row.id, ())) for row in rows
+ )
+ except BaseException:
+ if entry.filling is fill:
+ entry.filling = None
+ metrics().increment(DELIVERY_CACHE_FILLS, {"outcome": "failed"})
+ raise
+ if entry.filling is fill:
+ entry.filling = None
+ metrics().increment(DELIVERY_CACHE_ROWS_READ, {}, float(len(snapshots)))
+ if snapshots and snapshots[0].seq <= after:
+ # `list_for_room` reads `seq > after` in order, so this cannot
+ # happen; if it ever does, holding these rows would break the
+ # range the entry claims.
+ logger.error(
+ "A fill for room %s returned seq %d at or below %d; discarding",
+ room_id,
+ snapshots[0].seq,
+ after,
+ )
+ metrics().increment(DELIVERY_CACHE_FILLS, {"outcome": "discarded"})
+ return
+ fill.rows = snapshots
+ if self._entries.get(key) is not entry or entry.ceiling != after:
+ # Evicted, unwatched or invalidated while reading. Publishing into
+ # a dropped entry would resurrect it outside the limits; its
+ # waiters may still take the rows (`_from_dropped`).
+ metrics().increment(DELIVERY_CACHE_FILLS, {"outcome": "discarded"})
+ return
+ self._publish(key, entry, fill, snapshots)
+ metrics().increment(DELIVERY_CACHE_FILLS, {"outcome": "ok"})
+
+ def _publish(
+ self,
+ key: CacheKey,
+ entry: _Entry,
+ fill: _Fill,
+ snapshots: tuple[CachedRow, ...],
+ ) -> None:
+ """Make a fill's rows visible. No await, so nobody sees half of it."""
+ now = time.monotonic()
+ entry.rows = entry.rows + snapshots
+ entry.born = entry.born + (now,) * len(snapshots)
+ if snapshots:
+ entry.ceiling = snapshots[-1].seq
+ if len(snapshots) < self._page:
+ # It read to the end of the room as of its snapshot.
+ entry.complete_as_of = max(entry.complete_as_of, fill.started)
+ added = sum(row.size for row in snapshots)
+ entry.bytes += added
+ self._bytes += added
+ excess = len(entry.rows) - self._limits.max_rows_per_room
+ if excess > 0:
+ self._trim(entry, excess)
+ metrics().increment(DELIVERY_CACHE_EVICTIONS, {"reason": "rows"})
+ while self._bytes > self._limits.max_bytes and self._entries:
+ self._drop(next(iter(self._entries)), "bytes")
+
+ # ── Letting go ────────────────────────────────────────────────────────────
+
+ def _expire(self, key: CacheKey, entry: _Entry) -> None:
+ """Trim rows past the age limit. Lazy, on read: the byte and room
+ limits are what bound memory, this only stops a quiet room holding
+ rows nobody will ask for again."""
+ horizon = time.monotonic() - self._limits.max_age_seconds
+ stale = bisect.bisect_left(entry.born, horizon)
+ if stale:
+ self._trim(entry, stale)
+ metrics().increment(DELIVERY_CACHE_EVICTIONS, {"reason": "age"})
+
+ def _trim(self, entry: _Entry, count: int) -> None:
+ """Drop the oldest `count` rows; the range now starts above them."""
+ dropped = entry.rows[:count]
+ entry.floor = dropped[-1].seq
+ entry.rows = entry.rows[count:]
+ entry.born = entry.born[count:]
+ freed = sum(row.size for row in dropped)
+ entry.bytes -= freed
+ self._bytes -= freed
+
+ def _drop(self, key: CacheKey, reason: str) -> None:
+ entry = self._entries.pop(key, None)
+ if entry is None:
+ return
+ entry.dropped = reason
+ self._bytes -= entry.bytes
+ metrics().increment(DELIVERY_CACHE_EVICTIONS, {"reason": reason})
+
+ # ── Counting ──────────────────────────────────────────────────────────────
+
+ def _count(self, outcome: str) -> None:
+ metrics().increment(DELIVERY_CACHE_READS, {"outcome": outcome})
+
+
+def _seq_of(row: CachedRow) -> int:
+ return row.seq
+
+
+def _retrieve(task: asyncio.Task[None]) -> None:
+ if not task.cancelled():
+ task.exception()
diff --git a/core/switch_core/transport/stored.py b/core/switch_core/transport/stored.py
index ca9e93932..6c9aaf528 100644
--- a/core/switch_core/transport/stored.py
+++ b/core/switch_core/transport/stored.py
@@ -2,7 +2,7 @@
from __future__ import annotations
-from typing import Any
+from typing import TYPE_CHECKING, Any
from switch_core.attachments import ATTACHMENT_GROUP_KEY
from switch_core.db.models import Message, MessageAttachment
@@ -16,10 +16,20 @@ from switch_core.transport.types import (
InboundMessage,
)
+if TYPE_CHECKING:
+ from collections.abc import Sequence
+
+ from switch_core.transport.room_cache import CachedAttachment, CachedRow
+
+ # A row as delivery reads it: straight from the table, or the shared copy
+ # the room cache holds. `to_inbound` reads the same attributes off either.
+ DeliveryRow = Message | CachedRow
+ DeliveryAttachments = Sequence[MessageAttachment] | Sequence[CachedAttachment]
+
def to_inbound(
- row: Message,
- attachments: list[MessageAttachment],
+ row: DeliveryRow,
+ attachments: DeliveryAttachments,
*,
transport_room_id: str,
) -> InboundEvent:
@@ -28,6 +38,9 @@ def to_inbound(
Which of the four inbound shapes a row becomes is read off the row itself:
an arrival, a file, a `com.switch.*` payload, or a message. The row keeps the
whole content dict, so nothing is reconstructed here that was not sent.
+
+ A `CachedRow` is shared by every client in the room, so its `content` is
+ parsed afresh on each access and every event built here owns its dict.
"""
content = dict(row.content)
room_id = transport_room_id
diff --git a/core/tests/benchmarks/server.py b/core/tests/benchmarks/server.py
index a87f3e662..e1e7768e2 100644
--- a/core/tests/benchmarks/server.py
+++ b/core/tests/benchmarks/server.py
@@ -73,6 +73,7 @@ from switch_core.sessions.contract import ApprovalResult
from switch_core.tenant_context import bind_tenant_id, tenant_scope
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
from tests.benchmarks.instrumentation import (
RequestCounter,
TracingMiddleware,
@@ -474,6 +475,17 @@ async def _serve(
invites=invites,
)
+ room_cache = RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=session_env.message_store,
+ limits=RoomCacheLimits(
+ max_bytes=config.room_delivery_cache_max_bytes,
+ max_rooms=config.room_delivery_cache_max_rooms,
+ max_rows_per_room=config.room_delivery_cache_max_rows_per_room,
+ max_age_seconds=config.room_delivery_cache_max_age_seconds,
+ ),
+ )
+
client_factory = ClientFactory(
client_store=session_env.client_store,
session_factory=session_factory,
@@ -484,6 +496,7 @@ async def _serve(
listener=message_listener,
invites=invites,
ephemeral=ephemeral,
+ room_cache=room_cache,
)
client_factory.register(
"agent",
@@ -524,6 +537,7 @@ async def _serve(
collab_bridge_store=session_env.bridge_store,
resource_service=resource_service,
session_factory=session_factory,
+ room_cache=room_cache,
)
app, protocol = create_agent_bridge_app(
diff --git a/core/tests/conftest.py b/core/tests/conftest.py
index e335daff4..70b5692d4 100644
--- a/core/tests/conftest.py
+++ b/core/tests/conftest.py
@@ -29,6 +29,7 @@ from switch_core.db.models import TENANT_ZERO_ID, Tenant
from switch_core.db.runtime_role import grant_runtime_role
from switch_core.keys import Keyring
from switch_core.tenant_context import tenant_scope
+from tests.switch_core.pool_checkouts import PoolCheckouts
# Production configures this in `main.run()`; every test that touches a
# connection config needs the same, and none needs a particular key.
@@ -174,6 +175,23 @@ async def session_factory(
await engine.dispose()
+def pool_checkouts(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> Iterator[PoolCheckouts]:
+ """Connections borrowed from `session_factory`'s pool, counted per request.
+
+ Wrap an app under test with `pool_checkouts.wrap(app)`, or a block with
+ `pool_checkouts.scope(label)`, and assert `pool_checkouts.over() == []`:
+ no request held two connections at once. See `pool_checkouts.py`.
+ """
+ tracker = PoolCheckouts(session_factory.kw["bind"])
+ try:
+ yield tracker
+ finally:
+ tracker.close()
+
+
@dataclass(frozen=True)
class RLSHarness:
"""Two ways into the same database, for row-level-security tests that
diff --git a/core/tests/integration/conftest.py b/core/tests/integration/conftest.py
index 600b85dc9..5ddde7db0 100644
--- a/core/tests/integration/conftest.py
+++ b/core/tests/integration/conftest.py
@@ -88,6 +88,7 @@ from switch_core.room_service import RoomService
from switch_core.tenant_context import tenant_scope
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
# ── Mirrors deploy/local/docker-compose.yml — keep in sync ──────────────────────
POSTGRES_IMAGE = "postgres:16-alpine"
@@ -554,6 +555,17 @@ async def harness(session_env: SessionEnv) -> AsyncIterator[Harness]:
invites=invites,
)
+ room_cache = RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=session_env.message_store,
+ limits=RoomCacheLimits(
+ max_bytes=config.room_delivery_cache_max_bytes,
+ max_rooms=config.room_delivery_cache_max_rooms,
+ max_rows_per_room=config.room_delivery_cache_max_rows_per_room,
+ max_age_seconds=config.room_delivery_cache_max_age_seconds,
+ ),
+ )
+
client_factory = ClientFactory(
client_store=session_env.client_store,
session_factory=session_factory,
@@ -565,6 +577,7 @@ async def harness(session_env: SessionEnv) -> AsyncIterator[Harness]:
listener=message_listener,
invites=invites,
ephemeral=ephemeral,
+ room_cache=room_cache,
)
client_factory.register(
"agent",
@@ -605,6 +618,7 @@ async def harness(session_env: SessionEnv) -> AsyncIterator[Harness]:
collab_bridge_store=session_env.bridge_store,
resource_service=resource_service,
session_factory=session_factory,
+ room_cache=room_cache,
)
protocol = AgentCore(
diff --git a/core/tests/switch_core/bridges/agent/api/test_event_stream_compat.py b/core/tests/switch_core/bridges/agent/api/test_event_stream_compat.py
index 8e7ed15e1..6e4b6430a 100644
--- a/core/tests/switch_core/bridges/agent/api/test_event_stream_compat.py
+++ b/core/tests/switch_core/bridges/agent/api/test_event_stream_compat.py
@@ -34,6 +34,7 @@ from switch_core.bridges.agent.protocol.agent_connections import (
)
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload
+from switch_core.observability.metrics import MetricsRegistry, install, uninstall
AGENT_ID = "agent-1"
ROOM = "room-a"
@@ -201,6 +202,63 @@ async def test_closing_the_stream_detaches_it_and_keeps_the_connection() -> None
conn = protocol.connections.get("c1")
assert conn is not None
assert not conn.stream_attached
+ assert protocol.connections.live_agents_by_transport() == {
+ ("detached", "agent-runtime"): 1
+ }
+
+
+async def test_a_stream_attached_over_sse_is_reported_as_sse() -> None:
+ protocol = _Protocol()
+ stream = _Reader((await _open(protocol)).body_iterator)
+ await stream.frame()
+
+ assert protocol.connections.live_agents_by_transport() == {
+ ("sse", "agent-runtime"): 1
+ }
+ await stream.close()
+
+
+async def test_a_client_that_moves_to_the_socket_is_reported_on_the_socket() -> None:
+ """The transport is the last attach's: an upgraded Console reattaching the
+ same connection over the socket stops counting as an old client."""
+ protocol = _Protocol()
+ stream = _Reader((await _open(protocol)).body_iterator)
+ await stream.frame()
+
+ protocol.connections.open(
+ agent_id=AGENT_ID,
+ connection_id="c1",
+ scope="single",
+ delivery_filter="all",
+ spawn_capable=False,
+ cursor=0,
+ declaration=ClientDeclaration(speaks=8, accepts=1, artifact="agent-runtime"),
+ expected_generation=None,
+ transport="websocket",
+ )
+
+ assert protocol.connections.live_agents_by_transport() == {
+ ("websocket", "agent-runtime"): 1
+ }
+ await stream.close()
+
+
+async def test_a_refused_stream_is_counted_like_a_refused_socket() -> None:
+ registry = MetricsRegistry()
+ install(registry)
+ try:
+ with pytest.raises(HTTPException) as caught:
+ await _open(_Protocol(), protocol_version=99, protocol_accepts=99)
+ finally:
+ uninstall()
+
+ assert caught.value.status_code == 409
+ refused = next(
+ p for p in registry.collect() if p.name == "switch.agent.connections_refused"
+ )
+ assert [(dict(p.attributes), p.value) for p in refused.numbers] == [
+ ({"reason": "protocol"}, 1.0)
+ ]
# ── The declared ranges still meet ──────────────────────────────────────────
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_cost.py b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_cost.py
new file mode 100644
index 000000000..444d1c50f
--- /dev/null
+++ b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_cost.py
@@ -0,0 +1,237 @@
+"""What the agent event buffer costs per delivered event, and what it logs.
+
+An agent past the cap drops an event on every enqueue, and used to log a
+warning on every one: a line per delivered message for an agent that had read
+everything. The warning is now at most once per interval per agent, while the
+drop counter and the gap a reader is told about stay exact.
+
+The buffer answers reads by walking the agent's retained events, so the walk is
+measured: how many events, and how long, per call.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Iterator
+
+import pytest
+
+from switch_core.bridges.agent.protocol import event_buffer as buffer_module
+from switch_core.bridges.agent.protocol.event_buffer import (
+ OVERFLOW_WARNING_INTERVAL_SECONDS,
+ CursorExpiredError,
+ EventBuffer,
+ Reader,
+)
+from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload
+from switch_core.observability.metrics import MetricsRegistry, install, uninstall
+from switch_core.observability.otlp import HistogramPoint
+
+_LOGGER = "switch_core.bridges.agent.protocol.event_buffer"
+
+
+def registry() -> Iterator[MetricsRegistry]:
+ registry = MetricsRegistry()
+ install(registry)
+ yield registry
+ uninstall()
+
+
+class _Clock:
+ def __init__(self) -> None:
+ self.now = 1000.0
+
+ def __call__(self) -> float:
+ return self.now
+
+
+def clock(monkeypatch: pytest.MonkeyPatch) -> _Clock:
+ clock = _Clock()
+ monkeypatch.setattr(buffer_module.time, "monotonic", clock)
+ return clock
+
+
+def _event(index: int, *, addressed: bool = True) -> AgentEvent:
+ return AgentEvent(
+ type="message",
+ room_id="room-1",
+ payload=MessagePayload(
+ message_id=f"m{index}",
+ sender="@someone:test",
+ sender_name="someone",
+ body=f"hello {index}",
+ addressed=addressed,
+ timestamp=index,
+ ),
+ )
+
+
+def _overflow_warnings(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:
+ return [
+ record
+ for record in caplog.records
+ if record.name == _LOGGER and "exceeded" in record.getMessage()
+ ]
+
+
+def _dropped(registry: MetricsRegistry) -> dict[tuple, float]:
+ payload = next(
+ (p for p in registry.collect() if p.name == "switch.agent.events_dropped"),
+ None,
+ )
+ if payload is None:
+ return {}
+ return {
+ tuple(sorted(point.attributes.items())): point.value
+ for point in payload.numbers
+ }
+
+
+def _histograms(registry: MetricsRegistry, name: str) -> dict[str, HistogramPoint]:
+ payload = next((p for p in registry.collect() if p.name == name), None)
+ assert payload is not None, f"{name} was not recorded"
+ return {point.attributes["operation"]: point for point in payload.histograms}
+
+
+def test_overflow_warns_once_per_interval_per_agent(clock, registry, caplog):
+ caplog.set_level(logging.WARNING, logger=_LOGGER)
+ buffer = EventBuffer(
+ max_events_per_agent=3, retention_seconds=3600, sequence_base=0
+ )
+
+ for index in range(50):
+ buffer.enqueue("agent-1", "room-1", _event(index))
+ clock.now += 0.1
+
+ # 47 drops inside five seconds: one line, not 47.
+ assert len(_overflow_warnings(caplog)) == 1
+ assert _dropped(registry) == {(("reason", "overflow"),): 47.0}
+
+
+def test_overflow_warning_resumes_after_the_interval_with_a_count(
+ clock, registry, caplog
+):
+ caplog.set_level(logging.WARNING, logger=_LOGGER)
+ buffer = EventBuffer(
+ max_events_per_agent=3, retention_seconds=3600, sequence_base=0
+ )
+
+ for index in range(4):
+ buffer.enqueue("agent-1", "room-1", _event(index))
+ for index in range(4, 14):
+ clock.now += 1
+ buffer.enqueue("agent-1", "room-1", _event(index))
+ clock.now += OVERFLOW_WARNING_INTERVAL_SECONDS
+ buffer.enqueue("agent-1", "room-1", _event(14))
+
+ warnings = _overflow_warnings(caplog)
+ assert len(warnings) == 2
+ # The ten drops between the two lines are counted into the second.
+ assert "plus 10 dropped since the last warning" in warnings[1].getMessage()
+ assert _dropped(registry) == {(("reason", "overflow"),): 12.0}
+
+
+def test_overflow_warning_is_per_agent(clock, caplog):
+ caplog.set_level(logging.WARNING, logger=_LOGGER)
+ buffer = EventBuffer(
+ max_events_per_agent=1, retention_seconds=3600, sequence_base=0
+ )
+
+ for index in range(5):
+ buffer.enqueue("agent-1", "room-1", _event(index))
+ buffer.enqueue("agent-2", "room-1", _event(index))
+
+ warned = sorted(
+ record.getMessage().split()[1] for record in _overflow_warnings(caplog)
+ )
+ assert warned == ["agent=agent-1", "agent=agent-2"]
+
+
+def test_a_quiet_warning_still_leaves_the_gap_for_readers(clock, caplog):
+ caplog.set_level(logging.WARNING, logger=_LOGGER)
+ buffer = EventBuffer(
+ max_events_per_agent=3, retention_seconds=3600, sequence_base=0
+ )
+
+ for index in range(20):
+ buffer.enqueue("agent-1", "room-1", _event(index))
+
+ assert len(_overflow_warnings(caplog)) == 1
+ # A reader from the start is told about every drop, logged or not.
+ with pytest.raises(CursorExpiredError) as exc:
+ buffer.read_from("agent-1", 0)
+ assert exc.value.oldest == 18
+ assert exc.value.rooms == ("room-1",)
+ assert [item.seq for item in buffer.read_from("agent-1", 17)] == [18, 19, 20]
+
+
+def test_removing_an_agent_forgets_its_warning_state(clock, caplog):
+ caplog.set_level(logging.WARNING, logger=_LOGGER)
+ buffer = EventBuffer(
+ max_events_per_agent=1, retention_seconds=3600, sequence_base=0
+ )
+
+ for index in range(3):
+ buffer.enqueue("agent-1", "room-1", _event(index))
+ buffer.remove("agent-1")
+ for index in range(3):
+ buffer.enqueue("agent-1", "room-1", _event(index))
+
+ assert len(_overflow_warnings(caplog)) == 2
+
+
+def test_a_read_records_how_many_events_it_walked(registry):
+ buffer = EventBuffer(
+ max_events_per_agent=100, retention_seconds=3600, sequence_base=0
+ )
+ for index in range(10):
+ buffer.enqueue("agent-1", "room-1", _event(index))
+
+ buffer.read_from("agent-1", 0)
+ # Stopping at the limit walks only as far as the limit was reached.
+ buffer.read_from("agent-1", 0, limit=3)
+
+ scanned = _histograms(registry, "switch.agent.buffer.scanned")["read"]
+ assert scanned.count == 2
+ assert scanned.total == 13.0
+
+
+def test_a_read_and_an_unread_count_record_their_duration(registry):
+ buffer = EventBuffer(
+ max_events_per_agent=100, retention_seconds=3600, sequence_base=0
+ )
+ reader = Reader(id="session-1", is_session=True)
+ buffer.caught_up("agent-1", reader, "room-1", 0, arrived_on=None)
+ for index in range(4):
+ buffer.enqueue("agent-1", "room-1", _event(index, addressed=index == 3))
+
+ buffer.read_from("agent-1", 0)
+ assert buffer.unread("agent-1", "room-1", 4).count == 3
+
+ collected = registry.collect()
+ durations = next(
+ p for p in collected if p.name == "switch.agent.buffer.scan.duration"
+ )
+ scanned = next(p for p in collected if p.name == "switch.agent.buffer.scanned")
+ assert {p.attributes["operation"]: p.count for p in durations.histograms} == {
+ "read": 1,
+ "unread": 1,
+ }
+ assert all(p.total >= 0 for p in durations.histograms)
+ assert {p.attributes["operation"]: p.total for p in scanned.histograms} == {
+ "read": 4.0,
+ "unread": 4.0,
+ }
+
+
+def test_an_unread_count_with_nothing_to_count_records_no_scan(registry):
+ buffer = EventBuffer(sequence_base=0)
+ buffer.enqueue("agent-1", "room-1", _event(0))
+
+ # No baseline: answered without walking the buffer.
+ assert buffer.unread("agent-1", "room-1", 1).count is None
+
+ names = {p.name for p in registry.collect()}
+ assert "switch.agent.buffer.scanned" not in names
diff --git a/core/tests/switch_core/clients/test_client_tenant_plumbing.py b/core/tests/switch_core/clients/test_client_tenant_plumbing.py
index 266ad4ea1..d58d88d0b 100644
--- a/core/tests/switch_core/clients/test_client_tenant_plumbing.py
+++ b/core/tests/switch_core/clients/test_client_tenant_plumbing.py
@@ -65,6 +65,7 @@ def _factory(session_factory: async_sessionmaker[AsyncSession]) -> ClientFactory
listener=MagicMock(),
invites=InviteBus(),
ephemeral=EphemeralBus(),
+ room_cache=MagicMock(),
)
factory.register("user", Actor)
return factory
diff --git a/core/tests/switch_core/clients/test_enqueue_debug_logging.py b/core/tests/switch_core/clients/test_enqueue_debug_logging.py
new file mode 100644
index 000000000..2dfd54cce
--- /dev/null
+++ b/core/tests/switch_core/clients/test_enqueue_debug_logging.py
@@ -0,0 +1,135 @@
+"""The debug line for an enqueued event costs nothing unless DEBUG is on.
+
+It pretty-prints the whole event, and an argument to `logger.debug` is built
+before the level is checked. That ran once per agent per message: on a busy
+server, thousands of serialisations a second for a line nobody was logging.
+"""
+
+from __future__ import annotations
+
+import logging
+from types import SimpleNamespace
+
+import pytest
+
+from switch_core.bridges.agent.protocol.types import AgentEvent
+from switch_core.clients.agent_consumer import AgentConsumer
+from switch_core.transport import InboundMedia, InboundMessage, RoomRef
+
+_LOGGER = "switch_core.clients.agent_consumer"
+
+
+def _consumer(enqueued: list[AgentEvent]) -> SimpleNamespace:
+ """An unaddressed delivery: no store is consulted, the event is enqueued."""
+
+ async def _resolve_room_meta(_room_id: str) -> SimpleNamespace:
+ return SimpleNamespace(
+ room_id="room-1", bridge_id=None, channel_type="channel_public"
+ )
+
+ return SimpleNamespace(
+ agent=SimpleNamespace(id="agent-1", name="helper"),
+ _resolve_room_meta=_resolve_room_meta,
+ _addressed_without_lookup=lambda _event, _meta: False,
+ _event_buffer=SimpleNamespace(
+ enqueue=lambda _agent, _room, event: enqueued.append(event)
+ ),
+ )
+
+
+def _message() -> InboundMessage:
+ return InboundMessage(
+ room_id="!room:server",
+ event_id="$evt",
+ sender="@someone:switch.local",
+ timestamp=1,
+ content={"body": "hello"},
+ body="hello",
+ sender_name="someone",
+ )
+
+
+def dumps(monkeypatch: pytest.MonkeyPatch) -> list[int]:
+ calls: list[int] = []
+ real = AgentEvent.model_dump_json
+
+ def counting(self: AgentEvent, **kwargs: object) -> str:
+ calls.append(1)
+ return real(self, **kwargs) # type: ignore[arg-type]
+
+ monkeypatch.setattr(AgentEvent, "model_dump_json", counting)
+ return calls
+
+
+async def test_an_event_is_not_serialised_for_a_debug_line_nobody_logs(dumps, caplog):
+ caplog.set_level(logging.INFO, logger=_LOGGER)
+ enqueued: list[AgentEvent] = []
+
+ await AgentConsumer.on_message(
+ _consumer(enqueued), # type: ignore[arg-type]
+ RoomRef(room_id="!room:server"),
+ _message(),
+ )
+
+ assert len(enqueued) == 1
+ assert dumps == []
+ assert not [r for r in caplog.records if "Enqueuing event" in r.getMessage()]
+
+
+async def test_the_debug_line_is_still_logged_when_debug_is_on(dumps, caplog):
+ caplog.set_level(logging.DEBUG, logger=_LOGGER)
+ enqueued: list[AgentEvent] = []
+
+ await AgentConsumer.on_message(
+ _consumer(enqueued), # type: ignore[arg-type]
+ RoomRef(room_id="!room:server"),
+ _message(),
+ )
+
+ assert len(enqueued) == 1
+ assert dumps == [1]
+ lines = [r for r in caplog.records if "Enqueuing event" in r.getMessage()]
+ assert len(lines) == 1
+ assert '"body": "hello"' in lines[0].getMessage()
+
+
[email protected]("level", [logging.INFO, logging.DEBUG])
+async def test_a_media_event_is_serialised_only_when_debug_is_on(dumps, caplog, level):
+ caplog.set_level(level, logger=_LOGGER)
+ enqueued: list[AgentEvent] = []
+ media = InboundMedia(
+ room_id="!room:server",
+ event_id="$media",
+ sender="@someone:switch.local",
+ timestamp=1,
+ content={"body": "a file"},
+ body="a file",
+ uri="mxc://server/file",
+ )
+
+ await AgentConsumer._emit_media(
+ _consumer(enqueued), # type: ignore[arg-type]
+ RoomRef(room_id="!room:server"),
+ media,
+ SimpleNamespace( # type: ignore[arg-type]
+ room_id="room-1", bridge_id=None, channel_type="channel_public"
+ ),
+ False,
+ "someone",
+ None,
+ [],
+ "a file",
+ )
+
+ assert len(enqueued) == 1
+ logged = [r for r in caplog.records if "Enqueuing media event" in r.getMessage()]
+ if level == logging.DEBUG:
+ assert dumps == [1]
+ assert len(logged) == 1
+ else:
+ assert dumps == []
+ assert logged == []
diff --git a/core/tests/switch_core/db/stores/test_delivery_cursor_store.py b/core/tests/switch_core/db/stores/test_delivery_cursor_store.py
index b640e6a74..7da3a46f1 100644
--- a/core/tests/switch_core/db/stores/test_delivery_cursor_store.py
+++ b/core/tests/switch_core/db/stores/test_delivery_cursor_store.py
@@ -60,8 +60,8 @@ class TestPosition:
async def test_an_agent_that_was_never_delivered_is_at_zero(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
- """Zero, not an error: `seq` starts at 1, so zero already says
- "behind everything"."""
+ """Zero, not an error: live messages are numbered from 1, so zero
+ already says "behind every live message"."""
async with session_factory() as session:
agent_id, room_id = await _make_agent_and_room(session)
await session.commit()
diff --git a/core/tests/switch_core/db/stores/test_message_store.py b/core/tests/switch_core/db/stores/test_message_store.py
index 182f4b93b..195bc8700 100644
--- a/core/tests/switch_core/db/stores/test_message_store.py
+++ b/core/tests/switch_core/db/stores/test_message_store.py
@@ -369,8 +369,8 @@ class TestListForRoom:
async def test_after_seq_from_another_room_does_not_skip_rows(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
- """`seq` is global, so a cursor is only meaningful within its own room;
- rows interleaved from another room must not consume the page."""
+ """`seq` is per room, so a cursor is only meaningful within its own
+ room; rows interleaved from another room must not consume the page."""
store = MessageStore()
async with session_factory() as session:
room_a = await _make_room(session, "alpha")
diff --git a/core/tests/switch_core/fixtures/agent_controllers/controller_beat_request.json b/core/tests/switch_core/fixtures/agent_controllers/controller_beat_request.json
deleted file mode 100644
index da7deae60..000000000
--- a/core/tests/switch_core/fixtures/agent_controllers/controller_beat_request.json
+++ /dev/null
@@ -1,5 +0,0 @@
-{
- "connection_id": "00000000-0000-4000-8000-0000000000e1",
- "generation": 3141592653,
- "cursors": { "00000000-0000-4000-8000-0000000000a1": 42 }
-}
diff --git a/core/tests/switch_core/fixtures/agent_controllers/controller_beat_response.json b/core/tests/switch_core/fixtures/agent_controllers/controller_beat_response.json
deleted file mode 100644
index 0ed1b2e97..000000000
--- a/core/tests/switch_core/fixtures/agent_controllers/controller_beat_response.json
+++ /dev/null
@@ -1 +0,0 @@
-{ "agents": ["00000000-0000-4000-8000-0000000000a1"] }
diff --git a/core/tests/switch_core/fixtures/agent_controllers/controller_pong.json b/core/tests/switch_core/fixtures/agent_controllers/controller_pong.json
new file mode 100644
index 000000000..9a13c948e
--- /dev/null
+++ b/core/tests/switch_core/fixtures/agent_controllers/controller_pong.json
@@ -0,0 +1,4 @@
+{
+ "type": "pong",
+ "cursors": { "00000000-0000-4000-8000-0000000000a1": 42 }
+}
diff --git a/core/tests/switch_core/fixtures/agent_controllers/status_request.json b/core/tests/switch_core/fixtures/agent_controllers/status_request.json
index fd2ceec86..452ae818c 100644
--- a/core/tests/switch_core/fixtures/agent_controllers/status_request.json
+++ b/core/tests/switch_core/fixtures/agent_controllers/status_request.json
@@ -1,7 +1,7 @@
{
"seq": 42,
"observed_at": "2026-01-01T00:00:00Z",
- "controller": { "version": "0.1.0", "protocol": 1, "assignment_revision": 3 },
+ "controller": { "version": "0.1.0", "protocol": 2, "assignment_revision": 3 },
"machine": {
"platform": { "os": "linux", "arch": "x64", "os_version": "6.1.0" },
"disk_free_bytes": 53687091200,
diff --git a/core/tests/switch_core/gateway/test_directory_fallback.py b/core/tests/switch_core/gateway/test_directory_fallback.py
index 13a973dc3..c4af87eb1 100644
--- a/core/tests/switch_core/gateway/test_directory_fallback.py
+++ b/core/tests/switch_core/gateway/test_directory_fallback.py
@@ -84,6 +84,14 @@ class _StubUserStore:
return []
+class _StubSession:
+ """The route commits before asking the platform, so it is not holding a
+ connection while the platform answers."""
+
+ async def commit(self) -> None:
+ return None
+
+
async def _search(
*,
adapter_result: list[DirectoryUser] | Exception | None,
@@ -93,7 +101,7 @@ async def _search(
return await search_bridge_directory(
"b1",
query,
- None, # type: ignore[arg-type]
+ _StubSession(), # type: ignore[arg-type]
_StubBridgeStore(), # type: ignore[arg-type]
_StubExternalUserStore(known), # type: ignore[arg-type]
_StubUserStore(), # type: ignore[arg-type]
diff --git a/core/tests/switch_core/gateway/test_held_connections.py b/core/tests/switch_core/gateway/test_held_connections.py
new file mode 100644
index 000000000..b95037ba8
--- /dev/null
+++ b/core/tests/switch_core/gateway/test_held_connections.py
@@ -0,0 +1,698 @@
+"""No gateway request holds two pooled connections at once (CHOO-3198).
+
+A request that keeps its first connection while it waits for a second one from
+the same pool is the shape that starves the pool under load: every connection
+ends up held by a request waiting for another, nobody can give one back, and
+after the pool timeout they all fail together. The same goes for a connection
+held while the request waits on something that is not the database at all: a
+messaging platform, Matrix, or bcrypt.
+
+These drive the real routes through the real authentication and admin
+dependencies, against real Postgres, and count connections where every path
+meets: the pool (`tests/switch_core/pool_checkouts.py`). Two contracts:
+
+- no request ever holds more than one connection at a time;
+- every platform call, Matrix call and bcrypt call happens holding none, and
+ bcrypt runs off the event loop.
+"""
+
+from __future__ import annotations
+
+import secrets
+import threading
+from collections.abc import AsyncIterator, Callable
+from types import SimpleNamespace
+from typing import Any
+from unittest.mock import MagicMock
+
+import httpx
+import pytest
+from fastapi import Depends, FastAPI
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ Agent,
+ ApiKey,
+ Client,
+ CollaborationBridge,
+ Room,
+ TenantMember,
+ User,
+)
+from switch_core.db.stores.agent_store import AgentStore
+from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
+from switch_core.db.stores.external_user_store import ExternalUserStore
+from switch_core.db.stores.room_role_store import RoomRoleStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.user_store import UserStore
+from switch_core.gateway import auth as gateway_auth
+from switch_core.gateway import dependencies as gw_deps
+from switch_core.gateway.agents import router as agents_router
+from switch_core.gateway.auth import (
+ create_jwt,
+ get_current_user,
+ get_tenant_is_admin,
+ hash_password,
+)
+from switch_core.gateway.auth_routes import router as auth_router
+from switch_core.gateway.collaborations import router as bridges_router
+from switch_core.gateway.hosted_launches import hosted_settings
+from switch_core.gateway.rooms import router as rooms_router
+from switch_core.keys import Keyring
+from switch_core.room_service import RoomService
+from tests.switch_core.pool_checkouts import PoolCheckouts
+
+# Made for each run, so nothing secret-looking is committed.
+_SECRET = secrets.token_hex(32)
+_KEYRING = Keyring.parse("test:" + _SECRET, legacy_secret=None)
+_PASSWORD = secrets.token_urlsafe(16)
+
+
+class _Calls:
+ """What the fakes below were asked, with the connections held at the time."""
+
+ def __init__(self, tracker: PoolCheckouts) -> None:
+ self._tracker = tracker
+ self.held: list[tuple[str, int]] = []
+
+ def record(self, what: str) -> None:
+ self.held.append((what, self._tracker.held_now()))
+
+
+class _Provisioning:
+ """Matrix, as `RoomService` sees it."""
+
+ def __init__(self, calls: _Calls) -> None:
+ self._calls = calls
+
+ async def invite_to_room(self, transport_room_id: str, user_id: str) -> None:
+ self._calls.record("matrix invite")
+
+ async def kick_user(self, transport_room_id: str, user_id: str) -> None:
+ self._calls.record("matrix kick")
+
+ async def delete_room(self, transport_room_id: str) -> None:
+ self._calls.record("matrix delete")
+
+
+class _RunningClients:
+ def __init__(self, by_agent: dict[str, Any]) -> None:
+ self._by_agent = by_agent
+
+ def get_by_agent_id(self, agent_id: str) -> Any:
+ return self._by_agent.get(agent_id)
+
+ def get(self, client_id: str) -> Any:
+ return next(
+ (c for c in self._by_agent.values() if c.client_id == client_id), None
+ )
+
+ def get_by_type(self, client_type: str, tenant_id: str) -> list[Any]:
+ return []
+
+
+class _Adapter:
+ """A messaging platform's adapter: every call is a network round trip."""
+
+ def __init__(self, calls: _Calls) -> None:
+ self._calls = calls
+
+ async def search_directory_users(self, query: str) -> list[Any]:
+ self._calls.record("platform directory search")
+ return [
+ SimpleNamespace(
+ external_user_id="U1",
+ username=query,
+ display_name=query,
+ email=None,
+ )
+ ]
+
+ async def home_deeplink(self) -> str | None:
+ return None
+
+ async def install_links(self) -> list[Any]:
+ return []
+
+ async def install_note(self) -> str | None:
+ return None
+
+ async def channel_deeplink(self, external_channel_id: str) -> str | None:
+ self._calls.record("platform channel deeplink")
+ return None
+
+
+class _CollabLifecycle:
+ def __init__(self, calls: _Calls) -> None:
+ self._calls = calls
+ self.calls = calls
+ self._adapter = _Adapter(calls)
+
+ def get_adapter(self, bridge_id: str) -> _Adapter:
+ return self._adapter
+
+ def get(self, bridge_id: str) -> Any:
+ async def ensure_external_user(**_: Any) -> Any:
+ self._calls.record("provision external user")
+ raise ValueError("refused by the test")
+
+ return SimpleNamespace(ensure_external_user=ensure_external_user)
+
+ def supports_channel_creation(self, bridge_type: str) -> bool:
+ return True
+
+ def supports_directory_search(self, bridge_type: str) -> bool:
+ return True
+
+ async def check_edited_connection_config(self, **_: Any) -> None:
+ self._calls.record("platform credential check")
+
+ async def check_start_guards(self, **_: Any) -> None:
+ self._calls.record("start guards")
+
+ async def restart(self, bridge_id: str) -> None:
+ self._calls.record("bridge restart")
+
+ def note_preconfigured(self, bridge_id: str, value: bool) -> None:
+ pass
+
+ async def remove(self, bridge_id: str) -> None:
+ self._calls.record("bridge remove")
+
+
+class _Protocol:
+ """The parts of `AgentCore` the routes under test reach.
+
+ `delete_agent` does what the real one does to the pool: opens a session of
+ its own, then calls out to every bridge.
+ """
+
+ def __init__(
+ self, calls: _Calls, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ self.room_role_store = RoomRoleStore()
+ self.agent_store = AgentStore()
+ self.connections = SimpleNamespace(live_connection_ids=lambda: set())
+ self._calls = calls
+ self._session_factory = session_factory
+
+ async def delete_agent(
+ self, *, agent_id: str | None = None, agent_name: str | None = None
+ ) -> None:
+ async with self._session_factory() as session:
+ await AgentStore().get(session, agent_id or "")
+ self._calls.record("bridge identity removal")
+
+ async def get_agent_statuses_by_ids_in_session(
+ self, session: AsyncSession, room_id: str, agent_ids: list[str]
+ ) -> dict[str, Any]:
+ return {}
+
+
+def _config() -> SimpleNamespace:
+ return SimpleNamespace(
+ keyring=_KEYRING,
+ gateway_cookie_secure=False,
+ gateway_tenant_choice_enabled=False,
+ gateway_password_login_enabled=True,
+ gateway_signup_open=True,
+ gateway_signup_max_per_hour=100,
+ )
+
+
+def _app(
+ session_factory: async_sessionmaker[AsyncSession],
+ tracker: PoolCheckouts,
+ *,
+ room_service: RoomService,
+ collab: _CollabLifecycle,
+ extra: Callable[[FastAPI], None] | None = None,
+) -> Any:
+ calls = collab.calls
+
+ async def _session() -> AsyncIterator[AsyncSession]:
+ async with session_factory() as session:
+ yield session
+
+ app = FastAPI()
+ app.include_router(rooms_router, prefix="/rooms")
+ app.include_router(bridges_router, prefix="/bridges")
+ app.include_router(auth_router)
+ app.include_router(agents_router, prefix="/agents")
+ if extra is not None:
+ extra(app)
+ overrides: dict[Callable[..., Any], Callable[..., Any]] = {
+ gw_deps.get_session: _session,
+ gw_deps.get_system_session: _session,
+ gw_deps.get_session_factory: lambda: session_factory,
+ gw_deps.get_user_store: UserStore,
+ gw_deps.get_room_store: RoomStore,
+ gw_deps.get_bridge_store: CollaborationBridgeStore,
+ gw_deps.get_external_user_store: ExternalUserStore,
+ gw_deps.get_room_service: lambda: room_service,
+ gw_deps.get_collab_lifecycle: lambda: collab,
+ gw_deps.get_protocol: lambda: _Protocol(calls, session_factory),
+ gw_deps.get_agent_store: AgentStore,
+ gw_deps.get_config: _config,
+ hosted_settings: lambda: None,
+ }
+ app.dependency_overrides.update(overrides)
+ return tracker.wrap(app)
+
+
+def _client(app: Any, user: User) -> httpx.AsyncClient:
+ token = create_jwt(user.id, user.email, user.role, _KEYRING, TENANT_ZERO_ID)
+ return httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=app),
+ base_url="http://test",
+ cookies={"switch_auth": token},
+ )
+
+
+async def _admin(session_factory: async_sessionmaker[AsyncSession]) -> User:
+ """A workspace owner: a plain user whose admin bit comes from membership,
+ so `get_tenant_is_admin` has to read it."""
+ async with session_factory() as session:
+ user = User(
+ name="owner",
+ role="user",
+ password_hash=hash_password(_PASSWORD),
+ )
+ session.add(user)
+ await session.flush()
+ session.add(
+ TenantMember(tenant_id=TENANT_ZERO_ID, user_id=user.id, role="owner")
+ )
+ await session.commit()
+ return user
+
+
+async def _agent(session: AsyncSession, owner: User, name: str) -> tuple[Agent, Any]:
+ key = ApiKey(
+ user_id=owner.id,
+ key_hash=f"hash-{name}",
+ encrypted_key="enc",
+ label=name,
+ type="agent",
+ )
+ client = Client(transport_user_id=f"@{name}:test", display_name=name, type="agent")
+ session.add_all([key, client])
+ await session.flush()
+ agent = Agent(
+ name=name,
+ description="d",
+ agent_type="session_addressable",
+ connector_type="claude_code",
+ integration_profile={},
+ client_id=client.id,
+ api_key_id=key.id,
+ owner_id=owner.id,
+ )
+ session.add(agent)
+ await session.flush()
+ running = SimpleNamespace(
+ client_id=client.id, transport_user_id=client.transport_user_id
+ )
+ return agent, running
+
+
+async def _room(session: AsyncSession, owner: User, name: str) -> Room:
+ room = Room(
+ transport_room_id=f"!{name}:test",
+ name=name,
+ description="d",
+ owner_id=owner.id,
+ )
+ session.add(room)
+ await session.flush()
+ return room
+
+
+def _room_service(
+ session_factory: async_sessionmaker[AsyncSession],
+ calls: _Calls,
+ collab: _CollabLifecycle,
+ running: dict[str, Any],
+) -> RoomService:
+ return RoomService(
+ provisioning=_Provisioning(calls), # type: ignore[arg-type]
+ room_store=RoomStore(),
+ agent_store=AgentStore(),
+ client_lifecycle=_RunningClients(running), # type: ignore[arg-type]
+ collab_lifecycle=collab, # type: ignore[arg-type]
+ collab_bridge_store=CollaborationBridgeStore(),
+ resource_service=None, # type: ignore[arg-type]
+ session_factory=session_factory,
+ room_cache=MagicMock(),
+ )
+
+
+def _assert_one_at_a_time(tracker: PoolCheckouts, calls: _Calls) -> None:
+ assert tracker.scopes, "no request was counted; the tracker is not wired"
+ assert [(s.label, s.peak) for s in tracker.over()] == []
+ assert [(what, held) for what, held in calls.held if held] == []
+
+
+class TestTheGuardItself:
+ async def test_it_catches_a_request_that_takes_a_second_connection(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ ) -> None:
+ # The shape this whole file exists to forbid, written out on purpose:
+ # read on the request's session, then open another before committing.
+ # If this stops reporting 2, every other assertion here is vacuous.
+ user = await _admin(session_factory)
+ calls = _Calls(pool_checkouts)
+ collab = _CollabLifecycle(calls)
+
+ def held_and_wait(app: FastAPI) -> None:
+ @app.get("/held-and-wait")
+ async def held_and_wait(
+ session: AsyncSession = Depends(gw_deps.get_session),
+ _user: User = Depends(get_current_user),
+ _is_admin: bool = Depends(get_tenant_is_admin),
+ ) -> dict[str, bool]:
+ await RoomStore().get(session, "nothing")
+ async with session_factory() as other:
+ await RoomStore().get(other, "nothing")
+ return {"ok": True}
+
+ app = _app(
+ session_factory,
+ pool_checkouts,
+ room_service=_room_service(session_factory, calls, collab, {}),
+ collab=collab,
+ extra=held_and_wait,
+ )
+ async with _client(app, user) as client:
+ assert (await client.get("/held-and-wait")).status_code == 200
+
+ assert [(s.label, s.peak) for s in pool_checkouts.over()] == [
+ ("GET /held-and-wait", 2)
+ ]
+
+
+class TestRoomRoutes:
+ async def test_room_writes_hold_one_connection_at_a_time(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ ) -> None:
+ user = await _admin(session_factory)
+ async with session_factory() as session:
+ room = await _room(session, user, "main")
+ doomed = await _room(session, user, "doomed")
+ bulk = await _room(session, user, "bulk")
+ agent, running = await _agent(session, user, "helper")
+ await session.commit()
+ calls = _Calls(pool_checkouts)
+ collab = _CollabLifecycle(calls)
+ app = _app(
+ session_factory,
+ pool_checkouts,
+ room_service=_room_service(
+ session_factory, calls, collab, {agent.id: running}
+ ),
+ collab=collab,
+ )
+
+ async with _client(app, user) as client:
+ responses = [
+ await client.patch(f"/rooms/{room.id}", json={"name": "renamed"}),
+ await client.put(
+ f"/rooms/{room.id}/protection", json={"protection_config": {}}
+ ),
+ await client.put(
+ f"/rooms/{room.id}/observe", json={"observe_config": {}}
+ ),
+ await client.post(
+ f"/rooms/{room.id}/agents", json={"agent_ids": [agent.id]}
+ ),
+ await client.patch(
+ f"/rooms/{room.id}/agents/{agent.id}",
+ json={"receives_join_events": True},
+ ),
+ await client.delete(f"/rooms/{room.id}/agents/{agent.id}"),
+ await client.post(f"/rooms/{room.id}/archive"),
+ await client.post(f"/rooms/{room.id}/unarchive"),
+ await client.get(f"/rooms/{room.id}"),
+ await client.post(
+ "/rooms/bulk-archive",
+ json={"room_ids": [bulk.id], "archived": True},
+ ),
+ await client.delete(f"/rooms/{doomed.id}"),
+ await client.post("/rooms/bulk-delete", json={"room_ids": [bulk.id]}),
+ ]
+
+ assert [r.status_code for r in responses] == [200] * len(responses), [
+ r.text for r in responses if r.status_code != 200
+ ]
+ # The Matrix side was really reached, so "held nothing" means something.
+ assert {what for what, _ in calls.held} >= {
+ "matrix invite",
+ "matrix kick",
+ "matrix delete",
+ }
+ _assert_one_at_a_time(pool_checkouts, calls)
+
+
+class TestAgentRoutes:
+ async def test_deleting_an_agent_holds_one_connection_at_a_time(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ ) -> None:
+ user = await _admin(session_factory)
+ async with session_factory() as session:
+ by_id, _ = await _agent(session, user, "by-id")
+ await _agent(session, user, "by-name")
+ await session.commit()
+ calls = _Calls(pool_checkouts)
+ collab = _CollabLifecycle(calls)
+ app = _app(
+ session_factory,
+ pool_checkouts,
+ room_service=_room_service(session_factory, calls, collab, {}),
+ collab=collab,
+ )
+
+ async with _client(app, user) as client:
+ responses = [
+ await client.delete(f"/agents/{by_id.id}"),
+ await client.delete("/agents/by-name/by-name"),
+ ]
+
+ assert [r.status_code for r in responses] == [200, 200]
+ assert [what for what, _ in calls.held] == ["bridge identity removal"] * 2
+ _assert_one_at_a_time(pool_checkouts, calls)
+
+
+class TestBridgeRoutes:
+ async def _bridge(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> CollaborationBridge:
+ async with session_factory() as session:
+ client = Client(
+ transport_user_id="@bridge:test", display_name="bridge", type="bridge"
+ )
+ session.add(client)
+ await session.flush()
+ bridge = CollaborationBridge(
+ type="slack",
+ display_name="Slack",
+ connection_config={"bot_token": "xoxb-old"},
+ client_id=client.id,
+ status="running",
+ )
+ session.add(bridge)
+ await session.commit()
+ return bridge
+
+ async def test_platform_calls_are_made_holding_no_connection(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ ) -> None:
+ user = await _admin(session_factory)
+ bridge = await self._bridge(session_factory)
+ calls = _Calls(pool_checkouts)
+ collab = _CollabLifecycle(calls)
+ app = _app(
+ session_factory,
+ pool_checkouts,
+ room_service=_room_service(session_factory, calls, collab, {}),
+ collab=collab,
+ )
+
+ async with _client(app, user) as client:
+ edited = await client.patch(
+ f"/bridges/{bridge.id}",
+ json={
+ "agent_greetings_enabled": False,
+ "connection_config": {"bot_token": "xoxb-new"},
+ },
+ )
+ searched = await client.get(
+ f"/bridges/{bridge.id}/directory", params={"query": "ada"}
+ )
+ claimed = await client.post(
+ f"/bridges/{bridge.id}/identities",
+ json={"external_user_id": "U1", "username": "ada"},
+ )
+ deleted = await client.delete(f"/bridges/{bridge.id}")
+
+ assert edited.status_code == 200, edited.text
+ assert searched.status_code == 200, searched.text
+ # The fake refuses to provision, so the claim stops at 409 — after the
+ # directory search and the provisioning attempt, which is what counts.
+ assert claimed.status_code == 409, claimed.text
+ assert deleted.status_code == 200, deleted.text
+ assert [what for what, _ in calls.held] == [
+ "platform credential check",
+ "start guards",
+ "bridge restart",
+ "platform directory search",
+ "platform directory search",
+ "provision external user",
+ "bridge remove",
+ ]
+ _assert_one_at_a_time(pool_checkouts, calls)
+
+ async def test_a_refused_credential_edit_writes_nothing(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ ) -> None:
+ # The credential check now runs after the read's transaction ends, so
+ # the writes that used to precede it inside that transaction moved after
+ # it. A refusal must still leave every field as it was.
+ from switch_core.bridges.collaboration.models import BridgeCredentialError
+
+ user = await _admin(session_factory)
+ bridge = await self._bridge(session_factory)
+ calls = _Calls(pool_checkouts)
+ collab = _CollabLifecycle(calls)
+
+ async def refuse(**_: Any) -> None:
+ raise BridgeCredentialError("invalid_auth")
+
+ collab.check_edited_connection_config = refuse # type: ignore[method-assign]
+ app = _app(
+ session_factory,
+ pool_checkouts,
+ room_service=_room_service(session_factory, calls, collab, {}),
+ collab=collab,
+ )
+
+ async with _client(app, user) as client:
+ response = await client.patch(
+ f"/bridges/{bridge.id}",
+ json={
+ "agent_greetings_enabled": False,
+ "connection_config": {"bot_token": "xoxb-bad"},
+ },
+ )
+
+ assert response.status_code == 400
+ async with session_factory() as session:
+ stored = await CollaborationBridgeStore().get(session, bridge.id)
+ assert stored is not None
+ assert stored.agent_greetings_enabled is bridge.agent_greetings_enabled
+ assert stored.connection_config == {"bot_token": "xoxb-old"}
+
+
+class TestPasswordRoutes:
+ @pytest.fixture
+ def bcrypt_calls(
+ self, monkeypatch: pytest.MonkeyPatch, pool_checkouts: PoolCheckouts
+ ) -> list[tuple[str, bool, int]]:
+ """Each bcrypt call: which, whether it ran on the event loop's thread,
+ and the connections the request held at the time."""
+ seen: list[tuple[str, bool, int]] = []
+ real_hash = gateway_auth.hash_password
+ real_verify = gateway_auth.verify_password
+
+ def on_loop() -> bool:
+ return threading.current_thread() is threading.main_thread()
+
+ def hash_password(password: str) -> str:
+ seen.append(("hash", on_loop(), pool_checkouts.held_now()))
+ return real_hash(password)
+
+ def verify_password(password: str, password_hash: str | None) -> bool:
+ seen.append(("verify", on_loop(), pool_checkouts.held_now()))
+ return real_verify(password, password_hash)
+
+ monkeypatch.setattr(gateway_auth, "hash_password", hash_password)
+ monkeypatch.setattr(gateway_auth, "verify_password", verify_password)
+ return seen
+
+ async def test_bcrypt_runs_off_the_loop_holding_no_connection(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ bcrypt_calls: list[tuple[str, bool, int]],
+ ) -> None:
+ user = await _admin(session_factory)
+ async with session_factory() as session:
+ operator = User(name="op", email="[email protected]", role="admin")
+ session.add(operator)
+ await session.flush()
+ session.add(
+ TenantMember(
+ tenant_id=TENANT_ZERO_ID, user_id=operator.id, role="member"
+ )
+ )
+ await session.commit()
+ calls = _Calls(pool_checkouts)
+ collab = _CollabLifecycle(calls)
+ app = _app(
+ session_factory,
+ pool_checkouts,
+ room_service=_room_service(session_factory, calls, collab, {}),
+ collab=collab,
+ )
+
+ async with _client(app, user) as client:
+ login = await client.post(
+ "/auth/login", json={"email": user.email, "password": _PASSWORD}
+ )
+ wrong = await client.post(
+ "/auth/login", json={"email": user.email, "password": "nope-nope"}
+ )
+ changed = await client.put(
+ "/auth/me/password",
+ json={"current_password": _PASSWORD, "new_password": "a new one!"},
+ )
+ signed_up = await client.post(
+ "/auth/signup",
+ json={"email": "[email protected]", "password": "a password!"},
+ )
+ async with _client(app, operator) as client:
+ created = await client.post(
+ "/users",
+ json={
+ "name": "made",
+ "email": "[email protected]",
+ "password": "a password!",
+ "role": "user",
+ },
+ )
+
+ assert login.status_code == 200, login.text
+ assert wrong.status_code == 401, wrong.text
+ assert changed.status_code == 200, changed.text
+ assert signed_up.status_code == 201, signed_up.text
+ assert created.status_code == 200, created.text
+ assert bcrypt_calls == [
+ ("verify", False, 0),
+ ("verify", False, 0),
+ ("verify", False, 0),
+ ("hash", False, 0),
+ ("hash", False, 0),
+ ("hash", False, 0),
+ ]
+ _assert_one_at_a_time(pool_checkouts, calls)
diff --git a/core/tests/switch_core/management/harness.py b/core/tests/switch_core/management/harness.py
index c1b5c2aee..f39bc50c7 100644
--- a/core/tests/switch_core/management/harness.py
+++ b/core/tests/switch_core/management/harness.py
@@ -40,6 +40,10 @@ from switch_core.bridges.agent.auth import BearerAuthMiddleware, ControllerPrinc
from switch_core.bridges.agent.controller_auth_cache import ControllerAuthCache
from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
from switch_core.bridges.agent.protocol.agent_core import AgentCore
+from switch_core.bridges.agent.protocol.controller_presence import (
+ ControllerConnectionError,
+)
+from switch_core.bridges.agent.protocol.controller_stream import IDLE
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
from switch_core.db.models import (
TENANT_ZERO_ID,
@@ -59,6 +63,7 @@ from switch_core.gateway import dependencies as gw_deps
from switch_core.gateway.auth import create_jwt
from switch_core.keys import Keyring
from switch_core.management import controller_routes
+from switch_core.management.errors import ManagementError
from switch_core.management.wiring import Management, build_management
TEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
@@ -432,26 +437,20 @@ async def add_room(
return room.id
-def parse_frame(raw: bytes) -> tuple[str, dict[str, Any]]:
- event, data = "", "{}"
- for line in raw.decode().strip().splitlines():
- if line.startswith("event: "):
- event = line[len("event: ") :]
- elif line.startswith("data: "):
- data = line[len("data: ") :]
- return event, json.loads(data)
+def parse_frame(frame: dict[str, Any]) -> tuple[str, dict[str, Any]]:
+ return frame["event"], frame["data"]
async def take(
- stream: AsyncIterator[bytes], count: int, timeout: float = 3.0
+ stream: AsyncIterator[dict[str, Any]], count: int, timeout: float = 3.0
) -> list[tuple[str, dict[str, Any]]]:
- """The next `count` frames off a stream, keepalives skipped."""
+ """The next `count` frames off a stream, idle ticks skipped."""
frames: list[tuple[str, dict[str, Any]]] = []
async def pump() -> None:
while len(frames) < count:
raw = await anext(stream)
- if raw.startswith(b":"):
+ if raw is IDLE:
continue
frames.append(parse_frame(raw))
@@ -459,6 +458,30 @@ async def take(
return frames
+@dataclass
+class Answer:
+ """What the socket would have said, shaped like an HTTP response: a beat
+ or an attach either succeeds or is refused with the contract's envelope."""
+
+ status_code: int
+ body: dict[str, Any]
+
+ @property
+ def text(self) -> str:
+ return json.dumps(self.body)
+
+ def json(self) -> dict[str, Any]:
+ return self.body
+
+
+def _principal(controller: EnrolledController) -> ControllerPrincipal:
+ return ControllerPrincipal(
+ controller_id=controller.controller_id,
+ owner_id=controller.owner.id,
+ tenant_id=TENANT_ZERO_ID,
+ )
+
+
async def open_connection(
client: httpx.AsyncClient,
controller: EnrolledController,
@@ -477,22 +500,57 @@ async def open_stream(
harness: Harness,
controller: EnrolledController,
opened: dict[str, Any],
-) -> AsyncIterator[bytes]:
- """The controller's event stream, from the route itself.
-
- Called directly rather than over HTTP: the in-process transport collects a
- response whole, and this one never ends by itself.
- """
- response = await controller_routes.controller_events(
- principal=ControllerPrincipal(
- controller_id=controller.controller_id,
- owner_id=controller.owner.id,
- tenant_id=TENANT_ZERO_ID,
- ),
+) -> AsyncIterator[dict[str, Any]]:
+ """The frames the controller's socket would send, from the route's own
+ stream: the socket adds only pings around them."""
+ return await controller_routes.attach_stream(
+ principal=_principal(controller),
management=harness.management.service,
protocol=harness.protocol,
session_factory=harness.session_factory,
connection_id=opened["connection_id"],
generation=opened["generation"],
)
- return response.body_iterator # type: ignore[return-value]
+
+
+async def attach(
+ harness: Harness,
+ controller: EnrolledController,
+ connection_id: str,
+ generation: int,
+) -> Answer:
+ """An attach, answered as the socket's `refused` frame would be."""
+ try:
+ stream = await open_stream(
+ harness,
+ controller,
+ {"connection_id": connection_id, "generation": generation},
+ )
+ except ManagementError as error:
+ return Answer(error.status_code, error.body())
+ await stream.aclose() # type: ignore[attr-defined]
+ return Answer(200, {})
+
+
+def beat(
+ harness: Harness,
+ controller: EnrolledController,
+ *,
+ connection_id: str,
+ generation: int,
+ cursors: dict[str, int],
+) -> Answer:
+ """A pong, answered as the socket would: the agents on success, the
+ refusal its `evicted` frame would name otherwise."""
+ try:
+ agents = controller_routes.record_beat(
+ principal=_principal(controller),
+ protocol=harness.protocol,
+ connection_id=connection_id,
+ generation=generation,
+ cursors=cursors,
+ )
+ except ControllerConnectionError as exc:
+ error = controller_routes._connection_refusal(exc)
+ return Answer(error.status_code, error.body())
+ return Answer(200, {"agents": agents})
diff --git a/core/tests/switch_core/management/test_connection_socket.py b/core/tests/switch_core/management/test_connection_socket.py
new file mode 100644
index 000000000..6ab15a04f
--- /dev/null
+++ b/core/tests/switch_core/management/test_connection_socket.py
@@ -0,0 +1,192 @@
+"""The controller's connection over one WebSocket: the stream's frames down,
+pongs up as beats.
+
+What the stream carries, and what a beat does, are tested against Postgres in
+`test_stream.py` through the same `attach_stream` and `record_beat` the socket
+calls. These cover what is particular to the socket: framing, the ping the
+server drives, and how refusals and evictions are told.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections.abc import AsyncGenerator, Iterator
+from typing import Any
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+from starlette.types import ASGIApp, Receive, Scope, Send
+from starlette.websockets import WebSocketDisconnect
+
+from switch_core.bridges.agent.auth import ControllerPrincipal
+from switch_core.bridges.agent.dependencies import get_protocol
+from switch_core.bridges.agent.protocol.controller_presence import (
+ ControllerTakenOverError,
+ UnknownControllerConnectionError,
+)
+from switch_core.management import controller_routes
+from switch_core.management.dependencies import (
+ get_management,
+ get_management_session_factory,
+)
+
+CONTROLLER = "controller-1"
+PATH = f"/v1/controllers/{CONTROLLER}/connection/ws?connection_id=c1&generation=7"
+
+
+class _AuthenticatedAs:
+ """What the bearer middleware leaves in scope for a controller token."""
+
+ def __init__(self, app: ASGIApp) -> None:
+ self.app = app
+
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+ scope["controller"] = ControllerPrincipal(
+ controller_id=CONTROLLER, owner_id="owner-1", tenant_id="tenant-1"
+ )
+ await self.app(scope, receive, send)
+
+
+class _Calls:
+ def __init__(self) -> None:
+ self.frames: list[dict[str, Any]] = [
+ {"event": "connection_state", "data": {"connection_id": "c1"}},
+ {"event": "agent.attached", "data": {"agent_id": "a1", "from_seq": 4}},
+ ]
+ self.attach_refusal: Exception | None = None
+ self.beat_refusal: Exception | None = None
+ self.beats: list[dict[str, Any]] = []
+
+
+def calls() -> _Calls:
+ return _Calls()
+
+
+def client(calls: _Calls, monkeypatch: pytest.MonkeyPatch) -> Iterator[TestClient]:
+ # Pings every 50 ms rather than every heartbeat interval.
+ monkeypatch.setattr(controller_routes, "HEARTBEAT_INTERVAL_SECONDS", 0.05)
+
+ async def attach_stream(**kwargs: Any) -> AsyncGenerator[dict[str, Any]]:
+ if calls.attach_refusal is not None:
+ raise calls.attach_refusal
+
+ async def frames() -> AsyncGenerator[dict[str, Any]]:
+ for frame in calls.frames:
+ yield frame
+ while True: # the stream stays open until the socket goes
+ yield controller_routes.IDLE
+ await _sleep()
+
+ return frames()
+
+ def record_beat(**kwargs: Any) -> list[str]:
+ calls.beats.append(kwargs)
+ if calls.beat_refusal is not None:
+ raise calls.beat_refusal
+ return ["a1"]
+
+ monkeypatch.setattr(controller_routes, "attach_stream", attach_stream)
+ monkeypatch.setattr(controller_routes, "record_beat", record_beat)
+ app = FastAPI()
+ app.include_router(controller_routes.router)
+ app.dependency_overrides[get_protocol] = lambda: None
+ app.dependency_overrides[get_management] = lambda: None
+ app.dependency_overrides[get_management_session_factory] = lambda: None
+ with TestClient(_AuthenticatedAs(app)) as test_client:
+ yield test_client
+
+
+async def _sleep() -> None:
+ await asyncio.sleep(0.01)
+
+
+def _until(socket: Any, event: str) -> dict[str, Any]:
+ for _ in range(50):
+ frame: dict[str, Any] = socket.receive_json()
+ if frame["event"] == event:
+ return frame
+ raise AssertionError(f"no {event} frame")
+
+
+def test_the_stream_comes_down_as_json_and_the_server_pings(client: TestClient) -> None:
+ with client.websocket_connect(PATH) as socket:
+ first = socket.receive_json()
+ second = socket.receive_json()
+ ping = _until(socket, "ping")
+
+ assert first == {"event": "connection_state", "data": {"connection_id": "c1"}}
+ assert second["event"] == "agent.attached"
+ assert ping == {"event": "ping", "data": {}}
+
+
+def test_a_pong_is_the_beat_with_every_agents_cursor(
+ client: TestClient, calls: _Calls
+) -> None:
+ with client.websocket_connect(PATH) as socket:
+ _until(socket, "ping")
+ socket.send_json({"type": "pong", "cursors": {"a1": 9, "a2": -1, "a3": "x"}})
+ _until(socket, "ping")
+
+ assert calls.beats[0]["connection_id"] == "c1"
+ assert calls.beats[0]["generation"] == 7
+ # A cursor that is not a non-negative integer is left out, not trusted.
+ assert calls.beats[0]["cursors"] == {"a1": 9}
+
+
+def test_a_refused_beat_is_told_as_evicted_then_the_socket_closes(
+ client: TestClient, calls: _Calls
+) -> None:
+ calls.beat_refusal = ControllerTakenOverError("c1")
+ with client.websocket_connect(PATH) as socket:
+ _until(socket, "ping")
+ socket.send_json({"type": "pong", "cursors": {}})
+ evicted = _until(socket, "evicted")
+ with pytest.raises(WebSocketDisconnect) as closed:
+ socket.receive_json()
+
+ assert evicted["data"]["code"] == "taken_over"
+ assert closed.value.code == 1000
+
+
+def test_a_refused_attach_is_a_refused_frame_then_4000_plus_the_status(
+ client: TestClient, calls: _Calls
+) -> None:
+ calls.attach_refusal = controller_routes._connection_refusal(
+ UnknownControllerConnectionError("c1")
+ )
+ with client.websocket_connect(PATH) as socket:
+ refused = socket.receive_json()
+ with pytest.raises(WebSocketDisconnect) as closed:
+ socket.receive_json()
+
+ assert refused["event"] == "refused"
+ assert refused["data"]["status"] == 404
+ assert refused["data"]["detail"]["code"] == "unknown_connection"
+ assert closed.value.code == 4404
+
+
+def test_a_controller_speaking_protocol_1_is_refused(client: TestClient) -> None:
+ with client.websocket_connect(
+ PATH, headers={"Switch-Controller-Protocol": "1"}
+ ) as socket:
+ refused = socket.receive_json()
+ with pytest.raises(WebSocketDisconnect) as closed:
+ socket.receive_json()
+
+ assert refused["data"]["detail"]["code"] == "protocol_unsupported"
+ assert closed.value.code == 4426
+
+
+def test_another_controllers_socket_is_forbidden(client: TestClient) -> None:
+ with client.websocket_connect(
+ "/v1/controllers/someone-else/connection/ws?connection_id=c1&generation=7"
+ ) as socket:
+ refused = socket.receive_json()
+ with pytest.raises(WebSocketDisconnect) as closed:
+ socket.receive_json()
+
+ assert refused["data"]["detail"]["code"] == "forbidden"
+ assert closed.value.code == 4403
diff --git a/core/tests/switch_core/management/test_contract_fixtures.py b/core/tests/switch_core/management/test_contract_fixtures.py
index 550214b47..fc78a651f 100644
--- a/core/tests/switch_core/management/test_contract_fixtures.py
+++ b/core/tests/switch_core/management/test_contract_fixtures.py
@@ -25,8 +25,8 @@ from switch_core.bridges.agent.commands import room_control_frame
from switch_core.bridges.agent.protocol.event_buffer import Reader
from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload
from switch_core.db.stores.agent_store import AgentStore
+from switch_core.management import controller_routes
from switch_core.management.schemas import (
- ControllerBeatRequest,
ControllerConnectionRequest,
EnrollRequest,
OperationResultRequest,
@@ -38,6 +38,7 @@ from tests.switch_core.management.harness import (
Harness,
add_member,
add_room,
+ beat,
build_harness,
cookies_for,
create_managed_agent,
@@ -96,7 +97,6 @@ class TestRequestFixturesParse:
ControllerConnectionRequest.model_validate(
_fixture("controller_connection_request.json")
)
- ControllerBeatRequest.model_validate(_fixture("controller_beat_request.json"))
class TestEnrollmentAndTokens:
@@ -259,16 +259,15 @@ class TestTheControllerConnection:
)
stream = await open_stream(harness, controller, opened.json())
await take(stream, 2)
- beat_body = _fixture("controller_beat_request.json")
- beat_body.update(
+ pong = _fixture("controller_pong.json")
+ beaten = beat(
+ harness,
+ controller,
connection_id=opened.json()["connection_id"],
generation=opened.json()["generation"],
- cursors={agent_id: 0},
- )
- beat = await client.post(
- f"/v1/controllers/{controller.controller_id}/connection/beat",
- json=beat_body,
- headers=controller.headers,
+ cursors=controller_routes._cursors_of(
+ {**pong, "cursors": {agent_id: 0}}
+ ),
)
present = harness.protocol.connections.controllers.live_rooms(agent_id)
await stream.aclose()
@@ -277,14 +276,11 @@ class TestTheControllerConnection:
assert_same_shape(
opened.json(), _fixture("controller_connection_response.json")
)
- assert beat.status_code == 200, beat.text
- assert_same_shape(beat.json(), _fixture("controller_beat_response.json"))
+ assert beaten.status_code == 200, beaten.text
assert harness.protocol.connections.controllers.live_rooms(agent_id) == set()
assert present == {room_id}
- async def test_placements_are_refused_on_open_and_beat(
- self, harness: Harness
- ) -> None:
+ async def test_placements_are_refused_on_open(self, harness: Harness) -> None:
owner = await add_member(harness.session_factory, "ada")
async with harness.client() as client:
controller = await enroll_console(harness, client, owner)
@@ -292,36 +288,15 @@ class TestTheControllerConnection:
room_id = await add_room(harness.session_factory, agent_id)
body = _fixture("controller_connection_request.json")
body["cursors"] = {agent_id: 0}
- refused_open = await client.post(
+ refused = await client.post(
f"/v1/controllers/{controller.controller_id}/connection",
json={**body, "placements": {agent_id: [room_id]}},
headers=controller.headers,
)
- opened = await client.post(
- f"/v1/controllers/{controller.controller_id}/connection",
- json=body,
- headers=controller.headers,
- )
- stream = await open_stream(harness, controller, opened.json())
- await take(stream, 2)
- beat_body = _fixture("controller_beat_request.json")
- beat_body.update(
- connection_id=opened.json()["connection_id"],
- generation=opened.json()["generation"],
- cursors={agent_id: 0},
- placements={agent_id: [room_id]},
- )
- refused_beat = await client.post(
- f"/v1/controllers/{controller.controller_id}/connection/beat",
- json=beat_body,
- headers=controller.headers,
- )
- await stream.aclose()
- for refused in (refused_open, refused_beat):
- assert refused.status_code == 422, refused.text
- assert refused.json()["error"]["code"] == "validation_error"
- assert "placements" in refused.json()["error"]["message"]
+ assert refused.status_code == 422, refused.text
+ assert refused.json()["error"]["code"] == "validation_error"
+ assert "placements" in refused.json()["error"]["message"]
class _Outcomes:
diff --git a/core/tests/switch_core/management/test_controller_routes.py b/core/tests/switch_core/management/test_controller_routes.py
index 74162feca..4248b302f 100644
--- a/core/tests/switch_core/management/test_controller_routes.py
+++ b/core/tests/switch_core/management/test_controller_routes.py
@@ -440,18 +440,18 @@ class TestStatus:
response = await client.put(
f"/v1/management/controllers/{controller.controller_id}/status",
json=fixture("status_request.json"),
- headers={**controller.headers, "Switch-Controller-Protocol": "2"},
+ headers={**controller.headers, "Switch-Controller-Protocol": "1"},
)
accepted = await client.put(
f"/v1/management/controllers/{controller.controller_id}/status",
json=fixture("status_request.json"),
- headers={**controller.headers, "Switch-Controller-Protocol": "1"},
+ headers={**controller.headers, "Switch-Controller-Protocol": "2"},
)
assert response.status_code == 426
assert response.json()["error"]["code"] == "protocol_unsupported"
- assert response.headers["Switch-Controller-Protocol-Accepts"] == "1-1"
+ assert response.headers["Switch-Controller-Protocol-Accepts"] == "2-2"
assert accepted.status_code == 200
- assert accepted.headers["Switch-Controller-Protocol-Accepts"] == "1-1"
+ assert accepted.headers["Switch-Controller-Protocol-Accepts"] == "2-2"
async def _placed_agent(
diff --git a/core/tests/switch_core/management/test_middleware.py b/core/tests/switch_core/management/test_middleware.py
index 47729dac1..e3a68ec49 100644
--- a/core/tests/switch_core/management/test_middleware.py
+++ b/core/tests/switch_core/management/test_middleware.py
@@ -245,7 +245,6 @@ class TestRefusals:
"POST",
f"/v1/management/controllers/{second.controller_id}/credential/rotate",
),
- ("GET", f"/v1/controllers/{second.controller_id}/events"),
]:
response = await client.request(method, path, headers=first.headers)
assert response.status_code == 403, (method, path, response.text)
diff --git a/core/tests/switch_core/management/test_stream.py b/core/tests/switch_core/management/test_stream.py
index ee6f90b98..2f5adb9cf 100644
--- a/core/tests/switch_core/management/test_stream.py
+++ b/core/tests/switch_core/management/test_stream.py
@@ -39,6 +39,8 @@ from tests.switch_core.management.harness import (
Harness,
add_member,
add_room,
+ attach,
+ beat,
build_harness,
cookies_for,
enroll_console,
@@ -415,50 +417,36 @@ class TestTheConnection:
evicted = await take(old_stream, 1)
with pytest.raises(StopAsyncIteration):
await anext(old_stream)
- beat_path = f"/v1/controllers/{controller.controller_id}/connection/beat"
- stale = await client.post(
- beat_path,
- json={
- "connection_id": old["connection_id"],
- "generation": old["generation"],
- "cursors": {},
- },
- headers=controller.headers,
+ stale = beat(
+ harness,
+ controller,
+ connection_id=old["connection_id"],
+ generation=old["generation"],
+ cursors={},
)
- wrong_generation = await client.post(
- beat_path,
- json={
- "connection_id": new["connection_id"],
- "generation": new["generation"] + 1,
- "cursors": {},
- },
- headers=controller.headers,
+ wrong_generation = beat(
+ harness,
+ controller,
+ connection_id=new["connection_id"],
+ generation=new["generation"] + 1,
+ cursors={},
)
- unknown = await client.post(
- beat_path,
- json={
- "connection_id": "nope",
- "generation": 1,
- "cursors": {},
- },
- headers=controller.headers,
+ unknown = beat(
+ harness,
+ controller,
+ connection_id="nope",
+ generation=1,
+ cursors={},
)
- no_stream = await client.post(
- beat_path,
- json={
- "connection_id": new["connection_id"],
- "generation": new["generation"],
- "cursors": {},
- },
- headers=controller.headers,
+ no_stream = beat(
+ harness,
+ controller,
+ connection_id=new["connection_id"],
+ generation=new["generation"],
+ cursors={},
)
- old_events = await client.get(
- f"/v1/controllers/{controller.controller_id}/events",
- params={
- "connection_id": old["connection_id"],
- "generation": old["generation"],
- },
- headers=controller.headers,
+ old_events = await attach(
+ harness, controller, old["connection_id"], old["generation"]
)
assert new["generation"] != old["generation"]
@@ -500,20 +488,18 @@ class TestTheConnection:
assert not presence.is_live(agent_id)
stream = await open_stream(harness, controller, opened)
await take(stream, 2)
- beat = await client.post(
- f"/v1/controllers/{controller.controller_id}/connection/beat",
- json={
- "connection_id": opened["connection_id"],
- "generation": opened["generation"],
- "cursors": {agent_id: 99, "not-mine": 3},
- },
- headers=controller.headers,
+ beaten = beat(
+ harness,
+ controller,
+ connection_id=opened["connection_id"],
+ generation=opened["generation"],
+ cursors={agent_id: 99, "not-mine": 3},
)
live = presence.is_live(agent_id)
await stream.aclose()
- assert beat.status_code == 200, beat.text
- assert beat.json() == {"agents": [agent_id]}
+ assert beaten.status_code == 200, beaten.text
+ assert beaten.json() == {"agents": [agent_id]}
assert live
binding = presence.binding(agent_id)
assert binding is not None
@@ -541,16 +527,14 @@ class TestTheConnection:
first = await open_stream(harness, controller, opened)
# state, two attaches, three events for `confirmed`
await take(first, 6)
- beat = await client.post(
- f"/v1/controllers/{controller.controller_id}/connection/beat",
- json={
- "connection_id": opened["connection_id"],
- "generation": opened["generation"],
- "cursors": {confirmed: 2},
- },
- headers=controller.headers,
+ beaten = beat(
+ harness,
+ controller,
+ connection_id=opened["connection_id"],
+ generation=opened["generation"],
+ cursors={confirmed: 2},
)
- assert beat.status_code == 200, beat.text
+ assert beaten.status_code == 200, beaten.text
# The socket drops; events keep arriving; the same connection is
# streamed again.
buffer.enqueue(
@@ -649,7 +633,7 @@ class TestTheConnection:
async def test_a_lapsed_beat_ends_the_stream_and_its_agents_are_not_live(
self, harness: Harness, monkeypatch: pytest.MonkeyPatch
) -> None:
- monkeypatch.setattr(controller_routes, "KEEPALIVE_INTERVAL_SECONDS", 0.05)
+ monkeypatch.setattr(controller_routes, "IDLE_INTERVAL_SECONDS", 0.05)
owner = await add_member(harness.session_factory, "ada")
presence = harness.protocol.connections.controllers
async with harness.client() as client:
@@ -663,14 +647,12 @@ class TestTheConnection:
conn.last_beat = time.monotonic() - HEARTBEAT_TTL_SECONDS - 1
assert not presence.is_live(agent_id)
evicted = await take(stream, 1)
- beat = await client.post(
- f"/v1/controllers/{controller.controller_id}/connection/beat",
- json={
- "connection_id": opened["connection_id"],
- "generation": opened["generation"],
- "cursors": {},
- },
- headers=controller.headers,
+ beaten = beat(
+ harness,
+ controller,
+ connection_id=opened["connection_id"],
+ generation=opened["generation"],
+ cursors={},
)
assert evicted == [
@@ -683,7 +665,7 @@ class TestTheConnection:
},
)
]
- assert (beat.status_code, beat.json()["error"]["code"]) == (
+ assert (beaten.status_code, beaten.json()["error"]["code"]) == (
404,
"unknown_connection",
)
@@ -734,4 +716,4 @@ async def test_a_waiting_stream_wakes_promptly_for_an_event(harness: Harness) ->
)
raw = await asyncio.wait_for(pending, timeout=2)
await stream.aclose()
- assert b"agent.event" in raw
+ assert raw["event"] == "agent.event"
diff --git a/core/tests/switch_core/observability/test_agent_protocol_metrics.py b/core/tests/switch_core/observability/test_agent_protocol_metrics.py
index 23dcdad70..35b7a1a74 100644
--- a/core/tests/switch_core/observability/test_agent_protocol_metrics.py
+++ b/core/tests/switch_core/observability/test_agent_protocol_metrics.py
@@ -133,3 +133,98 @@ def test_a_quiet_sweep_counts_nothing(registry):
assert connections.sweep() == []
assert _counts(registry, "switch.agent.connections_expired") == {}
+
+
+def _open(connections: AgentConnectionRegistry, connection_id: str = "c1") -> None:
+ connections.open(
+ agent_id="agent-1",
+ connection_id=connection_id,
+ scope="single", # type: ignore[arg-type]
+ delivery_filter="all", # type: ignore[arg-type]
+ spawn_capable=False,
+ cursor=0,
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+
+
+def test_a_first_open_is_fresh(registry):
+ _open(AgentConnectionRegistry())
+
+ assert _counts(registry, "switch.agent.connections_opened") == {
+ (("kind", "fresh"),): 1.0
+ }
+
+
+def test_a_socket_coming_back_to_a_live_connection_is_a_reattach(registry):
+ connections = AgentConnectionRegistry()
+ _open(connections)
+ _open(connections)
+
+ assert _counts(registry, "switch.agent.connections_opened") == {
+ (("kind", "fresh"),): 1.0,
+ (("kind", "reattach"),): 1.0,
+ }
+
+
+def test_a_return_after_a_lapse_is_told_apart(registry, monkeypatch):
+ """The reconnect storm: agents lapse, then come straight back."""
+ connections = AgentConnectionRegistry()
+ _open(connections)
+ later = time_module.monotonic() + conn_module.HEARTBEAT_TTL_SECONDS + 1
+ monkeypatch.setattr(conn_module.time, "monotonic", lambda: later)
+ assert connections.sweep()
+
+ _open(connections)
+
+ assert _counts(registry, "switch.agent.connections_opened") == {
+ (("kind", "fresh"),): 1.0,
+ (("kind", "after_lapse"),): 1.0,
+ }
+
+
+def test_a_return_after_any_other_close_is_after_close(registry):
+ connections = AgentConnectionRegistry()
+ _open(connections)
+ connections.close("c1", conn_module.TAKEN_OVER)
+ _open(connections)
+
+ assert (
+ _counts(registry, "switch.agent.connections_opened")[(("kind", "after_close"),)]
+ == 1.0
+ )
+
+
+def test_the_memory_of_closed_connections_is_bounded(monkeypatch):
+ monkeypatch.setattr(conn_module, "_RECENTLY_CLOSED_LIMIT", 2)
+ connections = AgentConnectionRegistry()
+ for index in range(4):
+ _open(connections, f"c{index}")
+ connections.close(f"c{index}", conn_module.HEARTBEAT_LAPSED)
+
+ assert list(connections._recently_closed) == ["c2", "c3"]
+
+
+def test_a_lapse_noticed_on_next_use_is_counted_too(registry, monkeypatch):
+ """The sweep is one of three ways a lapse is noticed. Counting only it
+ under-reported lapses against the reconnects that follow them."""
+ connections = AgentConnectionRegistry()
+ _open(connections)
+ later = time_module.monotonic() + conn_module.HEARTBEAT_TTL_SECONDS + 1
+ monkeypatch.setattr(conn_module.time, "monotonic", lambda: later)
+
+ with pytest.raises(conn_module.UnknownConnectionError):
+ connections.require("agent-1", "c1")
+ _open(connections)
+
+ # One collection: reading the registry resets it.
+ payloads = {p.name: p for p in registry.collect()}
+ expired = sum(
+ point.value for point in payloads["switch.agent.connections_expired"].numbers
+ )
+ opened = {
+ point.attributes["kind"]: point.value
+ for point in payloads["switch.agent.connections_opened"].numbers
+ }
+ assert expired == 1.0
+ assert opened == {"fresh": 1.0, "after_lapse": 1.0}
diff --git a/core/tests/switch_core/observability/test_bootstrap.py b/core/tests/switch_core/observability/test_bootstrap.py
index bc677f899..02edafaf9 100644
--- a/core/tests/switch_core/observability/test_bootstrap.py
+++ b/core/tests/switch_core/observability/test_bootstrap.py
@@ -7,11 +7,14 @@ import json
from contextlib import asynccontextmanager
import pytest
+from sqlalchemy.ext.asyncio import create_async_engine
+from sqlalchemy.pool import NullPool
from switch_core.config import SwitchConfig
from switch_core.observability.bootstrap import RuntimeProbes, start_observability
from switch_core.observability.metrics import metrics, uninstall
from switch_core.observability.pool import PoolStats
+from switch_core.transport.room_cache import RoomCacheStats
BASE_ENV = {
"DB_HOST": "localhost",
@@ -74,8 +77,12 @@ def _probes(**overrides) -> RuntimeProbes:
consumers_running=lambda: 5,
connectors_running=lambda: 2,
connectors_configured=lambda: 2,
- agents_connected=lambda: 3,
+ agents_connected=lambda: {
+ ("websocket", "agent-runtime"): 2,
+ ("websocket", "unknown"): 1,
+ },
pool_stats=lambda: PoolStats(in_use=4, size=30, overflow=0),
+ room_cache_stats=lambda: RoomCacheStats(bytes=0, rooms=0, rows=0),
)
return RuntimeProbes(**{**defaults, **overrides})
@@ -163,7 +170,14 @@ async def test_an_endpoint_installs_the_registry_and_reports_state(monkeypatch):
for name, payload in payloads.items()
if payload.numbers and not payload.numbers[0].attributes
}
- assert values["switch.agents.connected"] == 3.0
+ connected = {
+ (point.attributes["transport"], point.attributes["client"]): point.value
+ for point in payloads["switch.agents.connected"].numbers
+ }
+ assert connected == {
+ ("websocket", "agent-runtime"): 2.0,
+ ("websocket", "unknown"): 1.0,
+ }
assert values["switch.consumers.running"] == 5.0
assert values["switch.connectors.running"] == 2.0
assert values["switch.db.pool.in_use"] == 4.0
@@ -181,6 +195,52 @@ async def test_an_endpoint_installs_the_registry_and_reports_state(monkeypatch):
await observability.aclose()
+async def test_the_room_cache_reports_what_it_holds(monkeypatch):
+ config = _config(
+ monkeypatch,
+ OTLP_ENDPOINT="https://collector.example",
+ DEPLOYMENT_ID=DEPLOYMENT_ID,
+ OTLP_EXPORT_INTERVAL_SECONDS="3600",
+ )
+ observability = start_observability(
+ config=config,
+ version="1.0.0",
+ session_factory=_session_factory(),
+ probes=_probes(
+ room_cache_stats=lambda: RoomCacheStats(bytes=4096, rooms=3, rows=12)
+ ),
+ )
+ try:
+ payloads = {p.name: p for p in metrics().collect()}
+ assert payloads["switch.delivery_cache.bytes"].numbers[0].value == 4096.0
+ assert payloads["switch.delivery_cache.rooms"].numbers[0].value == 3.0
+ finally:
+ await observability.aclose()
+
+
+async def test_a_start_is_counted_once_so_restarts_line_up(monkeypatch):
+ observability = start_observability(
+ config=_config(
+ monkeypatch,
+ OTLP_ENDPOINT="https://collector.example",
+ DEPLOYMENT_ID=DEPLOYMENT_ID,
+ OTLP_EXPORT_INTERVAL_SECONDS="3600",
+ ),
+ version="1.0.0",
+ session_factory=_session_factory(),
+ probes=_probes(),
+ )
+ try:
+ starts = next(
+ p for p in metrics().collect() if p.name == "switch.runtime.starts"
+ )
+ assert [point.value for point in starts.numbers] == [1.0]
+ finally:
+ await observability.aclose()
+
+
@pytest.mark.asyncio
async def test_a_pool_that_reports_nothing_produces_no_reading(monkeypatch):
"""A zero would draw an idle pool; absence draws nothing, which is true."""
@@ -262,3 +322,72 @@ async def test_closing_stops_the_loops_and_uninstalls(monkeypatch):
assert metrics().enabled is False
assert all(task.done() for task in observability._tasks)
+
+
+async def test_the_database_sampler_starts_with_the_metrics_exporter(monkeypatch):
+ built: list[object] = []
+
+ def engine():
+ built.append(object())
+ return create_async_engine(
+ "postgresql+asyncpg://u:[email protected]:1/x", poolclass=NullPool
+ )
+
+ observability = start_observability(
+ config=_config(
+ monkeypatch,
+ OTLP_ENDPOINT="https://collector.example",
+ DEPLOYMENT_ID=DEPLOYMENT_ID,
+ OTLP_EXPORT_INTERVAL_SECONDS="3600",
+ ),
+ version="1.0.0",
+ session_factory=_session_factory(),
+ probes=_probes(),
+ db_server_engine=engine,
+ )
+ try:
+ await asyncio.sleep(0.05)
+ names = {task.get_name() for task in observability._tasks}
+ assert "db-server-sampler" in names
+ assert built, "the sampler never built its engine"
+ finally:
+ await observability.aclose()
+
+
+async def test_a_connection_kind_that_empties_out_is_reported_as_zero(monkeypatch):
+ """Otherwise the dashboard keeps drawing its last count, which is exactly
+ the line that has to reach zero before an old transport can be removed."""
+ connected = {("websocket", "agent-runtime"): 2, ("detached", "agent-runtime"): 1}
+ observability = start_observability(
+ config=_config(
+ monkeypatch,
+ OTLP_ENDPOINT="https://collector.example",
+ DEPLOYMENT_ID=DEPLOYMENT_ID,
+ OTLP_EXPORT_INTERVAL_SECONDS="3600",
+ ),
+ version="1.0.0",
+ session_factory=_session_factory(),
+ probes=_probes(agents_connected=lambda: dict(connected)),
+ )
+ try:
+
+ def reading() -> dict[tuple[str, str], float]:
+ payload = {p.name: p for p in metrics().collect()}[
+ "switch.agents.connected"
+ ]
+ return {
+ (n.attributes["transport"], n.attributes["client"]): n.value
+ for n in payload.numbers
+ }
+
+ assert reading()[("detached", "agent-runtime")] == 1.0
+ del connected[("detached", "agent-runtime")]
+ assert reading() == {
+ ("websocket", "agent-runtime"): 2.0,
+ ("detached", "agent-runtime"): 0.0,
+ ("websocket", "unknown"): 0.0,
+ }
+ finally:
+ await observability.aclose()
diff --git a/core/tests/switch_core/observability/test_connection_metrics.py b/core/tests/switch_core/observability/test_connection_metrics.py
new file mode 100644
index 000000000..a4dd62d50
--- /dev/null
+++ b/core/tests/switch_core/observability/test_connection_metrics.py
@@ -0,0 +1,347 @@
+"""How agents are connected, and why a connection was refused.
+
+The rollout question is which clients are still connecting the old way, so the
+connected gauge carries the transport and the client, and the refusals carry a
+reason. Both labels come from fixed sets: a client's own string never reaches a
+metric.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from typing import Any
+
+import pytest
+from fastapi import FastAPI, HTTPException
+from fastapi.testclient import TestClient
+from starlette.types import ASGIApp, Receive, Scope, Send
+from starlette.websockets import WebSocketDisconnect
+
+from switch_core.bridges.agent.api import handlers
+from switch_core.bridges.agent.api.session_reporter import SessionReporter
+from switch_core.bridges.agent.api_key_cache import ApiKeyCache
+from switch_core.bridges.agent.auth import BearerAuthMiddleware
+from switch_core.bridges.agent.dependencies import get_config, get_protocol
+from switch_core.bridges.agent.protocol.agent_connections import (
+ CONTROLLER_LABEL,
+ PROTOCOL_VERSION,
+ AgentConnectionRegistry,
+ ClientDeclaration,
+ client_label,
+)
+from switch_core.bridges.agent.protocol.controller_presence import Binding
+from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+from switch_core.db.models import Agent
+from switch_core.db.stores.agent_store import AgentStore
+from switch_core.db.stores.api_key_store import ApiKeyStore
+from switch_core.observability.bootstrap import RuntimeProbes, _state_readings
+from switch_core.observability.metrics import MetricsRegistry, install, uninstall
+from switch_core.transport.room_cache import RoomCacheStats
+
+AGENT_ID = "agent-1"
+
+
+def registry() -> Iterator[MetricsRegistry]:
+ registry = MetricsRegistry()
+ install(registry)
+ yield registry
+ uninstall()
+
+
+def _counts(registry: MetricsRegistry, name: str) -> dict[tuple, float]:
+ payload = next((p for p in registry.collect() if p.name == name), None)
+ if payload is None:
+ return {}
+ return {
+ tuple(sorted(point.attributes.items())): point.value
+ for point in payload.numbers
+ }
+
+
+def _open(
+ connections: AgentConnectionRegistry,
+ agent_id: str,
+ connection_id: str,
+ artifact: str | None,
+ version: str | None = None,
+) -> Any:
+ return connections.open(
+ agent_id=agent_id,
+ connection_id=connection_id,
+ scope="all",
+ delivery_filter="all",
+ spawn_capable=False,
+ cursor=0,
+ declaration=ClientDeclaration(
+ speaks=PROTOCOL_VERSION, artifact=artifact, version=version
+ ),
+ expected_generation=None,
+ )
+
+
+# ── Connected, by transport and client ───────────────────────────────────────
+
+
+ ("artifact", "label"),
+ [
+ ("agent-runtime", "agent-runtime"),
+ ("switch-console", "switch-console"),
+ (None, "unknown"),
+ ("", "unknown"),
+ ("my-homemade-runtime/9.9", "other"),
+ ],
+)
+def test_the_client_label_is_a_registered_name_or_a_bucket(
+ artifact: str | None, label: str
+) -> None:
+ assert client_label(ClientDeclaration(artifact=artifact)) == label
+
+
+def test_agents_are_counted_per_transport_and_client() -> None:
+ connections = AgentConnectionRegistry()
+ _open(connections, "a1", "c1", "agent-runtime", version="0.7.2")
+ # One agent, two sockets from the same client: still one agent.
+ _open(connections, "a1", "c2", "agent-runtime", version="0.7.3")
+ _open(connections, "a2", "c3", None)
+ _open(connections, "a3", "c4", "something-else")
+ detached = _open(connections, "a4", "c5", "agent-runtime")
+ connections.detach_stream(detached, detached.stream_generation)
+
+ assert connections.live_agents_by_transport() == {
+ ("websocket", "agent-runtime"): 1,
+ ("websocket", "unknown"): 1,
+ ("websocket", "other"): 1,
+ ("detached", "agent-runtime"): 1,
+ }
+
+
+def test_agents_run_by_a_live_controller_count_under_the_controller() -> None:
+ connections = AgentConnectionRegistry()
+ _open(connections, "a1", "c1", "agent-runtime")
+ controllers = connections.controllers
+ for agent_id in ("m1", "m2"):
+ controllers.bind(
+ Binding(
+ agent_id=agent_id,
+ controller_id="ctl",
+ tenant_id="t",
+ controller_name="the machine",
+ running=True,
+ )
+ )
+ conn = controllers.open(controller_id="ctl", tenant_id="t", resume_cursors={})
+ controllers.attach_stream(conn)
+
+ assert connections.live_agents_by_transport() == {
+ ("websocket", "agent-runtime"): 1,
+ CONTROLLER_LABEL: 2,
+ }
+
+
+def test_an_agent_connected_two_ways_counts_under_each() -> None:
+ connections = AgentConnectionRegistry()
+ _open(connections, "a1", "c1", "agent-runtime")
+ _open(connections, "a1", "c2", None)
+
+ assert connections.live_agents_by_transport() == {
+ ("websocket", "agent-runtime"): 1,
+ ("websocket", "unknown"): 1,
+ }
+
+
+def _probes(agents_connected: Any) -> RuntimeProbes:
+ return RuntimeProbes(
+ listener_connected=lambda: True,
+ session_activity_listener_connected=lambda: True,
+ bridges_running=lambda: 0,
+ bridges_running_by_platform=lambda: {},
+ bridges_configured=lambda: 0,
+ consumers_running=lambda: 0,
+ connectors_running=lambda: 0,
+ connectors_configured=lambda: 0,
+ agents_connected=agents_connected,
+ pool_stats=lambda: None,
+ room_cache_stats=lambda: RoomCacheStats(bytes=0, rooms=0, rows=0),
+ )
+
+
+def _connected_readings(probes: RuntimeProbes) -> dict[tuple, float]:
+ return {
+ tuple(sorted(reading.attributes.items())): reading.value
+ for reading in _state_readings(probes)()
+ if reading.spec.name == "switch.agents.connected"
+ }
+
+
+def test_the_gauge_carries_transport_and_client_but_never_the_version() -> None:
+ connections = AgentConnectionRegistry()
+ _open(connections, "a1", "c1", "agent-runtime", version="0.7.2")
+ _open(connections, "a2", "c2", "agent-runtime", version="0.0.1-dirty")
+
+ readings = _connected_readings(_probes(connections.live_agents_by_transport))
+
+ # The zero baseline is always there; no series carries a version.
+ assert readings == {
+ (("client", "agent-runtime"), ("transport", "websocket")): 2.0,
+ (("client", "unknown"), ("transport", "websocket")): 0.0,
+ }
+
+
+def test_the_gauge_still_reports_zero_with_nobody_connected() -> None:
+ """A panel over no series reads "no data", which is not the same as none."""
+ readings = _connected_readings(_probes(lambda: {}))
+
+ assert list(readings.values()) == [0.0]
+
+
+# ── Refused ──────────────────────────────────────────────────────────────────
+
+
+async def test_the_event_stream_is_served_so_never_refused_as_removed(
+ registry,
+) -> None:
+ """Kept for old clients for a compatibility window, so `transport_removed`
+ does not fire; a stream refused for another reason is counted for it."""
+ with pytest.raises(HTTPException):
+ await handlers.poll_events(
+ agent_id=AGENT_ID,
+ agent=Agent(id=AGENT_ID, name="agent"),
+ protocol=None, # type: ignore[arg-type]
+ config=None, # type: ignore[arg-type]
+ accept="text/event-stream",
+ )
+
+ assert _counts(registry, "switch.agent.connections_refused") == {
+ (("reason", "other"),): 1.0
+ }
+
+
+def test_a_stream_attached_over_sse_is_counted_as_sse() -> None:
+ connections = AgentConnectionRegistry()
+ connections.open(
+ agent_id="a1",
+ connection_id="c1",
+ scope="all",
+ delivery_filter="all",
+ spawn_capable=False,
+ cursor=0,
+ declaration=ClientDeclaration(speaks=6, accepts=1, artifact="agent-runtime"),
+ expected_generation=None,
+ transport="sse",
+ )
+ _open(connections, "a2", "c2", "agent-runtime")
+
+ readings = _connected_readings(_probes(connections.live_agents_by_transport))
+
+ assert readings[(("client", "agent-runtime"), ("transport", "sse"))] == 1.0
+ assert readings[(("client", "agent-runtime"), ("transport", "websocket"))] == 1.0
+
+
+class _Protocol:
+ def __init__(self) -> None:
+ self.event_buffer = EventBuffer(sequence_base=0)
+ self.connections = AgentConnectionRegistry()
+ self.approval_outcomes = None
+ self.sessions = SessionReporter(None)
+
+ async def record_client_declaration(self, *args: Any) -> None:
+ return None
+
+ async def require_room_member(self, agent_id: str, room_id: str) -> None:
+ return None
+
+
+class _AuthenticatedAs:
+ def __init__(self, app: ASGIApp, agent: Agent) -> None:
+ self.app = app
+ self.agent = agent
+
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+ scope["agent"] = self.agent
+ await self.app(scope, receive, send)
+
+
+def client() -> Iterator[TestClient]:
+ app = FastAPI()
+ app.include_router(handlers.router, prefix="/agents")
+ protocol = _Protocol()
+ app.dependency_overrides[get_protocol] = lambda: protocol
+ app.dependency_overrides[get_config] = lambda: None
+ agent = Agent(id=AGENT_ID, name="agent")
+ with TestClient(_AuthenticatedAs(app, agent)) as client: # type: ignore[arg-type]
+ yield client
+
+
+def _refused_on_socket(client: TestClient, query: str) -> int:
+ with client.websocket_connect(f"/agents/{AGENT_ID}/connection/ws?{query}") as ws:
+ assert ws.receive_json()["event"] == "refused"
+ with pytest.raises(WebSocketDisconnect) as closed:
+ ws.receive_json()
+ return closed.value.code
+
+
+def test_a_protocol_mismatch_on_the_socket_is_counted_as_protocol(
+ registry, client: TestClient
+) -> None:
+ code = _refused_on_socket(
+ client, f"connection_id=c1&protocol={PROTOCOL_VERSION + 1}"
+ )
+
+ assert code == 4409
+ assert _counts(registry, "switch.agent.connections_refused") == {
+ (("reason", "protocol"),): 1.0
+ }
+
+
+def test_any_other_refusal_on_the_socket_is_counted_as_other(
+ registry, client: TestClient
+) -> None:
+ code = _refused_on_socket(client, "connection_id=c1&scope=everywhere")
+
+ assert code == 4400
+ assert _counts(registry, "switch.agent.connections_refused") == {
+ (("reason", "other"),): 1.0
+ }
+
+
+async def _unauthenticated_socket(path: str) -> None:
+ async def app(scope: Any, receive: Any, send: Any) -> None:
+ raise AssertionError("an unauthenticated socket reached the app")
+
+ middleware = BearerAuthMiddleware(
+ app,
+ agent_store=AgentStore(),
+ api_key_store=ApiKeyStore(),
+ api_key_cache=ApiKeyCache(ttl_seconds=5, max_entries=8),
+ session_factory=None, # type: ignore[arg-type]
+ )
+ inbox = [{"type": "websocket.connect"}]
+
+ async def receive() -> dict:
+ return inbox.pop(0)
+
+ async def send(message: dict) -> None:
+ return None
+
+ await middleware(
+ {"type": "websocket", "path": path, "headers": [], "query_string": b""},
+ receive,
+ send,
+ )
+
+
+async def test_an_unauthenticated_connection_socket_is_counted(registry) -> None:
+ await _unauthenticated_socket(f"/agents/{AGENT_ID}/connection/ws")
+
+ assert _counts(registry, "switch.agent.connections_refused") == {
+ (("reason", "unauthorized"),): 1.0
+ }
+
+
+async def test_an_unauthenticated_socket_elsewhere_is_not(registry) -> None:
+ await _unauthenticated_socket("/mcp")
+
+ assert _counts(registry, "switch.agent.connections_refused") == {}
diff --git a/core/tests/switch_core/observability/test_db_server.py b/core/tests/switch_core/observability/test_db_server.py
new file mode 100644
index 000000000..b533c02a0
--- /dev/null
+++ b/core/tests/switch_core/observability/test_db_server.py
@@ -0,0 +1,155 @@
+"""The database's own view of this server, against a real Postgres.
+
+The sampler exists for the moment the pool is exhausted, so the test that
+matters most holds the application pool's only connection and samples anyway.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+
+import pytest
+from sqlalchemy import text
+from sqlalchemy.engine import make_url
+from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.observability.db_server import DbServerSampler
+from switch_core.observability.metrics import MetricsRegistry, install, uninstall
+
+
+def registry() -> Iterator[MetricsRegistry]:
+ registry = MetricsRegistry()
+ install(registry)
+ yield registry
+ uninstall()
+
+
+def _unpooled(url: str) -> AsyncEngine:
+ return create_async_engine(url, poolclass=NullPool)
+
+
+def _restricted_url(rls_harness) -> str:
+ return rls_harness.restricted_engine.url.render_as_string(hide_password=False)
+
+
+def _states(sampler: DbServerSampler) -> dict[str, float]:
+ return {r.attributes["state"]: r.value for r in sampler.readings()} # type: ignore[misc]
+
+
+async def test_it_samples_while_the_app_pool_is_exhausted(postgres_url: str) -> None:
+ app_engine = create_async_engine(postgres_url, pool_size=1, max_overflow=0)
+ sampler = DbServerSampler(lambda: _unpooled(postgres_url))
+ try:
+ async with app_engine.connect() as held:
+ # The pool's only connection, checked out and left in a transaction.
+ await held.execute(text("SELECT 1"))
+
+ sample = await sampler.sample_once()
+
+ assert app_engine.pool.checkedout() == 1 # type: ignore[attr-defined]
+ assert sample is not None
+ assert sum(sample.connections.values()) >= 1
+ assert sample.connections["idle_in_transaction"] >= 1
+ assert sample.transactions >= 0
+ states = _states(sampler)
+ assert set(states) == {"active", "idle_in_transaction", "idle", "other"}
+ assert states["idle_in_transaction"] >= 1
+ finally:
+ await sampler.aclose()
+ await app_engine.dispose()
+
+
+async def test_it_does_not_count_its_own_session(rls_harness) -> None:
+ sampler = DbServerSampler(lambda: _unpooled(_restricted_url(rls_harness)))
+ try:
+ sample = await sampler.sample_once()
+ finally:
+ await sampler.aclose()
+
+ # A role with no session but the sampler's own.
+ assert sample is not None
+ assert sum(sample.connections.values()) == 0
+
+
+async def test_transactions_are_recorded_as_the_rise_between_samples(
+ postgres_url: str, registry: MetricsRegistry
+) -> None:
+ sampler = DbServerSampler(lambda: _unpooled(postgres_url))
+ other = _unpooled(postgres_url)
+ try:
+ await sampler.sample_once()
+ # The first sample is only a baseline.
+ assert not [
+ p for p in registry.collect() if p.name == "switch.db.server.transactions"
+ ]
+
+ async with other.begin() as conn:
+ await conn.execute(text("SELECT 1"))
+ await sampler.sample_once()
+
+ payload = next(
+ p for p in registry.collect() if p.name == "switch.db.server.transactions"
+ )
+ assert payload.numbers[0].value >= 1
+ finally:
+ await sampler.aclose()
+ await other.dispose()
+
+
+async def test_a_restricted_role_sees_only_its_own_sessions(rls_harness) -> None:
+ """The runtime role is not a superuser, and the numbers must still be true."""
+ sampler = DbServerSampler(lambda: _unpooled(_restricted_url(rls_harness)))
+ try:
+ async with rls_harness.owner_engine.connect() as owner:
+ await owner.execute(text("SELECT 1"))
+ before = await sampler.sample_once()
+
+ async with rls_harness.restricted_engine.connect() as held:
+ await held.execute(text("SELECT 1"))
+ during = await sampler.sample_once()
+ finally:
+ await sampler.aclose()
+
+ assert before is not None and during is not None
+ # The owner's open transaction is not this role's.
+ assert before.connections["idle_in_transaction"] == 0
+ assert during.connections["idle_in_transaction"] == 1
+
+
+async def test_an_unreachable_database_reports_nothing_and_keeps_trying(
+ postgres_url: str,
+) -> None:
+ unreachable = make_url(postgres_url).set(host="127.0.0.1", port=1)
+ sampler = DbServerSampler(
+ lambda: _unpooled(unreachable.render_as_string(hide_password=False))
+ )
+ try:
+ assert await sampler.sample_once() is None
+ assert list(sampler.readings()) == []
+ assert not sampler.disabled
+ finally:
+ await sampler.aclose()
+
+
+async def test_a_sampler_that_switches_itself_off_stops_reporting(
+ postgres_url: str, registry: MetricsRegistry
+) -> None:
+ """The last good reading must not stay on the dashboard after it stops."""
+ from sqlalchemy.exc import ProgrammingError
+
+ sampler = DbServerSampler(lambda: _unpooled(postgres_url))
+ try:
+ assert await sampler.sample_once() is not None
+ assert list(sampler.readings())
+
+ async def unreadable() -> None:
+ raise ProgrammingError("SELECT ...", {}, Exception("permission denied"))
+
+ sampler._read = unreadable # type: ignore[method-assign]
+ assert await sampler.sample_once() is None
+ assert sampler.disabled
+ assert list(sampler.readings()) == []
+ finally:
+ await sampler.aclose()
diff --git a/core/tests/switch_core/observability/test_pool_holds.py b/core/tests/switch_core/observability/test_pool_holds.py
new file mode 100644
index 000000000..5d8ec4c17
--- /dev/null
+++ b/core/tests/switch_core/observability/test_pool_holds.py
@@ -0,0 +1,187 @@
+"""Who holds a pooled connection, for how long, and how long others wait.
+
+The pool peak says the pool filled. These say which code was holding it and
+what that cost everyone queued behind it, which is what a pool timeout during
+a reconnect storm needs before anyone can fix it.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Iterator
+from types import SimpleNamespace
+
+import pytest
+from sqlalchemy import create_engine, text
+from sqlalchemy import exc as sqlalchemy_exc
+from sqlalchemy.ext.asyncio import create_async_engine
+from sqlalchemy.pool import QueuePool
+from sqlalchemy.util import greenlet_spawn
+
+from switch_core.observability import pool as pool_module
+from switch_core.observability.metrics import MetricsRegistry, install, uninstall
+from switch_core.observability.pool import (
+ UNKNOWN_CALLER,
+ WaitTimedQueuePool,
+ _Hold,
+ _LongHoldLog,
+ describe_borrower,
+ install_hold_timer,
+)
+
+# Code compiled under a switch_core path, so the borrower lookup takes it for
+# ours: the test module itself is deliberately not counted as a caller.
+_FAKE_FILE = "/srv/switch/core/switch_core/rooms/fake_borrower.py"
+
+
+def registry() -> Iterator[MetricsRegistry]:
+ registry = MetricsRegistry()
+ install(registry)
+ yield registry
+ uninstall()
+
+
+def _histograms(registry: MetricsRegistry, name: str) -> dict[tuple, tuple[int, float]]:
+ payload = next((p for p in registry.collect() if p.name == name), None)
+ if payload is None:
+ return {}
+ return {
+ tuple(sorted(point.attributes.items())): (point.count, point.total)
+ for point in payload.histograms
+ }
+
+
+def _ours(source: str) -> dict[str, object]:
+ namespace: dict[str, object] = {
+ "describe_borrower": describe_borrower,
+ "greenlet_spawn": greenlet_spawn,
+ }
+ exec(compile(source, _FAKE_FILE, "exec"), namespace)
+ return namespace
+
+
+class TestTheBorrower:
+ def test_names_the_innermost_switch_core_function(self) -> None:
+ borrow = _ours("def borrow():\n return describe_borrower()\n")["borrow"]
+
+ caller, stack = borrow() # type: ignore[operator]
+
+ assert caller == "switch_core.rooms.fake_borrower:borrow"
+ assert stack[0].startswith("switch_core.rooms.fake_borrower:borrow:")
+
+ async def test_finds_the_coroutine_behind_sqlalchemys_greenlet(self) -> None:
+ """An async checkout runs in a greenlet whose own stack starts at the
+ greenlet. The coroutine that awaited it is only reachable through the
+ parent, and it is the one worth naming."""
+ fetch = _ours(
+ "async def fetch_members():\n"
+ " return await greenlet_spawn(describe_borrower)\n"
+ )["fetch_members"]
+
+ caller, _ = await fetch() # type: ignore[operator]
+
+ assert caller == "switch_core.rooms.fake_borrower:fetch_members"
+
+ def test_code_outside_switch_core_is_unknown(self) -> None:
+ assert describe_borrower() == (UNKNOWN_CALLER, ())
+
+ def test_a_test_tree_under_a_switch_core_path_is_not_ours(self) -> None:
+ source = "def borrow():\n return describe_borrower()\n"
+ namespace: dict[str, object] = {"describe_borrower": describe_borrower}
+ test_file = "/srv/switch/core/tests/switch_core/rooms/test_fake.py"
+ exec(compile(source, test_file, "exec"), namespace)
+
+ assert namespace["borrow"]() == (UNKNOWN_CALLER, ()) # type: ignore[operator]
+
+
+class TestHoldTime:
+ def test_every_checkout_is_timed_by_its_caller(self, registry) -> None:
+ engine = create_engine("sqlite://", poolclass=QueuePool)
+ install_hold_timer(SimpleNamespace(sync_engine=engine)) # type: ignore[arg-type]
+
+ with engine.connect() as conn:
+ conn.execute(text("select 1"))
+
+ holds = _histograms(registry, "switch.db.pool.hold.duration")
+ assert list(holds) == [(("caller", UNKNOWN_CALLER),)]
+ count, total = holds[(("caller", UNKNOWN_CALLER),)]
+ assert count == 1
+ assert total >= 0.0
+
+ def test_a_long_hold_is_logged_with_its_stack(
+ self, registry, caplog, monkeypatch
+ ) -> None:
+ monkeypatch.setattr(pool_module, "LONG_HOLD_SECONDS", 0.0)
+ engine = create_engine("sqlite://", poolclass=QueuePool)
+ install_hold_timer(SimpleNamespace(sync_engine=engine)) # type: ignore[arg-type]
+
+ with caplog.at_level(logging.WARNING, logger=pool_module.__name__):
+ with engine.connect():
+ pass
+
+ assert "A database connection was held for" in caplog.text
+
+
+class TestTheLongHoldLog:
+ def test_one_warning_per_caller_per_window(self, caplog, monkeypatch) -> None:
+ clock = iter([100.0, 110.0, 200.0])
+ monkeypatch.setattr(pool_module.time, "monotonic", lambda: next(clock))
+ log = _LongHoldLog()
+ hold = _Hold(started=0.0, caller="switch_core.rooms:slow", stack=())
+
+ with caplog.at_level(logging.WARNING, logger=pool_module.__name__):
+ log.report(hold, 2.0)
+ log.report(hold, 3.0) # inside the window: counted, not logged
+ log.report(hold, 4.0) # after it: logged, with the one it held back
+
+ warnings = [r.getMessage() for r in caplog.records]
+ assert len(warnings) == 2
+ assert "(0 more long holds" in warnings[0]
+ assert "(1 more long holds" in warnings[1]
+
+
+class TestWaitTime:
+ async def test_every_request_for_a_connection_is_timed(self, registry) -> None:
+ engine = create_async_engine(
+ "postgresql+asyncpg://u:p@localhost/db", poolclass=WaitTimedQueuePool
+ )
+ pool = engine.pool
+
+ def get_and_return() -> None:
+ # The pool's own handover, without a server to connect to: a
+ # placeholder entry the queue hands straight back.
+ pool._pool.put(object()) # type: ignore[attr-defined]
+ pool._do_get() # type: ignore[attr-defined]
+
+ await greenlet_spawn(get_and_return)
+
+ waits = _histograms(registry, "switch.db.pool.wait.duration")
+ assert waits[()][0] == 1
+ await engine.dispose()
+
+ async def test_a_wait_that_times_out_is_timed_too(self, registry) -> None:
+ engine = create_async_engine(
+ "postgresql+asyncpg://u:p@localhost/db",
+ poolclass=WaitTimedQueuePool,
+ pool_size=1,
+ max_overflow=0,
+ pool_timeout=0.01,
+ )
+ pool = engine.pool
+ # Every connection the pool may have is out: the request can only wait.
+ pool._overflow = pool._max_overflow # type: ignore[attr-defined]
+
+ with pytest.raises(sqlalchemy_exc.TimeoutError):
+ await greenlet_spawn(pool._do_get) # type: ignore[attr-defined]
+
+ waits = _histograms(registry, "switch.db.pool.wait.duration")
+ assert waits[()][0] == 1
+ await engine.dispose()
+
+ def test_the_class_survives_a_rebuild(self) -> None:
+ engine = create_async_engine(
+ "postgresql+asyncpg://u:p@localhost/db", poolclass=WaitTimedQueuePool
+ )
+
+ assert isinstance(engine.pool.recreate(), WaitTimedQueuePool)
diff --git a/core/tests/switch_core/pool_checkouts.py b/core/tests/switch_core/pool_checkouts.py
new file mode 100644
index 000000000..c89e398c4
--- /dev/null
+++ b/core/tests/switch_core/pool_checkouts.py
@@ -0,0 +1,105 @@
+"""Count the pooled connections a request holds, at the pool itself.
+
+A request that holds one connection while it waits for a second from the same
+pool can starve it: under load every connection ends up held by a request
+waiting for one more, and they all time out together. The ways a request ends
+up doing that are many (a service that opens its own session, a lookup helper,
+a dependency that reads and never commits), so counting sessions at any one of
+them misses the rest. This counts where they all meet, the pool's `checkout`
+and `checkin` events, and attributes each connection to the request that
+borrowed it.
+
+A request is a `CheckoutScope`, entered with `PoolCheckouts.scope()` or, for an
+app under test, `PoolCheckouts.wrap(app)`. The scope lives in a context
+variable, which SQLAlchemy carries into the greenlet a checkout runs in, and
+which tasks the request starts inherit, so their connections count as the
+request's too. A checkin is matched back to its checkout by connection record
+rather than by context, because a connection can be returned from somewhere
+else, a session closed by the dependency's teardown for one.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass
+from typing import Any
+
+from sqlalchemy import event
+from sqlalchemy.ext.asyncio import AsyncEngine
+
+
+@dataclass
+class CheckoutScope:
+ """One request's connections: how many it holds now, and the most at once."""
+
+ label: str
+ held: int = 0
+ peak: int = 0
+
+
+_current: ContextVar[CheckoutScope | None] = ContextVar(
+ "pool_checkout_scope", default=None
+)
+
+
+class PoolCheckouts:
+ def __init__(self, engine: AsyncEngine) -> None:
+ self._pool = engine.sync_engine.pool
+ self._owners: dict[int, CheckoutScope] = {}
+ self.scopes: list[CheckoutScope] = []
+ event.listen(self._pool, "checkout", self._on_checkout)
+ event.listen(self._pool, "checkin", self._on_checkin)
+
+ def close(self) -> None:
+ event.remove(self._pool, "checkout", self._on_checkout)
+ event.remove(self._pool, "checkin", self._on_checkin)
+
+ def _on_checkout(self, _dbapi: Any, record: Any, _proxy: Any) -> None:
+ scope = _current.get()
+ if scope is None:
+ return
+ scope.held += 1
+ scope.peak = max(scope.peak, scope.held)
+ self._owners[id(record)] = scope
+
+ def _on_checkin(self, _dbapi: Any, record: Any) -> None:
+ scope = self._owners.pop(id(record), None)
+ if scope is not None:
+ scope.held -= 1
+
+ @contextmanager
+ def scope(self, label: str) -> Iterator[CheckoutScope]:
+ scope = CheckoutScope(label)
+ self.scopes.append(scope)
+ token = _current.set(scope)
+ try:
+ yield scope
+ finally:
+ _current.reset(token)
+
+ def held_now(self) -> int:
+ """Connections the current scope holds at this moment; 0 outside one.
+
+ For a fake standing in for a platform, Matrix or bcrypt to record when
+ it is called: the answer there should always be 0.
+ """
+ scope = _current.get()
+ return 0 if scope is None else scope.held
+
+ def wrap(self, app: Any) -> Any:
+ """`app`, with every HTTP request in a scope of its own."""
+
+ async def asgi(scope: dict[str, Any], receive: Any, send: Any) -> None:
+ if scope["type"] != "http":
+ await app(scope, receive, send)
+ return
+ with self.scope(f"{scope['method']} {scope['path']}"):
+ await app(scope, receive, send)
+
+ return asgi
+
+ def over(self, limit: int = 1) -> list[CheckoutScope]:
+ """Every scope that held more than `limit` connections at once."""
+ return [s for s in self.scopes if s.peak > limit]
diff --git a/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py b/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py
index 77f394b20..d4b2b3dc6 100644
--- a/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py
+++ b/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py
@@ -49,6 +49,7 @@ def _room_service() -> RoomService:
collab_bridge_store=None, # type: ignore[arg-type]
resource_service=None, # type: ignore[arg-type]
session_factory=_broken_session_factory, # type: ignore[arg-type]
+ room_cache=None, # type: ignore[arg-type]
telemetry=object(), # type: ignore[arg-type]
)
diff --git a/core/tests/switch_core/test_room_service_delete_room.py b/core/tests/switch_core/test_room_service_delete_room.py
index 7c8a02674..40e557079 100644
--- a/core/tests/switch_core/test_room_service_delete_room.py
+++ b/core/tests/switch_core/test_room_service_delete_room.py
@@ -70,6 +70,14 @@ class _FakeProvisioning:
self._events.append(("delete_room", transport_room_id))
+class _FakeRoomCache:
+ def __init__(self) -> None:
+ self.invalidated: list[tuple[str, str]] = []
+
+ def invalidate(self, tenant_id: str, room_id: str) -> None:
+ self.invalidated.append((tenant_id, room_id))
+
+
def _build_service(
*,
room: Any,
@@ -85,6 +93,7 @@ def _build_service(
svc._collab_lifecycle = _FakeLifecycle(bridges) # type: ignore[assignment]
svc._client_lifecycle = _FakeClientLifecycle(clients) # type: ignore[assignment]
svc._provisioning = _FakeProvisioning(events) # type: ignore[assignment]
+ svc._room_cache = _FakeRoomCache() # type: ignore[assignment]
return svc, room_store
@@ -198,3 +207,23 @@ class TestDeleteRoom:
with pytest.raises(ValueError, match="Room not found"):
await svc.delete_room("nope")
+
+ async def test_drops_the_rooms_shared_delivery_reads(self) -> None:
+ """Members still running here are kicked, which empties the room's
+ cache entry; this is the call that covers any that were not."""
+ events: list[Any] = []
+ room = SimpleNamespace(
+ id="room-1",
+ tenant_id="room-tenant",
+ transport_room_id="!mx:switch.local",
+ bridge_id=None,
+ external_channel_id=None,
+ )
+ svc, room_store = _build_service(
+ room=room, client_ids=[], clients={}, bridges={}, events=events
+ )
+
+ await svc.delete_room("room-1")
+
+ assert room_store.deleted == ["room-1"]
+ assert svc._room_cache.invalidated == [("room-tenant", "room-1")] # type: ignore[attr-defined]
diff --git a/core/tests/switch_core/test_room_service_pool_holds.py b/core/tests/switch_core/test_room_service_pool_holds.py
index 4275d80b3..e0f15cf3a 100644
--- a/core/tests/switch_core/test_room_service_pool_holds.py
+++ b/core/tests/switch_core/test_room_service_pool_holds.py
@@ -16,6 +16,7 @@ written until the transport side is in place.
from __future__ import annotations
+from types import SimpleNamespace
from typing import Any
import pytest
@@ -151,6 +152,9 @@ def _service(
svc._provisioning = provisioning # type: ignore[assignment]
svc._client_lifecycle = _RunningClients(running) # type: ignore[assignment]
svc._collab_lifecycle = _RunningClients({}) # type: ignore[assignment]
+ svc._room_cache = SimpleNamespace( # type: ignore[assignment]
+ invalidate=lambda tenant_id, room_id: None
+ )
return svc, provisioning
diff --git a/core/tests/switch_core/transport/test_postgres_transport.py b/core/tests/switch_core/transport/test_postgres_transport.py
index ac14d89aa..394ee2464 100644
--- a/core/tests/switch_core/transport/test_postgres_transport.py
+++ b/core/tests/switch_core/transport/test_postgres_transport.py
@@ -8,6 +8,9 @@ it.
What is deliberately not implemented is tested too: a transport that quietly
returns nothing would look like a working deployment with a silent room, which
is the one failure this codebase refuses to ship.
+
+Every transport here shares the room delivery cache, as in production. The
+cache must not change anything a test here can observe.
"""
from __future__ import annotations
@@ -50,9 +53,42 @@ from switch_core.transport import (
)
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
-from switch_core.transport.postgres import _DELIVERY_PAGE, PostgresTransport
+from switch_core.transport.postgres import DELIVERY_PAGE, PostgresTransport
+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
from tests.conftest import RLSHarness
+# The room cache `_transport` hands out. One per session factory, because that
+# is what the process has: one, shared by every transport built on the same
+# pool. Emptied per test by `_fresh_caches`.
+_CACHES: dict[int, RoomDeliveryCache] = {}
+
+
[email protected](autouse=True)
+def _fresh_caches() -> Iterator[None]:
+ _CACHES.clear()
+ yield
+ _CACHES.clear()
+
+
+def _room_cache(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> RoomDeliveryCache:
+ cache = _CACHES.get(id(session_factory))
+ if cache is None:
+ cache = RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=MessageStore(),
+ limits=RoomCacheLimits(
+ max_bytes=64 * 1024 * 1024,
+ max_rooms=1000,
+ max_rows_per_room=1000,
+ max_age_seconds=300.0,
+ ),
+ page=DELIVERY_PAGE,
+ )
+ _CACHES[id(session_factory)] = cache
+ return cache
+
async def _make_room(session: AsyncSession) -> tuple[str, str, str, str]:
"""Insert the Room and Client a send depends on.
@@ -202,6 +238,7 @@ def _transport(
listener=listener or _FakeListener(),
invites=invites or InviteBus(),
ephemeral=ephemeral or EphemeralBus(),
+ room_cache=_room_cache(session_factory),
)
@@ -592,6 +629,7 @@ class TestMedia:
listener=_FakeListener(),
invites=InviteBus(),
ephemeral=EphemeralBus(),
+ room_cache=_room_cache(session_factory),
)
@@ -2071,7 +2109,7 @@ class TestBeingRemovedFromARoom:
async def _kick_on_the_page_boundary(_room_ref, event) -> None:
delivered.append(event.body)
- if len(delivered) == _DELIVERY_PAGE:
+ if len(delivered) == DELIVERY_PAGE:
await provisioning.kick_user(room, leaving[1])
removed = _transport(
@@ -2090,14 +2128,14 @@ class TestBeingRemovedFromARoom:
await _watched_room(member)
# More than one page, so the loop would come back for another.
- for i in range(_DELIVERY_PAGE + 25):
+ for i in range(DELIVERY_PAGE + 25):
await member.send_message(
room, f"line {i}", sender_name="agent one", metered=True
)
await listener.announce(room_id)
# The whole of page one, and none of page two.
- assert delivered == [f"line {i}" for i in range(_DELIVERY_PAGE)]
+ assert delivered == [f"line {i}" for i in range(DELIVERY_PAGE)]
assert room_id not in removed._watching
assert room_id not in removed._cursors
assert "Delivery failed" not in caplog.text, (
diff --git a/core/tests/switch_core/transport/test_room_cache.py b/core/tests/switch_core/transport/test_room_cache.py
new file mode 100644
index 000000000..5eea47266
--- /dev/null
+++ b/core/tests/switch_core/transport/test_room_cache.py
@@ -0,0 +1,721 @@
+"""The room delivery cache on its own, against a store a test can steer.
+
+These pin the rules the cache rests on without a database in the way: what
+range an entry claims, when a shared read counts for a reader, what is
+published together, and what happens at every limit. The same rules are
+proved again through `PostgresTransport` against Postgres in
+`test_room_delivery_cache.py`; here a test can stop a read half-way and look.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import random
+import uuid
+from collections.abc import AsyncIterator, Collection
+from datetime import UTC, datetime
+from typing import Any
+
+import pytest
+
+from switch_core.db.models import Message, MessageAttachment
+from switch_core.tenant_context import current_tenant_id
+from switch_core.transport.room_cache import (
+ CachedAttachment,
+ CachedPage,
+ RoomCacheLimits,
+ RoomDeliveryCache,
+)
+
+TENANT = "tenant-a"
+ROOM = "room-1"
+PAGE = 5
+
+
+class _Sessions:
+ """Stands in for the session factory: `tenant_session` only enters it."""
+
+ def __call__(self) -> contextlib.AbstractAsyncContextManager[object]:
+ return self._session()
+
+ @contextlib.asynccontextmanager
+ async def _session(self) -> AsyncIterator[object]:
+ yield object()
+
+
+class _Store:
+ """The two reads a fill makes, over rows a test commits.
+
+ A read takes its snapshot when it is called, like a statement does, and
+ can then be held at `gate` so a test can act while it is in flight.
+ """
+
+ def __init__(self) -> None:
+ self.rows: dict[tuple[str, str], list[Message]] = {}
+ self.files: dict[str, list[MessageAttachment]] = {}
+ self.reads = 0
+ self.tenants: list[str | None] = []
+ self.gate: asyncio.Event | None = None
+ self.attachments_gate: asyncio.Event | None = None
+ self.fail_next = False
+ self.yield_randomly: random.Random | None = None
+
+ def commit(
+ self,
+ room_id: str = ROOM,
+ *,
+ tenant_id: str = TENANT,
+ body: str | None = None,
+ files: int = 0,
+ ) -> Message:
+ log = self.rows.setdefault((tenant_id, room_id), [])
+ seq = (log[-1].seq if log else 0) + 1
+ row = Message(
+ id=uuid.uuid4().hex,
+ seq=seq,
+ room_id=room_id,
+ transport_event_id=f"sw_{uuid.uuid4().hex}",
+ sender_id="@a:test",
+ sender_name="a",
+ event_type="m.room.message",
+ msgtype="m.file" if files else "m.text",
+ body=body or f"line {seq}",
+ formatted_body=None,
+ thread_root_event_id=None,
+ content={"body": body or f"line {seq}", "nested": {"n": seq}},
+ sent_at=datetime.now(UTC),
+ )
+ log.append(row)
+ self.files[row.id] = [
+ MessageAttachment(
+ message_id=row.id,
+ position=i,
+ uri=f"switch-media://{row.id}/{i}",
+ filename=f"f{i}.txt",
+ mimetype="text/plain",
+ size=10 + i,
+ )
+ for i in range(files)
+ ]
+ return row
+
+ async def _maybe_yield(self) -> None:
+ if self.yield_randomly is not None:
+ for _ in range(self.yield_randomly.randint(0, 3)):
+ await asyncio.sleep(0)
+
+ async def list_for_room(
+ self, session: object, room_id: str, *, after_seq: int, limit: int
+ ) -> list[Message]:
+ self.reads += 1
+ tenant = current_tenant_id()
+ self.tenants.append(tenant)
+ await self._maybe_yield()
+ if self.fail_next:
+ self.fail_next = False
+ raise RuntimeError("pool timeout")
+ log = self.rows.get((tenant or "", room_id), [])
+ snapshot = [row for row in log if row.seq > after_seq][:limit]
+ if self.gate is not None:
+ await self.gate.wait()
+ await self._maybe_yield()
+ return snapshot
+
+ async def attachments_for(
+ self, session: object, message_ids: Collection[str]
+ ) -> dict[str, list[MessageAttachment]]:
+ if self.attachments_gate is not None:
+ await self.attachments_gate.wait()
+ await self._maybe_yield()
+ return {i: self.files[i] for i in message_ids if self.files.get(i)}
+
+
+def _limits(**overrides: Any) -> RoomCacheLimits:
+ values: dict[str, Any] = {
+ "max_bytes": 10_000_000,
+ "max_rooms": 100,
+ "max_rows_per_room": 50,
+ "max_age_seconds": 300.0,
+ }
+ values.update(overrides)
+ return RoomCacheLimits(**values)
+
+
+def _cache(store: _Store, page: int = PAGE, **limits: Any) -> RoomDeliveryCache:
+ return RoomDeliveryCache(
+ session_factory=_Sessions(), # type: ignore[arg-type]
+ message_store=store, # type: ignore[arg-type]
+ limits=_limits(**limits),
+ page=page,
+ )
+
+
+def _seqs(page: CachedPage | None) -> list[int]:
+ assert page is not None
+ return [row.seq for row in page.rows]
+
+
+async def _read(
+ cache: RoomDeliveryCache,
+ after_seq: int,
+ woken_at: int,
+ *,
+ tenant_id: str = TENANT,
+ room_id: str = ROOM,
+) -> CachedPage | None:
+ return await cache.read(
+ tenant_id, room_id, after_seq=after_seq, woken_at=woken_at, limit=PAGE
+ )
+
+
+async def _until(predicate: Any) -> None:
+ for _ in range(200):
+ if predicate():
+ return
+ await asyncio.sleep(0)
+ raise AssertionError("never happened")
+
+
+class TestOneFillPerRoom:
+ async def test_concurrent_reads_for_one_room_collapse_into_one_fill(self) -> None:
+ store = _Store()
+ cache = _cache(store)
+ watchers = [object() for _ in range(6)]
+ for watcher in watchers:
+ cache.attach(TENANT, ROOM, watcher)
+ for _ in range(3):
+ store.commit()
+ store.gate = asyncio.Event()
+ woken = cache.tick()
+
+ readers = [asyncio.create_task(_read(cache, 0, woken)) for _ in watchers]
+ await _until(lambda: store.reads == 1)
+ for _ in range(5):
+ await asyncio.sleep(0)
+ assert store.reads == 1, "a second read started while one was in flight"
+ store.gate.set()
+ pages = await asyncio.gather(*readers)
+
+ assert store.reads == 1
+ assert all(_seqs(page) == [1, 2, 3] for page in pages)
+ assert all(page is not None and page.done for page in pages)
+
+ async def test_a_reader_cancelled_while_waiting_does_not_cancel_the_fill(
+ self,
+ ) -> None:
+ """The fill is the room's, not the reader's that happened to start it."""
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ cache.attach(TENANT, ROOM, "b")
+ store.commit()
+ store.gate = asyncio.Event()
+ woken = cache.tick()
+
+ starter = asyncio.create_task(_read(cache, 0, woken))
+ await _until(lambda: store.reads == 1)
+ other = asyncio.create_task(_read(cache, 0, woken))
+ await asyncio.sleep(0)
+ starter.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await starter
+ store.gate.set()
+
+ assert _seqs(await other) == [1]
+ assert store.reads == 1
+
+ async def test_a_failed_fill_reaches_its_waiters_and_the_next_read_retries(
+ self,
+ ) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ store.commit()
+ store.fail_next = True
+ woken = cache.tick()
+
+ with pytest.raises(RuntimeError, match="pool timeout"):
+ await _read(cache, 0, woken)
+ assert _seqs(await _read(cache, 0, woken)) == [1]
+
+
+class TestTheHandoff:
+ """The range an entry claims, and when a shared read counts for a reader."""
+
+ async def test_a_reader_inside_the_range_gets_exactly_the_rows_after_it(
+ self,
+ ) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ for _ in range(4):
+ store.commit()
+ woken = cache.tick()
+ assert _seqs(await _read(cache, 0, woken)) == [1, 2, 3, 4]
+
+ for cursor in range(4):
+ page = await _read(cache, cursor, woken)
+ assert _seqs(page) == list(range(cursor + 1, 5))
+ assert page is not None and page.done
+ assert store.reads == 1
+
+ async def test_a_fill_that_began_before_a_wake_does_not_answer_for_it(
+ self,
+ ) -> None:
+ """The missed-notification window, closed.
+
+ A read that took its snapshot before row 2 committed cannot say there
+ is nothing after 1 to a reader woken for row 2. It is waited for, not
+ trusted, and a fresh fill finds the row.
+ """
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ cache.attach(TENANT, ROOM, "b")
+ store.commit()
+ store.gate = asyncio.Event()
+ early = asyncio.create_task(_read(cache, 0, cache.tick()))
+ await _until(lambda: store.reads == 1)
+
+ store.commit() # row 2, committed after the fill's snapshot
+ woken_for_two = cache.tick()
+ late = asyncio.create_task(_read(cache, 0, woken_for_two))
+ await asyncio.sleep(0)
+ store.gate.set()
+
+ assert _seqs(await early) == [1]
+ first = await late
+ assert _seqs(first) == [1]
+ assert first is not None and not first.done, "told it was caught up"
+ second = await _read(cache, 1, woken_for_two)
+ assert _seqs(second) == [2]
+ assert second is not None and second.done
+ assert store.reads == 2
+
+ async def test_caught_up_means_a_fill_after_the_wake_reached_the_end(
+ self,
+ ) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ store.commit()
+ before = cache.tick()
+ assert _seqs(await _read(cache, 0, before)) == [1]
+
+ # Nothing new, and the last fill started after this wake: no read.
+ assert _seqs(await _read(cache, 1, before)) == []
+ assert store.reads == 1
+
+ # Woken again: the old fill cannot vouch for it, so one more read.
+ store.commit()
+ later = cache.tick()
+ assert _seqs(await _read(cache, 1, later)) == [2]
+ assert store.reads == 2
+
+ async def test_a_reader_below_the_range_is_sent_to_the_database(self) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ for _ in range(3):
+ store.commit()
+ woken = cache.tick()
+ # The first reader is at 2, so the entry starts there.
+ assert _seqs(await _read(cache, 2, woken)) == [3]
+
+ assert await _read(cache, 1, woken) is None
+ assert await _read(cache, 0, woken) is None
+ # Once its own reads bring it to the range, it is served from it.
+ assert _seqs(await _read(cache, 2, woken)) == [3]
+
+ async def test_a_reader_ahead_of_the_entry_extends_it_without_a_gap(
+ self,
+ ) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ store.commit()
+ assert _seqs(await _read(cache, 0, cache.tick())) == [1]
+ for _ in range(3):
+ store.commit()
+
+ woken = cache.tick()
+ # This reader read 2 and 3 itself; the entry extends from 1, not 3.
+ assert _seqs(await _read(cache, 3, woken)) == [4]
+ assert _seqs(await _read(cache, 0, woken)) == [1, 2, 3, 4]
+
+ async def test_a_long_room_is_read_a_page_at_a_time(self) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ for _ in range(PAGE * 2 + 1):
+ store.commit()
+ woken = cache.tick()
+
+ cursor, seen = 0, []
+ while True:
+ page = await _read(cache, cursor, woken)
+ assert page is not None
+ seen.extend(row.seq for row in page.rows)
+ if page.rows:
+ cursor = page.rows[-1].seq
+ if page.done:
+ break
+ assert seen == list(range(1, PAGE * 2 + 2))
+
+
+class TestWhatIsPublished:
+ async def test_rows_are_published_only_with_their_attachments(self) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ cache.attach(TENANT, ROOM, "b")
+ store.commit(files=2)
+ store.attachments_gate = asyncio.Event()
+ woken = cache.tick()
+
+ first = asyncio.create_task(_read(cache, 0, woken))
+ await _until(lambda: store.reads == 1)
+ # Rows read, attachments not yet: nothing is visible to anyone.
+ assert cache.stats().rows == 0
+ second = asyncio.create_task(_read(cache, 0, woken))
+ await asyncio.sleep(0)
+ assert not second.done()
+ store.attachments_gate.set()
+
+ for page in (await first, await second):
+ assert page is not None
+ (row,) = page.rows
+ assert [f.uri for f in row.attachments] == [
+ f"switch-media://{row.id}/0",
+ f"switch-media://{row.id}/1",
+ ]
+
+ async def test_attachments_are_metadata_only(self) -> None:
+ """Media bytes live in `media_blobs` and are never read or held."""
+ assert set(CachedAttachment.__slots__) == {
+ "uri",
+ "filename",
+ "mimetype",
+ "size",
+ }
+
+ async def test_each_access_to_content_is_a_fresh_dict(self) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ store.commit(body="original")
+ page = await _read(cache, 0, cache.tick())
+ assert page is not None
+ (row,) = page.rows
+
+ mine = row.content
+ mine["body"] = "edited"
+ mine["nested"]["n"] = -1
+ theirs = row.content
+ assert theirs["body"] == "original"
+ assert theirs["nested"] == {"n": 1}
+
+ async def test_a_cached_row_cannot_be_changed(self) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ store.commit()
+ page = await _read(cache, 0, cache.tick())
+ assert page is not None
+ with pytest.raises(AttributeError):
+ page.rows[0].body = "edited" # type: ignore[misc]
+
+
+class TestKeys:
+ async def test_an_entry_is_never_served_to_another_tenant(self) -> None:
+ """Same room id, two tenants: two entries, each filled under its own
+ tenant. A room id alone never finds another tenant's rows."""
+ store = _Store()
+ cache = _cache(store)
+ cache.attach("tenant-a", ROOM, "a")
+ cache.attach("tenant-b", ROOM, "b")
+ store.commit(tenant_id="tenant-a", body="for a")
+ woken = cache.tick()
+
+ mine = await _read(cache, 0, woken, tenant_id="tenant-a")
+ theirs = await _read(cache, 0, woken, tenant_id="tenant-b")
+
+ assert mine is not None and [r.body for r in mine.rows] == ["for a"]
+ assert theirs is not None and theirs.rows == ()
+ assert store.tenants == ["tenant-a", "tenant-b"]
+ assert cache.stats().rooms == 2
+
+ async def test_a_room_nobody_is_watching_is_not_cached(self) -> None:
+ store = _Store()
+ cache = _cache(store)
+ store.commit()
+ assert await _read(cache, 0, cache.tick()) is None
+ assert cache.stats().rooms == 0
+ assert store.reads == 0
+
+
+class TestLettingGo:
+ async def _filled(self, cache: RoomDeliveryCache, store: _Store) -> None:
+ cache.attach(TENANT, ROOM, "a")
+ store.commit()
+ assert _seqs(await _read(cache, 0, cache.tick())) == [1]
+
+ async def test_the_last_watcher_leaving_purges_the_room(self) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "b")
+ await self._filled(cache, store)
+
+ cache.detach(TENANT, ROOM, "a")
+ assert cache.stats().rooms == 1, "one member is still watching"
+ cache.detach(TENANT, ROOM, "b")
+ assert cache.stats() == cache.stats().__class__(bytes=0, rooms=0, rows=0)
+
+ async def test_detaching_what_never_attached_is_harmless(self) -> None:
+ cache = _cache(_Store())
+ cache.detach(TENANT, ROOM, "never")
+
+ async def test_invalidate_drops_the_entry(self) -> None:
+ """The hook for a future edit or redaction, and for room deletion."""
+ store = _Store()
+ cache = _cache(store)
+ await self._filled(cache, store)
+
+ cache.invalidate(TENANT, ROOM)
+
+ assert cache.stats().rooms == 0
+ assert cache.stats().bytes == 0
+ # Still watched, so the next read fills afresh from the database.
+ assert _seqs(await _read(cache, 0, cache.tick())) == [1]
+ assert store.reads == 2
+
+ async def test_invalidate_during_a_fill_sends_its_waiters_to_the_database(
+ self,
+ ) -> None:
+ store = _Store()
+ cache = _cache(store)
+ cache.attach(TENANT, ROOM, "a")
+ store.commit()
+ store.gate = asyncio.Event()
+ reader = asyncio.create_task(_read(cache, 0, cache.tick()))
+ await _until(lambda: store.reads == 1)
+
+ cache.invalidate(TENANT, ROOM)
+ store.gate.set()
+
+ assert await reader is None
+ assert cache.stats().rooms == 0, "a discarded fill came back to life"
+
+ async def test_an_eviction_during_a_fill_still_serves_its_waiters(
+ self,
+ ) -> None:
+ """The rows were read once already; a waiter reading them again
+ because the entry was evicted would multiply the load under churn."""
+ store = _Store()
+ cache = _cache(store, max_rooms=1)
+ cache.attach(TENANT, ROOM, "a")
+ cache.attach(TENANT, "other", "b")
+ store.commit()
+ store.commit("other")
+ store.gate = asyncio.Event()
+ woken = cache.tick()
+ waiters = [asyncio.create_task(_read(cache, 0, woken)) for _ in range(3)]
+ await _until(lambda: store.reads == 1)
+ # Another room's read takes the only slot and evicts this one mid-fill.
+ other = asyncio.create_task(_read(cache, 0, cache.tick(), room_id="other"))
+ await _until(lambda: store.reads == 2)
+
+ store.gate.set()
+
+ for waiter in waiters:
+ page = await waiter
+ assert page is not None, "an evicted fill sent its waiter to the database"
+ assert _seqs(page) == [1]
+ assert page.done
+ assert _seqs(await other) == [1]
+ assert cache.stats().rooms == 1, "the evicted room came back"
+ assert store.reads == 2
+
+ async def test_the_byte_limit_evicts_the_least_recently_read_room(
+ self,
+ ) -> None:
+ store = _Store()
+ probe = _cache(store)
+ probe.attach(TENANT, "probe", "p")
+ store.commit("probe")
+ await _read(probe, 0, probe.tick(), room_id="probe")
+ one_row = probe.stats().bytes
+
+ cache = _cache(store, max_bytes=one_row * 2 + one_row // 2)
+ for room in ("r1", "r2", "r3"):
+ cache.attach(TENANT, room, "a")
+ store.commit(room)
+ woken = cache.tick()
+ await _read(cache, 0, woken, room_id="r1")
+ await _read(cache, 0, woken, room_id="r2")
+ await _read(cache, 0, woken, room_id="r1") # r2 is now the oldest
+ await _read(cache, 0, cache.tick(), room_id="r3")
+
+ assert cache.stats().rooms == 2
+ assert cache.stats().bytes <= one_row * 2 + one_row // 2
+ reads = store.reads
+ await _read(cache, 0, woken, room_id="r1")
+ assert store.reads == reads, "r1 should have survived"
+ await _read(cache, 0, cache.tick(), room_id="r2")
+ assert store.reads == reads + 1, "r2 should have been evicted"
+
+ async def test_the_room_limit_evicts_the_least_recently_read_room(
+ self,
+ ) -> None:
+ store = _Store()
+ cache = _cache(store, max_rooms=2)
+ for room in ("r1", "r2", "r3"):
+ cache.attach(TENANT, room, "a")
+ store.commit(room)
+ await _read(cache, 0, cache.tick(), room_id=room)
+ assert cache.stats().rooms == 2
+
+ async def test_the_row_limit_raises_the_floor(self) -> None:
+ store = _Store()
+ cache = _cache(store, max_rows_per_room=PAGE)
+ cache.attach(TENANT, ROOM, "a")
+ for _ in range(PAGE):
+ store.commit()
+ woken = cache.tick()
+ assert len(_seqs(await _read(cache, 0, woken))) == PAGE
+ store.commit()
+ store.commit()
+ woken = cache.tick()
+ assert _seqs(await _read(cache, PAGE, woken)) == [PAGE + 1, PAGE + 2]
+
+ assert cache.stats().rows == PAGE
+ # Rows 1 and 2 were let go: a reader that still needs them reads them.
+ assert await _read(cache, 0, woken) is None
+ assert await _read(cache, 1, woken) is None
+ assert _seqs(await _read(cache, 2, woken))[0] == 3
+
+ async def test_the_age_limit_raises_the_floor(
+ self, monkeypatch: pytest.MonkeyPatch
+ ) -> None:
+ clock = [1000.0]
+ monkeypatch.setattr(
+ "switch_core.transport.room_cache.time.monotonic", lambda: clock[0]
+ )
+ store = _Store()
+ cache = _cache(store, max_age_seconds=60.0)
+ cache.attach(TENANT, ROOM, "a")
+ store.commit()
+ woken = cache.tick()
+ await _read(cache, 0, woken)
+ clock[0] += 61.0
+ store.commit()
+ woken = cache.tick()
+ assert _seqs(await _read(cache, 1, woken)) == [2]
+
+ assert await _read(cache, 0, woken) is None, "row 1 outlived its age"
+ assert cache.stats().rows == 1
+
+ def test_a_room_must_hold_at_least_one_page(self) -> None:
+ with pytest.raises(ValueError, match="max_rows_per_room"):
+ _cache(_Store(), page=200, max_rows_per_room=199)
+
+
+class TestEveryRowOnceInOrder:
+ """Readers, writes, evictions and invalidations, interleaved at random.
+
+ Each reader behaves as a transport does: it is woken after a commit, keeps
+ its own cursor, takes what the cache gives it and reads the store itself
+ when the cache says no. Whatever the interleaving, each must see every
+ row exactly once and in order.
+ """
+
+ @pytest.mark.parametrize("seed", range(40))
+ async def test_random_interleavings(self, seed: int) -> None:
+ rng = random.Random(seed)
+ store = _Store()
+ store.yield_randomly = rng
+ page = rng.choice([1, 2, 3, PAGE])
+ cache = _cache(
+ store,
+ page=page,
+ max_rows_per_room=page + rng.randint(0, 4),
+ max_bytes=rng.choice([10_000_000, 3000, 1500]),
+ )
+ readers = 4
+ total = 40
+ received: list[list[int]] = [[] for _ in range(readers)]
+ wakes = [asyncio.Event() for _ in range(readers)]
+ woken = [0] * readers
+ cursors = [0] * readers
+ idle = [asyncio.Event() for _ in range(readers)]
+
+ async def own_read(after: int) -> tuple[list[int], bool]:
+ # A transport's own database read: snapshot, then a suspension.
+ rows = [r.seq for r in store.rows.get((TENANT, ROOM), [])]
+ await asyncio.sleep(0)
+ found = [s for s in rows if s > after][:page]
+ return found, len(found) < page
+
+ async def reader(i: int) -> None:
+ cache.attach(TENANT, ROOM, i)
+ while True:
+ await wakes[i].wait()
+ wakes[i].clear()
+ at = woken[i]
+ while True:
+ shared = await cache.read(
+ TENANT, ROOM, after_seq=cursors[i], woken_at=at, limit=page
+ )
+ if shared is None:
+ seqs, done = await own_read(cursors[i])
+ else:
+ seqs, done = [r.seq for r in shared.rows], shared.done
+ for seq in seqs:
+ assert seq == cursors[i] + 1, (i, cursors[i], seqs)
+ cursors[i] = seq
+ received[i].append(seq)
+ # Reader 3 is slow, so it falls out of the held range.
+ for _ in range(rng.randint(0, 6) if i == 3 else 0):
+ await asyncio.sleep(0)
+ if rng.random() < 0.3:
+ await asyncio.sleep(0)
+ if done or not seqs:
+ break
+ if not wakes[i].is_set():
+ idle[i].set()
+
+ def wake_all() -> None:
+ for i in range(readers):
+ woken[i] = cache.tick()
+ idle[i].clear()
+ wakes[i].set()
+
+ tasks = [asyncio.create_task(reader(i)) for i in range(readers)]
+ try:
+ for _ in range(total):
+ store.commit()
+ wake_all()
+ for _ in range(rng.randint(0, 4)):
+ await asyncio.sleep(0)
+ if rng.random() < 0.05:
+ cache.invalidate(TENANT, ROOM)
+ for i in range(readers):
+ settled = asyncio.ensure_future(idle[i].wait())
+ done, _ = await asyncio.wait(
+ [settled, tasks[i]],
+ timeout=5,
+ return_when=asyncio.FIRST_COMPLETED,
+ )
+ settled.cancel()
+ if tasks[i] in done:
+ tasks[i].result() # a reader's own assertion, raised here
+ assert done, f"reader {i} never settled, seed {seed}"
+ finally:
+ for task in tasks:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+
+ expected = list(range(1, total + 1))
+ for i in range(readers):
+ assert received[i] == expected, f"reader {i}, seed {seed}"
diff --git a/core/tests/switch_core/transport/test_room_delivery_cache.py b/core/tests/switch_core/transport/test_room_delivery_cache.py
new file mode 100644
index 000000000..c2f10b304
--- /dev/null
+++ b/core/tests/switch_core/transport/test_room_delivery_cache.py
@@ -0,0 +1,675 @@
+"""The room delivery cache through `PostgresTransport`, against Postgres.
+
+`test_room_cache.py` pins the cache's rules with a store a test can stop
+half-way. These prove the same guarantees where they matter: real transports,
+real rows, real row-level security, and in one case the real LISTEN.
+
+`test_postgres_transport.py` runs every existing transport test on the cache
+too.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from collections.abc import Iterator
+from typing import Any
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.db.models import Client, ClientRoom, Message, Room, Tenant
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.messages.notify import MessageListener
+from switch_core.provisioning.postgres import PostgresProvisioning
+from switch_core.transport import InboundMedia, InboundMessage
+from switch_core.transport.invites import InviteBus
+from switch_core.transport.postgres import DELIVERY_PAGE, PostgresTransport
+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
+from tests.conftest import RLSHarness
+from tests.switch_core.transport.test_postgres_transport import (
+ _FakeListener,
+ _make_client,
+ _make_room,
+ _Received,
+ _settled,
+ _transport,
+ _watched_room,
+)
+
+DELIVERY_TIMEOUT = 10.0
+
+
+def _cache(
+ session_factory: async_sessionmaker[AsyncSession], **limits: Any
+) -> RoomDeliveryCache:
+ values: dict[str, Any] = {
+ "max_bytes": 64 * 1024 * 1024,
+ "max_rooms": 1000,
+ "max_rows_per_room": 1000,
+ "max_age_seconds": 300.0,
+ }
+ values.update(limits)
+ return RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=MessageStore(),
+ limits=RoomCacheLimits(**values),
+ page=DELIVERY_PAGE,
+ )
+
+
+def _with(transport: PostgresTransport, cache: RoomDeliveryCache) -> PostgresTransport:
+ """The transport `_transport` built, on the cache this test chose.
+
+ `_transport` hands out that module's shared cache; these tests each decide
+ for themselves which cache, and its limits.
+ """
+ transport._room_cache = cache
+ return transport
+
+
+class _Room:
+ """One room with `size` members, each a receiving transport."""
+
+ def __init__(self) -> None:
+ self.room_id = ""
+ self.transport_room_id = ""
+ self.members: list[PostgresTransport] = []
+ self.received: list[_Received] = []
+ self.tasks: list[asyncio.Task[None]] = []
+
+ @classmethod
+ async def open(
+ cls,
+ session_factory: async_sessionmaker[AsyncSession],
+ size: int,
+ *,
+ cache: RoomDeliveryCache,
+ listener: Any,
+ invites: InviteBus | None = None,
+ ) -> _Room:
+ room = cls()
+ async with session_factory() as session:
+ room_id, transport_room_id, first_id, first_user = await _make_room(session)
+ clients = [(first_id, first_user)]
+ for n in range(size - 1):
+ clients.append(await _make_client(session, f"member{n}"))
+ for client_id, _ in clients:
+ # Recorded directly, so no arrival row is in the way.
+ await RoomStore().add_client(session, client_id, room_id)
+ await session.commit()
+ room.room_id, room.transport_room_id = room_id, transport_room_id
+ for client_id, user_id in clients:
+ transport = _with(
+ _transport(
+ session_factory,
+ client_id=client_id,
+ user_id=user_id,
+ listener=listener,
+ invites=invites,
+ ),
+ cache,
+ )
+ received = _Received()
+ transport.register_handlers(received.handlers())
+ room.members.append(transport)
+ room.received.append(received)
+ room.tasks.append(asyncio.create_task(transport.receive_forever()))
+ for transport in room.members:
+ await _watched_room(transport)
+ return room
+
+ def close(self) -> None:
+ for task in self.tasks:
+ task.cancel()
+
+ async def settled(self) -> None:
+ for transport in self.members:
+ await _settled(transport)
+
+
+class _ReadCounter:
+ """Counts `list_for_room` calls, which is what a delivery read costs."""
+
+ def __init__(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ self.calls = 0
+ original = MessageStore.list_for_room
+
+ async def _counting(store, session, room_id, **kwargs): # type: ignore[no-untyped-def]
+ self.calls += 1
+ return await original(store, session, room_id, **kwargs)
+
+ monkeypatch.setattr(MessageStore, "list_for_room", _counting)
+
+
+async def _event_ids_in_order(
+ session_factory: async_sessionmaker[AsyncSession], room_id: str
+) -> list[str]:
+ async with session_factory() as session:
+ rows = await session.execute(
+ select(Message.transport_event_id)
+ .where(Message.room_id == room_id, Message.seq > 0)
+ .order_by(Message.seq)
+ )
+ return list(rows.scalars())
+
+
+def _ids(received: _Received) -> list[str]:
+ return [event.event_id for event in received.events] # type: ignore[attr-defined]
+
+
+def rooms() -> Iterator[list[_Room]]:
+ opened: list[_Room] = []
+ yield opened
+ for room in opened:
+ room.close()
+
+
+class TestEveryMemberGetsEveryRowOnce:
+ async def test_a_room_of_six_reads_each_new_message_once(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ monkeypatch: pytest.MonkeyPatch,
+ rooms: list[_Room],
+ ) -> None:
+ """The case the cache exists for, and the numbers for the PR.
+
+ Six members, twenty messages, each announced as it is sent. Every
+ member must get every message once and in order, and the room is read
+ once per message instead of once per member.
+ """
+ cache = _cache(session_factory)
+ listener = _FakeListener()
+ room = await _Room.open(session_factory, 6, cache=cache, listener=listener)
+ rooms.append(room)
+ reads = _ReadCounter(monkeypatch)
+
+ messages = 20
+ for n in range(messages):
+ await room.members[n % 6].send_message(
+ room.transport_room_id, f"m{n}", sender_name="a", metered=False
+ )
+ await listener.announce(room.room_id)
+
+ expected = await _event_ids_in_order(session_factory, room.room_id)
+ assert len(expected) == messages
+ for received in room.received:
+ assert _ids(received) == expected
+ per_message = reads.calls / messages
+ print( # noqa: T201 - the figure the PR quotes, read with -s
+ f"\nroom of 6, {messages} messages: {reads.calls} list_for_room "
+ f"calls, {per_message:.2f} per message"
+ )
+ assert reads.calls == messages
+
+ async def test_a_burst_over_the_real_listener_arrives_once_in_order(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ postgres_schema: str,
+ monkeypatch: pytest.MonkeyPatch,
+ rooms: list[_Room],
+ ) -> None:
+ """The real LISTEN, the real fan-out, three writers at once.
+
+ Announcements coalesce and land mid-read here, the way they do in
+ production; nothing is stepping the transports one at a time.
+ """
+ listener = MessageListener(
+ lambda: create_async_engine(postgres_schema, poolclass=NullPool)
+ )
+ await listener.start()
+ try:
+ await asyncio.wait_for(listener.connected.wait(), DELIVERY_TIMEOUT)
+ cache = _cache(session_factory)
+ room = await _Room.open(session_factory, 6, cache=cache, listener=listener)
+ rooms.append(room)
+ await room.settled()
+ reads = _ReadCounter(monkeypatch)
+
+ async def write(member: int, count: int) -> None:
+ for n in range(count):
+ await room.members[member].send_message(
+ room.transport_room_id,
+ f"{member}-{n}",
+ sender_name="a",
+ metered=False,
+ )
+
+ await asyncio.gather(write(0, 25), write(1, 25), write(2, 25))
+ expected = await _event_ids_in_order(session_factory, room.room_id)
+ assert len(expected) == 75
+
+ async def all_arrived() -> None:
+ while any(len(r.events) < 75 for r in room.received):
+ await asyncio.sleep(0.02)
+
+ await asyncio.wait_for(all_arrived(), DELIVERY_TIMEOUT)
+ await asyncio.sleep(0.2) # long enough for a duplicate to show
+ for received in room.received:
+ assert _ids(received) == expected
+ print( # noqa: T201 - the figure the PR quotes, read with -s
+ f"\nburst over LISTEN, room of 6, 75 messages from 3 writers: "
+ f"{reads.calls} list_for_room calls"
+ )
+ finally:
+ await listener.stop()
+
+ async def test_members_at_different_cursors_each_get_their_own_sequence(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ rooms: list[_Room],
+ ) -> None:
+ """One member far behind, one in the middle, one at the head.
+
+ Whoever reads first decides where the held range starts; the others
+ are below it (their own read until they reach it) or above it (the
+ range is extended up to them). Each gets exactly what follows its own
+ cursor.
+ """
+ cache = _cache(session_factory)
+ listener = _FakeListener()
+ room = await _Room.open(session_factory, 3, cache=cache, listener=listener)
+ rooms.append(room)
+ for n in range(12):
+ await room.members[0].send_message(
+ room.transport_room_id, f"m{n}", sender_name="a", metered=False
+ )
+ expected = await _event_ids_in_order(session_factory, room.room_id)
+
+ for start in ([0, 5, 11], [11, 0, 5], [5, 11, 0]):
+ cache.invalidate(room.members[0].tenant_id, room.room_id)
+ for member, received in zip(room.members, room.received, strict=True):
+ received.events.clear()
+ for member, cursor in zip(room.members, start, strict=True):
+ member._cursors[room.room_id] = cursor
+ await listener.announce(room.room_id)
+ for received, cursor in zip(room.received, start, strict=True):
+ assert _ids(received) == expected[cursor:], start
+
+ async def test_evictions_mid_stream_lose_and_repeat_nothing(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ rooms: list[_Room],
+ ) -> None:
+ """A byte limit smaller than one row, and invalidations as it runs.
+
+ Every fill is thrown away the moment it lands, so every read falls
+ back to the transport's own. Slower, and still exactly once.
+ """
+ cache = _cache(session_factory, max_bytes=1)
+ listener = _FakeListener()
+ room = await _Room.open(session_factory, 4, cache=cache, listener=listener)
+ rooms.append(room)
+
+ for n in range(15):
+ await room.members[n % 4].send_message(
+ room.transport_room_id, f"m{n}", sender_name="a", metered=False
+ )
+ if n % 5 == 0:
+ cache.invalidate(room.members[0].tenant_id, room.room_id)
+ await listener.announce(room.room_id)
+
+ expected = await _event_ids_in_order(session_factory, room.room_id)
+ for received in room.received:
+ assert _ids(received) == expected
+ assert cache.stats().bytes <= 1
+
+
+class TestWhatAMemberIsHanded:
+ async def test_files_arrive_with_their_rows(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ rooms: list[_Room],
+ ) -> None:
+ cache = _cache(session_factory)
+ listener = _FakeListener()
+ room = await _Room.open(session_factory, 3, cache=cache, listener=listener)
+ rooms.append(room)
+
+ await room.members[0].send_media(
+ room.transport_room_id,
+ "switch-media://abc",
+ "notes.txt",
+ "text/plain",
+ 12,
+ sender_name="a",
+ msgtype="m.file",
+ metered=False,
+ )
+ await listener.announce(room.room_id)
+
+ for received in room.received:
+ (event,) = received.events
+ assert isinstance(event, InboundMedia)
+ assert (event.uri, event.filename, event.mimetype, event.size) == (
+ "switch-media://abc",
+ "notes.txt",
+ "text/plain",
+ 12,
+ )
+
+ async def test_changing_a_delivered_message_changes_nobody_elses(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ rooms: list[_Room],
+ ) -> None:
+ cache = _cache(session_factory)
+ listener = _FakeListener()
+ room = await _Room.open(session_factory, 3, cache=cache, listener=listener)
+ rooms.append(room)
+
+ async def vandal(_room: object, event: InboundMessage) -> None:
+ event.content["body"] = "vandalised"
+ event.content["nested"]["n"] = -1 # type: ignore[index]
+ room.received[0].events.append(event)
+
+ room.members[0].register_handlers(room.received[0].handlers(on_message=vandal))
+ for _ in range(2): # twice, so a later reader follows an earlier vandal
+ await room.members[1].send_message(
+ room.transport_room_id,
+ "original",
+ sender_name="a",
+ metered=False,
+ extra_content={"nested": {"n": 1}},
+ )
+ await listener.announce(room.room_id)
+
+ for received in room.received[1:]:
+ assert [e.content["body"] for e in received.events] == ["original"] * 2 # type: ignore[attr-defined]
+ assert [e.content["nested"] for e in received.events] == [{"n": 1}] * 2 # type: ignore[attr-defined]
+
+
+class TestMembershipIsStillEachClients:
+ async def test_a_removed_member_gets_nothing_after_removal(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ monkeypatch: pytest.MonkeyPatch,
+ rooms: list[_Room],
+ ) -> None:
+ """Removed while the room's shared read was in flight.
+
+ The read completes and the remaining member is handed the row; the
+ removed one was waiting on the same read and gets nothing from it.
+ """
+ cache = _cache(session_factory)
+ listener = _FakeListener()
+ invites = InviteBus()
+ room = await _Room.open(
+ session_factory, 2, cache=cache, listener=listener, invites=invites
+ )
+ rooms.append(room)
+ leaving, staying = room.members
+
+ await staying.send_message(
+ room.transport_room_id, "held", sender_name="a", metered=False
+ )
+ gate = asyncio.Event()
+ original = MessageStore.list_for_room
+
+ async def held(store, session, room_id, **kwargs): # type: ignore[no-untyped-def]
+ await gate.wait()
+ return await original(store, session, room_id, **kwargs)
+
+ monkeypatch.setattr(MessageStore, "list_for_room", held)
+ for waker in list(listener.wakers[room.room_id]):
+ await waker(room.room_id)
+ for _ in range(20):
+ await asyncio.sleep(0)
+ assert leaving._delivering and staying._delivering
+
+ provisioning = PostgresProvisioning(
+ session_factory=session_factory,
+ room_store=RoomStore(),
+ client_store=ClientStore(),
+ message_store=MessageStore(),
+ invites=invites,
+ )
+ await provisioning.kick_user(room.transport_room_id, leaving.user_id)
+ gate.set()
+ await room.settled()
+
+ assert room.received[0].events == []
+ assert [e.body for e in room.received[1].events] == ["held"] # type: ignore[attr-defined]
+
+ # And nothing after it, while the room is still cached for the other.
+ await staying.send_message(
+ room.transport_room_id, "after", sender_name="a", metered=False
+ )
+ await listener.announce(room.room_id)
+ assert room.received[0].events == []
+ assert [e.body for e in room.received[1].events] == ["held", "after"] # type: ignore[attr-defined]
+
+ async def test_a_room_everyone_has_left_is_not_held(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ rooms: list[_Room],
+ ) -> None:
+ """Deleting a room removes its members first, so this is also what
+ deleting one leaves behind: nothing."""
+ cache = _cache(session_factory)
+ listener = _FakeListener()
+ invites = InviteBus()
+ room = await _Room.open(
+ session_factory, 2, cache=cache, listener=listener, invites=invites
+ )
+ rooms.append(room)
+ await room.members[0].send_message(
+ room.transport_room_id, "hello", sender_name="a", metered=False
+ )
+ await listener.announce(room.room_id)
+ assert cache.stats().rooms == 1
+
+ provisioning = PostgresProvisioning(
+ session_factory=session_factory,
+ room_store=RoomStore(),
+ client_store=ClientStore(),
+ message_store=MessageStore(),
+ invites=invites,
+ )
+ for member in room.members:
+ await provisioning.kick_user(room.transport_room_id, member.user_id)
+
+ assert cache.stats().rooms == 0
+ assert cache.stats().bytes == 0
+
+ async def test_another_tenants_rows_are_never_served(
+ self, rls_harness: RLSHarness
+ ) -> None:
+ """A transport of tenant B pointed at tenant A's room, by force.
+
+ The schema makes this impossible to reach honestly (a client cannot be
+ a member of another tenant's room), so the subscription is written
+ straight into the transport, standing in for some future bug that
+ reaches it. A's rows are already in the cache under A's key. B's read
+ is keyed by B and filled under B's tenant, so row-level security
+ answers it, and the answer is nothing.
+ """
+ owner, restricted = rls_harness.owner, rls_harness.restricted
+ tenant_a, tenant_b = (f"tenant-{uuid.uuid4().hex[:8]}" for _ in range(2))
+ async with owner() as session:
+ session.add_all(
+ [Tenant(id=t, slug=t, name=t) for t in (tenant_a, tenant_b)]
+ )
+ await session.flush()
+ suffix = uuid.uuid4().hex[:8]
+ a_client = Client(
+ tenant_id=tenant_a,
+ transport_user_id=f"@a-{suffix}:test",
+ display_name="a",
+ type="agent",
+ )
+ b_client = Client(
+ tenant_id=tenant_b,
+ transport_user_id=f"@b-{suffix}:test",
+ display_name="b",
+ type="agent",
+ )
+ room = Room(
+ tenant_id=tenant_a,
+ transport_room_id=f"!room-{suffix}:test",
+ name="a's room",
+ description="",
+ )
+ session.add_all([a_client, b_client, room])
+ await session.flush()
+ session.add(
+ ClientRoom(tenant_id=tenant_a, client_id=a_client.id, room_id=room.id)
+ )
+ await session.commit()
+ room_id, transport_room_id = room.id, room.transport_room_id
+ a_ids = (a_client.id, a_client.transport_user_id)
+ b_ids = (b_client.id, b_client.transport_user_id)
+
+ cache = _cache(restricted)
+ listener = _FakeListener()
+ a = _with(
+ _transport(
+ restricted,
+ client_id=a_ids[0],
+ user_id=a_ids[1],
+ tenant_id=tenant_a,
+ listener=listener,
+ ),
+ cache,
+ )
+ b = _with(
+ _transport(
+ restricted,
+ client_id=b_ids[0],
+ user_id=b_ids[1],
+ tenant_id=tenant_b,
+ listener=listener,
+ ),
+ cache,
+ )
+ a_got, b_got = _Received(), _Received()
+ a.register_handlers(a_got.handlers())
+ b.register_handlers(b_got.handlers())
+ tasks = [
+ asyncio.create_task(a.receive_forever()),
+ asyncio.create_task(b.receive_forever()),
+ ]
+ try:
+ await _watched_room(a)
+ await a.send_message(
+ transport_room_id, "for tenant a", sender_name="a", metered=False
+ )
+ await listener.announce(room_id)
+ assert [e.body for e in a_got.events] == ["for tenant a"] # type: ignore[attr-defined]
+ assert cache.stats().rooms == 1
+
+ # The forced subscription: what `_watch` would record, from 0.
+ b._room_ids[transport_room_id] = room_id
+ b._watching[room_id] = transport_room_id
+ b._cursors[room_id] = 0
+ cache.attach(tenant_b, room_id, b)
+ b._mark_pending(room_id)
+ await _settled(b)
+
+ assert b_got.events == []
+ assert cache.stats().rooms == 2, "B shared A's entry"
+ async with tenant_session(restricted, tenant_a) as session:
+ rows = await MessageStore().list_for_room(
+ session, room_id, after_seq=0, limit=10
+ )
+ assert [row.body for row in rows] == ["for tenant a"]
+ finally:
+ for task in tasks:
+ task.cancel()
+
+
+class TestTheListenerIsNeverHeldUp:
+ async def test_the_fan_out_finishes_while_a_fill_is_blocked(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """The single fan-out task wakes every room; a fill must not be in it.
+
+ The room's read is held. The fan-out still wakes both members, moves
+ on to the next room, and drains its queue.
+ """
+ listener = MessageListener(lambda: None) # type: ignore[arg-type,return-value]
+ listener._running = True
+ fan_out = asyncio.create_task(listener._fan_out_forever())
+ cache = _cache(session_factory)
+ tasks: list[asyncio.Task[None]] = []
+ try:
+ async with session_factory() as session:
+ room_id, room, first_id, first_user = await _make_room(session)
+ second_id, second_user = await _make_client(session, "second")
+ for client_id in (first_id, second_id):
+ await RoomStore().add_client(session, client_id, room_id)
+ await session.commit()
+ members = [
+ _with(
+ _transport(
+ session_factory,
+ client_id=client_id,
+ user_id=user_id,
+ listener=listener, # type: ignore[arg-type]
+ ),
+ cache,
+ )
+ for client_id, user_id in (
+ (first_id, first_user),
+ (second_id, second_user),
+ )
+ ]
+ for member in members:
+ member.register_handlers(_Received().handlers())
+ tasks.append(asyncio.create_task(member.receive_forever()))
+ for member in members:
+ await _watched_room(member)
+ for member in members:
+ await _settled(member)
+
+ gate = asyncio.Event()
+ fills = 0
+ original = MessageStore.list_for_room
+
+ async def held(store, session, room, **kwargs): # type: ignore[no-untyped-def]
+ nonlocal fills
+ fills += 1
+ await gate.wait()
+ return await original(store, session, room, **kwargs)
+
+ monkeypatch.setattr(MessageStore, "list_for_room", held)
+ other_room_woken = asyncio.Event()
+
+ async def other_room(_room_id: str) -> None:
+ other_room_woken.set()
+
+ listener.subscribe("some-other-room", other_room)
+ await asyncio.wait_for(other_room_woken.wait(), DELIVERY_TIMEOUT)
+ other_room_woken.clear()
+
+ # Wake the room; its one shared read starts and is held.
+ listener._mark(room_id, 1)
+ for _ in range(500):
+ if fills and all(m._delivering for m in members):
+ break
+ await asyncio.sleep(0.002)
+ assert fills == 1, "both members should be waiting on one read"
+
+ # With that read still held, the fan-out goes on waking rooms.
+ listener._mark("some-other-room", 1)
+ await asyncio.wait_for(other_room_woken.wait(), DELIVERY_TIMEOUT)
+ assert not gate.is_set()
+ assert fills == 1
+ assert listener._pending == {}
+ assert all(m._delivering for m in members)
+ assert not fan_out.done()
+ gate.set()
+ for member in members:
+ await _settled(member)
+ finally:
+ for task in tasks:
+ task.cancel()
+ fan_out.cancel()
diff --git a/deploy/observability/README.md b/deploy/observability/README.md
index c53391e6c..4d6adf18e 100644
--- a/deploy/observability/README.md
+++ b/deploy/observability/README.md
@@ -45,7 +45,9 @@ done
## Check these before you trust the dashboard
**Histogram panels need two things, not one.** Request latency, delivery lag,
-database query duration and platform call duration are OTLP histograms, and the
+database query duration, pool hold and wait time (`switch.db.pool.hold.duration`,
+`switch.db.pool.wait.duration`), platform call duration and the agent buffer's
+scan length and duration are OTLP histograms, and the
`p95:`/`p99:` queries here assume the collector exports them to Datadog as
*distributions*. Check the collector's
histogram mode — in the older `histograms` mode they arrive as separate
@@ -55,9 +57,10 @@ Then check the second thing, which is easy to miss because it is on Datadog's
side rather than the collector's: **percentile aggregations are off by default
on a distribution metric and are billed separately.** Enable them per metric in
Metrics Summary, and add the tag each panel groups by to that metric's
-configured tag set — `route` for requests, `operation` for database queries,
-`bridge` and `platform` for bridge calls — or `p95: … by {…}` returns nothing on a fresh
-account. Empty is the honest outcome either way; a panel is never silently
+configured tag set (`route` for requests, `operation` for database queries and
+the agent buffer, `caller` for pool hold time but not wait time, `bridge` and
+`platform` for bridge calls), or
+`p95: … by {…}` returns nothing on a fresh account. Empty is the honest outcome either way; a panel is never silently
switched to a different statistic.
The two latency **monitors** depend on the same thing, and fail more quietly
diff --git a/deploy/observability/dashboard.json b/deploy/observability/dashboard.json
index 51d5ed459..2a07d7448 100644
--- a/deploy/observability/dashboard.json
+++ b/deploy/observability/dashboard.json
@@ -71,7 +71,7 @@
"title": "Agents connected",
"requests": [
{
- "q": "avg:switch.agents.connected{$env,$service}",
+ "q": "sum:switch.agents.connected{$env,$service}",
"aggregator": "last"
}
],
@@ -714,6 +714,323 @@
"width": 12,
"height": 4
}
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Agents connected, by transport and client",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "sum:switch.agents.connected{$env,$service} by {transport,client}",
+ "display_type": "line"
+ }
+ ]
+ },
+ "layout": {
+ "x": 0,
+ "y": 67,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Agent connections refused per second, by reason",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "sum:switch.agent.connections_refused{$env,$service} by {reason}.as_rate()",
+ "display_type": "bars",
+ "style": {
+ "palette": "warm"
+ }
+ }
+ ]
+ },
+ "layout": {
+ "x": 6,
+ "y": 67,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Database sessions for this server's role, by state (the database's view)",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "max:switch.db.server.connections{$env,$service} by {state}",
+ "display_type": "area"
+ }
+ ]
+ },
+ "layout": {
+ "x": 0,
+ "y": 71,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Database transactions per second (the database's view)",
+ "show_legend": false,
+ "requests": [
+ {
+ "q": "avg:switch.db.server.transactions{$env,$service}.as_rate()",
+ "display_type": "line"
+ }
+ ]
+ },
+ "layout": {
+ "x": 6,
+ "y": 71,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Agent buffer events walked per scan, p50 / p95 by operation (cap 2000), needs distribution mapping, see README",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "p50:switch.agent.buffer.scanned{$env,$service} by {operation}",
+ "display_type": "line"
+ },
+ {
+ "q": "p95:switch.agent.buffer.scanned{$env,$service} by {operation}",
+ "display_type": "line"
+ }
+ ]
+ },
+ "layout": {
+ "x": 0,
+ "y": 75,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Agent buffer scan duration p95 / p99 (ms) by operation, needs distribution mapping, see README",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "p95:switch.agent.buffer.scan.duration{$env,$service} by {operation}",
+ "display_type": "line"
+ },
+ {
+ "q": "p99:switch.agent.buffer.scan.duration{$env,$service} by {operation}",
+ "display_type": "line"
+ }
+ ]
+ },
+ "layout": {
+ "x": 6,
+ "y": 75,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Database pool: connection hold p95 / p99 (ms) by caller. Needs distribution mapping, see README",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "p95:switch.db.pool.hold.duration{$env,$service} by {caller}",
+ "display_type": "line"
+ },
+ {
+ "q": "p99:switch.db.pool.hold.duration{$env,$service} by {caller}",
+ "display_type": "line"
+ }
+ ]
+ },
+ "layout": {
+ "x": 0,
+ "y": 79,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Database pool: wait for a connection p95 / p99 (ms). Needs distribution mapping, see README",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "p95:switch.db.pool.wait.duration{$env,$service}",
+ "display_type": "line"
+ },
+ {
+ "q": "p99:switch.db.pool.wait.duration{$env,$service}",
+ "display_type": "line"
+ }
+ ]
+ },
+ "layout": {
+ "x": 6,
+ "y": 79,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Agent connections opened per second, by kind (after_lapse = returning from a lapse)",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "sum:switch.agent.connections_opened{$env,$service} by {kind}.as_rate()",
+ "display_type": "bars"
+ }
+ ]
+ },
+ "layout": {
+ "x": 0,
+ "y": 83,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Process starts (deploys and restarts)",
+ "show_legend": false,
+ "requests": [
+ {
+ "q": "sum:switch.runtime.starts{$env,$service}.as_count()",
+ "display_type": "bars"
+ }
+ ]
+ },
+ "layout": {
+ "x": 6,
+ "y": 83,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "note",
+ "content": "**Shared room reads.** Each fill is one database read standing in for one per member of the room, so fills against deliveries is the saving. A fallback (behind, evicted, unwatched, gave_up) is a transport reading the database itself: it costs a read, never a message.",
+ "background_color": "gray",
+ "font_size": "14",
+ "text_align": "left",
+ "vertical_align": "top",
+ "show_tick": false,
+ "tick_pos": "50%",
+ "tick_edge": "left",
+ "has_padding": true
+ },
+ "layout": {
+ "x": 0,
+ "y": 87,
+ "width": 12,
+ "height": 2
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Room cache reads per second, by outcome (hit and filled are served from memory)",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "sum:switch.delivery_cache.reads{$env,$service} by {outcome}.as_rate()",
+ "display_type": "bars"
+ }
+ ]
+ },
+ "layout": {
+ "x": 0,
+ "y": 89,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Room cache fills and rows read per second, against messages delivered",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "sum:switch.delivery_cache.fills{$env,$service} by {outcome}.as_rate()",
+ "display_type": "line"
+ },
+ {
+ "q": "sum:switch.delivery_cache.rows_read{$env,$service}.as_rate()",
+ "display_type": "line"
+ },
+ {
+ "q": "sum:switch.messages.delivered{$env,$service}.as_rate()",
+ "display_type": "line"
+ }
+ ]
+ },
+ "layout": {
+ "x": 6,
+ "y": 89,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Room cache memory (bytes) and rooms held",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "max:switch.delivery_cache.bytes{$env,$service}",
+ "display_type": "line"
+ },
+ {
+ "q": "max:switch.delivery_cache.rooms{$env,$service}",
+ "display_type": "line"
+ }
+ ]
+ },
+ "layout": {
+ "x": 0,
+ "y": 93,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Room cache evictions per second, by reason",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "sum:switch.delivery_cache.evictions{$env,$service} by {reason}.as_rate()",
+ "display_type": "bars"
+ }
+ ]
+ },
+ "layout": {
+ "x": 6,
+ "y": 93,
+ "width": 6,
+ "height": 4
+ }
}
]
}
diff --git a/docs/design/agent-controllers-v1.md b/docs/design/agent-controllers-v1.md
index 2edbd418c..88ba97923 100644
--- a/docs/design/agent-controllers-v1.md
+++ b/docs/design/agent-controllers-v1.md
@@ -386,10 +386,9 @@ stream. The flag and everything else above stay as they are.
### Agent hosts in the controller's process, and the local relay
- Each agent's definition says where its agent host runs: `isolation: "shared"` (the default)
- in the controller's process, as below; `isolated` in a process of its own, served the
- per-agent agent protocol by the relay (its own event stream, heartbeat, placements and room
- claims), as before agent hosts moved in-process. Cloud machines run every agent isolated, as
- a systemd unit. Changing an agent's isolation restarts it the other way.
+ in the controller's process, as below; `isolated` in a process of its own, which reaches
+ the same hub over a WebSocket on the relay's port (`/hub`). Cloud machines run every agent
+ isolated, as a systemd unit. Changing an agent's isolation restarts it the other way.
- One upstream stream. Each agent's agent host runs inside the controller's process and is handed
its events from that stream directly (`AgentHub`): in order, filtered to what addresses the
agent, with gaps, resets, room controls and approval outcomes. Events that arrive while its
@@ -397,13 +396,15 @@ stream. The flag and everything else above stay as they are.
event. The agent host states its sessions' rooms in memory, for the relay and for routing
session commands; none of it is sent upstream.
- The shared agent host code (`runAgentHost`) takes the function that opens its event stream: the
- controller passes its hub, while Console and the shared daemon pass the agent host's own
- connection to Switch.
+ controller passes its hub, an isolated agent host the hub over its WebSocket
+ (`openHubStream`, chosen by `SWITCH_AGENT_HUB` in its credentials), while Console passes the
+ agent host's own connection to Switch. Only the controller holds the agent protocol's
+ connection to Switch; nothing on the machine imitates it.
- A loopback HTTP relay (`127.0.0.1`, a per-agent bearer token minted locally) is what each
agent host uses as `SWITCH_API_ENDPOINT` for its calls to Switch. It forwards everything with the
controller access token, the `X-Switch-Agent-Id` header, and `X-Switch-Room-Id` resolved from
- the placements. It serves no event stream and no connection bookkeeping. The credentials file
- names the relay and its local token, never a Switch credential.
+ the placements. It serves no event stream and no connection bookkeeping of its own. The
+ credentials file names the relay, its hub and its local token, never a Switch credential.
### Core implementation notes (decisions the spec left open)
diff --git a/docs/design/controller-contract-v1.md b/docs/design/controller-contract-v1.md
index 2ca401772..45727b353 100644
--- a/docs/design/controller-contract-v1.md
+++ b/docs/design/controller-contract-v1.md
@@ -7,10 +7,11 @@ Step 1 of the roadmap turns this into OpenAPI and JSON Schema files plus recorde
## 0. Conventions
-- **Transport:** HTTPS with JSON bodies (`application/json`), and one SSE stream (`text/event-stream`).
+- **Transport:** HTTPS with JSON bodies (`application/json`), and one WebSocket per controller carrying JSON frames.
- **Base paths:** Management is `/v1/management/...`. The Core controller surface is `/v1/controllers/...`. Agent actions use the existing `/agents/{agent_id}/...` routes.
-- **Version:** every request sends `Switch-Controller-Protocol: 1`.
- - The server answers with the range it supports in `Switch-Controller-Protocol-Accepts: 1-1`.
+- **Version:** every request, and the socket's opening request, sends `Switch-Controller-Protocol: 2`.
+ - The server answers with the range it supports in `Switch-Controller-Protocol-Accepts: 2-2`.
+ - Protocol 2 moved the stream and the beat onto one WebSocket (§6). Protocol 1 (an SSE stream and a beat POST) is no longer accepted.
- An unsupported version returns `426` with code `protocol_unsupported`.
- **Forward compatibility:** receivers ignore unknown fields. An unknown enum value is read as `unknown`, never as an error.
- **Time:** RFC 3339 UTC strings. **IDs:** opaque strings.
@@ -222,22 +223,26 @@ POST /v1/controllers/{id}/connection
→ 201 { connection_id: string, generation: number, heartbeat_interval_s: number,
attached: string[] } // agents Core attached to this connection
-GET /v1/controllers/{id}/events?connection_id=…&generation=…
- Accept: text/event-stream
- → SSE stream (frames below). The server sends ": ping" every 15s
-
-POST /v1/controllers/{id}/connection/beat
- { connection_id: string, generation: number, cursors: Record<agent_id, number> }
- → 200 { attached: string[] }
- → 409 { code: "taken_over" | "stale_generation" } // taken_over is terminal for this client
- → 404 { code: "unknown_connection" } // reopen
+WS /v1/controllers/{id}/connection/ws?connection_id=…&generation=…
+ Authorization: Bearer <access token>
+ ← { event: string, data: object } // frames below, one JSON text message each
+ ← { event: "ping", data: {} } // every heartbeat_interval_s
+ → { type: "pong", cursors: Record<agent_id, number> }
+ // the beat: alive, and how far each agent has read
+ ← { event: "refused", data: { status, detail: { code, message } } }, then close 4000 + status
+ // 404 unknown_connection, 409 taken_over | stale_generation,
+ // 401 controller_revoked | token_expired, 426 protocol_unsupported
+ ← { event: "evicted", data: { code, reason } }, then close
+ // a pong refused: taken_over is terminal for this client,
+ // the rest (no_stream, unknown_connection, …) reopen
+ close 1012 // the server is restarting: reconnect within a second
```
**Attachment is automatic.** Core attaches the agents currently bound to this controller (§7), and attaches or detaches them as bindings change. There's no per-agent subscribe call.
-**Core knows only whether an agent is connected.** A controller-backed agent is connected while its controller's stream is attached and beating, its owner has it `running`, and the controller is not revoked. Core does not know where the agent's sessions are, or whether one is running: the open and the beat carry no session map, and a body that still sends `placements` is refused with `422 validation_error` (the one exception to ignoring unknown fields, so a controller still reporting sessions is noticed rather than silently dropped). When a connected agent is addressed, Core delivers the message and posts nothing; if a session has to start, the agent's host posts "Starting a session…" itself. When it is not connected, Core tells the room why: stopped by its owner, its machine removed, or its machine offline.
+**Core knows only whether an agent is connected.** A controller-backed agent is connected while its controller's socket is attached and answering pings, its owner has it `running`, and the controller is not revoked. Core does not know where the agent's sessions are, or whether one is running: the open and the pong carry no session map, and a body that still sends `placements` is refused with `422 validation_error` (the one exception to ignoring unknown fields, so a controller still reporting sessions is noticed rather than silently dropped). When a connected agent is addressed, Core delivers the message and posts nothing; if a session has to start, the agent's host posts "Starting a session…" itself. When it is not connected, Core tells the room why: stopped by its owner, its machine removed, or its machine offline.
-**SSE frames.** `event:` is the type and `data:` is JSON.
+**Frames.** `event` is the type and `data` is the JSON object below without its `type`.
```ts
type StreamEvent =